Signer-Dialect Robustness in Landmark-Only Sign Recognition: A 21-Fold Leave-One-Signer-Out Study
Abstract
A single cross-signer accuracy number hides who a sign-recognition model actually serves. We run a 21-fold leave-one-signer-out study on the Google ISLR 250-sign dataset, holding out each of the 21 signers in turn and reporting the full per-signer distribution rather than a pooled average. Top-1 accuracy ranges from 25.6% on the worst-served signer to 64.2% on the best — a 38.6 percentage-point spread — with a mean of 41.7% and a per-signer standard deviation of 11.4 points, more than ten times the seed-to-seed deviation of the same architecture. Five of twenty-one signers fall below 30%. We pre-registered three hypotheses: that the spread would exceed 25 points (confirmed, 38.6), that the worst signer would fall below 30% (confirmed, 25.6%), and that handshape complexity would predict per-signer degradation (refuted, r-squared 0.008). The degradation is signer-driven, not sign-feature-driven: the same sign swings 60 to 82 points between the best and worst signers. We argue the per-signer distribution, not the pooled mean, is the honest unit of report for sign recognition, and that signer-level metadata must be captured in any future data collection.

1. Introduction
A companion study established that the honest ceiling for landmark-only isolated-sign recognition, measured on held-out signers, is about 45% top-1 rather than the 80 to 90% reported on signer-mixed splits [7]. That number is a pooled average across a held-out test set. This study asks the question the average hides: when the model meets a signer it has never seen, how much does performance depend on which signer it is?
The answer is: enormously. Running the model against each of the 21 signers in turn, top-1 accuracy ranges from 25.6% on the worst-served signer to 64.2% on the best. The mean of 41.7% describes almost none of them. For a technology whose entire purpose is to work for deaf people in the wild, a 38-point spread across signers is not a footnote to the accuracy number. It is the more important number.
We pre-registered three hypotheses before running the folds, and the one that failed is the most informative. We expected handshape complexity to predict which signers the model struggles with. It does not. The variable that predicts whether a sign is recognized is who is signing it, not how hard the handshape is.
2. Methods
2.1 Leave-one-signer-out protocol
The Google ISLR dataset [1] contains 250 signs from 21 signers, provided as per-frame MediaPipe landmarks [2] rather than video. We run 21 folds: in each fold, one signer is held out entirely for test, a second is held out for validation, and the model trains on the remaining 19. This is the strict form of signer-independent evaluation that fixed splits approximate but rarely guarantee [3].
The architecture is fixed: the frame transformer that was the best stable model in the companion ceiling study, trained with the same recipe in every fold. Holding the model constant isolates the signer as the variable. We pre-registered three hypotheses in the notebook contract before any fold ran: H1, that the per-signer spread would exceed 25 points; H2, that the worst signer would fall below 30%; and H3, that a handshape-complexity score would correlate with per-signer accuracy at r-squared of at least 0.30.
3. Results
3.1 The per-signer distribution
Figure 1 shows every fold. The distribution is wide and roughly continuous from 25.6% to 64.2%, a spread of 38.6 points (H1 confirmed). The worst-served signer sits at 25.6% (H2 confirmed). Five of the 21 signers fall below 30%, meaning the model recognizes fewer than three signs in ten for nearly a quarter of the people it was tested against. The pooled mean is 41.7%, but the per-signer standard deviation is 11.4 points — more than ten times the roughly one-point seed-to-seed deviation the same architecture shows when the signers are fixed. The variance lives in the population, not in the training noise.
| Value | |
|---|---|
| Folds (held-out signers) | 21 |
| Mean top-1 | 41.7% |
| Per-signer std | 11.4 pts |
| Worst / best signer | 25.6% / 64.2% |
| Spread (worst to best) | 38.6 pts |
| Signers below 30% | 5 of 21 |
| Signers at or above 50% | 6 of 21 |
| H1 — spread > 25 pts | CONFIRMED (38.6) |
| H2 — worst signer < 30% | CONFIRMED (25.6%) |
| H3 — handshape complexity predicts degradation | REFUTED (r² = 0.008) |
Hypotheses were registered in the notebook contract before the folds were run. H3's refutation is the load-bearing finding: the thing that predicts whether a sign is recognized is who signs it, not how hard the handshape is.
3.2 The same sign, very different signers
The spread is even sharper per sign. Figure 2 and Table 2 give the eight to ten most signer-degraded signs, comparing the three best-served signers against the three worst-served. The sign "feet" is recognized 86% of the time for the best signers and 3.7% for the worst, a gap of 82.5 points. "orange," "scissors," "cry," and "cowboy" all swing more than 70 points. These per-sign gaps are larger than the overall per-signer spread, which is the clearest possible demonstration that a single accuracy number cannot describe them.
| Best-3 | Worst-3 | Gap | Category | |
|---|---|---|---|---|
| feet | 86.2% | 3.7% | 82.5 pts | face-adjacent |
| orange | 91.7% | 10.0% | 81.7 pts | face-adjacent |
| scissors | 81.7% | 8.7% | 73.0 pts | small handshape diff |
| cry | 78.0% | 5.4% | 72.6 pts | one-handed neutral |
| cowboy | 78.9% | 8.3% | 70.6 pts | face-adjacent |
| fine | 93.5% | 24.6% | 68.9 pts | one-handed neutral |
| awake | 79.3% | 11.1% | 68.1 pts | face-adjacent |
| any | 80.0% | 11.9% | 68.1 pts | face-adjacent |
| minemy | 74.5% | 7.1% | 67.4 pts | small handshape diff |
| owl | 95.0% | 30.0% | 65.0 pts | face-adjacent |
best-3 and worst-3 refer to the three highest- and lowest-accuracy held-out signers overall, not per sign.
3.3 The hypothesis that failed
We expected the degraded signs to share a property — small handshape differences, or face-adjacent positions that landmarks capture poorly — and for a handshape-complexity score to predict the per-signer pattern. It does not. The correlation came back at r-squared 0.008, effectively zero (H3 refuted). The informal categories in Table 2 are real groupings, but they are not the cause. The thing that predicts whether a signer is well served is the signer, not the signs. That points the finger at signing style, recording conditions, and signer representation in the training set, not at any property of the vocabulary.
4. What this implies
The honest unit of report for sign recognition is the per-signer distribution, not the pooled mean. A model that averages 42% while failing a quarter of its signers below 30% is not the same product as a model that averages 42% evenly, and the average cannot tell them apart. Any benchmark that reports a single number is hiding this, and for an accessibility technology the hidden part is exactly the part that matters: who gets left out.
The practical consequence for our own roadmap is concrete. Any future data collection has to capture signer-level metadata and report results per signer, because a model tuned to a pooled average will keep optimizing for the signers it already serves. Closing a 38-point gap is a different engineering problem than raising a mean, and you cannot work on it if your evaluation cannot see it.
5. Limitations
This is 21 signers from one dataset; the absolute spread will differ elsewhere, though the existence of a large spread is consistent with the signer-independent literature [5][6]. It is isolated-sign classification on pre-extracted landmarks, not continuous signing or raw video, and the landmark modality discards facial grammar that carries meaning. The handshape-complexity score in H3 is one operationalization; a different complexity metric could correlate where ours did not, though the effect would have to be large to overturn an r-squared of 0.008. The measured contributions are the 38.6-point spread, the five sub-30% signers, and the refutation of the handshape-complexity explanation.
References
- Google. (2023). Isolated Sign Language Recognition (Google - ASL Signs). Kaggle competition. https://www.kaggle.com/competitions/asl-signs
- Lugaresi, C., Tang, J., Nash, H., et al. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv:1906.08172.
- Mukushev, M., et al. (2022). FluentSigners-50: A signer independent benchmark dataset for sign language processing. PLOS ONE, 17(9). [Signer-independent evaluation.]
- Li, D., Rodriguez, C., Yu, X., & Li, H. (2020). Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison. WACV.
- Joze, H. R. V., & Koller, O. (2019). MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language. BMVC.
- Improving Signer Independent Sign Language Recognition for Low Resource Languages. ACL Anthology. [Signer-independent generalization in low-resource settings.]
- TruPath Labs Research. (2026). The Honest Ceiling on Landmark-Only Isolated-Sign Recognition: A Signer-Holdout Audit. TruPath Labs, TPL-2026-023. [Companion paper establishing the 44.7% stable ceiling.]