Skip to content
SearchSubscribe
TPL-2026-024·preprint·2026-05-31

Signer-Dialect Robustness in Landmark-Only Sign Recognition: A 21-Fold Leave-One-Signer-Out Study

CVsign-languagefairnesssigner-holdoutevaluation

Abstract

A single cross-signer accuracy number hides who a sign-recognition model actually serves. We run a 21-fold leave-one-signer-out study on the Google ISLR 250-sign dataset, holding out each of the 21 signers in turn and reporting the full per-signer distribution rather than a pooled average. Top-1 accuracy ranges from 25.6% on the worst-served signer to 64.2% on the best — a 38.6 percentage-point spread — with a mean of 41.7% and a per-signer standard deviation of 11.4 points, more than ten times the seed-to-seed deviation of the same architecture. Five of twenty-one signers fall below 30%. We pre-registered three hypotheses: that the spread would exceed 25 points (confirmed, 38.6), that the worst signer would fall below 30% (confirmed, 25.6%), and that handshape complexity would predict per-signer degradation (refuted, r-squared 0.008). The degradation is signer-driven, not sign-feature-driven: the same sign swings 60 to 82 points between the best and worst signers. We argue the per-signer distribution, not the pooled mean, is the honest unit of report for sign recognition, and that signer-level metadata must be captured in any future data collection.

The 38-point signer fairness gap: per-signer top-1 ranges from 25.6% to 64.2%; five of 21 signers fall below 30%; the per-signer standard deviation dwarfs the seed-to-seed deviation; and a pre-registered hypothesis that handshape complexity drives degradation was refuted, so the degradation is signer-driven, not sign-feature-driven.
Visual summary of the per-signer fairness gap. Synthesized from the TruPath Labs sign-language research substrate.

1. Introduction

A companion study established that the honest ceiling for landmark-only isolated-sign recognition, measured on held-out signers, is about 45% top-1 rather than the 80 to 90% reported on signer-mixed splits [7]. That number is a pooled average across a held-out test set. This study asks the question the average hides: when the model meets a signer it has never seen, how much does performance depend on which signer it is?

The answer is: enormously. Running the model against each of the 21 signers in turn, top-1 accuracy ranges from 25.6% on the worst-served signer to 64.2% on the best. The mean of 41.7% describes almost none of them. For a technology whose entire purpose is to work for deaf people in the wild, a 38-point spread across signers is not a footnote to the accuracy number. It is the more important number.

We pre-registered three hypotheses before running the folds, and the one that failed is the most informative. We expected handshape complexity to predict which signers the model struggles with. It does not. The variable that predicts whether a sign is recognized is who is signing it, not how hard the handshape is.

2. Methods

2.1 Leave-one-signer-out protocol

The Google ISLR dataset [1] contains 250 signs from 21 signers, provided as per-frame MediaPipe landmarks [2] rather than video. We run 21 folds: in each fold, one signer is held out entirely for test, a second is held out for validation, and the model trains on the remaining 19. This is the strict form of signer-independent evaluation that fixed splits approximate but rarely guarantee [3].

The architecture is fixed: the frame transformer that was the best stable model in the companion ceiling study, trained with the same recipe in every fold. Holding the model constant isolates the signer as the variable. We pre-registered three hypotheses in the notebook contract before any fold ran: H1, that the per-signer spread would exceed 25 points; H2, that the worst signer would fall below 30%; and H3, that a handshape-complexity score would correlate with per-signer accuracy at r-squared of at least 0.30.

3. Results

3.1 The per-signer distribution

Figure 1 shows every fold. The distribution is wide and roughly continuous from 25.6% to 64.2%, a spread of 38.6 points (H1 confirmed). The worst-served signer sits at 25.6% (H2 confirmed). Five of the 21 signers fall below 30%, meaning the model recognizes fewer than three signs in ten for nearly a quarter of the people it was tested against. The pooled mean is 41.7%, but the per-signer standard deviation is 11.4 points — more than ten times the roughly one-point seed-to-seed deviation the same architecture shows when the signers are fixed. The variance lives in the population, not in the training noise.

Figure 10%10%20%30%40%50%60%70%TOP-1 ACCURACY (HELD-OUT SIGNER)26%127%228%329%430%530%633%734%834%941%1043%1146%1247%1348%1448%1552%1653%1753%1854%1956%2064%21
Figure 1 — Per-signer top-1 accuracy across all 21 leave-one-signer-out folds, sorted ascending. Each bar is one held-out signer (anonymized, rank order). Amber bars are the five signers below 30%; green bars are the six at or above 50%. The spread from worst (25.6%) to best (64.2%) is 38.6 points. The pooled mean of 41.7% describes almost none of the individual signers well.
Value
Folds (held-out signers)21
Mean top-141.7%
Per-signer std11.4 pts
Worst / best signer25.6% / 64.2%
Spread (worst to best)38.6 pts
Signers below 30%5 of 21
Signers at or above 50%6 of 21
H1 — spread > 25 ptsCONFIRMED (38.6)
H2 — worst signer < 30%CONFIRMED (25.6%)
H3 — handshape complexity predicts degradationREFUTED (r² = 0.008)
Table 1 — Fold-level summary across 21 leave-one-signer-out folds (frame transformer, the best stable architecture from the companion ceiling study), and the three pre-registered hypotheses with their outcomes. The per-signer standard deviation (11.4 points) is more than 10x the seed-to-seed deviation of the same architecture (~1.0 point), which means the variance is in the population of signers, not in the training noise.

Hypotheses were registered in the notebook contract before the folds were run. H3's refutation is the load-bearing finding: the thing that predicts whether a sign is recognized is who signs it, not how hard the handshape is.

3.2 The same sign, very different signers

The spread is even sharper per sign. Figure 2 and Table 2 give the eight to ten most signer-degraded signs, comparing the three best-served signers against the three worst-served. The sign "feet" is recognized 86% of the time for the best signers and 3.7% for the worst, a gap of 82.5 points. "orange," "scissors," "cry," and "cowboy" all swing more than 70 points. These per-sign gaps are larger than the overall per-signer spread, which is the clearest possible demonstration that a single accuracy number cannot describe them.

Figure 20%25%50%75%100%ACCURACY (BEST-3 VS WORST-3 SIGNERS)86%4%feet92%10%orange82%9%scissors78%5%cry79%8%cowboy94%25%fine79%11%awake80%12%anybest-3worst-3
Figure 2 — The eight most signer-degraded signs. For each sign, accuracy averaged over the three best-served signers (blue) versus the three worst-served signers (amber). The same sign swings 60 to 82 points depending only on who is signing it. 'feet' is recognized 86% of the time for the best signers and 3.7% for the worst. These gaps are larger than the overall per-signer spread, which is why a pooled accuracy number cannot describe them.
Best-3Worst-3GapCategory
feet86.2%3.7%82.5 ptsface-adjacent
orange91.7%10.0%81.7 ptsface-adjacent
scissors81.7%8.7%73.0 ptssmall handshape diff
cry78.0%5.4%72.6 ptsone-handed neutral
cowboy78.9%8.3%70.6 ptsface-adjacent
fine93.5%24.6%68.9 ptsone-handed neutral
awake79.3%11.1%68.1 ptsface-adjacent
any80.0%11.9%68.1 ptsface-adjacent
minemy74.5%7.1%67.4 ptssmall handshape diff
owl95.0%30.0%65.0 ptsface-adjacent
Table 2 — The ten most signer-degraded signs, with accuracy for the three best-served and three worst-served signers, the gap between them, and the informal category each sign falls into. The categories (face-adjacent signs, small handshape differences, one-handed neutral-space signs) are observations, not the cause: hypothesis H3 found handshape complexity does not predict degradation. The pattern that holds is signer identity.

best-3 and worst-3 refer to the three highest- and lowest-accuracy held-out signers overall, not per sign.

3.3 The hypothesis that failed

We expected the degraded signs to share a property — small handshape differences, or face-adjacent positions that landmarks capture poorly — and for a handshape-complexity score to predict the per-signer pattern. It does not. The correlation came back at r-squared 0.008, effectively zero (H3 refuted). The informal categories in Table 2 are real groupings, but they are not the cause. The thing that predicts whether a signer is well served is the signer, not the signs. That points the finger at signing style, recording conditions, and signer representation in the training set, not at any property of the vocabulary.

4. What this implies

The honest unit of report for sign recognition is the per-signer distribution, not the pooled mean. A model that averages 42% while failing a quarter of its signers below 30% is not the same product as a model that averages 42% evenly, and the average cannot tell them apart. Any benchmark that reports a single number is hiding this, and for an accessibility technology the hidden part is exactly the part that matters: who gets left out.

The practical consequence for our own roadmap is concrete. Any future data collection has to capture signer-level metadata and report results per signer, because a model tuned to a pooled average will keep optimizing for the signers it already serves. Closing a 38-point gap is a different engineering problem than raising a mean, and you cannot work on it if your evaluation cannot see it.

5. Limitations

This is 21 signers from one dataset; the absolute spread will differ elsewhere, though the existence of a large spread is consistent with the signer-independent literature [5][6]. It is isolated-sign classification on pre-extracted landmarks, not continuous signing or raw video, and the landmark modality discards facial grammar that carries meaning. The handshape-complexity score in H3 is one operationalization; a different complexity metric could correlate where ours did not, though the effect would have to be large to overturn an r-squared of 0.008. The measured contributions are the 38.6-point spread, the five sub-30% signers, and the refutation of the handshape-complexity explanation.

References

  1. Google. (2023). Isolated Sign Language Recognition (Google - ASL Signs). Kaggle competition. https://www.kaggle.com/competitions/asl-signs
  2. Lugaresi, C., Tang, J., Nash, H., et al. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv:1906.08172.
  3. Mukushev, M., et al. (2022). FluentSigners-50: A signer independent benchmark dataset for sign language processing. PLOS ONE, 17(9). [Signer-independent evaluation.]
  4. Li, D., Rodriguez, C., Yu, X., & Li, H. (2020). Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison. WACV.
  5. Joze, H. R. V., & Koller, O. (2019). MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language. BMVC.
  6. Improving Signer Independent Sign Language Recognition for Low Resource Languages. ACL Anthology. [Signer-independent generalization in low-resource settings.]
  7. TruPath Labs Research. (2026). The Honest Ceiling on Landmark-Only Isolated-Sign Recognition: A Signer-Holdout Audit. TruPath Labs, TPL-2026-023. [Companion paper establishing the 44.7% stable ceiling.]