Skip to content
SearchSubscribe
Expert · Lesson 20 — ASR ground-truth pipelines
E20Expert
Expert · Lesson 20● live

ASR ground-truth pipelines

Building an ASR eval set from scratch when no public benchmark fits. The Parley methodology for sign language and speech recognition.

25 min read · 2 hours applyprereq: Expert 14 (AR research sprints)

Benchmark gap

Parley’s Phase-1 COMPLETE milestone as of April 26 produced the finding that defines the project’s research contribution. The top architecture (frame_transformer) achieves 0.4467 ± 0.0097 on signer-holdout evaluation, roughly 35 percentage points below the random-split Kaggle leaderboard scores for the same dataset. Phase 1 shipped Notebooks 00, 01, and 02. Seven architectures, 3 seeds each, full ladder on Google ISLR.

The 35-point gap isn’t a failure. It’s the finding. Random-split evaluation on ISLR (which is what the Kaggle leaderboard measures) allows training data from a signer to leak into the test set. If Signer 12 appears in both the training and test portions, the model memorizes Signer 12’s style and gets evaluated on it. That isn’t generalization. Signer-holdout evaluation holds out all clips from a subset of signers entirely. The model never sees those signers during training. The 0.4467 measures what the field hasn’t been measuring: generalization to novel signers.

Same gap exists in speech recognition. LibriSpeech clean produces WER numbers that don’t translate to conversational speech in ambient environments. The Parley use case (real-time transcription of conversational speech rendered on an AR HUD for a deaf wearer) is acoustically harder than any standard ASR benchmark. The rest of this lesson covers building the ground-truth pipeline that measures real performance instead of benchmark performance.

Where benchmarks lie

Standard ASR and sign language recognition benchmarks lie in two systematic ways. Train-test data leakage, and test-condition mismatch. Understanding both is the prerequisite to building a ground-truth pipeline that doesn’t.

Benchmark failure modeHow it inflates the numberThe fix
Train-test leakageRandom split allows the same signer/speaker to appear in both training and test. Model memorizes individual style; test measures memorization.Signer/speaker holdout: all clips from held-out individuals stay out of training entirely.
Condition mismatchTest conditions (studio recording, read speech, controlled lighting) are easier than deployment conditions (ambient noise, conversational speech, AR camera placement).Build a deployment-realistic eval tier. Acknowledge the gap between benchmark and deployment conditions explicitly.
Population mismatchBenchmark population (ISLR signers, LibriSpeech readers) doesn’t match deployment population (diverse signing dialects, conversational speakers in ambient environments).Stratify eval by demographic and dialect diversity. Measure per-group accuracy, not just aggregate.

For Parley, all three failure modes are present in the existing benchmarks. The signer-holdout protocol addresses leakage. The tiered eval set (Tier 1 controlled, Tier 2 conversational, Tier 3 AR-hardware-realistic) addresses condition mismatch. The Deaf-community advisor engagement, currently waiting on the Kanban, is the population mismatch fix. Native signers reviewing the evaluation methodology and the published numbers before community distribution.

Building a signer-holdout eval set

Methodology Parley uses

The signer-holdout protocol in Parley’s Notebooks 00-02 works like this. Identify all unique signers in the Google ISLR dataset. Select a holdout group (in Phase-1 work, this is implemented as a full leave-one-signer-out cross-validation approach). Train on all remaining signers. Test exclusively on the held-out signers. Report accuracy on the holdout set as the primary metric.

The 7-architecture ladder from Notebook 02 (BiGRU, TCNDilated, SPOTER, ConformerSmall, SqueezeformerSmall, FrameTransformer, GcnLite) all ran under the same signer-holdout protocol with 3 seeds each. The Q1.5 finding, that 5 of 7 architectures collapse or lottery-ticket under a shared training recipe, raised the sub-question driving Notebook 03. Is the ceiling set by the architecture or by the recipe?

The WLASL Phase B workstream (kicked off in commit 926b7b6 on April 26) adds a secondary dataset. WLASL is larger and more diverse. Running the same signer-holdout protocol on WLASL produces two data points: does the ~35pp gap replicate on a different dataset, and does the specific architecture ranking hold? If both replicate, the finding is solid. If not, the dataset matters, which is itself an important finding.

Building a speech ASR eval set for the Parley use case

The Whisper large-v3 evaluation for the Parley speech transcription component needs a different protocol because the failure modes are different. Key ASR failure modes for Parley deployment:

Disfluency handling.Conversational speech has hesitations, repairs, false starts. Whisper’s cleaned output may normalize these away. “I um, I wanted to” becomes “I wanted to.” For a deaf wearer reading real-time transcription on an AR HUD, normalized output may miss conversational cues. The eval set needs verbatim transcription ground truth, not cleaned transcription.

Proper noun and name accuracy. In deaf/hearing conversation, names and proper nouns are often fingerspelled by the deaf interlocutor and spoken by the hearing interlocutor. ASR error rates on low-frequency names are higher than on common vocabulary. The eval set should include name-heavy utterances.

Rendering latency budget. The eval should measure end-to-end latency from speech to rendered text, not just ASR latency. Rendering adds time. The total must fit within the conversational usability budget. This is hardware-dependent (Tier 3) for the AR glasses rendering step, but measurable now for the ASR plus processing step.

Agent roles in the eval pipeline

CORTEX builds the data pipelines: signlang/data/landmarks.py, the WLASL ingest scripts, the evaluation framework. LEXICON handles the ASR/NLP specialist work: model selection, evaluation protocol design, results interpretation. BEACON runs QA gates before any notebook is published. Methodology check, results check, responsible-findings review. SCRIBE documents the protocol in publishable form.

The Deaf-community advisor (currently waiting on the Kanban) isn’t an agent role. It’s a human role that gates Phase 2 community-facing publication and Phase 3 labeling work. No agent substitutes for a Deaf community advisor reviewing the evaluation methodology and the framing of results.

Three failure patterns

Where ASR and sign language recognition eval sets produce misleading numbers. Each failure is structural and shows up in the published literature.

№ 01

Random split validation masquerading as generalization

claim looks like“A sign language recognition model achieves 0.80 accuracy on a random 80/20 train-test split of the Google ISLR dataset. The team publishes this number. In field testing, accuracy drops to 0.45 on signers not represented in the training data. The 0.80 number measured memorization, not generalization.”
what’s missingSigner-holdout validation as the primary eval metric. Random split validation allows training data from a signer to leak into the test set. If Signer 12 appears in both the training set (80% of their clips) and the test set (20% of their clips), the model memorizes Signer 12's style and gets credit for it in evaluation. Real deployment involves signers the model has never seen.
the moveUse signer-holdout as the primary eval protocol. Hold out all clips from N signers (not N% of all clips). Train on all other signers. Test only on the held-out signers. This measures generalization across signers — which is what the product requires. The Parley Phase-1 finding: 0.4467 on signer-holdout vs. ~0.80 on random split. The gap (~35 percentage points) is the field's quiet failure mode.
№ 02

Benchmark accuracy conflated with product accuracy

claim looks like“A team publishes 0.95 WER on a standard speech recognition benchmark (LibriSpeech clean). They represent this to potential investors as: 'Our system achieves 95% accuracy.' The product use case is real-time conversation transcription in ambient environments. LibriSpeech clean is read speech in a professional recording environment. Field WER for the product use case is 0.68. The 0.95 is not a lie; the conflation is.”
what’s missingExplicit acknowledgment of the test-condition gap between benchmark and deployment. Every accuracy number needs a test condition: which dataset, which speakers, which acoustic environment, which task. 'Achieves 95% accuracy' without these conditions is meaningless. For Parley specifically, any published number must state: ISLR dataset, signer-holdout protocol, controlled video conditions — not ambient AR glasses deployment.
the movePublish numbers with test conditions, always. The format: '[metric] on [dataset] under [protocol] — [N] signers held out, [N] signers in training.' The deployment gap is acknowledged explicitly: 'Real deployment conditions (AR frame-mounted camera, ambient noise, novel signers) are expected to produce different results. Measuring the deployment gap is the goal of Phase 3.' This is more credible than an unqualified 95% — and more useful.
№ 03

Ground-truth labels from a single annotator

claim looks like“A ground-truth dataset for sign language recognition is labeled by a single hearing annotator using text transcripts. The annotator has basic ASL knowledge but is not fluent. Labels for ambiguous signs reflect the annotator's interpretation. Inter-annotator agreement is never measured. The dataset is published and used to train models. The models learn the single annotator's interpretation, not ASL.”
what’s missingMulti-annotator agreement measurement as a quality gate for ground-truth labels. A single annotator's labels are not ground truth — they're one person's interpretation. Ground truth requires either (a) multi-annotator agreement on ambiguous items, or (b) native signer validation of labels. Without this, models trained on the labels learn systematic annotation errors.
the moveFor any ground-truth dataset: measure inter-annotator agreement on at least 10% of the data. For labels where annotators disagree: either (a) seek a third annotation to break the tie, or (b) mark the label as ambiguous and exclude it from training. Publish the inter-annotator agreement score alongside the dataset. For Parley: the deaf-community advisor engagement (currently waiting on kanban) is the native-signer validation gate — label quality for Phase 2 ASR data depends on this advisor being in the loop.

All three produce overconfident numbers. The fix in all three is more rigor in the eval protocol, not in the model. The model is only as good as the eval that measures it.

Generalization check

Run this before publishing any accuracy number from an ASR or recognition system. Takes about 10 minutes. Determines whether the number measures generalization or memorization.

Generalization check
Run a generalization check on this eval result.

Model: [model name and version]
Task: [what the model does]
Result: [the accuracy / WER number you want to publish]

Answer these questions:
1. TRAIN-TEST OVERLAP: Does the test set contain any
   examples from the same source (speaker, signer, document,
   session) as the training set?
   - If yes: the result measures memorization, not
     generalization. Use a holdout protocol.

2. TEST CONDITIONS: Under what conditions was this result
   measured?
   - Dataset name and version
   - Recording conditions (studio / ambient / in-the-wild)
   - Speaker/signer pool (overlap with training? N subjects?)
   - Task type (read speech / conversational / isolated signs)

3. DEPLOYMENT GAP: How different are the test conditions
   from real deployment conditions?
   - What conditions will the model actually encounter?
   - What do you expect the deployment result to be
     relative to the test result? (higher / lower / unknown)

4. COMMUNITY STANDARD: For this specific user group
   (deaf users, patients, etc.) — what accuracy level
   is meaningful? What are the consequences of publishing
   an optimistic number that doesn't hold in deployment?

Output: a 'publish with these conditions' statement OR
a 'don't publish — run a better eval first' recommendation.

Checks for train-test leakage, names the test conditions, estimates the deployment gap, and evaluates community publication standards. Run before publishing any eval result.

Annotated transcript

Building the Parley ASR eval set from scratch and handling the responsible publication of the signer-holdout finding. Two turns. Eval set design, then responsible framing of the 35-point benchmark gap.

Building the Parley ASR eval set and publishing the benchmark gap

Two operator turns, two agent turns. Result: tiered eval protocol designed, signer-holdout finding framed as research contribution, r/deaf hold maintained pending advisor review.

  1. operator

    Parley needs to build an ASR ground-truth eval set for the Whisper model. There's no existing benchmark for the specific use case: real-time transcription of conversational speech for a deaf wearer seeing it rendered on AR glasses. How do I build the eval set from scratch?

  2. agent

    Build the eval set in three tiers, from most to least resource-intensive. The tiers stack — you can publish a useful eval set at Tier 1 while working toward Tier 3.

    Tier 1 — Controlled acoustic conditions, novel speakers.
    Collect: 200-400 speech samples from speakers not in any ASR training set. Conditions: indoor quiet, typical conversation distance, standard microphone. Content: conversational phrases, questions, numbers, names — the vocabulary of typical deaf/hearing interaction.
    Label: verbatim transcription, multi-annotator agreement on ambiguous items.
    This tier measures Whisper's generalization to novel speakers under favorable conditions. It's achievable in a single data collection weekend with 10-15 volunteer speakers.

    Tier 2 — Real-conversation conditions.
    Collect: recordings from real deaf/hearing conversations (with participant consent). Conditions: natural acoustic environment, normal conversation dynamics — interruptions, repairs, disfluencies, overlapping speech.
    Label: verbatim transcription + disfluency annotation (hesitations, repairs). Compare Whisper's cleaned output to verbatim ground truth.
    This tier measures the gap between Whisper's academic WER and its conversational WER — which is the Parley deployment condition.

    Tier 3 — AR-frame acoustic conditions (hardware-dependent).
    Collect: speech captured through a microphone placed at the AR glasses frame position (temporal, slightly forward).
    This tier is hardware-dependent: the exact microphone placement depends on the Maverick AI or equivalent. Flag as Phase 3 work.

    The Parley signer-holdout finding applies here: publish Tier 1 and Tier 2 results with explicit test conditions. The deployment gap (Tier 1 → Tier 3) is the research question — not a secret.

  3. operator

    The Parley Notebook 02 signer-holdout result is 0.4467. The Kaggle leaderboard random-split benchmark is around 0.80. What's the right way to represent the 35-point gap when we publish, given that the deaf community is the eventual user group?

  4. agent

    Publish the gap directly and explain what it means. The gap is the finding, not a failure to hide.

    The framing:

    'Our signer-holdout evaluation (all clips from N signers held out entirely from training) produces 0.4467 accuracy, compared to ~0.80 for random-split evaluation on the same dataset. The 35-point gap measures what the field has generally not measured: how well models generalize to signing styles they have never seen. This gap is the core challenge for any deployed sign language recognition system. Publishing it honestly is more useful to the field — and to the Deaf community — than publishing a random-split number that overstates real-world performance.'

    Two things to be explicit about:

    1. Test conditions. ISLR dataset, controlled video, signer-holdout protocol. Not ambient AR glasses, not real conversations, not Deaf community signers who weren't in ISLR. Those gaps are flagged as Phase 2+ research questions.

    2. What this doesn't mean. A 0.4467 signer-holdout result doesn't mean 'the system works 45% of the time in conversation.' It means: on ISLR-style controlled video, for signing styles the model hasn't seen, the model correctly classifies 45% of isolated signs. Real conversational ASR performance is a different measurement that hasn't been done yet.

    The r/deaf posting hold is correct: don't post these numbers to r/deaf until (a) a Deaf-community advisor has reviewed the framing, and (b) Notebook 03 is published with more complete methodology. The numbers are honest; the framing requires community validation before community posting.

Ground-truth pipeline checklist

Full checklist for building a ground-truth eval pipeline. Each section is a prerequisite for publishing results.

GROUND-TRUTH EVAL PIPELINE CHECKLIST

DATASET SELECTION
[ ] Dataset is identified and documented
    - Name, version, size, source
[ ] Number of unique individuals (signers / speakers) counted
[ ] Demographic metadata available? (if yes, plan stratification)
[ ] License confirmed for research and publication use

HOLDOUT PROTOCOL
[ ] Holdout is individual-based, not clip-based
    (all clips from held-out signers stay out of training)
[ ] Holdout size: >= 20% of unique individuals
[ ] Holdout selected: [random / stratified by demographic]
[ ] Document: which individuals are in holdout (log the IDs)
[ ] No holdout individual's data appears in training set
    (verify: no train-test leakage)

ANNOTATION QUALITY
[ ] Ground-truth labels reviewed by domain expert
    (for sign language: fluent signer or Deaf advisor)
[ ] Inter-annotator agreement measured on >= 10% of data
    - Agreement score: [report value]
    - Ambiguous items: flagged and excluded or adjudicated
[ ] Label format documented: [what does each label represent?]

EVAL METRICS
[ ] Primary metric defined before training
    (for sign language: accuracy on signer-holdout)
    (for speech: WER on speaker-holdout)
[ ] Secondary metric: random-split (for comparison only)
[ ] Both metrics measured and gap reported
[ ] Per-group breakdown if demographic metadata exists

TEST CONDITIONS
[ ] Recording conditions documented:
    - Equipment, environment, instructions to participants
[ ] Test conditions described in publication:
    - 'Controlled video, ISLR dataset, signer-holdout protocol'
    — not just the number
[ ] Deployment gap acknowledged:
    - How do test conditions differ from deployment?

PUBLICATION GATE
[ ] Generalization check passed
[ ] Responsible findings statement written
[ ] For community-facing publication:
    [ ] Domain advisor reviewed framing
    [ ] r/deaf (or equivalent sensitive community):
        on HOLD until advisor review + methodology complete
[ ] Results do not conflate benchmark accuracy
    with deployment capability

Prompt kit

Three prompts. Generalization check, signer-holdout eval protocol design, responsible findings statement. I keep mine in CORTEX’s or LEXICON’s agent file, or in the research notebook CLAUDE.md.

Generalization check
Run a generalization check on this eval result.

Model: [model name and version]
Task: [what the model does]
Result: [the accuracy / WER number you want to publish]

Answer these questions:
1. TRAIN-TEST OVERLAP: Does the test set contain any
   examples from the same source (speaker, signer, document,
   session) as the training set?
   - If yes: the result measures memorization, not
     generalization. Use a holdout protocol.

2. TEST CONDITIONS: Under what conditions was this result
   measured?
   - Dataset name and version
   - Recording conditions (studio / ambient / in-the-wild)
   - Speaker/signer pool (overlap with training? N subjects?)
   - Task type (read speech / conversational / isolated signs)

3. DEPLOYMENT GAP: How different are the test conditions
   from real deployment conditions?
   - What conditions will the model actually encounter?
   - What do you expect the deployment result to be
     relative to the test result? (higher / lower / unknown)

4. COMMUNITY STANDARD: For this specific user group
   (deaf users, patients, etc.) — what accuracy level
   is meaningful? What are the consequences of publishing
   an optimistic number that doesn't hold in deployment?

Output: a 'publish with these conditions' statement OR
a 'don't publish — run a better eval first' recommendation.
Design a signer-holdout eval protocol
Design a signer-holdout evaluation protocol for this
recognition task.

Dataset: [name, size, number of signers/speakers]
Model: [what you're evaluating]
Task: [isolated signs / continuous signing / speech / other]

The protocol must specify:
1. HOLDOUT SIZE: How many signers/speakers to hold out?
   - Minimum: 20% of unique individuals
   - Recommended: the N signers with the most diverse
     signing styles (if diversity metadata exists)

2. HOLDOUT SELECTION: How to select the holdout set?
   - Random selection? Stratified by signer demographics?
   - If metadata allows: stratify by age, dialect, fluency
     level to maximize diversity in the holdout

3. TRAIN/HOLDOUT SPLIT: All clips from each holdout
   signer go to holdout. No clip-level splitting of
   any individual.

4. REPORTING: Report the following metrics:
   - Signer-holdout accuracy / WER (the primary metric)
   - Random-split accuracy / WER (for comparison)
   - The gap between the two (this is the generalization gap)
   - N signers in holdout, N signers in training

Output: the eval protocol as a numbered procedure +
the reporting template.
Write a responsible findings statement
Write a responsible findings statement for this
evaluation result.

Finding: [the accuracy or WER number]
Protocol: [how you measured it — dataset, holdout protocol]
Deployment use case: [what the model will actually be used for]
Gap between test and deployment: [known / estimated / unknown]
User community: [who the eventual users are]

The statement must include:
1. THE NUMBER with its exact test conditions
2. WHAT IT MEANS: what does this number measure?
3. WHAT IT DOESN'T MEAN: what does this number not claim?
4. THE DEPLOYMENT GAP: how different is test from deployment?
5. NEXT STEPS: what needs to be measured to narrow the gap?

Tone: direct, honest, useful. Not hedged to uselessness;
not overclaiming. A researcher who reads this should be
able to replicate the eval and know exactly what they're
measuring.

If this will be published to a specific community
(accessibility users, medical users, etc.): add a
section on what this finding means for that community's
use of the technology.

Apply this — build one honest eval pipeline

~2-hour exercise. Generalization check, holdout design, annotation audit, responsible statement. Publish the real number — with its real conditions.

Your first signer-holdout eval pipeline

Each step takes 20-30 minutes. Progress saves automatically.

0/5
  1. 01Run the generalization check on your most recently published accuracy number. Does it measure generalization or memorization?The question 'does the test set contain examples from the same source as the training set?' is the core check. If the answer is yes, the result is a memorization test, not a generalization test.
  2. 02Design a signer-holdout or speaker-holdout eval protocol for your recognition task. Define the holdout size, selection, and reporting requirements.If you're working with Google ISLR: define which signers go in the holdout. The Parley standard is holding out all clips from a subset of signers, not a random % of all clips.
  3. 03Measure the gap between your random-split accuracy and your holdout accuracy. Publish both numbers, with the gap named explicitly.The gap is the finding. A 35-point gap says something important about the field. A 5-point gap says something different. Either way, the gap should be named.
  4. 04Write a responsible findings statement for your primary eval result. Name the test conditions, the deployment gap, and what's being measured next.For Parley: the signer-holdout finding (0.4467 on frame_transformer) is publication-ready with appropriate conditions. The r/deaf hold is until the community advisor reviews the framing — write the statement now, hold the community post.
  5. 05Define the inter-annotator agreement protocol for your ground-truth labels. Measure agreement on 10% of your eval set.If you've never measured inter-annotator agreement on your eval set, you don't know if your 'ground truth' is actually agreed-upon. The 10% sample takes a few hours and tells you whether the labels are reliable.
Expert tier · what's next

After this lesson