Curify engineering · October 13, 2026

Building an ASL Video Translator: Experiments, Failures and Evaluation

By the Curify engineering team. Based on our experiment records through October 12, 2026.

Building an ASL video translator taught us how easily plausible English can look like a successful translation. A fluent sentence, a tidy subtitle file and a repeatable response are all useful engineering properties. None establishes that the system understood the signing.

This is a record of what we tested and what changed our decisions. The experimental ASL video-to-English tool and the research models discussed below are separate. Our October results do not establish a production accuracy improvement or a validated replacement model.

1. Check whether the model is reading the answer

One early demo appeared to translate a signed clip well. Inspection showed that the model was reading captions already burned into the frames. We retracted the result as evidence of sign-language understanding.

The evaluation input must not contain the reference answer. That means checking visible text, prompts and how references were written, rather than assuming a video input makes a test independent. An internally produced translation with no recorded annotation method can help diagnose a failure, but it should not become an official accuracy score.

Our practical change was to distinguish diagnostic examples from independent evaluation clips. A successful-looking demo now raises a question: which visual evidence supports the predicted meaning?

2. Inspect the frames the recognizer actually received

Our frame inspector aligns sampled frames, batch boundaries, reference cues and saved predictions on one timeline. It lets us ask whether a cue had enough temporal coverage, whether a transition crossed a batch boundary, and whether the text follows the visible action.

The inspector must match the saved run. Recreating frames with a newer sampler or a different budget would explain a different input. We therefore preserve the sampling configuration alongside predictions. The historical inspection sheet used 96 frames; later duration-dependent sampling is not interchangeable with that record.

3. More frames did not establish better understanding

We tested denser sampling, uniform sampling, and overlapping batches with shared context. In our 12-clip diagnostic study, none of these frame-based variants recovered correct meaning. This is a result for that setup and sample, not proof that temporal modeling cannot help.

A lower word error rate can also conceal a worse result. In one batching comparison, WER improved while character overlap fell and output became shorter. Looking only at one metric would have rewarded the system for saying less.

Likewise, lowering generation temperature can make repeated runs more consistent without making their interpretation correct. Repeatability is valuable for debugging; it is not a substitute for meaning-level evaluation.

4. Compare data and representation under explicit controls

Our October landmark-model experiment held initialization and the 10,320-step compute budget fixed while increasing supervised training data. The frozen 147-clip evaluation did not show a sustained BLEU gain.

The best increase was not statistically significant in our bootstrap comparison. Each condition used one seed; larger datasets received fewer epochs at fixed compute. This does not show that more sign-language pretraining data is unhelpful.

A separate representation experiment held downstream supervision at 2,751 clips. Frozen Uni-Sign features improved BLEU from 1.023 to 3.249 and chrF from 11.121 to 17.049 on frozen-147. This comparison is separate from the same-initialization scaling table above.

That supports representation as a bottleneck in our recipe. It does not isolate architecture from pretraining: the encoder had much more prior training, and a timing-matched check showed that preprocessing explained part of the raw gain. Seeds were not replicated, and the dataset shares signers between training and test. We cannot call these numbers customer-video accuracy.

5. A benchmark gap can contain several different problems

For Uni-Sign, the authors’ released predictions scored 14.50 BLEU-4 on the full 2,349-clip How2Sign test and 9.59 on our harder 128-clip subset. Our inference path scored 5.39 on that same subset. Comparing 14.50 directly with 5.39 would mix subset difficulty with our implementation gap.

We found timing and crop differences worth testing. In 28 of those 128 clips, our original time interval and the officially realigned interval had no overlap. But these observations do not yet assign a causal share of the remaining 4.20 BLEU gap. A controlled path reproduction is still needed, and released predictions are not a fresh checkpoint run.

We credit Uni-Sign and its authors for the research baseline. Our evaluation findings describe our own integration experiments, not a claim that we reproduced every upstream result.

How should ASL-to-English translation accuracy be evaluated?

• Use independent references and audit visible-caption or prompt leakage.
• Freeze the evaluation split, model version and preprocessing; report sample sizes.
• Compare meaning, omissions, repetitions and timing alongside text-overlap metrics.
• Use fluent ASL review for correctness claims. Fluent English alone is not a quality check.
• Test the intended input distribution. A research-dataset gain does not establish performance on uploaded videos.

Our historical 128-clip production-shaped evaluation also used a different model from the deployed recognizer because of evaluation-account access. It must not be presented as a measurement of the exact live configuration. Keeping that distinction visible is part of the evaluation, not a footnote to remove from a chart.

What this means for the current tool

The October research work did not replace the production recognizer. A locally prepared alternative-backend switch is an integration option, not evidence that a better model has shipped. Repetition, long-input behavior and quality on customer footage remain deployment questions.

If you need to translate an ASL video into an English draft, check the current tool’s price and limitations before uploading. Review the result against the signing with a fluent ASL reviewer. Successful export of text or subtitles is not proof that the translation is correct.

For the practical steps, read our ASL video translation guide. For engineering work, the next useful experiment is one that can distinguish competing explanations—not another polished demo.

Sources and research context

The internal source record is Curify’s ASL engineering index, including its caption-leakage correction, frame-inspector notes, temporal ablations and October 12 consolidation of controlled experiments completed through October 11. The underlying private corpus is not a public benchmark and is not redistributed here.

• How2Sign: dataset and documentation — consult its terms before using or redistributing data.
• Uni-Sign: Toward Unified Sign Language Understanding at Scale — upstream research, separate from our integration results.