Sign language translation (SLT) systems face a persistent data bottleneck: paired sign-video and text corpora remain scarce, and real-world vocabularies exhibit heavy-tailed distributions. A new study from researchers Bianco, Pedro Dal, Reinhold, Jean Paul Nunes, Stanchi, Oscar, Quiroga, Facundo, Ronchetti, Franco, and Corrêa, Ulisses Brisolara explores a target-side augmentation strategy that leverages large language models (LLMs) to generate paraphrase variants of the reference spoken-language sentence while keeping the sign input unchanged. The work, published on arXiv, is the first to examine LLM-generated target-side paraphrases as an augmentation mechanism for SLT and the first to apply an LLM-as-a-Judge evaluation protocol to the field.
Data Scarcity in Sign Language Translation
According to the paper, SLT remains constrained by limited paired corpora and heavy-tailed vocabularies. Most existing approaches rely on a single reference translation per sign video, which can lead to overfitting to specific phrasing. The researchers propose to use GPT-4o to produce semantically faithful variants of the training targets. They train a Signformer-style pose-based Transformer under a two-stage schedule: pre-training on the augmented corpus followed by fine-tuning on the original references.
Experimental Setup and Datasets
The study evaluates the strategy on three datasets that span complementary challenges:
- PHOENIX14T (German Sign Language): a real-world corpus with moderate lexical diversity.
- Greek Sign Language Dataset (GSL): highly controlled, repetitive recordings.
- LSA-T (Argentinian Sign Language): a naturalistic corpus with a large vocabulary and severe long-tail sparsity.
This range allowed the researchers to characterize precisely when and why target-side augmentation is beneficial.
Results: BLEU-4 Improvement on PHOENIX14T
On PHOENIX14T, augmentation improved BLEU-4 from 9.56 to 10.33, demonstrating that paraphrastic exposure helps the decoder generalize beyond memorized reference phrasing. The researchers reported that this complementary evaluation revealed gains in semantic fidelity that lexical overlap metrics understate.
| Dataset | Language | BLEU-4 Before Augmentation | BLEU-4 After Augmentation |
|---|---|---|---|
| PHOENIX14T | German Sign Language | 9.56 | 10.33 |
Limitations on Controlled and Sparse Datasets
On the near-saturated GSL baseline and the extremely sparse LSA-T setting, the approach showed limits. The paper notes that in both cases, single-reference lexical overlap metrics are insufficient to capture the full picture, motivating the complementary semantic evaluation using an LLM-as-a-Judge protocol. This protocol, according to the authors, is a novel contribution to SLT evaluation.
Implications for Enterprise AI
While the study focuses on sign language, the technique of LLM-generated paraphrase augmentation has broader implications for any natural language processing task with limited paired data. For enterprise technology leaders, this research highlights how generative AI can address data scarcity in specialized domains. The use of GPT-4o as a paraphrase generator and the two-stage training schedule (pre-train on augmented data, fine-tune on original) offers a replicable template for improving model generalization when reference corpora are small. The LLM-as-a-Judge evaluation also provides a more holistic quality metric beyond traditional n-gram overlap, which could be adopted in other machine translation and content generation pipelines.