Code-switching—the alternation between two or more languages within a single conversation—poses a persistent challenge for automatic speech recognition (ASR) systems. Limited availability of high-quality code-switch (CS) text-speech pairs for training has kept error rates high, particularly for languages with frequent mixing such as Mandarin and English. According to a preprint on arXiv, researchers have introduced a novel framework that improves CS ASR by guiding synthetic speech generation toward better code-switching fidelity.
The proposed method, detailed in the paper "Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech," employs a code-mixing guided preference-learning framework. Instead of optimizing only reconstruction fidelity as in conventional text-to-speech (TTS) approaches, the framework explicitly enforces language-boundary consistency using the Code Mixing Index (CMI). The CMI measures the degree of language mixing in a speech segment, and the preference-learning step steers the TTS model to generate synthetic speech that aligns more closely with genuine code-switching patterns.
The Challenge of Code-Switching ASR
Standard ASR models are often trained on monolingual or well-segmented multilingual data. Code-switching introduces rapid language shifts that confuse acoustic and language models. As the paper notes, existing CS TTS approaches primarily optimize reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. The limited availability of natural CS speech corpora compounds the problem, making data augmentation a critical strategy.
A Preference-Learning Framework with CMI
The authors propose a preference-learning pipeline that leverages the Code Mixing Index (CMI) as a supervisory signal. The framework first generates synthetic CS speech using a TTS model, then evaluates each sample based on its CMI score. Samples with CMI values that better match natural CS distributions are preferred during training, effectively teaching the TTS model to produce more realistic code-switched utterances. This approach avoids the need for manual annotation of language boundaries.
Experimental Results on SEAME Corpus
Experiments were conducted on the SEAME (South East Asia Mandarin-English) conversational corpus, a widely used benchmark for code-switching ASR. The researchers fine-tuned OpenAI's Whisper Large model using synthetic speech generated by the proposed method. The results are reported as Mixed Error Rate (MER), which combines word and character errors across both languages.
| Test Set | Baseline MER (Whisper Large) | Fine-tuned MER (Proposed Method) |
|---|---|---|
| DevMAN | 12.1% | 8.9% |
| DevSGE | 17.8% | 14.2% |
According to the paper, the proposed approach reduces MER from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively. This represents a relative improvement of 26% on DevMAN and 20% on DevSGE.
Implications for Enterprise Speech Technology
For enterprise technology leaders, higher accuracy in code-switching ASR translates directly into better performance for multilingual customer service, voice assistants, and transcription services in markets where language mixing is common. While the work focuses on Mandarin-English, the framework's reliance on the CMI metric means it could be adapted to other language pairs. The use of synthetic data augmentation also reduces the dependence on costly manual collection of CS speech.
Further research and benchmarking on additional corpora will be needed to validate generalizability, but the initial results on SEAME demonstrate a clear path to improving ASR systems where code-switching is prevalent.
