Understanding how individual words in a style caption influence the acoustic output of a text-to-speech system is critical for diagnosing failure modes and improving controllability in expressive TTS. A new research paper, "How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech," published on arXiv and authored by Mathur, Nityanand; Sayed, Hamees; Madha, Wasim; Singh, Apoorv; Khurana, Sameer; Mandloi, Akshat; and Kamath, Sudarshan, proposes a method to address this gap by adapting the DAAM (Diffusion Attribution and Attention Mapping) framework to the speech domain for the first time.
The Challenge of Controlling Expressive TTS
Style-captioned TTS systems use natural language descriptions—such as "speak softly" or "enthusiastic tone"—to control voice characteristics. However, the relationship between caption tokens and acoustic output has been opaque. According to the paper, understanding this is essential for "diagnosing failure modes and improving controllability." The researchers apply their method to CapSpeech-TTS, a style-captioned speech diffusion model. Their technique extracts per-token heatmaps across 25 layers and 24 ODE steps, enabling fine-grained attribution of the model's attention to specific caption words.
Methodology: Adapting DAAM to Speech
The researchers analyzed 3,600 (style caption, text transcript) combinations, comprising 120 style captions conditioning the generation of 30 text transcripts each. This large-scale analysis revealed how caption tokens shape the resulting waveform. The method modifies DAAM, originally designed for image generation, to handle speech diffusion models. The heatmaps visualize the cross-attention weights between style caption tokens and the acoustic features of the generated speech.
Key Findings
The study reports four major results:
- Style tokens have lower temporal variance than content/function tokens, confirming that they provide global conditioning across the utterance.
- Style attention correlates with fundamental frequency (F0) and energy, two key prosodic features that convey speaking style.
- Style conditioning peaks in early ODE steps and deep layers of the diffusion model, suggesting that the network prioritizes style at specific stages of generation.
- Attention entropy reaches its minimum at layer 17, co-occurring with the style importance peak, indicating "maximal network selectivity at the most style-critical stage."
These findings are summarized in the table below:
| Finding | Description |
|---|---|
| Temporal variance | Style tokens have lower variance than content/function tokens, confirming global conditioning. |
| Acoustic correlation | Style attention correlates with F0 and energy. |
| Timing of style influence | Conditioning peaks in early steps and deep layers. |
| Network selectivity | Attention entropy minimum at layer 17 coincides with style importance peak. |
Significance for AI Research
This is the first study of how natural language influences cross-attention in speech diffusion models, according to the paper. By providing a method to attribute model behavior to specific caption words, the work offers a diagnostic tool for improving controllability and interpretability in expressive TTS systems. The researchers note that understanding these mechanisms can help in designing more reliable and nuanced voice synthesis for applications ranging from virtual assistants to accessibility tools. The findings also open avenues for further research into the interplay between textual conditioning and prosody generation in generative audio models.