Audio-informed multilingual disfluency correction
A three-stage cross-modal knowledge distillation framework that corrects speech disfluencies in Hindi, Bengali, Marathi and English by giving a text model access to what the audio knew.
- Languages
- 4
- Recordings collected
- 2,166
- Base models
- 3B × 2
- Adaptation
- LoRA
समस्या
Spontaneous speech is full of repairs, restarts and filled pauses, and a transcript records the words while discarding the evidence that a repair happened at all. Working from text alone, a correction model has to guess whether a repeated phrase is emphasis or a false start.
Indian languages make the data problem acute: there was no multilingual Indian disfluency corpus to train or evaluate on, so the framework and the dataset had to be built together.
निर्णय
The framework distils acoustic information into a text model in three stages, so the corrector benefits from audio cues at training time without needing the audio encoder at inference.
Adaptation used parameter-efficient fine-tuning — LoRA adapters over Qwen2.5-3B and Llama-3.2-3B — rather than full fine-tuning of either base model. With four languages and a small corpus, adapter-level adaptation was the difference between one experiment and a usable matrix of them.
Alongside it we constructed a multilingual Indian speech disfluency dataset of 2,166 spontaneous recordings across Hindi, Bengali, Marathi and English.
परिणाम
The distilled models improved BLEU and chrF2 over zero-shot baselines across the four languages. The paper is in revision, and the dataset is the reusable part: it is the evaluation set the next model gets measured on.