Pranav Sharma

AI Research Engineer

I build AI systems that have to hold up on real, messy data, and the evaluation that says whether they do. Most recently full-duplex speech models for Indic languages — training, data curation at scale, and benchmarks, end to end.

github
email
linkedin

Work

Human-1
Moshi / Mimi

Human-1: a full-duplex conversational system in Hindi

An open-source full-duplex speech-to-speech dialogue system for Hindi, adapting the Moshi/Mimi architecture to real-time turn-taking, overlapping speech, and backchannels.

Problem
Full-duplex architectures were built and evaluated on English. Hindi conversation carries different turn-taking behaviour, heavy code-mixing, and no tokenizer or stereo conversational corpus at the scale the architecture needs.
Decision
Adapted Moshi/Mimi rather than cascading ASR → LLM → TTS, built a 32k-token Hindi SentencePiece tokenizer for the text stream, and trained on real-world conversations rather than synthesised dialogue.
Result
naturalness 4.10/5.00 · 66.9% of generations tie with human speech · 26,000 h stereo corpus · 8× H100

Read the write-up →

Duplex eval
Full-Duplex-Bench

Tracing full-duplex failure modes back to their cause

Checkpoint-level evaluation for a fine-tuned English full-duplex model: surfacing the degradation patterns that aggregate metrics hide, and tying each one back to the training data or fine-tuning strategy that produced it.

Problem
A duplex model can score well on transcription metrics while being unusable in conversation — volume drifting, noise building up, hallucination loops, backchannelling at the wrong moments. None of that appears in a WER number.
Decision
Evaluated per checkpoint rather than per release, benchmarking overlap handling and turn-taking with Full-Duplex-Bench against the base Moshi model, and treating each named failure mode as something to attribute rather than merely report.
Result
4 degradation patterns isolated and attributed · benchmarked against base Moshi · eval signal converted into targeted fixes

Read the write-up →

Hear to Correct
Cross-modal KD

Audio-informed multilingual disfluency correction

A three-stage cross-modal knowledge distillation framework that corrects speech disfluencies in Hindi, Bengali, Marathi and English by giving a text model access to what the audio knew.

Problem
Disfluency correction from transcripts alone throws away the acoustic evidence that marks a repair — hesitation, prosody, restart. Text-only models therefore delete real words and keep false starts, and the problem compounds across Indic languages with little labelled data.
Decision
Distilled acoustic knowledge into text models across three stages, then fine-tuned with LoRA adapters on Qwen2.5-3B and Llama-3.2-3B rather than full fine-tuning, keeping the approach affordable per language.
Result
2,166 spontaneous recordings collected · BLEU and chrF2 improved over zero-shot baselines · 4 languages

Read the write-up →

Indic ASR
Whisper / Wav2Vec2

Cutting word error on low-resource Indic ASR

Error analysis, data curation and multi-stage fine-tuning of Whisper, Wav2Vec2 and comparable models, moving word error rate on multilingual holdout sets rather than on a leaderboard split.

Problem
Off-the-shelf ASR degrades sharply on Indic languages and Indian-accented English, and the usual response — more data, more epochs — spends compute without addressing what the model is actually getting wrong.
Decision
Led with error analysis and used it to drive data curation, then staged the fine-tuning rather than training once on everything, with distributed PyTorch training tuned for multi-GPU throughput.
Result
25—50% relative WER reduction across multilingual holdout sets · 100k+ hours curated · vLLM inference pipelines

Read the write-up →

Who spoke
diarization

An evaluation set for diarization that does not flatter the model

A speaker diarization evaluation dataset built end to end for in-person and online multi-party calls with up to 10 speakers — segmentation, annotation protocol and tooling, quality control, and the model evaluation it was built to support.

Problem
Diarization scores well on clean, turn-taking audio and falls apart on the calls people actually have: speakers talking over each other, cross-talk bleeding between microphones, and one-word backchannels too short for most systems to attribute.
Decision
Built the evaluation set around those cases rather than sampling audio at random — defining the boundary generation process, writing the annotation protocol and the tooling annotators worked in, and running quality control before any model touched it.
Result
up to 10 speakers per call · in-person and online conditions · overlap, cross-talk and backchannels labelled · annotation protocol and tooling built in-house

Read the write-up →

TTS bench
human eval

Benchmarking 15+ TTS systems on Indic speech, by ear

A human evaluation framework for text-to-speech across Indic languages and Indian-accented English — what raters score, how the sampling makes systems comparable, and a per-language ranking the team can act on.

Problem
TTS vendors publish English-centric numbers. On Indic languages the differences that matter — pronunciation of loanwords, code-mixed sentences, prosody over long-form text — are not captured by any automatic metric, and a single averaged opinion score hides which language a system is actually weak in.
Decision
Specified what raters score rather than asking for a general impression, built the rater workflow and sampling so results are comparable across 15+ commercial and open systems, and reported per-language rankings instead of one number.
Result
15+ commercial and open systems benchmarked · Indic languages and Indian-accented English · per-language ranking used for vendor selection

Read the write-up →

Publications

preprint
arXiv
co-author

Sharma, P., et al. Human-1: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations. arXiv:2604.23295, 2026.

arxiv.org/abs/2604.23295 · doi · open-source release

in revision
4 languages

Sharma, P., et al. Hear to Correct: Audio-Informed Multilingual Disfluency Correction via Cross-Modal Knowledge Distillation. In revision.

Hindi · Bengali · Marathi · English

Contact

Delhi, India · open to research roles and collaboration

github
linkedin
resume