Urdu Sukhan ASR: Accurate Urdu Speech Recognition at Half the Size of Whisper large-v3

What we built

Urdu Sukhan ASR v1 is an Urdu speech recognition model that also handles English and Urdu–English code-switching. It is a full fine-tune of OpenAI’s Whisper large-v3-turbo, so it keeps the turbo model’s size and speed: 809 million parameters, about half the 1.55 billion of Whisper large-v3. It was trained on about 110 hours of transcribed audio per pass: public Urdu corpora, two audiobooks read by two narrators with transcripts proofread by the author, Urdu–English clips we assembled and no-speech audio.

Against the base model with the same decoding, it makes fewer word errors on all seven held-out Urdu test sets, 39% fewer on average. This report describes the data, the six training rounds (two of which failed on quiet audio), how the released checkpoint was chosen, why Whisper models write a word on silence, speed on four machines in three runtimes, and a compression experiment that was not good enough to release. The full paper, with every table, is the PDF at the end of this page.

Against the base model

Words wrong (word error rate, WER) for the base model and ours, both decoded greedily with Hugging Face Transformers after the same text normalisation. None of these sets was used in training. The largest gains are on conversational speech from many speakers (IndicVoices, 59% fewer word errors), the single speaker of nzg73 (67%), one- or two-word replies (73%) and clips that switch from Urdu to English (63%). FLEURS, read Wikipedia text with many names and technical words, improves least among the public sets (23%), and English stays at 4.9%.

Examples

Error rates hide what the errors look like. We ran 47 public test clips with speech, and two without, through both models as an application would (whisper.cpp, q8_0, beam 5, temperature fallback). Over these clips, our model was better on 27, worse on 4 and the same on 16 (19.1% against 37.8% words wrong).

The table shows short clips that we picked by hand to show where the fine-tune helps most, from the top of a ranking of all clips by words fixed, and one of the 4 clips where it did worse; the model card on Hugging Face has them with their audio. The base model turns short replies into other words, even “music” for a spoken “hello” and “Allah, Allah” for “separately”. The largest gains were on clips that switch from Urdu to English, too long for the table: on one, the base model wrote four English words for a 28-word clip; on another, it translated the Urdu half into English instead of writing it down. Ours transcribed both, with two wrong words in all. It still loses on some word spacing, as the last example shows. On silence, allowing an empty result only helps our model: the base model still writes a word.

The clips and their reference transcripts come from public test sets: IndicVoices and UrduSpeech (AI4Bharat, CC BY 4.0), Common Voice (Mozilla, CC0) and FLEURS (Google, CC BY 4.0). The quiet, phone and Urdu-then-English clips are Common Voice and FLEURS clips that we turned down, passed through a phone filter or joined. The hiss is generated, and none of the examples comes from the two audiobooks.

Training data

The released model was trained on 59,165 clips in one pass (about 110 hours), sampled from 87,571 cleaned clips (about 151 hours). Every source except the two audiobooks is public.

The UAT dataset has no public licence and its transcripts were made by a machine, so we used only its audio, transcribed by our own model, by agreement with its developers. The nzg73 dataset by Muhammad Noman is used under its own licence, which allows training speech models with credit to its author and forbids misuse such as deepfakes. We reviewed and left out four further sets: a text-to-speech set with a non-commercial licence, a synthetic text-to-speech corpus, a podcast collection without a licence and an emotion corpus without transcripts.

Before the last round, a clip was removed when it was shorter than 0.5 or longer than 30 seconds, had no text, held more letters per second than speech can (or too few for its length), or was an Urdu clip whose text was less than 60% Urdu script. The last round also added 2,800 no-speech clips with empty transcripts (noise, instrumental music, digital silence, microphone hiss and mains hum) and 782 clips that join an Urdu clip and an English one, with the English part written in English letters.

The audiobooks and the proofreading loop

Two audiobooks, each read by a different narrator, were used: an Urdu translation of a self-help book (3.97 hours) and an Urdu novel (about 1.2 hours). They were cut at pauses into clips of at most 25 seconds and drafted by the base model. The author corrected the drafts in a browser tool built for the purpose, which marks uncertain words, plays any word on click and shows the drafts’ error rate on the clips finished so far. After every five finished clips, the corrections became rules for the remaining drafts: 765 rules by the end, each checked against every draft it would touch.

Each round’s model redrafted the unfinished clips. On the first book, the author changed 6.1% of the words of the base model’s drafts and 3.4% of the round-1 drafts. In all, 579 clips (2.85 hours) were proofread: 521 for training and 58 held out for testing. The final rule was “transcribe what the speaker says”: spelling, names and house style are corrected, but a speaker’s wording is never “improved”, so that the model does not learn to correct speakers.

Urdu has several accepted spellings for common words, and training on all of them teaches a model to be inconsistent. We applied one house style to every training and test sentence and to both sides before scoring, for example لیے (liye) rather than لئے, and compound verbs written as two words. This alone lowered the base model’s FLEURS error from 22.5% to 21.0%, so spelling variation was worth about 1.5 points.

Six training rounds

Rounds 1–3 ran on the laptop (Apple M4, 16 GB) as LoRA on the decoder only and taught the model the two audiobook voices. Rounds 4–6 ran on one rented RTX A4000 (16 GB). Round 4 trained LoRA on the encoder and the decoder with audio augmentation, and round 5 was the first full fine-tune of all 809 million weights. Round 6, the released model, added IndicVoices, UrduSpeech, English, the code-switch and no-speech clips and background noise, used half of round 5’s learning rate and ran a single pass in 4.5 GPU-hours.

Round 5 was the most accurate model so far on read speech, but it wrote runaway repeats on 21 of 260 Common Voice clips from unseen speakers and invented text on 30 of 30 silence clips. A 50/50 average of the round-4 and round-5 weights was our best model until round 6, but it invented text on 35 of 100 no-speech clips. Round 6 trained on five times as much data, with no-speech and code-switch clips, and is the first round that writes English words in English letters.

Choosing the checkpoint

Accuracy on speech improved up to step 2,775 of 3,697. Over the last 922 steps, while the learning rate decayed to zero, the model learned to hear words in faint audio: invented text on silence rose from 3 to 27 of 30 clips and on noise from 6 to 21 of 40, and quiet speech went from 23.5% to 271.6% words wrong. We release step 2,775. Weight averages of that step with the final step, and with the round-4 and round-5 average, each gained up to 2.5 points on the second audiobook and lost elsewhere, so none was kept.

Invented text on silence

Round 5 and the end of round 6 learned that faint hiss means speech. Measuring the level of the training audio explains why. FLEURS Urdu recordings are very quiet: median −45 dBFS, with one clip in twenty below −60 dBFS, which overlaps the hiss in our silence test (−66 to −52 dBFS). Two augmentation settings added more: a “very quiet” setting that lowered a fifth of the augmented clips by 12–30 dB, and a room-echo filter that lowered echoed clips by a further 20–25 dB. Together they put speech with full transcripts below the hiss of the silence test, so a model rewarded for transcribing it learns to transcribe hiss.

The fix is to set every clip to a normal level first, apply the final level after the echo filter, never go below −40 dBFS, and keep noise 15–35 dB below quiet speech.

The one-word answer is a decoding rule

By default, whisper.cpp (suppress_blank) and Transformers (begin_suppress_tokens) forbid the end-of-text token and a blank as the first generated token, so a Whisper model cannot return an empty transcript. On no-speech clips our model puts nearly all its probability on exactly those forbidden tokens, so by default it must write a word, typically اچھا (achha), “all right”, or ہاں (haan), “yes”. The usual two-word measure hides this: by default both the base model and ours write at least one word on every noise, music and silence clip.

With blank suppression off (one line in an application: params.suppress_blank = false), our model writes nothing on all 100 no-speech clips and every speech score is unchanged. Running a voice detector first also removes most invented text, but in our tests it dropped faint English speech as silence.

Names and English words

Names are the weakest point. The second audiobook’s test chapter has 23 name occurrences; the base model spelled 4 right and ours 11 with greedy decoding. A whisper.cpp prompt that lists the chapter’s names raised the count from 12 to 17 of 23. English inside Urdu speech is kept in English letters, which the author prefers for reading. It differs from the audiobook transcripts, which write English loanwords in Urdu script; the two are not yet reconciled.

Speed and deployment

The released model has exactly the layout of Whisper large-v3-turbo, so its speed is the base model’s. In whisper.cpp the encoder dominates CPU time: it always processes a full 30-second window, so a one-second reply costs as much encoder time as 30 seconds of speech. One window takes 1.3 s on the laptop’s GPU, 5.4 s on its CPU and 11.2 s on a 12-core cloud server.

On that server, language auto-detection ran the encoder twice, q4_0 was the fastest whisper.cpp format and q5_0 the slowest, and all 12 threads were slower than 8 while other services were running. faster-whisper transcribed a 27.4-minute recording in 15.6 minutes with greedy decoding (1.76 times real time). The whisper.cpp files take 874 MB (q8_0), 574 MB (q5_0) and 474 MB (q4_0).

Recommended settings: set the language when it is known, or detect it once per conversation or file; turn off blank suppression; run a voice detector first except where faint speech matters; decode without timestamps and without carried-over context; on x86 servers use q4_0 and about eight threads per job.

A compression experiment

To make the model faster on CPUs we distilled a student with 16 of the 32 encoder layers and all 4 decoder layers. After about 11 GPU-hours, the kept student was 1.9 times faster in the encoder on every machine and invented text on 0, 1 and 0 noise, music and silence clips, but it stayed 0.5–5.4 points behind on the test sets and was weak in English, so it is not released. Removing half of a 32-layer encoder discards more than 110 hours of audio can restore; more teacher-transcribed audio, not more steps, is the way forward.

Lessons

Evaluation code needs as much care as training code. A test file with duplicate rows, a loop rule that changed three times, a two-word rule that hid one-word answers, and two scoring jobs that deleted each other’s outputs all changed numbers we had already reported.

Audio level is part of the label. Faint speech with full transcripts taught two rounds to transcribe hiss.

Decoder defaults are part of the model. Timestamps, blank suppression, language detection and temperature fallback each changed results more than some training decisions did.

Test speed on the target machine. The GPU server’s 2012 processor was several times slower than the cloud server and gave the wrong picture, and the load of other services changed which thread count was fastest.

Limitations

The audiobook test sets are small (58 clips) and come from the same two narrators whose other chapters were used in training; the short-reply and mixed sets have 40 and 20 clips. We report no confidence intervals. IndicVoices is mostly Indian Urdu, women are under-represented in the Common Voice test set, and we have no test of spontaneous phone conversations by Pakistani speakers. The causes we give for the collapses fit the measurements but were not tested one at a time. We did not measure memory use or run the model on a phone.

Model, code and data

The model is on Hugging Face as Loudness-ai/urdu-sukhan-asr-v1 in Transformers, whisper.cpp (q8_0, q5_0, q4_0) and CTranslate2 formats under the CC BY-NC 4.0 licence for non-commercial use; access is granted on request, and for commercial use you can contact us. The example clips and their audio are in Loudness-ai/urdu-sukhan-asr-examples, and the test sets we can share (808 clips, about 96 minutes, from the Common Voice and FLEURS Urdu test sets and eight stress sets) are in Loudness-ai/urdu-sukhan-asr-test-sets, also on request. Once your request is approved, you can try the model in our Colab notebook, which needs a free Google account and runs in your own Colab session. The code, the result table of every evaluation and the scripts that build this report’s tables and figures are on GitHub at Loudness-ai/urdu-sukhan-asr. Installation and recommended settings are on the Urdu Sukhan ASR developer page.

The original full-precision checkpoint was lost when our training server was shut down at the end of the work; the Transformers weights were rebuilt from the q8_0 file and checked against the results reported here. The audiobook recordings and the transcribed UAT audio are not redistributed. The IndicVoices, UrduSpeech and nzg73 test sets and the other public datasets are available from their publishers under their own terms. This report is licensed under CC BY 4.0; the model and its code are under CC BY-NC 4.0.

Competing interests and funding

The author develops Loudness.ai, whose products will use the model described here. No external funding was received.

References

Radford, A. et al. (2023). Robust speech recognition via large-scale weak supervision. ICML 2023.

Arif, S. et al. (2024). WER we stand: benchmarking Urdu ASR models. arXiv:2409.11252.

Koenecke, A. et al. (2024). Careless Whisper: speech-to-text hallucination harms. ACM FAccT 2024.

Conneau, A. et al. (2023). FLEURS: few-shot learning evaluation of universal representations of speech. IEEE SLT 2023. Ardila, R. et al. (2020). Common Voice: a massively-multilingual speech corpus. LREC 2020. Javed, T. et al. (2024). IndicVoices. Findings of ACL 2024. Snyder, D., Chen, G. and Povey, D. (2015). MUSAN: a music, speech, and noise corpus. arXiv:1510.08484.

Park, D. S. et al. (2019). SpecAugment. Interspeech 2019. Hu, E. J. et al. (2022). LoRA. ICLR 2022. Wortsman, M. et al. (2022). Model soups. ICML 2022. Hinton, G., Vinyals, O. and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531. Gandhi, S., von Platen, P. and Rush, A. M. (2023). Distil-Whisper. arXiv:2311.00430.

Software: whisper.cpp 1.9.5, faster-whisper 1.2.1, Silero VAD and Chartist 2.1.0.

Paper and study files

Download paper (PDF)Read on Zenodo

Cite this report

Shehzad, Asad (2026). Urdu Sukhan ASR: Accurate Urdu Speech Recognition at Half the Size of Whisper large-v3. Loudness.ai, LA-RB-2026-004, version 1.0. https://loudness.ai/research/urdu-speech-recognition-whisper/ DOI: 10.5281/zenodo.23518703.

BibTeXRISResearch RSS feed