Accuracy: measured, not claimed
We do not publish one accuracy number. Every row below is one recording set and one model, with its clip count, minutes and measurement date, and it says plainly whether the audio is a real speaker or generated speech.
Rows labelled SYNTHETIC were measured on text-to-speech lines: they show the pipeline works and say nothing about accuracy with real speakers. Gulf, Egyptian and Levantine Arabic have not been measured with real speakers yet. Iraqi and Maghrebi Arabic are transcribed with no measured error rate from real speakers, and Iraqi has no dedicated preset yet.
Results
| Set | Audio source | Model | Dialect hint | Clips | Minutes | Word error rate | Character error rate | Measured |
|---|---|---|---|---|---|---|---|---|
| Modern Standard Arabic, read | Real speaker | Whisper baseResearch comparison, not in the app | No hint | 86 | 15.0 | 28.1% | 6.7% | |
| Modern Standard Arabic, read | Real speaker | Whisper baseResearch comparison, not in the app | With dialect hint | 86 | 15.0 | 27.8% | 6.2% | |
| Modern Standard Arabic, read | Real speaker | Whisper large-v3-turboUsed by Qawlak today | No hint | 86 | 15.0 | 7.2% | 2.4% | |
| Modern Standard Arabic, read | Real speaker | Whisper large-v3-turboUsed by Qawlak today | With dialect hint | 86 | 15.0 | 7.0% | 2.2% | |
| Modern Standard Arabic, read | Real speaker | oddadmix dialectalResearch comparison, not in the app | No hint | 86 | 15.0 | 10.2% | 3.0% | |
| Modern Standard Arabic, read | Real speaker | oddadmix dialectalResearch comparison, not in the app | With dialect hint | 86 | 15.0 | 9.8% | 3.1% | |
| Modern Standard Arabic, read | Real speaker | Audar-ASR-V1-FlashResearch comparison, not in the app | — | 86 | 15.0 | 8.8% | 3.1% | |
| Modern Standard Arabic, read | Real speaker | Cohere Transcribe ArabicResearch comparison, not in the app | — | 86 | 15.0 | 7.1% | 2.9% | |
| Modern Standard Arabic, read | Real speaker | cohere+vad+guardResearch comparison, not in the app | — | 86 | 15.0 | 7.0% | 2.8% | |
| Modern Standard Arabic, read | Real speaker | Qwen3-ASR-1.7BResearch comparison, not in the app | — | 86 | 15.0 | 8.3% | 1.8% | |
| Modern Standard Arabic, read | Real speaker | omniasrResearch comparison, not in the app | — | 86 | 15.0 | 8.0% | 2.9% | |
| Hindi, read | Real speaker | Whisper large-v3-turboBefore 15 Sep / research | As the app transcribes an upload with the language preset | 239 | 42.8 | 31.4% | 13.7% | |
| Hindi, read | Real speaker | AI4Bharat IndicConformerResearch comparison, not in the app | — | 239 | 42.8 | 12.1% | 5.4% | |
| Hindi, read | Real speaker | Qwen3-ASR-1.7BResearch comparison, not in the app | — | 239 | 42.8 | 14.5% | 6.9% | |
| Hindi, read | Real speaker | ARTPARK Whisper large-v3 Vaani HindiResearch comparison, not in the app | — | 239 | 42.8 | 13.0% | 6.2% | |
| Hindi, read | Real speaker | Srota (Qwen3-ASR-0.6B Hinglish)Research comparison, not in the app | — | 239 | 42.8 | 26.7% | 21.4% | |
| Hindi, read | Real speaker | Whisper large-v3-turbo, bare decode (control)Research comparison, not in the app | — | 239 | 42.8 | 33.2% | 15.7% | |
| Hindi, read | Real speaker | AI4Bharat IndicConformerUsed by Qawlak today | As the app transcribes an upload with the language preset | 239 | 42.8 | 12.8% | 6.1% | |
| Hindi, spontaneous speech | Real speaker | Whisper large-v3-turboBefore 15 Sep / research | As the app transcribes an upload with the language preset | 300 | 33.6 | 45.9% | 29.0% | |
| Hindi, spontaneous speech | Real speaker | AI4Bharat IndicConformerResearch comparison, not in the app | — | 300 | 33.6 | 16.7% | 7.6% | |
| Hindi, spontaneous speech | Real speaker | Qwen3-ASR-1.7BResearch comparison, not in the app | — | 300 | 33.6 | 23.4% | 12.1% | |
| Hindi, spontaneous speech | Real speaker | ARTPARK Whisper large-v3 Vaani HindiResearch comparison, not in the app | — | 300 | 33.6 | 21.6% | 11.3% | |
| Hindi, spontaneous speech | Real speaker | vasista22 whisper-hindi-large-v2Research comparison, not in the app | — | 300 | 33.6 | 34.2% | 23.7% | |
| Hindi, spontaneous speech | Real speaker | Srota (Qwen3-ASR-0.6B Hinglish)Research comparison, not in the app | — | 300 | 33.6 | 38.2% | 27.7% | |
| Hindi, spontaneous speech | Real speaker | Whisper large-v3-turbo, bare decode (control)Research comparison, not in the app | — | 300 | 33.6 | 42.7% | 25.5% | |
| Hindi, spontaneous speech | Real speaker | AI4Bharat IndicConformerUsed by Qawlak today | As the app transcribes an upload with the language preset | 300 | 33.6 | 16.9% | 7.8% | |
| Hindi, spontaneous speech | Real speaker | Whisper large-v3-turboUsed by Qawlak today | Auto: the app detects the language | 300 | 33.6 | 55.6% Not reliable yet | 41.6% | |
| Urdu, read | Real speaker | Whisper large-v3-turboBefore 15 Sep / research | As the app transcribes an upload with the language preset | 267 | 45.3 | 22.4% | 9.2% | |
| Urdu, read | Real speaker | AI4Bharat IndicConformer (Urdu)Research comparison, not in the app | — | 267 | 45.3 | 16.9% | 6.8% | |
| Urdu, read | Real speaker | AI4Bharat IndicConformer (Urdu)Used by Qawlak today | As the app transcribes an upload with the language preset | 267 | 45.3 | 17.2% | 7.1% | |
| Urdu, spontaneous speech | Real speaker | Whisper large-v3-turboBefore 15 Sep / research | As the app transcribes an upload with the language preset | 300 | 35.3 | 24.4% | 13.9% | |
| Urdu, spontaneous speech | Real speaker | AI4Bharat IndicConformer (Urdu)Research comparison, not in the app | — | 300 | 35.3 | 10.0% | 4.2% | |
| Urdu, spontaneous speech | Real speaker | AI4Bharat IndicConformer (Urdu)Used by Qawlak today | As the app transcribes an upload with the language preset | 300 | 35.3 | 10.3% | 4.4% | |
| Urdu, spontaneous speech | Real speaker | Whisper large-v3-turboUsed by Qawlak today | Auto: the app detects the language | 300 | 35.3 | 46.1% | 39.3% | |
| Hindi, synthetic lines | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 10 | 0.4 | 6.7% | 1.8% | |
| Urdu, synthetic lines | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 10 | 0.5 | 4.1% | 1.2% | |
| Hinglish, synthetic lines | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 3 | 0.1 | 0.0% | 0.0% | |
| Gulf mix: Arabic, Hindi and English | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 2 | 0.1 | 6.2% | 4.5% | |
| Egyptian Arabic, synthetic lines | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 10 | 0.5 | 8.1% | 0.7% | |
| Levantine Arabic, synthetic lines | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 10 | 0.5 | 15.5% | 2.7% | |
| Moroccan Arabic (Darija), synthetic lines | SYNTHETIC | Whisper large-v3-turboUsed by Qawlak today | As the app transcribes an upload with the language preset | 10 | 0.5 | 36.6% | 7.8% | |
| Gulf Arabic | Not yet measured | Waiting for real recordings with written consent | ||||||
| Egyptian Arabic | Not yet measured | Waiting for real recordings with written consent | ||||||
| Levantine Arabic | Not yet measured | Waiting for real recordings with written consent | ||||||
| Gulf Arabic mixed with English | Not yet measured | Waiting for real recordings with written consent | ||||||
| Hinglish from Gulf speakers | Not yet measured | Waiting for real recordings with written consent | ||||||
| Urdu-English from Gulf speakers | Not yet measured | Waiting for real recordings with written consent | ||||||
| Hindi and Urdu from Gulf speakers | Not yet measured | Waiting for real recordings with written consent | ||||||
MSA: clips from the Arabic Speech Corpus under CC BY 4.0, one studio reader, and not a dialect.
Hindi and Urdu: validation clips from FLEURS (read sentences) and IndicVoices (spontaneous speech) under CC BY 4.0, and no test split was opened.
How we measure
Word error rate = substituted, deleted and inserted words ÷ words in the reference transcript, over the whole set rather than averaged per clip. Text is normalised before comparison: no tashkeel or punctuation, alef and taa marbuta forms unified. The numbers come from the benchmark files themselves and are reproduced with one command:
python bench/asr_bench.py --set msa --models large-v3-turbo --seed both
python bench/asr_bench.py --set fleurs-hi-val --productWhat is missing
Recordings with written consent from real speakers: 30 minutes each of Gulf, Egyptian, Levantine and MSA, 10 minutes of Gulf Arabic mixed with English, and 30 minutes each of Hinglish and Urdu-English. When they land, the empty rows fill from the same command with a new date.