Speech recognition is a solved problem in English. Modern systems transcribe clean English audio with word error rates below 5% — accurate enough that most people no longer think about it. Dictation works. Captions work. Voice assistants understand you.
Somali is spoken by roughly 25 million people across Somalia, Somaliland, Djibouti, Ethiopia, and Kenya, plus a large global diaspora. For those speakers, none of this works.
We wanted to know exactly how badly it fails, so we measured it.
What we tested
We built a small evaluation set of 50 Somali audio clips, roughly ten seconds each, drawn from a publicly available Somali speech dataset with human-written reference transcripts. We ran three systems against it and computed word error rate — the standard measure of how often a system gets a word wrong through substitution, deletion, or insertion.
The results:
- faster-whisper large-v3 — 87.1% word error rate
- ElevenLabs Scribe v1 — 59.2% word error rate
- AssemblyAI Universal-2 — returned Arabic text on 100% of clips tested in an earlier round
A word error rate of 59% means that for every ten words spoken, roughly six come back wrong. At 87%, the output bears almost no relationship to what was said.
The AssemblyAI result is a different kind of failure and worth dwelling on. It did not produce bad Somali. It produced Arabic — confidently, on every single clip. The system does not recognise that Somali exists as a distinct language and routes the audio to the nearest thing in its training distribution.
Why this happens
The reason is not that Somali is unusually hard to transcribe. Somali has a straightforward Latin orthography, a relatively small phoneme inventory, and no tonal complexity of the kind that makes some languages genuinely difficult for automatic systems.
The reason is data.
Modern speech recognition systems learn from large quantities of paired audio and text. Whisper was trained on roughly 680,000 hours of audio. The overwhelming majority of that is English, with substantial coverage of the other major world languages — Spanish, Mandarin, French, German, Arabic, Hindi.
Languages outside that set are described in the literature as “low-resource,” which is a polite way of saying almost no training data exists. For Somali, the largest publicly available speech dataset we are aware of contains a few dozen hours. Compare that to the tens of thousands of hours behind English.
A model cannot learn what it has never heard. When Whisper encounters Somali it does its best with representations built almost entirely from other languages, and the result is what we measured.
The dialect problem underneath the data problem
There is a second layer that the aggregate numbers hide.
The small amount of Somali training data that does exist skews heavily toward standard written Somali — the register used in formal writing and education, closest to the Northern Somali on which the standard orthography was based. It does not represent the range of dialectal variation across the Somali-speaking world.
Djiboutian Somali differs from the standard in vocabulary, pronunciation, and rhythm. So does the Somali spoken in Somaliland, and in the Ogaden, and in the Kenyan northeast. A system trained on standard Somali and evaluated on Djiboutian broadcast speech faces a harder problem than the headline numbers suggest.
This matters because it means the gap cannot be closed simply by collecting more Somali data in general. The data has to match the dialect and the domain where the system will actually be used.
What we found when we fine-tuned
We took Whisper small — 244 million parameters, a fraction of the size of large-v3 — and fine-tuned it on Somali data for about an hour on a single free-tier T4 GPU.
It reached 36.8% word error rate on the validation split.
That is a 57.8% relative improvement over base Whisper large-v3 and 37.8% better than the strongest commercial system, achieved with a much smaller model and roughly a dollar of compute.
An important caveat, which we state clearly in our evaluation report: this result is in-domain. The fine-tuned model was trained and evaluated on the same dataset distribution, while the commercial systems were evaluated out-of-domain on data they had never seen. The numbers are not directly comparable and we do not claim they are.
What the result does show is narrower and more useful: the performance gap between general-purpose commercial systems and a domain-matched specialist model is very large, and it closes quickly with the right training data. Model size is not the binding constraint. Data match is.
Why this is worth solving
Every argument for speech recognition in English applies to Somali. Searchable archives. Accessible media. Documentation that does not require someone to type. Access to digital services for people who speak but do not write fluently in the administrative language of their country.
Somali speakers get none of it, not because the problem is hard but because the data required to solve it has never been assembled.
That is the problem we are working on.
Our full evaluation, including per-clip results, methodology, and limitations, is published as SASR Eval.