Building SASR Eval: A Benchmark for Somali Speech Recognition

Phoenix Rising Group · August 2026

You cannot improve what you do not measure. Before we could argue that Somali speech recognition was inadequate, we needed to establish how inadequate, with numbers that someone else could reproduce.

This is a description of how we built SASR Eval, what it found, and — equally important — what it does not establish.

Why a benchmark was needed

Before we started, our understanding of Somali ASR performance was anecdotal. People who had tried commercial transcription tools on Somali audio reported that the output was unusable. That is a useful signal but not a measurement, and it does not distinguish between systems, quantify the gap, or give anyone a target to improve against.

For most widely spoken languages, standard benchmarks exist and have existed for years. Researchers compare against them, improvements are measurable, and progress accumulates. For Somali there was no equivalent, which meant there was no way to make a specific claim about the state of the art or to demonstrate that a new approach was better.

The first attempt and why it was insufficient

Our first evaluation used real broadcast and lecture audio, which is the material we actually care about. It surfaced two clear failure modes: one commercial system returned Arabic script for Somali audio on every clip, and another produced repetition loops on longer religious lecture recordings.

Those are real findings. But the evaluation had a fundamental limitation: there were no ground-truth transcripts. Without a reference, you cannot compute word error rate. We could describe failures qualitatively and detect obvious script mismatches, but we could not produce a number.

A qualitative finding is arguable. A word error rate is not.

The methodology

For the second evaluation we used a publicly available Somali speech dataset with human-written transcripts, which gave us the ground truth the first attempt lacked.

We constructed 50 evaluation clips of roughly ten seconds each from the held-out test split, concatenating shorter samples with brief silence padding to reach a consistent length. Clips were exported as 16 kHz mono WAV. We ran three systems against them: a leading commercial API, an open-source large model running locally, and a model we fine-tuned ourselves.

Scoring used the standard jiwer library for word error rate and character error rate, with text normalisation applied to both hypothesis and reference before comparison — Unicode normalisation, apostrophe standardisation, lowercasing, and punctuation stripping.

We fixed one scoring bug in the process, which is worth mentioning because it is the kind of thing that silently distorts results: our earlier scorer treated Unicode curly apostrophes as punctuation and stripped them while preserving ASCII apostrophes. This had no effect on the dataset we used, which contains no apostrophes, but it would matter for Somali corpora that use apostrophes to mark pharyngeal consonants. We fixed it before publishing.

The results

Across the 50-clip set:

The commercial system outperformed the larger open-source model on 44 of 50 clips.

An observation worth noting: the commercial system had higher variance. Its best clip came in at 12.5% word error rate — genuinely usable — while its worst returned 100%. The open-source model was more consistently mediocre, with a higher floor and a higher ceiling. Character error rate tells a somewhat gentler story than word error rate throughout, which is expected when a system gets phonemes approximately right but word boundaries wrong.

The caveat that matters most

Our fine-tuned model’s 36.8% figure is in-domain. It was trained on the training split of the same dataset used for evaluation, while the commercial systems were evaluated out-of-domain on data they had never seen.

Comparing those numbers directly overstates our model’s advantage, and we say so explicitly in the report.

What the result supports is a narrower claim, which we think is the more interesting one: a small model — 244 million parameters, fine-tuned for roughly an hour on free-tier hardware — substantially outperforms a much larger general-purpose model on the same distribution. Scale is not the binding constraint here. Data match is.

Whether that advantage persists on out-of-domain broadcast audio, where neither system has a training advantage, is an open question. We expect it to, because domain-matched training data is exactly what the specialist has and the generalist does not. But expecting is not measuring, and the next evaluation is designed to settle it.

Other limitations

Beyond the in-domain confound, several limitations apply.

Concatenating short clips creates unnatural speaker and topic changes within a single clip, which production systems are not designed for. Results on continuous natural speech will differ.

Short references inflate word error rate — a single substitution on a one-word reference produces a WER of 1.0, and an insertion pushes it above 1.0. Character error rate is the more stable metric on this distribution.

We took the dataset’s transcripts as given rather than re-verifying them with native speakers. Roughly a third of the clips contained annotation artefacts — stray markup and syntactic labels that leaked into the reference text — though we measured their aggregate effect on the scores as negligible.

And the dataset represents standard Somali read speech rather than the Djiboutian broadcast dialect that is our actual target.

What comes next

The next evaluation addresses the central limitation: it will use real broadcast audio with native-speaker ground-truth transcription, evaluated out-of-domain, with dialect annotation. It will also include the first evaluation of any speech recognition system on Afar, in any domain.

We publish the methodology, per-clip results, and limitations in full because a benchmark that cannot be checked is not a benchmark. The evaluation set and results data are available on our site.