SASR Eval paper available. Methodology, per-clip data, and methodological notes.

Document	Description	Format
SASR Eval	Full evaluation paper, methodology, results, and analysis	PDF
SASR Eval results	Per-clip WER/CER, hypotheses, references, and summary statistics	JSON
SASR Eval per-clip data	Clip-by-clip scores and transcript text	CSV
SASR Eval findings	Detailed benchmark analysis and notable clip examples	Markdown
SASR Eval pilot results	Prior reference-free evaluation (June 2026)	JSON
SASR Eval pilot per-clip data	Reference-free pass clip data	CSV

Summary

Aggregate results.

Evaluated June 22, 2026 on 50 stitched clips (~10 seconds each, ~550 seconds total) from the skydheere/soomali-asr-dataset test split. Ground-truth transcriptions used for WER and CER scoring.

System	n	Mean WER	Mean CER	Parameters
Fine-tuned Whisper small (ours) Trained ~1 hr on T4 GPU, 500 steps	50	0.368	-	244M
ElevenLabs Scribe v1 Commercial ASR API	50	0.592	0.185	-
faster-whisper large-v3 Open-source baseline	50	0.871	0.266	1.5B

37.8%

Relative WER reduction
vs. ElevenLabs

57.8%

Relative WER reduction
vs. Whisper large-v3

44 / 50

Clips ElevenLabs wins
by WER (baseline systems)

81.5%

ElevenLabs char accuracy
Mean CER 0.185

Key wins

What the numbers show.

37.8%

Relative reduction vs. ElevenLabs

Our fine-tuned Whisper small achieves mean WER 0.368 versus ElevenLabs Scribe v1 at 0.592. Domain-specific training on Somali closes a gap that commercial systems leave open.

57.8%

Relative reduction vs. Whisper large-v3

Despite using a model 6× smaller (244M vs 1.5B parameters), our fine-tuned system outperforms the large-v3 baseline by a wide margin on this evaluation set.

In-domain

Performance on broadcast data

On public dataset clips, ElevenLabs wins 44 of 50 by WER. Our fine-tuned model is designed for in-domain broadcast data, SASR Eval broadcast validation (August 2026) will validate on regional archives.

~1 hour

Training efficiency

Single T4 GPU, 500 training steps. Demonstrates that low-resource language ASR improvement does not require massive compute and is relevant for rapid deployment in operational environments.

Notable clips

Per-clip examples.

Selected clips illustrating system behavior across the evaluation set. Full per-clip data in SASR Eval.

clip50: Best ElevenLabs performance

ElevenLabs WER: 0.125 Whisper WER: 0.875

Reference maalaa Guriga dadka soo dhisay ma aragtay? Waxaa yimid arday badan. Waxaa aan jeclaa Emma. gurigaygan

ElevenLabs Maalaa guriga dadka soo dhisay ma aragtay. Waxaay yimid arday badan waxa aan jeclaa emma gurigaygan.

Whisper maa laa guriga dadka saadisay maa raqday waha yimid ardaybadan waha ancha laa emma guriga igan

ElevenLabs near-perfectly transcribes a full sentence. Whisper mangles several words (“saadisay” for “soo dhisay”, “raqday” for “aragtay”).

clip36: Proper noun preservation

ElevenLabs WER: 0.263 Whisper WER: 0.842

Reference Hargeysa waa magaalo weyn. Kani waa kaanaga. Warsame! farasey! Waan aqaannaa ardayda oo idil. maxkamadda gobolka qorayaasha iyo boqorrada

ElevenLabs Hargeysa waa magaalo weyn. Kani waa kaannada. Warsame. Farasay. Waan aqaana ardayda oo idil. Maxkamadda gobolka. Qorshaha iyo boqorada.

Whisper Har geysa waa magaalaa weyn. Kani waa kaan naga. Huwar samay. Farasay. Waan aqaanaa radaayda oo eedil. Mahkamadda goblka. Urayaasha iyo baxar rada.

ElevenLabs preserves proper nouns (“Hargeysa”, “Warsame”, “Maxkamadda”) and sentence structure. Whisper splits tokens and garbles proper nouns.

clip11: Worst ElevenLabs performance

ElevenLabs WER: 1.000 Whisper WER: 0.867

Reference goror caanaha saca raggaa Saddexda wiil ayaad ammaantay. OSV Dhiso. maalmo afar ah oo horreysay

ElevenLabs Wuxuu u geystay qorayn ayaa ah goorar. Canaan saac. Ragga. Saddex daweyl ayaa dambe. Dhiso. Maalmo afar ah oo horaysay.

Whisper qorar aanaha saad ragga sada dhawiyil aya dhammaante diso maalmo qafar ah oo horay say

Annotation artifact (“OSV”) in the ground truth reference threw off ElevenLabs. Whisper echoed more of the phonetic surface form.

clip28: Largest performance gap

Whisper WER: 1.438 ElevenLabs WER: 0.875

Reference qalimmada silsiladahayga dukaanle Ninkii libaaxii qabtay miyuu la hadlay? S/SOVAV baaldiyo labo ganacsato oo waawanaagsan

ElevenLabs Xalammada silsiladaha iga dukaan leh ninkii libaaxi qabtay miyu la hadlay baal diyo laba ganacsi ugu wanaagsan.

Whisper Qalima dha. Silsila dhaha yiga. Du kaan la. Ninkin li baax hii qabtay. Miuu lahad la. Baal diyo. La baga naa sato'u buwan-buwan ahaksan.

Whisper fragments continuous speech into spurious sentence breaks, inflating WER past 1.0. ElevenLabs handles run-on sentence structure better.

Head-to-head

ElevenLabs vs. Whisper large-v3 (baseline systems).

Per-clip WER comparison between the two baseline commercial/open-source systems evaluated in SASR Eval. ElevenLabs wins 44 of 50 clips. 2 ties. Whisper wins 4.

Clip	Whisper WER	ElevenLabs WER	Winner
clip01	0.938	0.438	ElevenLabs
clip02	0.684	0.526	ElevenLabs
clip03	0.812	0.688	ElevenLabs
clip04	0.722	0.556	ElevenLabs
clip05	0.957	0.696	ElevenLabs
clip06	0.850	0.650	ElevenLabs
clip07	0.850	0.650	ElevenLabs
clip08	0.727	0.818	Whisper
clip09	0.882	0.588	ElevenLabs
clip10	0.905	0.857	ElevenLabs
clip11	0.867	1.000	Whisper
clip12	0.923	0.846	ElevenLabs
clip13	0.857	0.762	ElevenLabs
clip14	0.667	0.917	Whisper
clip15	0.900	0.500	ElevenLabs
clip16	0.824	0.529	ElevenLabs
clip17	0.824	0.647	ElevenLabs
clip18	0.667	0.500	ElevenLabs
clip19	1.231	0.462	ElevenLabs
clip20	0.917	0.500	ElevenLabs
clip21	0.809	0.714	ElevenLabs
clip22	0.842	0.737	ElevenLabs
clip23	0.765	0.882	Whisper
clip24	0.944	0.667	ElevenLabs
clip25	1.000	0.625	ElevenLabs
clip26	0.765	0.412	ElevenLabs
clip27	0.950	0.700	ElevenLabs
clip28	1.438	0.875	ElevenLabs
clip29	0.818	0.682	ElevenLabs
clip30	0.867	0.400	ElevenLabs
clip31	0.733	0.333	ElevenLabs
clip32	0.889	0.556	ElevenLabs
clip33	0.950	0.400	ElevenLabs
clip34	0.938	0.562	ElevenLabs
clip35	0.769	0.308	ElevenLabs
clip36	0.842	0.263	ElevenLabs
clip37	0.778	0.333	ElevenLabs
clip38	1.000	0.818	ElevenLabs
clip39	0.941	0.529	ElevenLabs
clip40	0.682	0.682	Tie
clip41	0.800	0.267	ElevenLabs
clip42	0.833	0.417	ElevenLabs
clip43	1.059	0.529	ElevenLabs
clip44	1.000	0.571	ElevenLabs
clip45	0.923	0.538	ElevenLabs
clip46	1.000	0.667	ElevenLabs
clip47	0.933	0.600	ElevenLabs
clip48	0.789	0.789	Tie
clip49	0.619	0.476	ElevenLabs
clip50	0.875	0.125	ElevenLabs

Next steps

SASR Eval broadcast validation (August 2026)

SASR Eval validated our fine-tuned model on public Somali speech data. SASR Eval broadcast validation will evaluate performance on in-domain broadcast recordings from regional archives. The data our platform is designed to process at scale.

June 22, 2026 (Complete)

SASR Eval

50 clips from skydheere/soomali-asr-dataset. Three systems evaluated. Fine-tuned Whisper small: WER 0.368.

August 2026

SASR Eval broadcast validation

Broadcast data validation from regional archives. Tests in-domain generalization and operational readiness.