Speech recognition vendors often lead with a single accuracy figure, usually a word error rate measured on a public benchmark. Those numbers are useful for comparing models under controlled conditions, but they rarely predict how a model will perform on the audio your product actually handles.

For data teams, the practical question is how to run a fair evaluation on in-house recordings. This article walks through how word error rate works, where it can mislead, and how to set up a test that gives a reliable answer.

What word error rate measures

Word error rate, or WER, compares a model’s transcript with a reference transcript written by a person. It counts three types of mistakes:

• Substitutions, where the model writes the wrong word
• Deletions, where the model misses a word
• Insertions, where the model adds a word that was not spoken

The total number of errors is divided by the number of words in the reference. A WER of 10 percent means roughly one word in ten is wrong, missing or extra.

Why benchmark results do not transfer

Public benchmarks are built from specific types of audio, such as read speech, earnings calls, parliamentary recordings or meetings. A model that performs well on one may struggle on another. Your own audio may include accents, background noise, domain vocabulary, short commands or overlapping speakers that are underrepresented in any public dataset.

Benchmarks are best treated as a shortlist tool. They tell you which models are worth testing, not which one will win on your data.

Build a representative test set

A good evaluation set does not need to be large, but it should reflect real usage. Aim for a few hours of audio drawn from the situations your product sees most often, plus the difficult cases you care about.

• Sample across channels, devices and recording environments
• Include a mix of speakers, accents and speaking speeds
• Keep examples of crosstalk and interruptions if your audio contains them
• Remove or mask personal data before sharing audio with any vendor

Reference transcripts should be produced carefully, ideally by two people with a process for resolving disagreements. Errors in the reference will show up as model errors.

Normalize before you score

Raw transcripts differ in ways that do not matter to users. One model writes “twenty five” and another writes “25”. One includes filler words and another removes them. Without normalization, these differences inflate WER.

Agree on a set of rules before scoring: lowercasing, punctuation removal, number formatting and how to treat fillers and hesitations. Apply the same rules to every model and to the reference.

Look beyond a single number

An overall WER hides useful detail. Break results down by segment type, such as clean speech, noisy speech and overlapping speech, and by speaker group. A model with a slightly higher average error rate may still be the better choice if it handles your most important segments well.

It also helps to measure the errors that matter most to your use case. For a support analytics product, getting product names and account terms right may matter more than small function words. A custom keyword error rate on a list of important terms can capture this.

Measure latency and stability

For streaming use cases, accuracy is only part of the picture. Measure how long the model takes to return final text, and how often partial results change before they settle. A model that revises its output heavily can be hard to use for live captions or voice agents.

Compare cost against quality

Once you have accuracy and latency results, compare them against cost at your expected volume. Include features that may be priced separately, such as speaker labels or language support. Plotting accuracy against cost often makes trade-offs much clearer than a table.

Use public comparisons as a starting point

Before running your own tests, it helps to see how leading models perform on public benchmarks and how they differ on latency, languages, diarization and pricing. A recent roundup of speech to text models compared on benchmark accuracy is a useful way to narrow a long list down to three or four candidates worth evaluating properly.

Make evaluation repeatable

Speech models improve quickly, and a model that wins today may be overtaken within months. Keep your test set, normalization rules and scoring scripts under version control so you can rerun the evaluation whenever a new model is released. A repeatable process turns model selection from a one-off decision into a routine check.