Image source: https://api.insert.link/gp-images/b897a580-1448-4f80-bf18-acf92761b416/hero-3-benchmark-method.png/
Public leaderboards will tell you which video model produces the prettiest clip. They will not tell you which one will blow up your pipeline at 2 a.m., double your cost per asset, or quietly time out a third of your jobs. Those answers require a different kind of benchmark, and after operating 475 generative media endpoints in production, we have converged on a methodology worth sharing.
Why lab benchmarks mislead production teams
Most generative media benchmarks share three properties that make them nearly useless for production decisions.
They score single outputs, usually cherry-picked, so they measure a model’s ceiling rather than its distribution. They run in controlled conditions, so they miss queueing, cold starts, and provider-side degradation, which is where real latency lives. And they ignore economics entirely: a model that scores 5 percent higher on aesthetic quality but takes three times as long and fails twice as often is a worse production model for most workloads, and no leaderboard will tell you that.
Production benchmarking starts from a different question. Not “which model is best?” but “which model gets this specific job to an accepted output at the lowest fully loaded cost?”
The three metrics that actually predict production behavior
Observed runtime percentiles, per endpoint. Not the vendor’s claimed speed: the median (p50) and tail (p95) of real jobs over a rolling window. Medians resist the outliers that plague generation workloads, and the p50-to-p95 gap tells you how much buffer your queue design needs.
Acceptance rate. The fraction of outputs a downstream consumer, human or automated, actually accepts. Generative work is probabilistic; a model that needs four attempts per accepted asset is, for budgeting purposes, four times its list price.
Cost per accepted output. The product of the two above plus pricing: total spend, including retries and failures, divided by shipped assets. This is the only number that survives contact with a CFO.
What the runtime data shows across 475 endpoints
We continuously measure median observed runtime across every model endpoint on the each::labs catalog: 230 video, 108 image, and 17 audio endpoints among them, 456 with sufficient live traffic to report. The distribution is more skewed than most teams assume when they design their first pipeline.
The median image endpoint returns in 20 seconds. The median video endpoint takes 120 seconds, six times longer. A quarter of video endpoints finish under 64 seconds, but the 75th percentile sits at 205 seconds, and the slowest endpoints run to 500 seconds for a single job.
Two findings matter most for system design. First, only about one video endpoint in four has a median runtime under a minute, so synchronous request patterns that feel fine in image workloads simply do not transfer to video. Second, the spread between the fastest and slowest video endpoints spans two orders of magnitude, which means model selection is not just a quality decision: it is the single largest latency and cost lever available to a pipeline.
The audio sample is smaller (17 endpoints), so we treat its 12-second median as directional rather than definitive. Honest benchmarking means saying so.
The methodology, step by step
- Instrument at the job level. Log submit time, completion time, terminal status, and cost for every generation, tagged by model endpoint and use case. Aggregates hide the tail; job-level data is non-negotiable.
- Report medians and p95s over a rolling window, never lifetime means. Model providers change infrastructure; last quarter’s latency is a different model in practice.
- Track acceptance separately from success. An API “success” that a reviewer rejects is a cost without an asset. Wire the accept/reject signal from your product back into the benchmark.
- Compute cost per accepted output per use case, not per model globally. The same model can be economical for thumbnails and ruinous for hero assets.
- Re-rank monthly. In our catalog data, new model releases now arrive weekly, and rankings on runtime and economics reshuffle far faster than quality rankings do.
We publish the per-endpoint medians openly in the each::labs model catalog, and we are expanding this into a continuously updated public benchmark of generative media inference.
Limitations worth stating
Runtime percentiles say nothing about output quality; a fast model that generates unusable footage has an infinite cost per accepted output. Acceptance rates embed human subjectivity and shift with brand guidelines. Traffic volumes differ per endpoint, so low-volume endpoints carry wider confidence intervals. A production benchmark is a decision tool, not a scoreboard: its job is to make the retry math and the tail risk visible before they reach your invoice.
FAQ
What is cost per accepted output?
Total generation spend, including retries and failed runs, divided by the number of outputs actually shipped. It is the production-grade replacement for price per generation.
Why use p50 instead of average latency for generative models?
Generation runtimes are heavily right-skewed. A handful of 500-second jobs drags a mean far above what a typical user experiences, while the median tracks the real center of the distribution.
How often should production teams re-benchmark models?
Monthly at minimum. Model releases now land weekly, and provider-side infrastructure changes shift runtime distributions without any announcement.
Bartu Varol works on marketing at each::labs, where the team operates 475 generative media model endpoints and measures their production behavior continuously.