Inference goodput

definition · version 0.1 (draft) · 2026-07-21 · gdput Research

Inference goodput is the rate of useful output an inference system delivers while satisfying all defined service-level objectives.

Formal definition

For a request set R over measurement interval T with SLO set S, inference goodput is the total output of the requests that satisfy every objective in S, divided by T.

Formula

goodput = Σ output_tokens(r) for r ∈ R_compliant ÷ T
R_compliant = { r ∈ R : r satisfies every objective in S }

Units

Output tokens per second (tok/s). Report the SLO set S alongside every figure.

Includes

Output tokens of requests that met all SLOs (e.g. TTFT, ITL/TBT percentile targets) within T.

Excludes

Output of non-compliant requests, discarded speculative tokens, retries, failed requests, warm-up traffic.

Example

A deployment produces 1000 tok/s aggregate throughput. 91.4% of requests meet the SLO set and carry 87% of output tokens. Inference goodput is 870 tok/s — the other 130 tok/s is paid-for but non-useful output.

Common misuse

Reporting aggregate throughput as goodput; comparing goodput numbers measured under different SLO sets; omitting the SLO definition from the result.

Limitations

Entirely dependent on the SLO set chosen — goodput is meaningless without S stated. Binary compliance grants no partial credit to near-misses.

Related terms

SLO attainment · inference effectiveness · throughput

Preferred citation

gdput Research. "Inference goodput." gdput definitions, v0.1, 2026.
https://gdput.com/definitions/goodput