Time per output token (TPOT)
Time per output token is the average time between successive output tokens of a request after the first token, governed by the decode phase.
- Formal definition
- For a request r emitting n(r) > 1 tokens, TPOT(r) = (t_last_token(r) − t_first_token(r)) ÷ (n(r) − 1). The per-gap variant — the time between each pair of consecutive tokens — is inter-token latency (ITL), also called time between tokens (TBT).
- Units
- Milliseconds per token (
ms/tok). Report percentiles over a stated request set; for streaming experience, report tail ITL/TBT as well — a mean TPOT hides stalls. - Distinguish
- TPOT is a per-request average; ITL/TBT is per-gap. A request can meet a mean-TPOT target while stalling visibly mid-stream. State which formulation a result uses; fluidity-style metrics exist precisely because averages hide stalls.
- What determines it
- Batch composition at each decode step (continuous batching changes it every iteration), KV-cache memory management, parallelism, engine kernels, and hardware. In tension with TTFT: scheduling that protects decode smoothness delays new prefills, and vice versa.
- Common misuse
- Using mean TPOT as a streaming-quality guarantee; conflating TPOT with ITL/TBT; comparing across different output-length distributions.
- Related terms
- time to first token · inference goodput · SLO attainment
- Preferred citation
gdput Research. "Time per output token (TPOT)." gdput definitions, v0.1, 2026. https://gdput.com/definitions/tpot