A user submits a prompt. The first token appears in 300 ms, so the interface feels instant. Then the answer crawls out in uneven bursts, and the same user who praised the “fast” model a moment ago now describes it as sluggish. Nothing changed in the model between those two reactions — what changed is which phase of generation the user was watching.

Inter token latency (ITL) is the metric that describes that second phase: how quickly tokens arrive after the first one. It answers a different question than time to first token (TTFT), and it is frequently confused with time per output token (TPOT), tokens per second (TPS), generation speed, decode latency, and throughput. This guide separates those terms, shows the formulas behind each, and explains why comparing only TTFT cannot represent the full streaming experience.

The First Token Arrived in 300 ms — So Why Does the Answer Still Feel Slow?

flowchart TD
    S0["Start: inter token latency"]
    S1["1. You tune a deployment, watch the dashbo…"]
    S0 --> S1
    S2["2. This guide walks through one pass of th…"]
    S1 --> S2
    S3["3. Client requests inter token latency"]
    S2 --> S3
    S4["4. Service validates inter token latency"]
    S3 --> S4
    S5["5. Network delivers inter token latency"]
    S4 --> S5
    Done["Outcome: The First Token Arrived in 300 ms — So…"]
    S5 --> Done

The familiar developer frustration: a fast first token but a bursty stream

You tune a deployment, watch the dashboard, and the first-token number looks healthy. Users still complain that the output “comes in chunks.” The reason is that TTFT and inter-token latency measure two different segments of the same request. TTFT captures how long the user waits before anything appears; ITL captures how smoothly everything appears after that. A model can win the first race and lose the second.

The core question: why does generation speed vary after the first token?

After the first token, the model generates tokens one at a time, and each new token depends on the ones before it. That sequential dependency is why the decode phase behaves differently from the prefill phase that produced the first token. Anyscale’s benchmarking documentation describes this phase as memory-bound and notes that it “may slow down if large prefills share the batch, competing for GPU compute and extending the model’s forward pass” [2]. In other words, the same infrastructure that made the first token fast can make later tokens slow, depending on what else the server is doing.

What this guide will clarify

This guide walks through one pass of the easily-confused terms — Inter-Token Latency, TPOT, Time Between Tokens, Tokens per Second, Generation Speed, Decode Latency, and Throughput — with the formulas and caveats that separate them. It covers measurement practice, the tasks that care most about generation speed, and the levers that improve it. It deliberately does not cover TTFT internals and optimization, full LLM latency optimization, model quality benchmarking, or a complete serving-capacity guide; those deserve their own treatments.

What Inter-Token Latency (ITL) Actually Measures

sequenceDiagram
    participant A as TTFT manages the start; ITL manages the…
    participant B as Inter-token latency is the average time…
    participant C as The decode phase is the part of inferen…
    A->>B: Request inter token latency
    B->>C: Execute What Inter-Token Latency (ITL) Actually…
    C-->>B: Return inter token latency status
    B-->>A: Confirm Two effects explain the 'slow after a f…

Definition of ITL

Inter-token latency is the average time between consecutive streamed tokens. NVIDIA’s NIM benchmarking documentation defines it precisely: “Inter-token latency (ITL) is defined as the average time between consecutive tokens and is also known as time per output token (TPOT)” [1]. The same page adds an important qualifier — “Although the definition is straightforward, tools differ on whether TTFT is included in the average” [1]. ITL covers the decode portion of a request only; it says nothing about queuing, prefill, or network time before the first token.

The decode phase explained

The decode phase is the part of inference that runs after prefill. During prefill, the model processes the entire input prompt and builds a key-value (KV) cache, which Anyscale describes as “its short-term memory for the session” [2]. After that, tokens are emitted one at a time, each conditioned on the cache. Because this step reuses the cache rather than reprocessing the prompt, it is not compute-bound in the same way prefill is — it is memory-bound, and the cost of each new token grows as the sequence lengthens.

Why output still feels slow after the first token

Two effects explain the “slow after a fast start” experience. First, the KV cache grows with every generated token, and attention cost grows with it. NVIDIA notes that “with longer output sequences, the KV cache and its memory cost grow” and that attention computation cost “also grows with each new token because this cost is linear in the length of the input and output sequence so far” [1]. Second, batching shares GPU resources across requests, so a heavy prefill from another user can extend your forward pass [2].

The practical anchor: TTFT manages the start; ITL manages the process. Consistent inter-token latencies indicate efficient memory management, memory bandwidth, and attention computation [1] — and consistency is often more important to perceived quality than the raw average.

Inter Token Latency vs TPOT — Highly Related but Not Identical

What TPOT means

TPOT is “time per output token.” For a single request, the two metrics collapse into one number. Anyscale gives the formula:

“ TPOT (1 request) = Avg(ITL) = (E2E latency − TTFT) / (#Output Tokens − 1) “

For a single request, “the mean of all ITLs equals TPOT, which is why the two are sometimes used interchangeably” [3]. The denominator subtracts one because the first token is already accounted for by TTFT; the equation “characterizes only the decoding part of request processing” [1].

Where ITL and TPOT diverge

The equivalence breaks down once you aggregate across requests. vLLM, as documented by Anyscale, reports them differently: “average TPOT is the simple mean of each request’s TPOT (all requests weighted equally), while average ITL is token-weighted (all tokens weighted equally)” [2]. The documentation illustrates the gap with two requests:

• Request A: 100 tokens at ~10 ms per token → TPOT ≈ 10 ms
• Request B: 2 tokens at ~50 ms per token → TPOT ≈ 50 ms
• Avg TPOT = (10 + 50) / 2 = 30 ms (treats A and B equally)
• Avg ITL ≈ (100×10 + 2×50) / (100 + 2) ≈ 10.8 ms (weights by tokens)

The same two requests produce 30 ms under one aggregation and roughly 10.8 ms under the other [2]. Neither number is wrong; they answer different questions about the same traffic.

Why tools compute them differently

Three definitional choices drive the differences. First, whether TTFT is included: NVIDIA’s AIPerf excludes TTFT from ITL, computing ITL = (e2elatency − TTFT) / (Totaloutput_tokens − 1) [1], while other tools may include it. Second, the denominator: subtracting one restricts the metric to the decode phase [1]. Third, aggregation weighting across requests, as shown above [2].

The literature itself acknowledges the ambiguity. Anyscale states plainly: “The distinction between TPOT and ITL varies across the literature, and some sources treat them as equivalent” [2]. NVIDIA’s guidance is equally direct: “Tool implementations vary, so compare results only when definitions align” [1]. BentoML’s handbook adds a practical rule: “When reading benchmark results, always check how TPOT and ITL are defined. Different frameworks and papers may calculate and use the metrics differently, and this can change how you should interpret performance numbers” [3].

The correct framing is therefore: ITL and TPOT are highly related token-generation performance metrics. For a single request they are effectively equivalent; across requests they are not, because the aggregation convention differs. Writing “ITL = TPOT” as a blanket statement ignores that distinction.

Tokens per Second, Generation Speed, and Throughput

Tokens per Second (TPS) and how it is calculated

Tokens per second describes how many output tokens are produced per second, but “which TPS” matters. NVIDIA distinguishes two definitions. Total TPS per system “represents total output token throughput across all simultaneous requests,” computed as total output tokens divided by the elapsed time between the first request and the last response [1]. TPS per user is “throughput from a single-client perspective,” defined as output sequence length divided by e2e latency for each request, “which asymptotically approaches 1/ITL as the output sequence length increases” [1]. BentoML adds a unit caveat: “always check whether TPS metrics refer to input, output, or a combined view” [3].

The relationship between ITL and Tokens per Second

Per-user TPS and ITL are inverses: lower ITL means more tokens per second for that user, and vice versa. BentoML states the relationship directly: “A lower TPOT means the model can produce tokens faster, leading to higher tokens per second” [3]. This is why low ITL usually produces smoother streaming — the output arrives at a steadier, faster cadence, closer to the pace at which a person reads.

Throughput vs single-user generation speed

Throughput is a system-level property, not a user-level one. Databricks defines it as “the number of output tokens per second an inference server can generate across all users and requests” [4]. The distinction matters because the two move in opposite directions under load. NVIDIA notes: “As the number of concurrent requests increases, total system TPS increases while TPS per user decreases as latency increases” [1]. Total system TPS rises until it “saturates the available GPU compute resources. Beyond that point, TPS can decrease” [1].

Two sub-distinctions are worth keeping straight:

• Token throughput (TPS) counts output tokens per second across the system.
• Request throughput (RPS) counts completed requests per second, computed as total completed requests divided by the measurement window [1].

A deployment can have high token throughput and modest per-user speed, or the reverse. Conflating them leads to the classic mistake of reading a system throughput number as if it described what a single user experiences.

TTFT vs ITL and TPOT — Reading the Four Quadrants

TTFT vs ITL and TTFT vs TPOT

TTFT covers everything before the first token: NVIDIA states it “generally includes request queuing time, prefill time, and network latency” [1]. ITL and TPOT cover everything after it. End-to-end latency ties them together. NVIDIA’s formula is e2elatency = TTFT + Generationtime, where generation time is the duration from the first token to the final token [1]. Databricks expresses the same relationship in metric terms: “latency = (TTFT) + (TPOT) * (the number of tokens to be generated)” [4]. Both say the same thing: the total wait is the start plus the process.

Fast TTFT but slow generation vs slow TTFT but fast generation

Consider two models measured under identical conditions:

• Model A: first token appears very quickly, but the gap between each subsequent token is long.
• Model B: first token appears slightly later, but subsequent tokens arrive quickly.

Model A has low TTFT and high ITL. Model B has higher TTFT and low ITL. Laid out as a quadrant:

Model A wins the metric most people check first and loses the one they feel second-by-second. Model B does the reverse.

Why comparing TTFT alone cannot represent the full streaming experience

TTFT answers “how long until something appears?” It does not answer “how smoothly does the rest arrive?” A user watching Model A sees the interface respond instantly and then stall repeatedly — often described as laggy despite a fast first token. A user watching Model B waits a beat and then reads a steady stream. Anyscale captures the consequence: “TPOT, and its token-weighted counterpart ITL, capture the smoothness and speed of generation. Low, consistent values give a fluid real-time feel, while high or variable values make output appear in bursts, degrading chat or streaming UX” [2]. Evaluating streaming quality requires both dimensions, not TTFT alone.

What Affects Generation Speed — Output Length, Model Size, Hardware, and Concurrency

Output length and model size

Longer outputs extend total time almost linearly, because each additional token adds another decode step. Growing output also enlarges the KV cache and raises attention cost per token [1]. On the model side, the general pattern is that smaller or more efficient models generate tokens faster than larger ones under the same conditions — a directional relationship, not a fixed ranking, since architecture, quantization, and serving stack all intervene.

Hardware and provider infrastructure

The same model served by different providers can show different ITL and TPS, because inference engine, batching strategy, and hardware differ. This is exactly why NVIDIA warns to “compare results only when definitions align” [1] — and, by extension, only when infrastructure and test conditions are comparable. A number measured on one stack does not transfer to another.

Batch size and concurrency

Batching raises aggregate throughput while lowering per-user speed. NVIDIA’s inverse relationship — total TPS up, TPS per user down as concurrency rises [1] — is the central trade-off. Streaming changes how that trade-off feels: even when per-user speed drops, incremental delivery lets the user start reading immediately, so perceived responsiveness depends more on steady ITL than on total completion time. In streaming mode, NVIDIA notes the de-tokenization step “can be done multiple times when partial results are returned to the user” [1], which is what makes incremental display possible.

How to Measure ITL and TPOT Correctly

A measurement checklist before you trust a number

Before comparing any two numbers, confirm the definitional choices that produced them:

1. Definition: Does ITL/TPOT include TTFT? (AIPerf excludes it; other tools may not) [1].
2. Aggregation: Across requests, is it request-weighted (Avg TPOT) or token-weighted (Avg ITL)? The values differ [2].
3. Denominator: Was (#Output Tokens − 1) used to isolate the decode phase? [1].
4. TPS unit: Does “TPS” mean input, output, or combined? [3].
5. Conditions: Were prompt length, output length, concurrency, batch size, and hardware/provider recorded? [2][3].
6. Distribution: Are mean, median, P95, and P99 reported, not just an average? [3].
7. Repetition: Was the test run multiple times to exclude single-run noise?

Why you should run multiple tests

A single run can be skewed by prompt length, output length, concurrency, batch size, and hardware state. Because ITL and TPOT respond to all of these, one measurement is a snapshot, not a property. Repeating the test under controlled conditions is the only way to separate a model’s behavior from the noise of a particular run.

Average vs percentile

The mean is easy to report and easy to distort. A single slow request can pull it upward, so the mean may not describe what a typical user experiences. The median reflects the typical case; P95 and P99 expose tail latency — the slowest 5% and 1% of requests. For real-time applications, tail behavior often matters more than the average, because a user notices the worst stall, not the median token. BentoML’s guidance to report metrics “such as mean TTFT, median TPOT, and P99 E2EL” reflects this need to capture different aspects of the experience [3].

Which Tasks Care Most About Token Generation Speed

Scenario comparison

Different workloads prioritize different metrics:

Anyscale states the split directly: “interactive applications such as chatbots or coding assistants benefit most from low time to first token (TTFT) and inter-token latency (ITL) to maintain a responsive feel. In contrast, large-scale batch processing or data generation pipelines might prioritize high tokens per second (TPS) or requests per second (RPS)” [2]. Any target figures are directional and must be calibrated to your own test conditions.

How to improve generation speed

Improving generation speed is a combined decision, not a single knob:

• Model selection: Confirm the task actually needs the largest model; choose a smaller or more efficient one when quality allows.
• Provider selection: Compare TTFT/ITL/TPS measured under the same definitions and conditions, not across mismatched numbers [1].
• Output length control: Cap max tokens and prompt for concise outputs to keep E2E latency bounded [4].
• Routing: Send each request to the model or provider best suited to its latency-versus-throughput needs, rather than forcing one model to serve every workload.

ITL / TPOT / TPS quick reference

• ITL: average gap between consecutive tokens (decode phase); lower is smoother.
• TPOT: average time per output token; for a single request, equals Avg(ITL).
• TPS: tokens per second; per-user ≈ 1/ITL, total = system throughput.
• Memory anchor: TTFT manages the “start,” ITL/TPOT manage the “process,” TPS/Throughput manage the “scale,” E2E manages the “whole.”

Core metric comparison:

Key Takeaways and Next Steps

The one-sentence anchor

TTFT manages the “start,” ITL/TPOT manage the “process,” TPS/Throughput manage the “scale,” and E2E manages the “whole.” Inter token latency is the metric that explains why a fast first token can still feel slow — and why a slightly slower first token can still feel smooth.

FAQ

Next steps

If you are evaluating model or provider performance, start by fixing your measurement definitions — then compare only like with like. For deeper background, see our guides on LLM performance metrics, time to first token, and how to choose an LLM. To put routing decisions into practice, review LLM routing strategies.