LLM Performance Metrics: TTFT, Latency, TPOT, and Throughput Explained

Imagine you pick a model that tops a reasoning leaderboard, wire it into your product through an API, and your users immediately complain that responses "take forever" and the spinner spins too long. The model is genuinely capable — but capability is not the same thing as responsiveness. What you are actually measuring, once a model is deployed or called through an API, belongs to a different family of numbers.

This guide is about that second family: LLM performance metrics. It is not a model ranking and it does not compare which model is "best." Instead, it explains the metrics you use to judge whether an LLM API feels fast, streams smoothly, and can carry the concurrency your product needs. We will define each metric, explain the differences in how tools compute them, and show you which metrics to watch for each use case.

Why a "Powerful" Model Can Still Feel Slow

flowchart TD
    Start["llm performance metrics"] --> Q{"Why a 'Powerful' Model Can Still Feel S…?"}
    Q -->|Yes| Y["LLM performance metrics"]
    Q -->|No| N["When a model is served through an API…"]
    Y --> Out["This article deliberately stays inside…"]
    N --> Alt["Mitigate llm performance metrics"]

Before any metric makes sense, you need to separate two questions that developers routinely conflate: How well does the model answer? and How fast does it respond when called? A leaderboard can tell you the first; it tells you almost nothing about the second.

The developer's real puzzle: a leaderboard-topping model that "takes forever" once you call its API

When a model is served through an API, its speed depends on a deployment stack — prefill, decode, batching, queueing, hardware, and network — none of which appear on a capability leaderboard. A system tuned purely for tokens-per-second throughput can still return a poor time to first token if requests sit in a queue before generation starts [1]. The inference stack between the model weights and your user is where perceived speed is actually decided.

Two orthogonal metric families: Model Quality Metrics vs. Inference / API Performance Metrics

Model quality benchmarks (MMLU, reasoning, coding suites) answer one question: can the model answer well? Inference and API performance metrics answer a different one: how fast, how smooth, and at what concurrency does the model behave once it is deployed or called? These two families are orthogonal — measuring one tells you nothing about the other [6]. A capable model can be slow to serve, and a lightweight model can stream tokens quickly but answer poorly. Both sets of numbers are needed; they just answer different questions.

Scope of this guide: only the second family matters here

This article deliberately stays inside the inference and API performance family. It does not rank models, does not compare which model is fastest, and does not walk through MMLU or coding quality benchmarks. Those topics deserve dedicated coverage. Here, the goal is a practical, developer-facing explanation of the LLM performance metrics you rely on when you deploy or call an LLM through an API.

What Are LLM Performance Metrics?

flowchart TD
    S0["Start: llm performance metrics"]
    S1["1. LLM performance metrics"]
    S0 --> S1
    S2["2. Quality benchmarks evaluate the output…"]
    S1 --> S2
    S3["3. llm performance metrics context"]
    S2 --> S3
    S4["4. What Are LLM Performance Me… mechanism"]
    S3 --> S4
    S5["5. llm performance metrics controls"]
    S4 --> S5
    Done["Outcome: What Are LLM Performance Metrics?"]
    S5 --> Done

A working definition: LLM performance metrics are the measurements that describe how a deployed model or an API call behaves in terms of latency, generation speed, and system throughput. They are the numbers you read when you want to know whether an endpoint is fast enough for your product.

Quality benchmarks measure "can it answer well"; inference performance measures "how fast, how smooth, and at what concurrency"

Quality benchmarks evaluate the output itself. Inference performance evaluates the serving behavior: the delay before the first token, the pace of the streamed response, and how many concurrent requests the system can sustain. The two are orthogonal and both are needed [6]. When you pick a model for a product, you are choosing on both axes — but you can only compare them if you keep them separate.

Why developers cannot rely on a single performance number

Each metric answers a different question about perceived responsiveness, generation experience, and system behavior. As one practitioners' guide puts it, "Inference performance is not one number. Practitioners generally track five distinct metrics, and conflating them is the most common source of misleading comparisons" [4]. TTFT tells you how responsive the first token feels; TPOT or tokens per second tell you how smoothly the rest streams; throughput tells you what the system can carry; percentiles tell you about the worst requests. Pick one and you are blind to the others.

The Core LLM Performance Metrics, Explained

Here is the set of metrics you will encounter in almost every LLM serving benchmark. The table below summarizes them; the subsections that follow explain each one and, critically, the differences in how tools compute them.

Time to First Token (TTFT): the first "gate" users perceive as responsiveness

Time to First Token is the delay from when a request is submitted to when the first output token arrives. Anyscale defines it as "initial latency before the first token appears" [2]. NVIDIA notes that TTFT generally includes request queuing time, prefill time, and network latency — and that longer prompts increase TTFT because the attention mechanism builds the KV cache over the full input before generation begins [1]. For interactive streaming experiences, TTFT is the single strongest signal of perceived responsiveness.

End-to-End Latency (Total Response Time): the full request-to-complete-response duration

End-to-end latency measures how long it takes from submitting a query to receiving the complete response, including queueing, batching, and network time [1]. It is the metric that captures the whole wait a user experiences for a complete answer. Because it grows with output length and reasoning, it is the metric to watch for one-shot calls where the user must wait for the full result before doing anything useful.

Time Per Output Token (TPOT) vs. Inter-Token Latency (ITL): aggregate vs. the literal gap

TPOT and ITL are closely related but not identical. TPOT is an end-to-end aggregate — roughly the total time from the first to the last generated token divided by the number of tokens. ITL is the literal gap between two consecutive output tokens, which reveals jitter and spikes in the stream [4][10]. Some tools treat the terms as interchangeable; others compute them differently. For example, vLLM calculates TPOT "once per request, excluding the first token, and then aggregated across requests" [4]. The practical takeaway: TPOT answers "how fast is generation on average," while ITL answers "is the stream smooth or does it stall in bursts."

Tokens per Second and Throughput: one stream vs. the whole system

Tokens per second measures generation speed on a single stream — how many output tokens arrive per second. Throughput measures system-wide capacity: tokens per second or requests per second processed across all concurrent requests under load [1][7]. OpenAI's operational guidance distinguishes a similar pair: "token velocity," which is independent of prompt size, and "request time," which is heavily influenced by output length and reasoning [9]. Throughput is the number that drives cost and concurrency planning; tokens per second is the number your end users feel on a single stream.

Terminology caveat: TPOT, ITL, and Throughput are computed differently across tools

Different tools — vLLM, NVIDIA NIM, Anyscale, BentoML — compute TPOT, ITL, and throughput in slightly different ways. As NVIDIA's benchmarking documentation warns, "Tool implementations vary, so compare results only when definitions align" [1]. Before you compare numbers across providers or tools, confirm that the definitions match. A TPOT from one benchmark may not be directly comparable to a TPOT from another if one excludes the first token and the other does not.

Why Latency Percentiles Beat a Single Average

A single average latency number can look perfectly healthy while a meaningful share of your requests are slow enough to drive users away. That is why production benchmarks report percentiles instead.

P50 / P95 / P99 defined: the median experience vs. the slowest-tail requests

P50 is the median — half your requests are faster. P95 and P99 describe the tail: 95% or 99% of requests complete at or below that latency [5]. The tail matters because a small fraction of slow requests is often exactly what causes user churn, timeouts, and support tickets. As the nrouter engineering team puts it, "An average latency number hides the requests that ruin your UX" [5].

How an average can look healthy while masking bad tail latency

Because LLM latency is structurally heavy-tailed, the mean can sit close to the median while a nontrivial share of requests land far out in the tail [4][5]. A benchmark that reports only an average masks exactly the requests that hurt user experience. Percentile reporting — p50, p95, p99 — is the standard practice in AI serving precisely because it exposes that distribution [4].

Setting SLOs the right way: layered budgets instead of one number

Rather than a single latency target, teams set layered budgets per percentile — for example, a P50 TTFT under 200 ms, a P95 TTFT under 1.5 s, and a P99 TTFT under 4 s [8]. (Treat these as illustrative thresholds, not universal requirements.) Layering forces you to care about the tail, not just the typical case. If you only set one number, you will optimize the middle and ignore the requests that actually break the experience.

What Actually Moves These Metrics

Latency and throughput are not fixed properties of a model. They shift with the request, the deployment, and the network. Understanding the levers is the prerequisite for choosing the right metric.

Input token length and output length: they directly change TTFT, TPOT, and total response time

Longer prompts increase TTFT because prefill processes the full input and builds the KV cache before generation begins [1]. Longer outputs extend the generation phase and therefore raise end-to-end latency and total tokens. Input and output length are the two variables that most directly scale the numbers you see [10][14]. A benchmark run on a short prompt and short output tells you little about a long-context workload.

Model size, provider infrastructure, and hardware: they shift the whole latency profile

Model size, quantization, batch load, and memory bandwidth all shape inference latency [3]. Larger models require more compute per token; quantized or smaller models generally serve faster. Provider infrastructure and hardware capacity set the ceiling on how fast and how many requests a deployment can handle [7]. These factors change the entire latency profile, not just one metric [14].

Region and network: physical distance changes TTFT and round-trip time

TTFT includes network latency, so the physical distance between your client and the serving region directly affects how quickly the first token streams back [1]. A call routed across continents adds meaningful round-trip time before generation even begins. When you compare providers or endpoints, you must hold region and network constant, or you are measuring geography, not the model.

Queueing, concurrency, streaming, and caching: how load redistributes latency across metrics

Under load, requests queue, which inflates TTFT even when generation itself is fast [1]. Higher concurrency raises throughput but can lengthen per-request latency as the system approaches saturation [7]. Streaming changes what users experience — they see the first token sooner even if total latency is unchanged. Prompt caching can cut prefill time on repeated prefixes, improving TTFT for cached inputs. These levers often move metrics in opposite directions: as one practitioners' note observes, optimizing for system throughput while ignoring TTFT can hurt the UX of workloads that wait on the first chunk [11].

Which Metrics to Watch for Your Use Case

The right metric depends on what your users are waiting for. This table maps common scenarios to the metrics that matter most.

Interactive streaming vs. one-shot calls vs. background batch: reading the same metrics differently

Interpret the same metrics differently by scenario. For interactive streaming (chat, copilots), TTFT and ITL dominate the experience because the user watches tokens appear. For one-shot calls that must return a complete answer, end-to-end latency is the number that matters. For background batch jobs, throughput and cost per unit of work matter more than the latency of any single request [12][7]. The metric set is the same; the emphasis is not.

Single-metric risk: what you miss when you watch only one number

• Only tokens per second: you miss TTFT and tail latency — a fast streamer can still feel slow if the first token takes too long.
• Only average latency: you miss the tail — slow requests that drive churn are hidden in the mean [5].
• Only a one-shot benchmark: you miss how throughput and latency degrade under concurrency and queueing [7].
• Only model quality scores: you miss actual API speed and serving capacity entirely [6].

How to Build a Basic LLM Performance Test

If you want numbers you can trust, you need a reproducible test — not a single ad-hoc cold-start call. The steps below give you a baseline you can compare across providers and over time.

A reproducible test recipe: fix model version, provider, region, network, and workload

Pin the environment before you measure. Fix the exact model version and provider, because model iterations change performance. Fix the region and endpoint and record the network environment, since geography changes TTFT and round-trip time [1]. Fix input and output length and the prompt template, because those directly determine generation time [10]. Change any one of these and you are no longer testing the same thing [13]. Reproducible benchmarks pin the exact dated model snapshot and test conditions so the numbers can be compared over time and across environments [13].

Run multiple times under target concurrency and record percentiles

Never trust a single cold-start number. Run multiple rounds at your target concurrency and report percentiles (P50, P95, P99), not just an average [4][5]. A concurrency sweep — repeating the benchmark at increasing numbers of simultaneous requests — reveals how throughput and latency change as the system approaches saturation [7]. This is the only way to see whether your endpoint degrades gracefully under real load.

Log the test method and tool, and never present third-party benchmarks as your own

Record the test method and tool so that TPOT, ITL, and throughput definitions match before you compare providers [1]. Because tools differ in how they compute these metrics, a number from one benchmark may not be comparable to one from another. And if you cite a third-party benchmark — from NVIDIA, Anyscale, vLLM, or anyone else — attribute it clearly. Do not present another organization's test as GoModelHub's own measurement.

Performance Metrics Checklist

Run this checklist before you evaluate any LLM endpoint:

• [ ] Am I judging "how fast is the deployed or API-served model," not "how capable is the model"? (If the latter, switch to a quality-benchmark article.)
• [ ] Am I evaluating latency with percentiles (P50/P95/P99), not a single average?
• [ ] Do I distinguish TTFT (first token), TPOT/ITL (generation phase), and end-to-end latency (total time)?
• [ ] Have I confirmed the tool's definition of TPOT/ITL/Throughput before comparing across providers or tools?
• [ ] Did I record Model Version, Provider, Region, Test Date, and Test Method?
• [ ] Did I run multiple times under my target concurrency, not a single cold-start call?
• [ ] Did I record input length, output length, model size, streaming on/off, and caching as known variables?
• [ ] Did I choose the metric that matches my scenario (interactive vs. batch vs. agent), rather than watching a single number?

Key Takeaways and Next Steps

Treat LLM performance metrics as a set, not a single score. TTFT, end-to-end latency, TPOT/ITL, tokens per second, throughput, and P50/P95/P99 each answer a different question about responsiveness, generation experience, and system capacity. Pick the ones that match how your users wait, and read them together.

Recap: choose the metric set by use case

For interactive chat, watch TTFT and its tail percentiles. For coding assistants, combine a fast TTFT with steady generation speed. For long-context or one-shot calls, track end-to-end latency and TPOT. For batch work, optimize throughput and cost. For agent workflows, add P99 because sequential calls accumulate latency. No single number is a verdict.

The path forward: run your own reproducible tests and compare only when definitions align

The most reliable way to understand an endpoint is to test it yourself under your own workload: pin the model version and provider, hold region and network constant, fix input and output length, run multiple rounds at target concurrency, and log percentiles and the test method. Compare providers only when the tool definitions align — otherwise you are comparing measurement conventions, not performance [1].

Related reading and action

To apply these metrics when choosing and routing models, explore available models and read the platform docs to understand how to measure latency, throughput, and cost across the models you evaluate. Explore Models · Read the Docs