LLM Latency: What It Is, What Causes It, and How to Reduce It

If you build chat, agent, coding, RAG, or any other real-time AI application, LLM latency is the number that quietly decides whether your product feels instant or broken. The same model can emit its first token in roughly 300 ms one moment and make a user wait four seconds the next. A model that streams at 90 tokens per second can still feel sluggish if the user has to stare at an empty screen first. Before you tune anything, it helps to understand what LLM latency actually is, where the time in a single request really goes, and which levers actually move user-visible waiting time.

This guide covers the definition, the causes, the measurement, and the optimization of LLM latency in one place — because treating them as separate problems is usually where slow AI products begin.

What Is LLM Latency and Why It Is Not a Single Number

LLM latency is the total time between a user request and the complete model response, and it is best understood as a stackable timeline of network, queueing, prefill, token generation, tool calls, and retrieval — not a single number. The moment you treat it as one metric, you lose the ability to fix it.

The developer's everyday puzzle: why the same model can emit a first token in 300ms one moment and make you wait 4 seconds the next

Every developer who ships an LLM feature has hit this: a request flies through, and the next identical one crawls. The reason is that what you measure as "latency" is really the sum of several stages, each with its own variable. A long prompt that has to be fully processed (prefill), a provider queue that is suddenly full, or a second of extra network round-trip can add seconds even when the model itself is unchanged.

Why "90 tokens per second" can still feel laggy: separating perceived speed from raw throughput

Throughput and perceived speed are different things. A model can generate quickly once it starts and still have a long, noticeable delay before that first token arrives if its prefill step — or an oversized prompt and context window — is slow [1]. In other words, quoting tokens per second tells you little about whether the user is staring at a blank screen. What the user feels is dominated by how fast the first token arrives and how smoothly the rest flows.

Core mental model: LLM latency is a stackable timeline, and end-to-end latency is what your user actually experiences

Hold this model in your head for the whole article: a single request is a stack of stages, and the time that matters is the end-to-end total. When you decompose the timeline, you can point at the exact stage that is slow. When you only watch the total, you are guessing.

Where the Time Goes: Decomposing a Single Request

From the moment a user clicks Send to the moment the complete response returns, time is spent across a sequence of stages — and the model's token generation is often only part of it.

The request timeline from Send to Final Response

A useful way to reason about a request is as a chain of stages. Each one can add latency independently:

``text User Request ↓ Network (round-trip to the API) ↓ Auth / Gateway processing ↓ Queueing (waiting for a free slot under load) ↓ Prompt / Prefill (processing all input tokens) ↓ First Token (TTFT) ↓ Token Generation (decode, token by token) ↓ Tool / Retrieval (if present) ↓ Final Response ``

The latency formula and the two-phase split

You can approximate the total with a simple formula: total ≈ TTFT + (output tokens ÷ tokens per second). The first token time (TTFT) covers everything up to the first token — network round-trip, provider queue, and prefill — while the generation phase after it is governed by how many output tokens you need and how fast the model produces them. As an example, a request with a 300 ms TTFT that generates a 240-token answer at 60 tokens per second takes roughly 4.3 seconds in total [2]. The split matters because the two phases have different causes and different fixes.

LLMs generate output one token at a time, and each token requires a forward pass through every layer of the model — so the generation phase is inherently serial and grows with output length [3].

Why tool calls, retrieval, and external API calls add serial waiting time on top of the model itself

In agent and RAG applications, the model is only one actor in a longer chain. Each tool call, each retrieval step, and each external API request runs sequentially on top of the model's own generation. That is why an agent request can be several times slower than a plain chat request even when every individual call is fast.

What Drives LLM Latency: Model-Side and Application-Side Causes

flowchart LR
    subgraph System["What Drives LLM Latency: Model-Side and…"]
      A["Prompt and context length"]
      B["Context window limits"]
      C["Output length"]
      D["Model size and model choice"]
    end
    A --> B
    B --> C
    C --> D
    Out["llm latency output"]
    D --> Out

The causes of LLM latency fall into two buckets: what you send and choose on the model side, and how your application and provider are set up on the other side.

Model-side levers: prompt and context length, context window limits, output length, and model size or model choice

What you send to the model and what you ask it to produce directly shape the timeline:

• Prompt and context length drive the prefill phase, because the model must process every input token before it can emit the first output token. Large prompts mean high GPU memory bandwidth and compute usage during prefill [4].
• Context window limits interact with prompt length: the closer you get to filling the window, the more prefill work every request does, and the slower the first token arrives.
• Output length drives the generation phase. Generation is serial and token-by-token, so longer outputs take proportionally longer — in many real workloads, output length, not prompt length, is what dominates end-to-end time [5].
• Model size and model choice set the ceiling on both phases. Larger, reasoning-oriented models tend to be slower to emit a first token; smaller models are typically faster but may sacrifice reasoning depth [6].

Provider-side levers: provider infrastructure and serving stack, region and geographic distance, and traffic or concurrency causing queueing

The model you call is not the whole story. A managed model's latency also includes the provider's serving stack, queueing, network path, and region [7]. Three provider-side factors stand out:

• Provider infrastructure and serving stack determine how efficiently requests are batched and served. Dynamic batching can raise throughput dramatically, but under heavy concurrency, requests queue and queueing cost starts to dominate high-percentile latency [8].
• Region and geographic distance add fixed round-trip time. Calling a provider endpoint on another continent adds tens to hundreds of milliseconds of network latency before any model work begins.
• Traffic and concurrency cause queueing. When a provider is overloaded, requests wait in line, which shows up as occasional very slow requests rather than a uniform slowdown.

Application/architecture amplifiers: agent workflows, multiple tool calls, and RAG retrieval each add waiting time per step

Your architecture can amplify latency far beyond the model. Multi-step agents with function calling accumulate latency at every hop — LLM calls, tool calls, and external API latency [9]. In one decomposed example, a total of 8.1 seconds came from roughly 4.2 seconds of LLM calls, 0.5 seconds of retrieval, 2.6 seconds of tool calls, and the rest from orchestration [10]. RAG adds similar overhead: agentic RAG is inherently slower than plain RAG because multiple LLM calls, parallel retrieval, and sufficiency checks add latency at every step [11].

How to Measure LLM Latency: TTFT, TPOT, and Why Percentiles Beat a Single Run

flowchart TD
    S0["Start: llm latency"]
    S1["1. Total generation time: how long the ful…"]
    S0 --> S1
    S2["2. End-to-end latency: the total your user…"]
    S1 --> S2
    S3["3. Time to first token (TTFT)"]
    S2 --> S3
    S4["4. Tokens per second / inter-token latency…"]
    S3 --> S4
    S5["5. Total generation time"]
    S4 --> S5
    Done["Outcome: How to Measure LLM Latency: TTFT, TPOT…"]
    S5 --> Done

To fix LLM latency, you need to measure each phase separately and track percentiles — because a single run or an average hides the tail that actually breaks the experience.

The metrics that matter for a request

Break a request into metrics you can act on:

• Time to first token (TTFT): how long until the first output token arrives. This is the strongest driver of perceived responsiveness.
• Tokens per second / inter-token latency (TPOT / ITL): how fast output tokens arrive after the first one. This determines whether streamed output feels smooth or stuttery [5].
• Total generation time: how long the full output takes, driven by output length.
• End-to-end latency: the total your user experiences, including all application stages.

Why you should not measure once, and why average latency hides the tail

A single measurement tells you almost nothing, because latency is distributed, not fixed. Average latency is especially misleading: it is pulled up by the same few slow outliers that a handful of users actually feel, and it hides the tail that breaks the experience. For each model and endpoint, you want to record TTFT, TPOT, total generation time, and end-to-end time as distributions [12].

Tracking p50 / p95 / p99 per model and endpoint, and treating error rate and cancelled requests as separate thresholds

Watch the percentiles — p50, p95, p99 — per model and per endpoint, not just the average. Also avoid a measurement trap: do not let fast failures beautify your latency numbers. Compute percentiles over successful responses only, treat error rate as an independent threshold, and never count requests the client cancelled as fast generations [13].

A symptom-to-cause diagnostic table

When something is slow, map the symptom to a likely cause and a metric to check:

How to Reduce LLM Latency: Engineering and Model Strategies

Most latency wins come from a small set of repeatable moves: send less, ask for less, pick the right model, stream output, and remove unnecessary round trips.

Trim what you send and what you ask for

The cheapest latency is the latency you never create:

• Cut unnecessary context. Ask whether the whole context really needs to go to the model. Trim history, summarize, or keep only the retrieved chunks that matter.
• Control output length. Set max_tokens to the smallest value the task needs, since the generation phase scales with output length.
• Keep prompts lean. Fewer input tokens mean less prefill work before the first token arrives.

Choose a faster or better-suited model, and stream so the first token starts reading immediately

Model choice is one of the highest-impact levers. For tasks that tolerate less reasoning, a smaller, faster model can cut response time dramatically [6]. For genuinely hard tasks, keep the larger model but reserve it for the requests that need it. Enable streaming whenever you can: it lets the user read the first token as it arrives instead of staring at a blank screen for the whole generation. Streaming can cut perceived latency by roughly 60–80% because users immediately see tokens rather than waiting for a complete response [14]. Human reading speed (roughly 200–250 words per minute) is often faster than many models generate, so as long as the first token arrives quickly, the streamed output can keep pace with reading [15].

Pick the right provider and region, enable caching for repeated prefixes, parallelize independent work, and route around slow or overloaded endpoints

• Provider and region selection. Latency varies between providers even for models of similar size, so trying a comparable model on a different provider — or a region closer to your users — can help [6][7].
• Caching. Prompt caching reuses the processing of stable prefixes across requests. OpenAI documents that prompt caching can reduce latency by up to 80% and input token costs by up to 90% on supported workloads [16]. Move static content — system prompt, instructions, examples — to the start of the prompt so it can be cached across requests.
• Parallel execution. Run independent work concurrently. Parallel tool calls can cut agent latency by up to 3.7x because multiple tools execute at once and results return in one batch [17].
• Routing. Give yourself a fallback path. When a provider is overloaded or slow, route around it to an available alternative instead of waiting in queue.

Reduce unnecessary tool calls and collapse agent steps where a single call can do the job

Every tool call is a serial round trip on top of the model. Audit your agents: can two calls become one? Are some tools being called when their results are not actually needed? Limiting the number of tools called per query and running the rest in parallel is a direct way to cut total waiting time [18].

Latency Trade-offs and What Each Application Can Tolerate

Faster is not always better — cutting LLM latency usually trades against quality and cost, and the right balance is specific to each workload.

Latency versus quality

A smaller, faster model can cut response time dramatically — in some cases from around 8 seconds down to 800 ms — but it tends to raise hallucination or reasoning risk. There is an optimum in that trade-off, but it is task-specific, not universal [19].

Latency versus cost

Many latency levers also reduce cost: caching lowers both latency and input-token spend, smaller models are cheaper per token, and tighter output limits reduce both generation time and billed output tokens. The balance differs by workload — a high-volume repeated query benefits most from caching, while a complex one-off task may justify a larger model.

Why acceptable latency differs by scenario

Different applications optimize different metrics and tolerate different amounts of waiting:

Putting It Into Practice: A Latency Optimization Checklist

Use these three checklists before you ship or when you are chasing a slow request.

Pre-ship: trim context, cap output, choose the right model, stream, and validate proximity

1. Do I really need to send the full context? Can I trim history, summarize, or keep only the necessary retrieved chunks?
2. Is the output token limit set too high? Can max_tokens be capped to the minimum the task needs?
3. Can this task be handled by a faster, better-suited model — leaving the larger model for genuinely complex requests?
4. Is streaming enabled so the user can start reading at the first token?
5. Is the provider and region as close to my users as possible? Have I priced in the geographic round-trip?

In production: cache, parallelize, reduce tool calls, and set up routing or failover

1. Have repeated static prefixes (system prompt, instructions, examples) been moved to the start of the prompt to hit the cache?
2. Are there tool calls in my agent that can run in parallel? Can I reduce the number of calls per turn?
3. Is there a routing or failover path so slow or overloaded providers can be bypassed?

Measuring: percentiles, not averages, with errors tracked separately

1. Am I monitoring TTFT, TPOT, total generation, and end-to-end time with p50 / p95 / p99 — not averages or single runs?
2. Are error rate and cancelled requests tracked as separate thresholds, not counted as fast responses?

Conclusion: Core Takeaways and Next Steps

Decompose the timeline, measure with percentiles, and apply the model-side and application-side strategies that fit your workload. That is the whole mental model in one sentence.

Start by measuring your own requests end to end — TTFT, generation time, and the p95/p99 tail. Then apply the highest-impact levers in order: streaming, leaner context, controlled output length, and the right model choice. Where your product uses agents or RAG, audit tool calls and retrieval for serial round trips you can parallelize or remove.

Explore Models to compare latency-relevant model options on the gomodelhub model catalog, and Read the Docs to wire them into your stack through one OpenAI-compatible API. When you can switch models without rebuilding your application, choosing the faster model for the right task becomes a routine decision instead of a migration.