What Is Time to First Token (TTFT) in LLMs?

For a streaming LLM application, "fast" is decided before the model has written a single full sentence. When a user clicks send in a chat interface, what they actually feel is the gap until the first streamed character appears — not the moment the final reply finishes. That gap is Time to First Token (TTFT), and it is frequently the difference between an app that feels instant and one that feels broken.

The same model, the same prompt, can return its first output token in roughly 300 ms or in several seconds. The spread is rarely caused by the model itself. It lives in the path the request travels before generation begins: network round-trips, queues, gateway processing, and the prefill pass over your prompt. This guide explains what TTFT measures, where its measurement boundary starts and ends, why it is the single most important latency metric for interactive and streaming apps, which factors move it, and how to measure, interpret, and reduce it in production.

What Is Time to First Token (TTFT)?

Time to First Token (TTFT) is the latency between a client sending a request and receiving the first generated output token back from the model [1][2]. In plain terms, it answers one question: after I hit send, how long until I see the first word of a response?

Why "fast" for a streaming app is decided by the first token, not the full reply

Before streaming APIs became standard, a user had to wait for the entire completion before any text appeared, which made total response time the only thing they could perceive [3]. Streaming changed that contract. Now the model can return one token at a time, and the perceived responsiveness of the whole interaction collapses onto a narrow window: the time to that first token [4]. A reply that streams smoothly after a healthy first token feels responsive even if the full answer takes a while. A reply that stalls before its first token feels slow no matter how fast the rest arrives.

A one-sentence definition and what this guide covers

TTFT is the elapsed time from the moment a request is sent (or received by the server, depending on the measurement baseline) until the first output token is generated [5][2]. This guide walks through the definition boundary, the phases a request passes through before its first token, the factors that move TTFT up or down, how to measure it reliably, and the practical levers for reducing it.

How TTFT Is Measured: From When to When

flowchart TD
    S0["Start: ttft"]
    S1["1. Network latency — the round-trip betwee…"]
    S0 --> S1
    S2["2. Request queueing — waiting for compute…"]
    S1 --> S2
    S3["3. Client-send baseline."]
    S2 --> S3
    S4["4. Server-receive baseline."]
    S3 --> S4
    S5["5. First-byte baseline (HTTP/routing level…"]
    S4 --> S5
    Done["Outcome: How TTFT Is Measured: From When to When"]
    S5 --> Done

TTFT sounds like a single number, but in practice it depends on where the clock starts and where it stops. Two teams reporting "TTFT" can be measuring different things and still both be correct.

The three measurement baselines

There is no single universal start point for TTFT, and the choice changes the number you get:

• Client-send baseline. The clock starts when the client finishes sending the request and stops when the first streaming chunk arrives. This is the view from IBM and DigitalOcean, and it is the number your users actually experience because it includes your network round-trip [1][2].
• Server-receive baseline. The clock starts when the serving engine receives the request. vLLM documents its TTFT metric this way, and it explicitly includes the time a request spends waiting in the pending queue before prefill begins [5][2].
• First-byte baseline (HTTP/routing level). Some platforms measure TTFT at the routing or HTTP layer, while inference engines such as vLLM, SGLang, and TensorRT-LLM measure it internally. NVIDIA's AIPerf documentation makes this distinction explicit: the same label applied at different layers produces different values [6].

Because these baselines differ, TTFT numbers from different tools and providers are not directly comparable. The practical rule is to always state which baseline you are using and to compare like with like.

Why the measurement layer matters

The measurement layer determines how much of the request path is included. A routing-level TTFT captures network hops and gateway processing on top of model compute, whereas an inference-engine TTFT only reflects what happens inside the server after the request arrives [6]. If you are debugging user-perceived slowness, the client-send or routing-level number is the one that matters; if you are tuning the serving stack, the engine-internal number is more useful.

The request timeline before the first token

Between "send" and "first token," a request typically passes through several stages, and each can add latency [1]:

1. Network latency — the round-trip between your client and the provider's endpoint.
2. Request queueing — waiting for compute if the serving system is saturated.
3. Authentication / gateway processing — key validation, routing decisions, and any middleware.
4. Prompt processing / prefill — the model reads and processes your entire input before it can generate anything.
5. Model loading / scheduling — on a cold or freshly scaled instance, weights must be loaded and the request scheduled onto a GPU.

Only after all of these does the first output token get generated and streamed back. TTFT is the sum of everything on this path up to that first token [1][7].

Why TTFT Matters: Perceived Speed and Streaming

flowchart TD
    S0["Start: ttft"]
    S1["1. Interactive chat"]
    S0 --> S1
    S2["2. Batch pipelines and offline processing"]
    S1 --> S2
    S3["3. Code generation and IDE autocomplete"]
    S2 --> S3
    S4["4. TTFT is not just one latency number amo…"]
    S3 --> S4
    S5["5. Not every application should optimize T…"]
    S4 --> S5
    Done["Outcome: Why TTFT Matters: Perceived Speed and S…"]
    S5 --> Done

TTFT is not just one latency number among many — it is the metric users feel first, and for streaming applications it is often the metric they feel most [4].

TTFT is the primary driver of perceived responsiveness

For interactive chat, voice, and agent interfaces, TTFT together with inter-token latency defines how "alive" the app feels [8]. A low TTFT tells the user the system understood the request and is working; a high one tells them the app is unresponsive. The counter-intuitive risk is that a slow TTFT combined with fast streaming can feel worse than no streaming at all: the user is shown a promise of immediate content ("typing…") and then left waiting for it [9].

Different workloads care about different clocks

Not every application should optimize TTFT above all else. The right metric depends on the workload [8]:

• Interactive chat values TTFT and inter-token latency, because users want a fast first token and smooth subsequent flow.
• Batch pipelines and offline processing value throughput — how many tokens or requests the system finishes per unit of time.
• Code generation and IDE autocomplete often care about end-to-end latency, because the developer needs the complete, valid result before acting on it.

The two-clock reality: TTFT is not the full answer time

A common mistake is to conflate TTFT with "how long the full answer takes." They are different clocks. A useful estimate of total completion time is roughly TTFT plus the number of output tokens divided by the tokens-per-second rate [10]. A model can have an excellent TTFT and still be slow to finish if its per-token generation is slow — and the reverse is possible too. TTFT only tells you when the answer starts, not when it ends.

TTFT vs. Related Latency Metrics

TTFT is one member of a family of latency metrics, each answering a different question. The table below summarizes where each one starts and stops [7][1][4].

Reading the table

The table makes one point concrete: TTFT answers "when does it start," while end-to-end latency, tokens per second, and TPOT describe what happens after it starts [1][7]. If you need to know how long a full answer takes, TTFT alone is not enough — you need the decode-side metrics as well. Do not treat TTFT as a proxy for total response time.

Where to go for the related metrics

This guide keeps the focus on TTFT, but the neighboring metrics deserve their own deep dives. For a fuller picture of the whole performance family, see our guides on LLM performance metrics, LLM latency, and inter-token latency / TPOT.

What Affects TTFT: Request-Side Factors

Some of the largest influences on TTFT are under your control before the request ever leaves your code.

Prompt and context length

The biggest single contributor to TTFT is the prefill phase, during which the model must process every token in your input before it can generate a single output token [7][3]. Prefill scales roughly linearly with the number of input tokens: each extra token adds processing time [11]. This is exactly why long-context requests feel slow — the model cannot start answering until it has read the whole context [12]. The more system prompt, conversation history, or retrieved context you send, the longer prefill runs and the higher TTFT climbs.

Model size and architecture

Model size is a second request-side lever. Larger models generally take longer to produce a first token, even as their per-token generation may be slower or faster depending on architecture [13]. A frontier model optimized for reasoning quality is not automatically the right choice when your requirement is a fast first token for a simple, real-time task.

Choosing for real-time scenarios

For interactive workloads, the "strongest" model is not always the best pick. If a task is straightforward and the user is waiting on a first token, a smaller or faster model can deliver a better experience than a larger one whose quality advantage is irrelevant to that particular request. This is where routing strategies and per-task model selection become valuable, a point we return to below.

What Affects TTFT: Infrastructure and Provider Factors

Even with a short prompt and a fast model, infrastructure and provider conditions can dominate TTFT.

Provider and region

Network round-trip time is baked into the TTFT your users experience, and it depends on where the provider's compute sits relative to your users or your service [1][3]. A provider endpoint on the other side of the world adds tens to hundreds of milliseconds of pure network latency before the model has done any work.

Cold starts

On a cold or freshly scaled instance, the model weights must be loaded, caches may be empty, and connections must be established before inference can begin. Cold-start TTFT can be many multiples of warm TTFT, producing sharp latency spikes on idle conversations and first requests after scale-down [14]. Warm TTFT is the steady-state number you should tune against; cold TTFT is a separate problem with separate levers.

High-concurrency queueing

Under load, a provider queues requests, and that wait happens before prefill even starts. On a saturated serving system, a request can wait hundreds of milliseconds in the queue before the model begins processing it [15]. A violation of your TTFT service-level objective is often not caused by slow prefill but by an overloaded serving system [15]. Worse, this queueing wait is frequently invisible in client-side timing that only records network and compute — the request is neither failing nor starting, it is simply waiting [13].

How to Measure and Interpret TTFT

Measuring TTFT reliably takes a little discipline, because a single sample is noisy and averages can hide the failures that matter.

A practical measurement method

The direct method is to record a start timestamp when you send the request, then capture the timestamp when the first streaming chunk that contains real generated text arrives, and subtract [16]. Two cautions apply: repeat the measurement many times and aggregate, because individual runs are noisy [16]; and do not count metadata or empty delimiters as the first token — wait for the first actual generated output [2].

Why a single test is not enough and averages can mislead

A single request tells you almost nothing about your real TTFT, because cold starts, queueing, and network jitter all vary run to run. Averages are also misleading: one slow outlier pulls an arithmetic mean up even when most users are fine. Percentiles describe the distribution far better.

Understanding p50, p90, and p95

• p50 (median) describes the typical user — half of requests are faster, half are slower [17].
• p90 and p95 capture the long tail and are the primary SLO targets for most user-facing services [17]. p95 is a common target for interactive chat applications.
• The gap between the median and p95 exposes how well a system handles load. If the median is low but p95 is high, a meaningful fraction of users — roughly 1 in 20 at p95 — is having a poor experience [18]. A provider could show a clean median while its p95 tells a very different story under concurrency.

When you track TTFT, watch both a central value and a tail value, and look at how the gap between them behaves as concurrency rises.

How to Reduce TTFT: A Practical Action Guide

Reducing TTFT is a matter of pulling the right levers for your situation. The table below maps each lever to its trade-off and best-fit scenario.

A pre-send TTFT self-check for developers

Before you deploy, run through this checklist to catch the most common causes of avoidable TTFT:

1. Is the request using streaming? Without it, perceived TTFT is effectively end-to-end latency.
2. Can the prompt or context be trimmed? Remove non-essential system instructions and history tokens — prefill is roughly linear in input length [11].
3. Is the model chosen for the real-time scenario, not just for maximum capability?
4. Is the provider or region close to your users or your service?
5. Is there a routing or fallback path that can land the request on the fastest healthy provider?
6. Are cold-start requests being distinguished from steady-state requests, so cold data does not distort your warm evaluation?
7. Are you tracking both p50 and p95, rather than a single sample or only an average?

How routing fits in

Latency-aware routing and automatic fallback can materially improve the tail of your TTFT. A router can send each request to the fastest healthy provider and retry elsewhere if that provider errors, which improves p95 and reduces outages without you writing failover logic [19]. Routing also lets you pick a model per request, improving latency, cost, and accuracy together rather than forcing one model to serve every call [20]. The trade-off is real: every routed request crosses an additional hop, so the routing layer itself must be low-latency or it can offset the gains it is meant to create [3].

Core Takeaways and Next Steps

Recap

• TTFT measures when an answer starts, not when it ends. Do not conflate it with total response time.
• Define your measurement baseline (client-send, server-receive, or routing-level) before comparing numbers across tools or providers.
• Track both p50 and p95, and watch the gap between them to understand how your system behaves under load.
• Optimize the fastest levers first: trim the prompt, choose a model suited to the real-time task, pick a nearby region, enable streaming, and use routing with fallback.

Next steps

If you are building an LLM application and want to compare models or apply routing to keep TTFT under control, explore the models available through the GoModelHub unified API and read the GoModelHub docs to see how one OpenAI-compatible endpoint can switch between models without reworking your stack.

Related reading

LLM performance metrics — the full metric family beyond TTFT.

LLM latency — a deeper look at the latency stack.

Inter-token latency / TPOT — what happens after the first token.

LLM routing strategies — latency-aware routing and fallback.

How to choose an LLM — matching model choice to your latency and quality needs.