Streaming (LLM Responses)

What is streaming?

LLMs generate tokens one at a time, and streaming forwards each one to the client as it’s produced — usually over server-sent events (SSE). A 30-second generation feels instant because the first words appear in under a second. Every serious chat UX streams; the “wait for the spinner” pattern died with GPT-3.5.

Why it’s a UX decision and an architecture decision

The metric streaming optimizes is time-to-first-token (TTFT) — perceived responsiveness — decoupling it from total generation time. But it ripples through your stack: every hop (gateway, backend, CDN) must pass chunks through without buffering (proxy buffering silently un-streams you), timeouts must accommodate minutes-long connections, and billing/logging happens at stream end, not request start.

The hard parts nobody mentions

• Structured output — you can’t JSON.parse half a JSON object; use incremental parsers or stream freeform text and emit structure at the end.
• Tool calls mid-stream — the stream pauses while your code executes tools, then resumes; UIs need states for “thinking”, “acting”, and “writing”.
• Guardrails on partial output — content you’ve already streamed can’t be unshown; moderation must run on the stream (with small buffers) or accept post-hoc retraction UX.
• Error mid-stream — a failure at token 500 needs retry-with-resume or a graceful “regenerate”, not a blank bubble.

What people get wrong

• Streaming everything. Machine-to-machine calls (pipelines, batch jobs) gain nothing; non-streaming is simpler and easier to retry.
• Judging speed by tokens/sec alone. Users perceive TTFT far more than throughput; optimizing the wrong one wastes money.
• Forgetting mobile networks. Long-lived SSE connections through flaky networks need reconnect logic; test on a train, not localhost.

Try it in code

Consuming a server-sent-events stream and measuring what matters (TTFT):

import time

def stream_chat(prompt):
    start = time.monotonic()
    first_token_at = None
    full = []
    for event in llm_stream(prompt):          # SSE iterator from any provider SDK
        if event.type == "token":
            if first_token_at is None:
                first_token_at = time.monotonic() - start   # <-- TTFT
            full.append(event.text)
            print(event.text, end="", flush=True)
    total = time.monotonic() - start
    print(f"\nTTFT: {first_token_at:.2f}s | total: {total:.2f}s "
          f"| {len(full)/max(total,0.001):.0f} tok/s")
    return "".join(full)

Log TTFT and total separately — they are optimized by entirely different levers.

Why Nearly Every Chat Application Streams

Without streaming, a user submitting a request to an LLM would see nothing at all until the entire response — potentially hundreds or thousands of tokens — finished generating, which for longer responses can mean a wait of many seconds with no visible feedback at all. Streaming instead sends each token (or small group of tokens) to the client as soon as it’s generated, letting the interface display text progressively as it arrives, which dramatically improves perceived responsiveness even though the total time to generate the full response is unchanged. This is why time-to-first-token, not total generation time, is often the latency metric application developers optimize most aggressively for chat-style interfaces — a fast first token combined with steady streaming afterward feels significantly more responsive to users than an equivalent total wait time delivered all at once at the end.