Continuous Batching

Technical Overview of Continuous Batching

Continuous Batching (also known as Iteration-level Batching) is an inference scheduling algorithm that groups active requests dynamically at the token iteration level.

Standard static batching groups multiple requests and processes them together. However, because requests have varying input and output lengths, the entire batch must wait for the longest request to finish generation, leaving GPU resources idle. Continuous Batching addresses this by dynamically adding and removing requests from the batch at each token generation step.

Key Architecture & Implementation

Unlike static batching, the scheduling loop in Continuous Batching executes at every step:

1. Pre-fill Phase: New requests are processed to generate their initial KV cache.
2. Decoding Steps: The scheduler executes a single token generation step for all active requests in the batch.
3. Dynamic Eviction: Requests that generate end-of-sequence (EOS) tokens are immediately removed from the batch.
4. Dynamic Insertion: Pending requests in the queue are inserted into the batch at the next step, utilizing the freed resources.

Static Batching:      [ Req A (50t) | Req B (10t) ] ---> Batch runs for 50 steps
Continuous Batching:  Step 10: Evict Req B, insert Req C ---> Batch size remains optimal

Core Parameters

• Latency Reduction: Significantly decreases queue time for requests.
• Throughput: Improves overall system serving throughput by up to 2x compared to static batching.

Real-world Applications

• Implemented in production serving engines like vLLM, Triton Inference Server, and TensorRT-LLM.