Speculative Decoding

Technical Overview of Speculative Decoding

Speculative Decoding is an inference acceleration technique that speeds up LLM token generation by using a small draft model to generate candidate tokens that are verified in parallel by a larger target model.

LLM inference is memory-bandwidth bound: generating each token requires loading all model parameters into GPU SRAM, regardless of the calculation size. Speculative Decoding addresses this by using a fast, low-parameter draft model to generate a sequence of tokens, which the larger target model validates in a single forward pass, reducing memory transfer overhead.

Key Architecture & Implementation

The process runs in a loop:

1. Draft Generation: A small draft model (e.g., 1B parameters) autoregressively generates $K$ candidate tokens. This step is fast due to the model’s small memory footprint.
2. Parallel Verification: The large target model (e.g., 70B parameters) evaluates all $K$ candidate tokens in a single forward pass, calculating the probability distribution for each position.
3. Acceptance Phase: The system accepts candidate tokens based on probability thresholds. If a token is rejected, the target model’s output at that position is used to generate the corrected token, and the draft model restarts from that point.

Draft Model (1B):  Token 1 --> Token 2 --> Token 3 (Fast Draft)
                                              |
Target Model (70B): Validate [1, 2, 3] in 1 step ---> Accept 1 & 2, Reject 3 (Correct and Restart)

Core Parameters

• Speedup Ratio: Typically yields a 1.5x to 2.5x latency improvement, depending on the similarity of the draft and target models.
• Acceptance Rate (lpha): The average fraction of draft tokens accepted by the target model.

Real-world Applications

• Used in low-latency production applications, such as code generation and chat interfaces.