Inference Systems Optimization

Speculative Decoding Speedup Calculator

Evaluate expected generation speedups when pairing large target models with small, high-speed draft models. Compute token latency, expected acceptance rates, and net compute savings.

Serving Variables

Adjust latency and acceptance configurations below.

Target Model Latency 80 ms

Single-token forward pass latency of the main LLM (e.g. Llama 3 70B).

Draft Model Latency 10 ms

Single-token forward pass latency of the draft LLM (e.g. Llama 3 8B).

Token Acceptance Rate (α) 0.70

Probability that the target model accepts a draft token. Dependent on semantic domain alignment.

Lookahead Steps (γ) 4

Number of draft tokens generated before submitting block verification to the target model.

Net Speedup 1.82x Faster token generation

Speculative Latency 44.0 ms Compared to baseline 80ms

Expected Tokens / Step 2.77 Generated per verification

Speculative Generation Step Trace

Draft Proposal Phase

Target Verification Outcome

⚙️ Speculative Decoding Math

The expected number of accepted tokens per verification block of length γ is: E = (1 - α^{γ + 1}) / (1 - α)

The net latency per generated token matches: T_token = (γ × T_draft + T_target) / E