Batch Inference

What is batch inference?

Batch inference submits many requests as a job and collects results later (typically within minutes to 24 hours) instead of holding a connection per request. Major providers price batch at ~50% of interactive rates, because they can schedule the work into idle capacity. For self-hosted inference, batching is even more fundamental — GPU throughput comes from processing many sequences simultaneously.

The mental shift

Teams default to real-time APIs because chat demos do — but audit your workload: classification backfills, embedding corpora, nightly summarization, eval runs, content moderation sweeps, data extraction pipelines, synthetic data for distillation — none of it has a human waiting. Moving the async majority to batch is often the single largest AI cost cut available: literally half off for changing an API call pattern.

Design patterns

Submit idempotent jobs with your own IDs (partial failures happen — reconcile, don’t resubmit blindly), poll or use webhooks for completion, and keep per-item error handling: one malformed request shouldn’t sink a 100k-item job. For recurring pipelines, treat batch jobs like any scheduled workload — monitoring, retries, dead-letter queues.

Cost reality

The 50% discount compounds with everything else: batch + prompt caching + a small routed model can put a classification pipeline at 1–2% of the naive “frontier model, real-time, one call at a time” cost. The trade is latency (up to 24h SLA, usually much faster) and slightly more plumbing.

What people get wrong

• Real-time habits in offline pipelines. The most common LLM cost mistake in data engineering.
• No reconciliation. Batch jobs complete mostly; pipelines that assume 100% success corrupt datasets silently.
• Ignoring rate-limit interplay. Batch quotas are separate from interactive quotas at most providers — use them to protect production traffic from your backfills.

Try it in code

The batch pattern: submit, poll, reconcile — never assume 100% success:

import time

def run_batch(items):
    requests = [{"custom_id": f"item-{i}", "prompt": p}   # your IDs, for reconciliation
                for i, p in enumerate(items)]
    job = batch_api.submit(requests)                      # ~50% cheaper than realtime

    while True:
        status = batch_api.status(job.id)
        if status.state in ("completed", "expired"):
            break
        time.sleep(60)

    results, failed = {}, []
    for r in batch_api.results(job.id):
        if r.error:
            failed.append(r.custom_id)                    # reconcile, don't ignore
        else:
            results[r.custom_id] = r.output
    if failed:
        print(f"retrying {len(failed)} failed items individually")
    return results

The custom_id reconciliation is the part teams skip — and the reason pipelines silently lose rows.