GPU (AI Accelerators)

What is a GPU in the AI context?

A GPU executes thousands of operations in parallel — originally for graphics, now the workhorse of AI because neural networks are mostly giant matrix multiplications. NVIDIA dominates (H100/H200/Blackwell class, with the CUDA software moat), while alternatives include Google TPUs, AWS Trainium/Inferentia, and AMD Instinct. “Accelerator” is the vendor-neutral term.

The two numbers that matter

For LLM work, a GPU is characterized by memory capacity (can the model + KV cache fit? an 80 GB card holds a 70B model only at 8-bit) and memory bandwidth (how fast weights stream — this, not raw FLOPS, bounds tokens/sec for inference). Training cares about interconnect too (NVLink, InfiniBand) since models span many GPUs.

How teams actually buy compute

• API models — someone else’s GPUs; zero GPU knowledge required. The right default.
• Cloud GPU rental — $1–10+/hr per card from hyperscalers or neo-clouds (CoreWeave, Lambda); spot/preemptible cuts 50–70% for interruptible work.
• Serverless GPU (Modal, RunPod, Cloud Run GPU) — per-second billing with cold starts; good for spiky inference.
• Buying hardware — justified only by sustained multi-year utilization; capacity planning and depreciation become your problem.

Cost reality

GPU spend is dominated by utilization, not price: a rented H100 idling at 15% utilization costs ~7× per useful token what a batched, saturated one does. Before shopping for cheaper cards, measure utilization — batching, quantization, and right-sizing models routinely beat hardware negotiations.

What people get wrong

• Sizing by FLOPS. LLM inference is memory-bound; a card with more bandwidth beats one with more compute for serving.
• Ignoring the software stack. Non-NVIDIA hardware can be cheaper per hour but cost more in engineering time if your stack assumes CUDA.
• Reserving before profiling. Committing to a year of capacity based on projected (not measured) load is how AI infra budgets die.