Auto Scaling

What is autoscaling?

Autoscaling adjusts compute capacity automatically in response to demand — adding instances/pods/containers under load, removing them when idle. It’s the mechanism behind cloud’s central pitch: pay for what you use, absorb spikes without over-provisioning for a peak that’s rare. But “automatic” hides real decisions — what signal to scale on, how fast to react, and what bounds to enforce — and getting those wrong turns autoscaling from a cost-saver into an outage source or a runaway bill.

The two axes and the metric problem

Horizontal scaling adds more instances (the cloud-native default — stateless services scale out cleanly); vertical scaling makes instances bigger (limited by max instance size, usually needs a restart). The harder question is the signal. CPU is the classic metric and often the wrong one: a service bottlenecked on database connections or queue depth won’t show CPU pressure while it drowns. Scaling on the metric that actually reflects load — request latency, queue depth, concurrent connections, or a custom business metric — is what separates responsive autoscaling from autoscaling that reacts to the wrong thing. Kubernetes HPA, cluster autoscaler, and KEDA (event-driven scaling on queue length, etc.) exist to broaden beyond CPU.

The failure modes

Reaction lag: scaling takes time (new instances boot, pods pull images, cold starts warm), so a fast spike can overwhelm you before capacity arrives — pre-scaling on schedules or predictive signals bridges the gap. Flapping: aggressive scale-in/scale-out oscillates, churning capacity and cost — cooldowns and stabilization windows damp it. Runaway scaling: a bug or attack drives infinite scale-out and an infinite bill — maximum limits are mandatory, the autoscaling equivalent of a circuit breaker. And scaling only works if the app is genuinely stateless and the downstream dependencies (database, connection pools) can absorb the added fleet.

What people get wrong

• Scaling on CPU when the real bottleneck is elsewhere — the service melts while CPU looks calm.
• No maximum bound — turning a traffic spike or bug into an unbounded cost event.
• Forgetting the database: scaling stateless tiers into a fixed-capacity datastore just moves the bottleneck and can take the DB down.

Primary source: AWS Auto Scaling documentation

Scaling Policies: Reactive vs. Predictive

Most autoscaling implementations use reactive scaling policies, adjusting capacity based on real-time metrics like CPU utilization or request queue length crossing a defined threshold — simple to configure but inherently lagging, since new capacity only comes online after demand has already increased and the metric has already crossed its threshold. Predictive scaling, offered by some cloud providers as a more advanced option, uses historical traffic patterns and machine learning forecasts to provision additional capacity proactively ahead of an anticipated demand increase (a known daily traffic peak, for example), reducing the lag inherent in purely reactive scaling for workloads with sufficiently predictable, recurring demand patterns — though for workloads with genuinely unpredictable traffic spikes, reactive scaling combined with a reasonable buffer remains the more broadly applicable approach.