Auto-Scaling
Auto-scaling automatically adds or removes compute instances in response to load metrics, keeping capacity aligned with demand without manual intervention.
Auto-scaling = a restaurant that hires extra cooks when the order line gets long, then sends them home when it's slow. Hiring on 'how sweaty the cook looks' (CPU) is too late -- hire when the ticket queue grows.
Auto-scaling policies trigger on metrics: CPU utilization, request queue depth, p99 latency, or custom business metrics. Reactive scaling (scale when CPU > 70%) has a lag of 3–5 minutes (boot time + warmup). Predictive scaling uses historical patterns to pre-provision capacity. Scale-in must be slower and more conservative than scale-out to avoid oscillation. Cooldown periods prevent thrashing.
Boot time is the primary constraint on reactive auto-scaling. Strategies to reduce it: pre-warmed instance pools, baked AMIs (dependencies pre-installed), containerized services with fast startup (< 5s for Go/Rust, longer for JVM). For JVM services, 'slow start' algorithms at the load balancer ramp traffic to new instances gradually, avoiding cold-start latency spikes. Stateful services (database, Kafka) cannot auto-scale horizontally without data rebalancing — auto-scaling is primarily for stateless app tiers. For databases, read replicas can be added automatically; write capacity requires pre-planning. Kubernetes HPA (Horizontal Pod Autoscaler) scales on CPU/memory or custom metrics via the metrics API.
I'd design auto-scaling with fast instance startup as the primary constraint. For app servers, I'd use containerized services (Docker/Kubernetes) with baked images to get startup under 30 seconds. I'd scale on a leading indicator — request queue depth or p95 latency — rather than CPU, which is a lagging indicator. I'd keep a minimum baseline capacity to avoid cold starts on traffic spikes and use predictive scaling for known traffic patterns (e.g., business hours).
Scaling on CPU alone is a lagging indicator. By the time CPU is saturated, users are already experiencing degradation. Scale on upstream metrics like queue depth or latency instead.