Autoscaling simulator

Simulated time runs at 30 times real time. Traffic needs a certain amount of CPU, which the ready pods share equally; each pod requests 500 millicores (half a core) and can use up to 1000. The HPA wants average usage at the target percentage of the request, between 2 and 20 pods. The dashed line shows how many pods the traffic needs at the target, the solid line what the HPA asked for, and the shaded area the pods actually ready. Each decision in the log shows the calculation behind it.

Try these:

• Run the Spike. Traffic jumps fivefold at 2 minutes. The HPA's first answer is capped by its rate limit, the next is damped because the new pods are still starting, and capacity catches up after about 80 seconds. Then switch to Slow start-up and compare how much demand goes unserved.
• Watch the end of the spike at 10 minutes. With the default 300-second window, pods stay for five more minutes. Set the window to 0 s and they go at the next check.
• Choose Burn CPU, so starting pods use their full CPU limit. Nothing changes, because the HPA ignores CPU readings from pods that are still starting.
• Raise the target to 90%. Fewer pods run, but there is little headroom, so spikes go unserved for longer.

The controller is a port of the Kubernetes source (the replica calculator and the stabilization and rate-limit functions), checked against the worked examples in the Kubernetes documentation. The traffic, start-up times and the way pods share load are a simple model, described in the article.

Autoscaling changes the amount of computing capacity a service runs on as its load changes. Horizontal autoscaling adds or removes copies of the service (pods, containers, virtual machines); vertical autoscaling gives each copy more or less CPU and memory. Horizontal scaling is the common case for stateless services, because copies can be added without restarting the ones already running.

Most autoscalers are feedback loops. They measure something that rises with load, such as CPU utilization, requests per second or queue length, compare it with a target, and change the number of copies to bring it back. The hard part is that capacity takes time to arrive: a new copy has to be scheduled, started and warmed up before it can take traffic, and by then the load may have moved on.

The Kubernetes Horizontal Pod Autoscaler is a control loop in the kube-controller-manager. Every 15 seconds it reads each pod's CPU usage from the metrics API, usually served by metrics-server, and expresses it as a percentage of the pod's CPU request. It averages that over the pods with usable readings and computes

desiredReplicas = ceil(currentReplicas × currentUtilization ÷ targetUtilization)

If the ratio of current to target is within 10% of 1.0, it does nothing, so small fluctuations do not cause churn. In the steady scenario, traffic needs 1.2 cores and each pod should run at 50% of half a core, so 4.8 pods would be exact; the HPA settles on 5, the next whole number. A pod without a CPU request has no utilization, and if any pod lacks one the HPA takes no action on CPU.

Two details from the source code show up in the log: utilization is rounded down to a whole percentage before the ratio is taken, and the multiplication uses the number of pods with usable metrics, not the number the Deployment asks for.

New pods are a problem for any CPU-based autoscaler. A pod that is still starting serves no traffic, so counting its low CPU would drag the average down and slow the scale-up; an application that burns CPU while it starts, such as a JVM compiling code, would push the average up and cause more scaling than needed.

The HPA handles this in two ways. First, for the 5 minutes after a pod starts (the CPU initialization period), it ignores the pod's CPU unless the pod is ready and the latest sample was collected entirely after it became ready. A pod that is still Pending, or not yet ready, is set aside. Second, if the remaining pods say "scale up", the set-aside pods are added back as if they used 0% of their request, and the ratio is recomputed. That damps the step while new pods are on their way: if they are already enough, the HPA waits instead of adding more.

Both effects appear in the first two decisions after the spike in the simulator. At 2 min 20 s the HPA wants 16 pods but the rate limit allows 8. At 2 min 35 s, with 0 pods still starting: "Utilization 200% against 50%. Starting pods count as 0% to damp the change: ratio 2.00 × 8 pods = 16.00, rounded up to 16."

With starting pods burning their full CPU limit, the spike scenario still peaks at 20 pods and catches up after 80 seconds, exactly as without the burn, because none of those readings are used. AWS takes a similar approach: an EC2 instance is not counted in the group's aggregated metrics until its warmup time has passed.

The raw recommendation is smoothed before it is applied. A stabilization window makes the HPA look back over its recent recommendations: when scaling down it uses the highest one in the window, and when scaling up the lowest. The defaults are 300 seconds for scaling down and none for scaling up, so the HPA adds pods at the next check but removes them only after load has stayed low for five minutes. This prevents flapping, where pods are removed during a short dip and then added back.

Scaling policies then limit how fast the count may change. Each policy allows a number of pods or a percentage of the current count per period, and by default the most permissive policy wins. The default scale-up policies allow doubling or adding four pods every 15 seconds, whichever is more; the default scale-down policy allows removing all excess pods at once. Both can be changed per HPA with the behavior field, which became stable in Kubernetes 1.23.

The documentation's example of a gentler scale-down allows 4 pods or 10% per minute. Run through the simulator's controller, 80 replicas wanting 10 go 80 → 72 → 64 → 57 → 51 → 45 → 40 → 36 → 32 → 28 → 24 → 20, one step per minute: 10% of the count while that is more than 4, then 4 at a time.

From the moment load rises, several delays add up before new capacity serves it: up to 15 seconds for the metrics window to reflect the change, up to 15 seconds until the next HPA check, the rate limit on how many pods each check may add, and the time each new pod takes to be scheduled, start and pass its readiness probe. In clusters without spare nodes, the node autoscaler adds a further delay to bring up machines. The table shows the fivefold spike in the simulator with three start-up times.

A metric works for this kind of proportional control if it falls in proportion as copies are added. Average CPU utilization usually does, for services where CPU is the bottleneck. Requests per second per pod works for many web services and responds faster to load than CPU does. AWS's documentation gives examples that do not work for target tracking: the total request count at the load balancer, which does not change when instances are added, and latency, which rises with load but not proportionally.

For workers that consume a queue, the backlog per worker is a better signal than CPU, which is what tools like KEDA scale on. Memory is rarely a good scaling metric: many applications do not release memory when load falls, so the autoscaler scales up and never back down.

The target sets the headroom. A 50% CPU target means each pod could absorb a doubling of load before it saturates, enough to ride out the lag above; 90% is cheaper but leaves almost nothing for a spike.

Amazon EC2 Auto Scaling launched in 2009, together with CloudWatch and Elastic Load Balancing, and scaled groups of instances on alarms. Target tracking policies, which pick the adjustments themselves, followed in July 2017. Kubernetes added the Horizontal Pod Autoscaler in version 1.1 in November 2015, scaling on CPU; the autoscaling/v2 API, with custom and external metrics and configurable scaling behavior, became stable in Kubernetes 1.23.