Static algorithms assume equal servers and equal requests, load-aware ones read in-flight work, and hashing trades balance for affinity. Pick the one whose assumption matches your traffic, then pair it with health checks and a warm-up period for new servers.
Decide what is balanced: connections or requests
A layer 4 balancer chooses a target per connection or flow. AWS documents that a Network Load Balancer selects a TCP target with a flow hash over the protocol, source and destination addresses and ports, and the TCP sequence number, and that each TCP connection stays on one target for its lifetime. A client that multiplexes many requests over one long-lived connection, as HTTP/2 allows, therefore sends all of those requests to a single backend.
A layer 7 balancer terminates the connection, parses HTTP, and chooses a target per request. That allows routing by host, path or header, per-request retries, and counting in-flight requests, at the price of TLS termination and parsing work on the balancer. Use layer 7 when per-request decisions matter, and layer 4 for non-HTTP protocols or when the balancer should stay out of the payload. The algorithms below apply at either layer; only the unit being placed changes.
Static algorithms: round robin, weighted, random
Round robin rotates through healthy servers. It is the default in NGINX, which distributes requests by weighted round robin, and in AWS Application Load Balancer target groups. AWS describes it as suited to requests of similar complexity and targets of similar processing capability. Weighted round robin keeps the rotation but sends proportionally more requests to bigger servers; in NGINX, each server's weight defaults to 1.
Random selection sends each request to a server chosen at random. Envoy notes that random generally performs better than round robin when no health checking policy is configured, because it avoids a bias toward the host that comes after a failed one. What these algorithms share is that they ignore current load. A server that slows down because of a noisy neighbor, a long garbage-collection pause, or a bad deploy keeps receiving its full share until a health check removes it.
Load-aware algorithms: least connections, outstanding requests, two choices
Least connections sends each request to the server with the fewest active connections, adjusted by weight in NGINX's least_conn. Least outstanding requests counts in-flight requests instead, which is what AWS ALB offers; AWS describes it as the usual choice when requests vary in complexity or targets vary in processing capability. The distinction matters when connections are not work: an idle keep-alive connection still counts as load, while one HTTP/2 connection may carry many concurrent requests.
Power of two choices samples two random servers and picks the less loaded one. NGINX exposes it as random two least_conn, and Envoy's least request balancer samples two hosts by default when weights are equal. Mitzenmacher's analysis found that two choices give an exponential improvement in expected time in the system over one random choice, while three choices help only by a constant factor. Envoy also cites P2C's resistance to herding, where many balancers pile onto the same least-loaded server.
Simulate one degraded server
The model is deliberately small. Servers A to C finish 3 requests per tick and D, degraded, finishes 1; 9 requests arrive per tick against a total capacity of 10, so the cluster as a whole is not overloaded. Each policy routes 100 ticks of arrivals using the current queue lengths, then the script prints every server's remaining backlog. Random choices use a fixed seed, so every run prints the same numbers on Python 3.12 and 3.14.
The printed backlogs for A, B, C and D were: round_robin [0, 0, 0, 125]; random [0, 0, 1, 126]; least_outstanding [0, 0, 0, 1]; two_choices [0, 0, 0, 2]. Round robin gives D 2.25 requests per tick while it finishes 1, so its queue grows by 1.25 per tick. Random does the same on average. Both load-aware policies stop feeding D. The model has one balancer with perfect, instant queue data, uniform request cost and no retries, so it shows the mechanism, not real latency.
import random
SPEED = (3, 3, 3, 1) # requests each server finishes per tick; D is degraded
def run(policy, ticks=100, arrivals=9, seed=1):
rng, queue, nxt = random.Random(seed), [0] * len(SPEED), 0
for _ in range(ticks):
for _ in range(arrivals):
if policy == "round_robin":
s, nxt = nxt, (nxt + 1) % len(SPEED)
elif policy == "random":
s = rng.randrange(len(SPEED))
elif policy == "least_outstanding":
s = min(range(len(SPEED)), key=lambda i: queue[i])
else: # power of two choices: sample two, keep the less busy one
a, b = rng.sample(range(len(SPEED)), 2)
s = a if queue[a] <= queue[b] else b
queue[s] += 1
queue = [max(0, q - c) for q, c in zip(queue, SPEED)]
return queue
for policy in ("round_robin", "random", "least_outstanding", "two_choices"):
print(f"{policy:<18} backlog after 100 ticks (A, B, C, D): {run(policy)}")Hashing for affinity, and what it costs
Hash-based policies send the same key to the same server. NGINX's ip_hash uses the first three octets of an IPv4 address or the whole IPv6 address. Its generic hash directive accepts any key, such as the request URI, and with the consistent parameter uses ketama so that only a few keys move when servers change; the consistent hashing guide measures that effect. Envoy's ring hash and Maglev balancers serve the same purpose. The usual reasons are cache locality and servers that keep per-client state.
The cost is balance. Hashing ignores load, and real keys are skewed: one large tenant, or thousands of users behind one NAT address, can pin a single server. Cookie stickiness has similar limits; AWS notes that after the first request, sticky sessions bypass the routing algorithm, and that weighted random cannot be combined with them. Treat affinity as a performance optimization, and store session state elsewhere so that losing a server never means losing a session.
Health checks, slow start, and where each algorithm breaks
Every algorithm needs to know which servers are alive. Passive checks count failures in real traffic: open-source NGINX marks a server unavailable after max_fails failures, default 1, within fail_timeout, default 10 seconds, while active periodic checks are part of its commercial subscription. AWS ALB runs active health checks per target group and, when too few targets are healthy, fails open by routing to all registered targets; by default that happens when no target is healthy, and the threshold is configurable. Decide in advance whether that behavior is what you want.
A new server is a special case. With least connections it starts at zero, so it can receive a rush while its caches are still cold. Slow start ramps its share instead: AWS ALB increases traffic linearly over a window of 30 to 900 seconds, though not with least outstanding requests or weighted random. Envoy supports slow start for round robin, least request and client-side weighted round robin, beginning the ramp when a host joins or, with active health checking, when it turns healthy.
| Algorithm | Signal used | Breaks when | Usual fix |
|---|---|---|---|
| Round robin | Rotation only | One server slows down | Load-aware policy |
| Weighted round robin | Static weights | Capacity drifts from weights | Dynamic weights |
| Least connections | Open connections | Connections are not work | Count in-flight requests |
| Least outstanding | In-flight requests | New server gets a rush | Slow start |
| Two random choices | Load of two samples | No load signal reported | Report in-flight counts |
| Hash or IP hash | Client or key | Keys or NAT are skewed | Stateless servers |
Answer the follow-ups
Prepare one-sentence answers to the usual probes. Why not always least connections? Connection counts can misrepresent work, and a fresh server gets rushed. How do you balance gRPC? Per request at layer 7, because a layer 4 balancer pins each long-lived connection. What happens during a deploy? Drain in-flight requests from old instances, then let health checks and slow start bring new ones in. Where does session state live? Outside the servers, so affinity stays optional.
Rerun the simulation with D at speed 2, or with 12 arrivals per tick, and predict the backlogs before you look; changing one parameter and explaining the result is better practice than memorizing names. Keep your notes as decision rules, such as varying request cost calls for counting in-flight work, rather than as a list of algorithms. The articles below cover the scaling questions that usually follow.
Quick answers
Frequently asked questions
Which load balancing algorithm is best?
None in general. Round robin is fine when requests and servers are uniform. Least outstanding requests or two random choices fit varying request cost, and consistent hashing fits cache affinity. Start from the shape of your traffic and the failure you most need to survive.
What is the difference between least connections and least outstanding requests?
Least connections counts open connections; least outstanding requests counts requests still being processed. With keep-alive a connection may sit idle, and with HTTP/2 one connection can carry many requests, so in-flight requests usually track actual work more closely.
Why sample two servers instead of picking the least loaded one?
Picking the global minimum needs fresh, global state and makes independent balancers converge on the same server. Sampling two is cheap, and the power-of-two-choices analysis shows it captures most of the benefit over pure random selection.
Is an AWS Network Load Balancer round robin?
No. For TCP it selects a target with a flow hash over the connection's protocol, addresses, ports and sequence number, and keeps each connection on one target. Distribution is per connection, so a few heavy connections can still load targets unevenly.
Source notes
References and review policy
Information checked on November 7, 2026. Section links identify sources for factual claims and technical explanations. Interpretations, practice scenarios and preparation recommendations are RecallDeck’s editorial work.
From reading to recall
Practice the full interview loop.
RecallDeck schedules the concepts you miss and keeps coding, design, and behavioral fundamentals available when the interviewer changes direction.