Load Balancing: The Art of Traffic Distribution for Million-Request Systems

Posted on: 4/20/2026 7:40:51 AM

What Is Load Balancing and Why Does It Matter?

Imagine you run a restaurant with 10 tables. If every customer ended up sitting at one table while the other 9 sat empty, the experience would be terrible. A Load Balancer is the restaurant host — distributing customers evenly across tables so nobody waits too long.

In software architecture, a Load Balancer is the component that distributes client traffic across multiple backend servers to optimize performance, increase availability, and ensure no single server is overloaded.

Layer 4 vs Layer 7: Two Schools of Load Balancing

This is the first architectural decision when choosing a Load Balancer. The difference lies in the OSI layer where the LB operates, and that directly affects routing capabilities, performance, and cost.

graph TB
    subgraph L4["Layer 4 — Transport"]
        A[Client Request] -->|TCP/UDP| B[L4 Load Balancer]
        B -->|IP + Port| C[Server A]
        B -->|IP + Port| D[Server B]
        B -->|IP + Port| E[Server C]
    end

    subgraph L7["Layer 7 — Application"]
        F[Client Request] -->|HTTP/gRPC| G[L7 Load Balancer]
        G -->|/api/*| H[API Server]
        G -->|/static/*| I[CDN/Static Server]
        G -->|/ws/*| J[WebSocket Server]
    end

    style L4 fill:#f8f9fa,stroke:#e0e0e0,color:#2c3e50
    style L7 fill:#f8f9fa,stroke:#e0e0e0,color:#2c3e50
    style B fill:#e94560,stroke:#fff,color:#fff
    style G fill:#e94560,stroke:#fff,color:#fff
    style C fill:#2c3e50,stroke:#fff,color:#fff
    style D fill:#2c3e50,stroke:#fff,color:#fff
    style E fill:#2c3e50,stroke:#fff,color:#fff
    style H fill:#2c3e50,stroke:#fff,color:#fff
    style I fill:#2c3e50,stroke:#fff,color:#fff
    style J fill:#2c3e50,stroke:#fff,color:#fff

The 6 Most Popular Load Balancing Algorithms

1. Round Robin — Simple but Effective

2. Weighted Round Robin — When Servers Aren't Equal

# NGINX config
upstream backend {
    server app1.example.com weight=5;  # 16 CPU, 64GB RAM
    server app2.example.com weight=3;  # 8 CPU, 32GB RAM
    server app3.example.com weight=1;  # 2 CPU, 8GB RAM
}

3. Least Connections — Adaptive to Real Load

4. IP Hash — Simple Session Affinity

# NGINX config
upstream backend {
    ip_hash;
    server app1.example.com;
    server app2.example.com;
    server app3.example.com;
}

5. Consistent Hashing — The King of Distributed Cache

graph TB
    subgraph Ring["Hash Ring — Consistent Hashing"]
        direction TB
        N1["Server A
position: 0°"]
        N2["Server B
position: 120°"]
        N3["Server C
position: 240°"]
        K1["Key 'user:42'
→ Server A"]
        K2["Key 'session:99'
→ Server B"]
        K3["Key 'cart:17'
→ Server C"]
    end

    K1 -.->|hash → 35°| N1
    K2 -.->|hash → 155°| N2
    K3 -.->|hash → 280°| N3

    style Ring fill:#f8f9fa,stroke:#e0e0e0,color:#2c3e50
    style N1 fill:#e94560,stroke:#fff,color:#fff
    style N2 fill:#e94560,stroke:#fff,color:#fff
    style N3 fill:#e94560,stroke:#fff,color:#fff
    style K1 fill:#2c3e50,stroke:#fff,color:#fff
    style K2 fill:#2c3e50,stroke:#fff,color:#fff
    style K3 fill:#2c3e50,stroke:#fff,color:#fff

Virtual Nodes is an important technique to improve even distribution. Instead of each server occupying a single position on the ring, you create 100-200 virtual positions per physical server. This helps:

• Distribute keys much more evenly
• When one server fails, its load spreads across many servers rather than piling on the next one
• Amazon DynamoDB, Apache Cassandra, and ScyllaDB all use this technique

// Consistent Hashing with virtual nodes — pseudo code
class ConsistentHash {
    private ring: SortedMap<int, string> = new SortedMap();
    private virtualNodes: int = 150;

    addServer(server: string) {
        for (let i = 0; i < this.virtualNodes; i++) {
            let hash = md5(`${server}:${i}`);
            this.ring.set(hash, server);
        }
    }

    getServer(key: string): string {
        let hash = md5(key);
        // Find the nearest node clockwise
        let entry = this.ring.ceilingEntry(hash);
        return entry ? entry.value : this.ring.firstEntry().value;
    }

    removeServer(server: string) {
        for (let i = 0; i < this.virtualNodes; i++) {
            this.ring.delete(md5(`${server}:${i}`));
        }
        // Only ~1/n of keys remap — no global impact
    }
}

6. Random Two Choices — The "Just Smart Enough" Algorithm

# NGINX Plus config
upstream backend {
    random two least_conn;
    server app1.example.com;
    server app2.example.com;
    server app3.example.com;
    server app4.example.com;
}

Algorithm Comparison at a Glance

Common Load Balancing Tools

NGINX — Reverse Proxy and Load Balancer

NGINX is the most popular choice for L7 Load Balancing thanks to its high performance (handling millions of concurrent connections), simple configuration, and rich module ecosystem.

# nginx.conf — Complete Load Balancing
http {
    upstream api_servers {
        least_conn;
        server 10.0.1.10:8080 weight=3 max_fails=3 fail_timeout=30s;
        server 10.0.1.11:8080 weight=3 max_fails=3 fail_timeout=30s;
        server 10.0.1.12:8080 weight=1 backup;  # Only used when the above 2 fail
    }

    server {
        listen 443 ssl http2;
        server_name api.example.com;

        # SSL Termination
        ssl_certificate     /etc/ssl/certs/api.crt;
        ssl_certificate_key /etc/ssl/private/api.key;

        # Implicit health check via max_fails/fail_timeout
        location /api/ {
            proxy_pass http://api_servers;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;

            # Timeout configuration
            proxy_connect_timeout 5s;
            proxy_read_timeout 60s;
            proxy_send_timeout 10s;
        }
    }
}

HAProxy — Dedicated Load Balancer

HAProxy is renowned for its L4 and L7 capabilities, purpose-built for load balancing with active health checks, detailed metrics, and impressive performance.

# haproxy.cfg
frontend http_front
    bind *:443 ssl crt /etc/ssl/api.pem
    mode http

    # Content-based routing
    acl is_api path_beg /api
    acl is_ws  hdr(Upgrade) -i websocket

    use_backend api_servers if is_api
    use_backend ws_servers  if is_ws
    default_backend static_servers

backend api_servers
    mode http
    balance leastconn
    option httpchk GET /health
    http-check expect status 200

    server api1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server api2 10.0.1.11:8080 check inter 5s fall 3 rise 2

backend ws_servers
    mode http
    balance source   # IP Hash to keep WebSocket sessions sticky
    timeout tunnel 3600s
    server ws1 10.0.2.10:8080 check
    server ws2 10.0.2.11:8080 check

Cloud Load Balancers — Managed and Auto-scaling

Health Checks and Failover — The Safety Net

A Load Balancer without health checks is like traffic without signals. Health checks let the LB automatically remove dead servers and bring them back when they recover.

sequenceDiagram
    participant LB as Load Balancer
    participant S1 as Server A (healthy)
    participant S2 as Server B (failing)
    participant S3 as Server C (healthy)

    loop Health Check (every 5s)
        LB->>S1: GET /health
        S1-->>LB: 200 OK ✓
        LB->>S2: GET /health
        S2-->>LB: 503 Error ✗
        LB->>S3: GET /health
        S3-->>LB: 200 OK ✓
    end

    Note over LB,S2: Server B fails 3 times in a row → marked DOWN

    LB->>S1: Route traffic (50%)
    LB->>S3: Route traffic (50%)
    Note over S2: Receives no traffic

    S2-->>LB: 200 OK ✓ (after 2 successful checks)
    Note over LB,S2: Server B recovers → returned to the pool

There are 3 common types of health checks:

• Active Health Check: The LB actively sends probes (HTTP GET /health, TCP connect, or custom scripts). HAProxy and cloud LBs support this by default.
• Passive Health Check: The LB observes real traffic responses — if a server returns errors continuously (e.g., 5× 5xx in 30s), mark it down automatically. NGINX Open Source only supports this.
• Deep Health Check: Verify dependencies too (database connection, disk space, memory). Return details via a /health/detailed endpoint.

// ASP.NET — Deep Health Check
// Program.cs
builder.Services.AddHealthChecks()
    .AddSqlServer(connectionString, name: "database")
    .AddRedis(redisConnection, name: "cache")
    .AddCheck("disk-space", () =>
    {
        var drive = new DriveInfo("C");
        return drive.AvailableFreeSpace > 1_073_741_824  // > 1GB
            ? HealthCheckResult.Healthy()
            : HealthCheckResult.Degraded("Low disk space");
    });

app.MapHealthChecks("/health", new HealthCheckOptions
{
    Predicate = _ => true,
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});

Real-World Deployment Architectures

Pattern 1: Single-tier LB — For Small and Medium Systems

graph LR
    Client[Client] --> LB[NGINX / HAProxy
L7 Load Balancer]
    LB --> S1[App Server 1]
    LB --> S2[App Server 2]
    LB --> S3[App Server 3]
    S1 --> DB[(Database)]
    S2 --> DB
    S3 --> DB

    style Client fill:#f8f9fa,stroke:#e94560,color:#2c3e50
    style LB fill:#e94560,stroke:#fff,color:#fff
    style S1 fill:#2c3e50,stroke:#fff,color:#fff
    style S2 fill:#2c3e50,stroke:#fff,color:#fff
    style S3 fill:#2c3e50,stroke:#fff,color:#fff
    style DB fill:#4CAF50,stroke:#fff,color:#fff

Pattern 2: Two-tier LB — For Large Systems

graph TB
    Client[Client] --> DNS[DNS / GeoDNS]
    DNS --> L4A[NLB - L4
Region A]
    DNS --> L4B[NLB - L4
Region B]

    L4A --> L7A1[ALB/NGINX - L7]
    L4A --> L7A2[ALB/NGINX - L7]

    L4B --> L7B1[ALB/NGINX - L7]

    L7A1 --> API1[API Pods]
    L7A2 --> WEB1[Web Pods]
    L7B1 --> API2[API Pods]

    style Client fill:#f8f9fa,stroke:#e94560,color:#2c3e50
    style DNS fill:#ff9800,stroke:#fff,color:#fff
    style L4A fill:#e94560,stroke:#fff,color:#fff
    style L4B fill:#e94560,stroke:#fff,color:#fff
    style L7A1 fill:#2c3e50,stroke:#fff,color:#fff
    style L7A2 fill:#2c3e50,stroke:#fff,color:#fff
    style L7B1 fill:#2c3e50,stroke:#fff,color:#fff
    style API1 fill:#4CAF50,stroke:#fff,color:#fff
    style WEB1 fill:#4CAF50,stroke:#fff,color:#fff
    style API2 fill:#4CAF50,stroke:#fff,color:#fff

Pattern 3: Global Load Balancing — For Worldwide Systems

When users are spread globally, you need load balancing at the DNS layer. GeoDNS or Anycast routing brings users to the nearest datacenter. Cloudflare, AWS Route 53, and Azure Traffic Manager all support this pattern.

Common Pitfalls and How to Fix Them

# HAProxy slow start
backend api_servers
    server api1 10.0.1.10:8080 check slowstart 60s

Load Balancing in Kubernetes

Kubernetes has its own load balancing system via Services and Ingress. Understanding how they work helps avoid duplication or conflicts with external LBs.

Load Balancer Deployment Checklist

Conclusion

Load Balancing isn't just "splitting requests evenly" — it's the art of distributing load so the system stays fast, stable, and resilient. There's no "best" algorithm — only the one that fits your specific context:

• Stateless APIs → Round Robin or Random Two Choices
• WebSockets / Long-running → Least Connections
• Distributed Cache → Consistent Hashing
• Legacy session-based apps → IP Hash (temporary; migrate to stateless)
• Multi-region → Global LB (GeoDNS) + Regional L4/L7

Start simple with Round Robin, add health checks, and only add complexity when you truly need it. Over-engineering load balancing from day one is one of the most common mistakes in system design.

References

• NGINX HTTP Load Balancing Documentation
• Understanding Load Balancing Algorithms and Strategies (2026)
• Consistent Hashing Explained — AlgoMaster
• Layer 4 vs Layer 7: Load Balancing and Why It Matters — CloudRPS
• ALB vs NLB: Which AWS Load Balancer Fits Your Needs?
• Edge Computing: Cloudflare Workers Development Guide 2026