Systems that need to scale have predictable failure modes at each scale inflection point. The engineering patterns that address scaling challenges are well understood for applications that have crossed the first few scale thresholds.
The first scaling inflection point for most applications is the relational database. A single database instance handles reads and writes for all application traffic. The first scaling interventions include adding a read replica for read traffic, implementing a connection pool like PgBouncer or HikariCP to reduce connection overhead, and adding database indexes for queries that generate table scans. These interventions can extend a single-database architecture significantly before sharding becomes necessary.
For example, I have seen a PostgreSQL database instance handling 5000 concurrent connections with a PgBouncer connection pool. The pool reduced the connection overhead from 300ms to 50ms, allowing the database to handle a 20% increase in traffic without adding more hardware. However, as the traffic grows, the connection pool's effectiveness decreases, and sharding becomes necessary. A common approach is to shard by user ID or geographic region, using a technique like consistent hashing to minimize the number of cache misses.
A cache acts as a force multiplier by reducing database load, serving frequently accessed data from memory. The cache hit rate is key to its effectiveness; a 90% hit rate can mean 10 times the read throughput with the same database. Key caching decisions involve what to cache – read-heavy, expensive-to-compute data with tolerable staleness – the eviction policy, such as LRU for general caches, and cache invalidation methods like time-to-live for eventually consistent data or explicit invalidation for consistency-sensitive data. Redis is the standard distributed cache.
In one instance, we used Redis to cache product recommendations for an e-commerce application. The cache hit rate was 92%, reducing the database load by 80% and allowing the application to handle a 50% increase in traffic without adding more database instances. However, we had to implement a cache invalidation mechanism to ensure that product recommendations were updated in real-time. We used a combination of time-to-live and explicit invalidation to achieve this.
Stateless services, where all state resides in the database or cache and not in local session state, scale horizontally by simply adding more instances behind a load balancer. Common statefulness traps include local in-memory session state, local filesystem caching, and background timers that depend on a specific instance remaining alive. Identifying and externalizing these state dependencies is a prerequisite for horizontal scaling.
High-volume write operations that are not latency-sensitive, such as analytics events, audit logs, email sends, or asynchronous notifications, should not write directly to the database from the request path. A message queue decouples the accept rate, the rate at which the service accepts requests, from the processing rate, the rate at which the database can handle writes. Load spikes are absorbed by the queue instead of cascading to the database.