How to Design Systems That Handle Millions of Users

How to design systems that handle millions of users is one of the most searched questions in backend engineering, and for good reason: I’ve sat in a system design interview, or worse, a production incident, and realized I didn’t have a clear mental model for what comes after “add more servers.” Designing systems that support millions of users isn’t a single architectural decision. It’s a staged sequence of choices, each one unlocking the next order-of-magnitude of capacity. Get the sequence wrong and you either over-engineer too early and kill your velocity, or under-engineer and scramble when traffic spikes.

In this article, I walk you through those stages from a single-server setup to a globally distributed architecture, with concrete decision checkpoints at every milestone. If you want to go deeper on any of the layers I cover here, Posts tagged “engineering”, imlucas.dev is where I publish detailed breakdowns on each one, from database sharding internals to observability strategy.

The six-stage scaling map: a framework for scalable system design

The most common mistake I see engineers make is designing for 10 million users on day one. Over-engineering at an early stage kills development velocity, wastes infrastructure budget, and introduces complexity you’re not equipped to operate yet. In my experience, premature sharding or microservices adoption harms early-stage velocity more than traffic spikes do, a pattern I’ve seen repeatedly in engineering postmortems across the industry. The staged model is how I decide when to invest in the next layer, not before.

The six stages progress like this: (1) single server, (2) separated database, (3) cached reads plus a load balancer, (4) CDN plus async queues, (5) sharded data with read replicas, (6) globally distributed with CQRS. Each stage is triggered by observable load thresholds, not arbitrary timelines or gut feelings. The transition from Stage 1 to Stage 2 happens when your database CPU crosses around 70%, within the generally accepted 70–85% warning band used in SRE practice, not when you hit a certain funding milestone.

The mechanism that makes this work in practice is the decision checkpoint: a set of defined metrics (CPU utilization, p95 latency, error rate) that tell me when my current architecture is approaching its ceiling. Reactive scaling, fixing things after they break in production, is avoidable when I build these checkpoints into my operational routine from the start. The teams I’ve worked with that define thresholds in advance tend to scale more smoothly; the ones that don’t often end up doing architecture work under pressure.

Stages 1 and 2: from a single server to a separated, cached setup (0 to ~10K users)

A single server is a legitimate starting point. It handles co-located compute, app logic, and database traffic efficiently at low scale, and it’s the simplest possible thing to operate. The signals I watch for to know it’s time to move on are specific: database CPU exceeding 70%, p95 query latency crossing 500ms, or any moment the database and app server are actively competing for memory on the same instance. None of these are vague. They’re measurable.

Moving the database to a dedicated host typically reduces resource contention significantly by eliminating the competition for CPU and memory that degrades performance on a shared instance. My next move is adding an in-memory cache like Redis to serve repeated reads without touching the database on every request. The 80/20 rule is reliable here: roughly 20% of my data (hot rows, frequently queried records) accounts for around 80% of read traffic. Caching those aggressively cuts database load by 60–80% before I ever touch horizontal scaling, a significant performance gain from a relatively contained infrastructure change.

Vertical scaling is cheap and fast to implement, but it has a hard ceiling. At some point, I’ve exhausted the cost-effective instance sizes and I’m still seeing degradation under load. That’s my checkpoint for Stage 3. The question I ask isn’t “do I feel ready to scale horizontally?” It’s “have I hit the ceiling of what a single, larger machine can cost-effectively deliver?” Those are very different questions, and the second one has a concrete answer.

Stages 3 and 4: horizontal scale, load balancing, and async processing (10K to 500K users)

Stateless services are the prerequisite for horizontal scaling. Any session state, local file storage, or in-process caching that ties a request to a specific server has to move out before I can treat individual instances as disposable. Once my web tier is stateless, I can scale it horizontally behind a load balancer (AWS ELB, NGINX, HAProxy) and gain fault tolerance for the first time. This is the stage where I eliminate the single point of failure in my architecture.

A CDN offloads static assets to edge nodes, cutting origin server load by 50–80% for media-heavy applications. Simultaneously, read replicas on the database distribute the roughly 80% of traffic that’s reads across multiple nodes while all writes still go to the primary. In my experience, this combination handles the majority of scaling challenges in the 10K–500K user range without requiring sharding. I’ve seen many B2B SaaS products and typical web applications never need to go further than this, though the point at which sharding becomes necessary depends heavily on write volume and dataset size.

Offloading async work: the queue decision

Inline synchronous processing becomes a serious bottleneck as concurrency grows. Image resizing, email sending, report generation: these operations have no business being in the critical request path. I offload them to message queues (SQS, RabbitMQ, Kafka) to decouple them entirely. The request returns immediately; a worker consumes the task asynchronously. The heuristic I use: if any synchronous operation in my critical path consistently adds 100ms or more to request latency, it’s worth evaluating as a queue candidate, though the right threshold depends on the specific SLOs.

Stages 5 and 6: designing systems that handle millions of users at data scale (500K to 10M+)

At this scale, the database becomes the constraint again, but for different reasons than Stage 1. I’m no longer fighting resource contention on a single host; I’m fighting write throughput and dataset size. Sharding vs partitioning vs replication is a useful primer I lean on when deciding how to split data: sharding splits data horizontally across nodes by a partition key (user ID, region), enabling write scalability and massive dataset growth. Replication copies the full dataset for read scaling and high availability, but doesn’t solve write bottlenecks. I treat them as complementary tools, not alternatives.

Distributed SQL databases like YugabyteDB and CockroachDB automate both sharding and replication. They shard internally, replicate per shard, and maintain ACID guarantees across nodes. For most teams I’ve worked with at this scale, distributed SQL eliminates the operational complexity of manual sharding while delivering availability that manual approaches rarely match without substantial additional investment. I’ve seen a well-configured distributed SQL cluster with high replication factors reach five-nines availability; achieving comparable uptime with hand-rolled sharding typically requires considerably more engineering overhead.

CQRS (Command Query Responsibility Segregation) separates the read and write paths entirely, letting me scale each independently. Write models handle mutations; read models serve queries from denormalized, precomputed views at high throughput. I reach for this pattern at Stage 6 when independent read scaling becomes a hard requirement, and it pairs naturally with event sourcing for temporal queries. Amazon and large e-commerce platforms use it precisely because the read-to-write ratio is so heavily skewed that treating both paths identically wastes resources on one end and bottlenecks the other.

Instagram’s scaling history is the most instructive real-world reference I’ve found. They scaled from 14 million users with 3 engineers to 400 million daily active users by making a handful of deliberate choices: sharding PostgreSQL by user ID, running Cassandra clusters per continent for feed data to manage latency, and rewriting Python hot paths in Cython before adding hardware. The most transferable lesson I take from them is their sequencing: optimize code and queries first, add infrastructure second. I’ve watched many teams reverse this and end up overprovisioning instead of profiling. For a deeper technical retrospective on how Instagram evolved its infrastructure, see this write-up on how Instagram scaled its infrastructure.

Capacity planning for millions of users: turning user counts into resource estimates

When I’m capacity planning for millions of users, I start with a back-of-envelope calculation that converts user counts into infrastructure requirements. The core assumptions I chain together:

1. Assume 1–10% of total users are daily active users (DAU).
2. Assume 1–5% of DAU are concurrently active during peak.
3. Multiply concurrent users by average requests per second per user to get peak RPS at the web tier.
4. Apply a downstream amplification factor of 2–4x for database queries per web request.

For 10 million total users, this typically yields 4,000–10,000 RPS at the API layer and 10,000–30,000 database operations per second.

Each tier has different resource constraints, and they require separate planning. The web tier is stateless and scales horizontally; target CPU below 70% per instance and provision for peak with 20–30% headroom. The API layer has roughly 3x amplification to the database, so it needs caching in front of it. The database layer is the hardest to scale: 10TB of data at 300K ops/sec requires a sharded setup with 5–20 nodes, tuned indexes, and hot data cached aggressively. For practical cache-tuning and optimization techniques when you reach this point, review this guide to cache optimization strategies.

The standard recommendation I follow is to provision 20–50% above estimated peak to absorb unexpected spikes. Traffic spikes can reach 10x average for event-driven products, and I treat my capacity plan as a starting point, not a contract. Real-world tuning after profiling always surfaces application-specific constraints that estimates miss. I use load testing tools like k6 or JMeter to validate estimates before they become production surprises. If you’re dealing with AI workloads and high-throughput inference, which carry distinct resource profiles from standard web services, my post How Does a Large Language Model Actually Work? An Engineering Perspective, imlucas.dev covers layer-specific capacity planning in more depth.

Observability and knowing when your architecture needs to evolve

Three metrics matter to me at every stage: latency (p95 and p99), throughput (RPS), and error rate. A common SLO target I use is error rate below 0.1%, though I’ve worked at organizations that use 1% as a critical alert threshold depending on product context. What matters is that I define the threshold explicitly and treat it as a trigger. CPU and memory utilization are lagging indicators; by the time they spike, users are already seeing degraded performance. I instrument my systems to track latency at the boundary of each tier (web, API, database, cache) so I know exactly which layer is the bottleneck before it becomes a production incident. Boundary-level instrumentation is the difference between a 30-minute diagnosis and a 3-hour war room.

I run load testing continuously in staging, not once before launch. I simulate estimated peak RPS, then push to 2–3x that figure to find where the architecture breaks. The failure mode matters to me as much as the threshold: does latency degrade gracefully, or does the system fall off a cliff at a specific concurrency level? Gradual degradation is acceptable engineering; cliff-edge failure is a design flaw I want to find in staging, not production.

I define decision checkpoints explicitly and document them as organizational examples calibrated to my workload, not universal rules. For instance: “When p95 latency on the API layer exceeds 200ms at peak load, we evaluate caching strategy.” “When database CPU sustains above 70% for three consecutive hours, we evaluate read replicas.” These function as triggers that kick off a structured architectural review. The teams I’ve seen build this discipline into their operational playbook scale deliberately instead of reacting to outages.

Build the foundation, then earn the next layer

The way I think about designing systems that handle millions of users is as a staged discipline, not a one-time architectural leap. You don’t need globally distributed architecture on day one; you need to know exactly when each layer becomes the constraint and what to reach for next. The six stages in this framework give you the map I use: start simple, instrument everything, and let your metrics drive the next architectural move rather than speculation about future scale. Apply that sequencing consistently and the path from a single server to a globally distributed system becomes a series of deliberate, well-timed decisions.

Each layer I covered here has real depth beneath the surface: sharding internals, distributed SQL trade-offs, CQRS implementation patterns, observability stack design. I publish detailed breakdowns on all of it at Hello World: Why I Started This Blog, imlucas.dev, written for engineers who are actually building these systems. If any section here left you wanting more specifics, that’s where I’d point you next.