Threads in Distributed Systems: How Servers Handle Many Requests at Once

A thread is a single stream of execution inside a process. Threads in the same process share memory, so passing data between them costs nothing. That is what makes them cheap compared with separate processes.

In a distributed system, threads solve a local problem. One node has many clients waiting on it. Threads let that one node work on many requests at the same time.

Here is the number that explains their value. Suppose a request spends 100 ms waiting on a database. One thread can then serve 10 requests per second. A pool of 200 threads serves about 2,000 requests per second on the same machine.

Nothing about that is distributed. The distributed part is what the threads do. Most of them are waiting for another machine to reply.

Threads, Processes and Nodes

These three words are often mixed up. They describe different levels.

A thread crash is not contained. If one thread corrupts shared memory, every thread in that process is affected. This is the main reason some systems keep risky work in a separate process.

How a Multithreaded Server Works

There are three common designs, and each suits a different workload.

Thread per request creates a new thread for every incoming call. It is simple and it breaks under load. Creating 10,000 threads costs about 10 GB of stack memory.

A thread pool creates a fixed number of threads at start and reuses them. Requests wait in a queue when all threads are busy. This is the standard server design.

An event loop uses one thread per CPU core and never blocks. Work is handed back through callbacks. It handles many idle connections cheaply, but one slow piece of code stalls everything on that core.

Most server frameworks use a pool. The hard question is how large to make it.

For work that is mostly waiting, the pool can be large. For work that is mostly computing, the pool should be near the core count. A pool of 500 threads on 8 cores spends most of its time switching between threads, not running them.

Design Issues Threads Create in Distributed Computing

Threads bring four problems that show up in almost every design review.

Race conditions come first. Two threads write the same variable and the result depends on timing. The fix is a lock, and locks bring their own costs.

Deadlock comes second. Thread A holds lock 1 and wants lock 2. Thread B holds lock 2 and wants lock 1. Neither moves. The usual fix is to always take locks in the same order.

Thread pool exhaustion comes third, and it is the most common outage. A downstream service slows from 50 ms to 5 seconds. Every thread ends up waiting on it. The server stops answering healthy requests too.

Two guards prevent that. Set a timeout on every remote call. Give each downstream service its own small pool, so one slow service cannot take all the threads.

Debugging is the fourth problem. A bug that depends on timing may appear once in a million runs, and it will not appear on your laptop.

Threads Are Not Distributed Coordination

A lock inside a process only protects that process. Two nodes running the same code do not see each other's locks.

To protect a resource across machines, you need a distributed lock. That is a network call, so it is thousands of times slower than a local lock, and it can fail.

Treat local threading and distributed coordination as two separate topics. Mixing them is a common mistake in interviews.

How to Prepare

• Define the term first. A thread is a stream of execution inside a process that shares memory with its siblings.
• Size a pool out loud. Say whether the work waits or computes, then give a number and the reason.
• Name thread pool exhaustion. Timeouts and separate pools per dependency are the answers interviewers listen for.
• Reuse instead of recreate. The same idea drives connection pooling for databases.
• Put threads in the wider picture. Read the challenges of designing a distributed system, then study Grokking System Design Fundamentals for the parts that surround a server.