0% completed
Key Characteristics of Distributed Systems
On This Page
Scalability
Two ways to add resources
Reliability
Availability
Efficiency
Manageability
Reliability and Availability Are Not the Same
How the Five Are Connected
Key Takeaways
A distributed system is a group of computers that work together. To the person using it, it looks like one system.
When you design one, you have to say how good it must be. Five words are used to describe this: scalability, reliability, availability, efficiency, and manageability.
This lesson explains each word. For each one you will see what it means, a small example, and how it is measured.
Scalability
Scalability means the system can handle more work when you give it more resources.
Imagine a website with 1,000 users today and 1 million users next year. If you can serve those users by adding more machines, the system is scalable. If the only answer is to write the system again, it is not.
More work can arrive in three different ways:
- More requests. More people open the app at the same time.
- More data. You store ten years of photos instead of one year.
- More traffic to one item. One video becomes popular and everyone asks for it.
These three are not the same problem, and they do not have the same fix. Adding machines helps with more requests. It does not help with one popular item, unless you also copy that item or keep it in a cache.
Two ways to add resources
Vertical scaling means you replace the machine with a bigger machine. More CPU, more memory, more disk.
Horizontal scaling means you add more machines and share the work between them.
Vertical scaling is simpler, because your code does not change. But it stops at some point. There is a largest machine you can buy, and machines that big are expensive.
Horizontal scaling has no single limit. The cost is that your code becomes harder. It must know which machine holds which data. It must also keep working when one machine stops.
Cassandra is built for horizontal scaling. You add a node, and the cluster shares the data again. MySQL is usually made bigger first. After that you add copies for reading. To split MySQL data across many machines, you need an extra layer, such as Vitess.
One more point about adding machines. Some work cannot be shared. If every write must pass through one machine, that machine sets the limit for writes. Adding more machines around it does not remove that limit. This is why the speed does not always grow when the number of machines grows.
How it is measured: the response time and the cost per user as the load grows.
Reliability
Reliability means the system keeps doing the correct thing over time, even when some parts fail.
Two ideas are inside this definition.
The first idea is correct. A system that answers quickly with wrong data is not reliable.
The second idea is over time. Reliability is measured over days and weeks, not at one moment.
Here is an example. A user adds an item to the shopping cart of an online store. The machine holding that cart stops working. If the item is still in the cart, the system is reliable. This works because the cart is saved on more than one machine.
Keeping more than one copy is called redundancy. It is the usual way to get reliability, and it is not free. Three copies use three times more storage. Writes also become slower, because a write is not finished until enough copies have the data.
Fault tolerance is a related term. A fault is one part breaking. Fault tolerance means the system keeps working when a part breaks. Fault tolerance is one of the ways you build a reliable system.
How it is measured: MTBF, which is short for mean time between failures. It is the average time the system runs before something breaks. A longer time is better.
Availability
Availability means the system is able to answer requests during the time it is expected to work.
Availability is written as a percentage. People say "three nines" for 99.9 percent and "four nines" for 99.99 percent.
A percentage is hard to imagine, so it helps to turn it into time.
Notice how large the difference is. Three nines allows about 43 minutes of downtime every month. Four nines allows about 4 minutes. With 43 minutes, a person can wake up and fix the problem. With 4 minutes, the system has to repair itself.
Two more rules are useful when you count availability.
When a request passes through several parts, every part must work. So the numbers multiply, and the result is worse than any single part. Three parts at 99.9 percent give you 99.7 percent together. This is why every extra step in the path has a price.
When you have two copies and only one of them needs to answer, the system fails only if both fail at the same time. So copies make the number better. Two machines at 99 percent give you 99.99 percent, but only if they fail for independent reasons and one machine alone can carry the whole load. A shared power feed, a shared network path, or one bad deployment sent to both removes most of that gain.
There is one detail that many people miss. Availability is measured against the time you promised, not always against 24 hours a day. If a bank system is only expected to run at night, it is not unavailable during the day. Nobody promised it would be running then.
One warning about this word. In the CAP Theorem lesson, availability means something different. There it means that every machine that is still working gives an answer. It is a yes or a no, not a percentage. Both meanings are common, so it helps to say which one you mean.
How it is measured: the percentage of the promised time that the system can answer.
Efficiency
Efficiency means how well the system performs for the resources it consumes.
Three numbers describe it, and they are not the same thing.
Latency is the time one request takes. It is usually written in milliseconds.
Throughput is the number of requests the system finishes in one second.
Cost is the money you pay for the machines, the storage, and the network.
Latency and throughput are different numbers. If the system handles many requests at the same time, throughput goes up. One request can still take the same time as before.
When you measure latency, do not look only at the average. The average hides the slow requests.
To describe the slow requests, engineers sort all the requests from fast to slow. If you have 100 requests, the 99th one is called the p99. It tells you what the slowest users are waiting for. The average tells you nothing about them.
How it is measured: latency including the slow requests, throughput, and cost.
Manageability
Manageability, also called serviceability, means how easy it is to operate the system, find problems, and fix them.
It answers three questions:
- How long until someone knows that something is wrong?
- How long to find which part is broken?
- How long to repair it?
Each question has its own tools. Alerts answer the first one. Logs and request tracing answer the second one. Automatic failover and a tested rollback plan answer the third one.
Manageability is easy to forget, because it does not change what the system does for the user. But it changes how long every problem lasts.
How it is measured: MTTR, which is short for mean time to repair. It is the average time to bring the system back after it breaks. A shorter time is better.
Reliability and Availability Are Not the Same
These two words are often mixed up, so it is worth separating them.
Availability asks one question: did the system answer?
Reliability asks two: was the answer correct, and will the system still answer tomorrow?
A cache that returns old data answers every request, so it is available. The answers are wrong, so it is not reliable.
A job that is promised to run every night, and that fails to start on some of those nights, is not available on those nights. If it never produces a wrong result when it does run, it is reliable.
You will often read the rule "if a system is reliable, then it is available". This is only true when the system is expected to run all the time.
How the Five Are Connected
The five words are not five separate ideas. Two of them decide a third.
The formula says that availability depends on two things: how long the system runs before it breaks, and how long the repair takes.
This gives you two ways to make a system more available.
The first way is to break less often. You add redundancy, you use better hardware, and you test more. This is reliability work.
The second way is to repair faster. You add automatic failover, faster alerts, and a rollback plan you have tested. This is manageability work.
The second way is often cheaper, and the difference is larger than people expect. A system that breaks every month and repairs itself in 10 seconds is available 99.9996 percent of the time. A system that breaks once a year, but needs one full day of manual work, is available 99.7 percent of the time. The second system breaks twelve times less often and is still much worse.
Key Takeaways
- Scalability is the ability to handle more work by adding resources. Work grows in three ways: more requests, more data, and more traffic to one item.
- Vertical scaling buys a bigger machine and keeps the code simple, but it has a limit. Horizontal scaling adds machines with no single limit, but the code becomes harder.
- Reliability is doing the correct thing over time, even when parts fail. You buy it with redundancy, and redundancy costs storage and slower writes. It is measured with MTBF.
- Availability is the percentage of the promised time that the system can answer. Parts in a line multiply their numbers and make it worse. Copies side by side make it better.
- Efficiency is latency, throughput, and cost. Always look at the slow requests, because the average hides them.
- Manageability is how fast you notice a problem, find the broken part, and repair it. It is measured with MTTR.
- Availability = MTBF / (MTBF + MTTR). Break less often, or repair faster. Repairing faster is usually cheaper.
In an interview, these five words are only useful with numbers next to them. "The system should be highly available" does not help anyone. "The system needs 99.9 percent availability, and reads can be up to five seconds old" is much better. It tells the interviewer what you are building, and what you are ready to pay for it.
Utkarsh Gupta
· 3 years ago
I think MySQL also allows to scale horizontally if you use it in a cluster mode. Isn't it?
Marvin Xu
· 4 years ago
I disagree on all systems that are reliable are available. For example the stock market is considered reliable but not very available since it doesn't operate at all hours. This is because something that does what it's expected to do is reliable even if it's not available. If it's not expected to be available 24/7 it can still be considered reliable since it hasn't failed to meet the expectations set.
I propose to redefine reliability as how much a system meets expectations.
Reading Progress
0%
On This Page
Scalability
Two ways to add resources
Reliability
Availability
Efficiency
Manageability
Reliability and Availability Are Not the Same
How the Five Are Connected
Key Takeaways