System Design Fundamentals

0% completed

Introduction to Caching

On This Page

What Caching Is

Why Caching Works

How a Cache Works

Key Terms

The Effect in Numbers

What Gets Cached, and Where

The Costs of Caching

Key Takeaways

Practice Questions

An online store has one very popular product. Its page is viewed 10,000 times every minute. Each view runs the same database query to load the product's name, price, and description, and each query takes 50 ms.

The product details almost never change. But the database repeats the same work 10,000 times a minute, and it slows down for every other request too.

The fix is to keep a copy of the answer somewhere much faster and reuse it. This lesson explains what that fast storage is, how it works, the terms used to describe it, and what it costs.

What Caching Is

A cache is a high-speed storage layer that sits between the application and the original source of the data. The original source can be a database, a file system, or a remote web service.

A cache holds temporary copies of data or computation results. It is designed for fast access and retrieval. Caching is the practice of storing those copies so that later requests can be answered from the cache instead of the slower source.

The goal of caching is to reduce how many times data must be fetched from its original source. This makes processing faster and reduces latency.

Why Caching Works

Two facts make caching useful in almost every large system.

Memory is much faster than disk or the network. Reading from memory takes about 100 nanoseconds. Reading from an SSD takes about 100 microseconds, and a round trip inside one data center takes about half a millisecond. The numbers you should know lesson lists these values.
Many requests ask for the same data. Often a small share of items, like popular products or trending posts, receives most of the requests. A cache that holds just those items can answer most requests.

How a Cache Works

When the application needs data, it follows the same steps every time.

Check the cache first.
If the data is found in the cache, return it to the application.
If the data is not found, fetch it from the original source. Store a copy in the cache for future use, and return the data to the application.

The application checks the cache first, returns the copy when it is found, and otherwise reads the original source and stores a copy for next time

Here is the same logic as simple code.

function getProduct(id):
    product = cache.get("product:" + id)
    if product is found:
        return product                       # cache hit
    product = database.findProduct(id)       # cache miss
    cache.set("product:" + id, product, ttl = 300 seconds)
    return product

This common pattern, where the application checks the cache and fills it on a miss, is called cache-aside. The cache read strategies lesson compares it with other patterns.

Key Terms

Cache. A temporary storage location for data or computation results, designed for fast access and retrieval.

Cache hit. A cache hit happens when the requested data or computation result is found in the cache.

Cache miss. A cache miss happens when the requested data is not found in the cache. It must then be fetched from the original data source or calculated again.

Hit rate. The share of requests that are cache hits. For example, if 900 of 1,000 requests are hits, the hit rate is 90 percent.

Cache eviction. Eviction is the process of removing data from the cache. It usually happens to make room for new data. A rule called an eviction policy decides which items to remove. For example, a common policy removes the item that was used least recently. The cache replacement policies lesson covers the common policies.

Cache staleness. Staleness means the data in the cache is outdated compared with the original data source. For example, a price changed in the database, but the cache still holds the old price.

TTL (time to live). A time limit set on a cached item. After the TTL passes, the item is treated as stale and must be fetched again. The cache invalidation lesson explains how systems keep cached data fresh.

A cache hit finds the data in the cache, a cache miss fetches it from the source, eviction removes data to make room, and staleness means the copy is older than the source

The Effect in Numbers

Consider the popular product page again. It gets 10,000 views per minute, and the database query takes 50 ms. A cache read takes about 1 ms.

Without a cache, the database runs 10,000 queries every minute, and each view waits about 50 ms for the data.

With a cache and a 90 percent hit rate:

9,000 views are cache hits. Each one takes about 1 ms.
1,000 views are cache misses. Each one checks the cache (1 ms), then queries the database (50 ms), for about 51 ms.
The average time is 0.9 x 1 + 0.1 x 51 = 0.9 + 5.1, which is about 6 ms, instead of 50 ms.
The database now runs only 1,000 queries per minute, instead of 10,000.

Without a cache the database answers all 10,000 views, while with a 90 percent hit rate it answers only the 1,000 misses and the average time falls to about 6 ms

The cache made the page about 8 times faster, and it removed 90 percent of the load from the database. Lower load also means the database can serve more users before it needs more hardware.

What Gets Cached, and Where

Caching can be used for many kinds of data.

Web pages and parts of pages.
Database query results, like the product details above.
API responses from internal or external services.
Images, videos, stylesheets, and scripts.
Computation results, like a recommendation list that took seconds to calculate.

Caches also exist at many places along the path of a request, from the user's device to the database.

Browser cache. The user's browser keeps copies of images, stylesheets, and scripts, so a repeat visit downloads less.
CDN cache. A content delivery network stores copies of files on servers near users, which reduces latency for people far from the origin server. The CDN chapter covers this.
Application cache. The application keeps data in memory, either in its own process or in a shared cache server, like Redis or Memcached.
Database cache. The database itself keeps recently used data in memory, so repeated reads avoid the disk.
Disk cache. Data can also be cached on local disk. Disk is slower than memory, but faster than fetching data from a remote source.

Caches sit at many layers of a request: the browser, the CDN edge, the application, and the database, in front of the original data on disk

A cache hit closer to the user saves more time, because the request travels a shorter distance. The types of caching lesson covers each kind in more detail.

The Costs of Caching

A cache also has costs. It brings new problems that a system must handle.

Stale data. Users may see old data until the cached copy expires or is removed. The system must decide how old is acceptable for each kind of data.
Limited space. Memory is expensive, so a cache cannot hold everything. It must evict some items, and the wrong choice lowers the hit rate.
A cold cache. After a restart, the cache is empty, so every request is a miss until the cache fills again. The sudden load on the database can cause problems.
More complexity. The cache is one more system to run, monitor, and scale.

Not all data should be cached. Data that changes on every request is a poor fit, because the cached copy is almost always stale. Data that must be exactly correct at a specific moment, like the stock count when a customer pays, should be read from the source. The caching challenges lesson covers the common problems and their fixes.

Key Takeaways

A cache is a high-speed storage layer that sits between the application and the original data source. The source can be a database, a file system, or a remote web service.
The application checks the cache first. If the data is found, it is returned. If not, it is fetched from the source, stored in the cache, and returned.
A cache hit means the data was found in the cache. A cache miss means it was not found and had to be fetched from the source.
Cache eviction removes data from the cache, usually to make room. Cache staleness means the cached data is outdated compared with the source.
Caching reduces latency and database load. With a 90 percent hit rate, a 50 ms query can become an average of about 6 ms.
Caches exist in browsers, CDNs, applications, and databases. They bring costs, like stale data, limited space, and extra complexity.

A cache uses some extra memory and accepts slightly older data to give much faster answers and much less work at the source. The next lesson, Why is Caching Important?, explains the main benefits of caching in more detail.

Practice Questions

Try each question first, then open the answer.

1. A cache receives 1,000 requests, and 850 of them are served from the cache. What is the hit rate, and how many requests must read the original source?