Caching

Think of it like this

Analogy

You're writing a report with a library of reference books down the hall. Walking to the library every time you need a fact is slow, so you keep the handful of books you use most on your desk. Most of the time the answer is right there. When it isn't, you walk to the library, and you bring that book back to your desk too.

Your desk only has room for a few books. When it's full and you bring a new one, one has to go back. Which one you return is your eviction policy.

The problem it solves

Databases keep data safe on disk and can answer complicated questions, but every query costs time and capacity. On a busy site, most reads ask for the same small set of things: the home page, the top products, the profile of whoever is logged in.

Without a cache:

• Every page view hits the database, even for data that changed hours ago.
• The database becomes the bottleneck long before your app servers do, and scaling a database is much harder than adding app servers.
• Users wait. A database query often takes several milliseconds or more; reading from an in-memory cache usually takes well under one.

A real example

Example

A product page during a flash sale

A product page needs the description, the price, the stock level and the reviews. During a flash sale it gets 20,000 views a minute.

The description and reviews are cached for 10 minutes; nobody minds if a new review shows up a little late. Stock is cached for only 5 seconds, and the final check at checkout always reads the database. With a 97% hit rate, the database handles about 30 times fewer reads.

Then the cached page expires at the peak, and thousands of requests miss at once and pile onto the database: a cache stampede. The fix is to let one request rebuild the value while the others keep getting the old copy for a moment.

How it works

A cache is a key-value store kept in memory, such as Redis, Valkey or Memcached. Before going to the database, the app asks the cache. If the data is there, that's a cache hit. If not, it's a cache miss: the app reads the database and usually stores the result in the cache for next time.

The most common pattern is cache-aside, where the app itself manages the cache:

The number that matters most is the hit rate: the share of requests answered from the cache. Going from 80% to 95% doesn't sound like much, but it cuts database reads by three quarters.

Try it below. Start traffic with “A few popular items” and watch the hit rate climb. Then change the cache size, switch eviction policies, and try the “A to J in order” pattern with a small LRU cache.

Writing: keeping the cache and database in step

Reads are the easy part. When data changes, you have to decide what happens to the cached copy.

Strategy	How a write works	Good	Watch out
Cache-aside	Write to the database, then delete the cache key	Simple; cache only holds data someone asked for	Brief window where a reader can see the old value
Write-through	Write to the cache and the database together	Cache is always fresh	Every write is slower; you cache data nobody may read
Write-back	Write to the cache now, database later in batches	Very fast writes	Data is lost if the cache dies before the flush

Most web apps start with cache-aside and a TTL on every key.

Eviction: what to throw out when it's full

Memory is limited, so a full cache must remove something to make room. That's eviction.

• LRU (least recently used) removes whatever hasn't been read for the longest time. A good default for most traffic.
• LFU (least frequently used) removes whatever has been read the fewest times. It protects long-term favorites, but can keep something that was popular yesterday and isn't now.
• FIFO (first in, first out) removes the oldest item regardless of use. Simple and rarely the best choice.
• TTL expiry isn't eviction exactly, but every key can carry a time to live, after which it's dropped no matter what.

Invalidation: the hard part

There's an old joke that the two hardest problems in computer science are naming things and cache invalidation. The cache doesn't know when the database changes. Your options:

• Expire on a timer (TTL). Simple and self-healing, but data can be stale for up to the TTL.
• Delete on write. The code that updates the database also deletes the cache key. Fresh, but every write path has to remember to do it.
• Versioned keys. Put a version in the key (product:42:v7). A change bumps the version, so old entries are never read again and simply age out.

Use deletes for freshness and a TTL as a safety net.

Where caches live

Caching happens at every layer, often all at once: the browser's cache, a CDN at the edge (lesson 15), memory inside each app server, a shared cache cluster like Redis, and the database's own memory buffer. The closer to the user, the faster, and the harder to invalidate.

Two failure modes to know

• Cache stampede. A very popular key expires, and a thousand requests miss at the same moment and all hit the database. Fixes: let one request rebuild the value while others wait or get the old value, or add a little randomness to TTLs so keys don't all expire together.
• Cold start. After a deploy or a cache restart, the hit rate is zero and the database takes the full load. Warm the cache with the most popular keys first, or bring cache nodes back gradually.

Trade-offs

What you gain

• Much faster reads, often 10× or more.
• Takes most of the read load off the database, so it lasts much longer before needing to scale.
• Absorbs traffic spikes on popular items.

What it costs you

• Users can see stale data.
• More moving parts: another service to run, monitor and pay for.
• Invalidation bugs are subtle and easy to miss in testing.
• If the cache fails, the database suddenly gets all the traffic.

When not to use it

• Data must always be exactly current, like an account balance during a payment. Read it from the source of truth.
• Reads are rarely repeated, so the hit rate would be low.
• The data is cheap to compute or fetch anyway; the cache just adds complexity.

In the cloud

In-memory cachingEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS2

• Amazon ElastiCache (opens vendor docs)

  Managed Valkey, Redis OSS or Memcached, as a serverless cache or a cluster of nodes you size.

• Amazon DynamoDB Accelerator (DAX) (opens vendor docs)

  A cache that sits in front of DynamoDB and answers repeat reads in microseconds.

Azure1

• Azure Managed Redis (opens vendor docs)

  Microsoft's managed Redis service, the replacement for the retiring Azure Cache for Redis.

Google Cloud2

• Memorystore for Valkey (opens vendor docs)

  Managed Valkey (an open-source Redis fork) with automatic failover across zones.

• Memorystore for Redis Cluster (opens vendor docs)

  Managed, horizontally scalable Redis with a 99.99% availability SLA.

App servers check the cache first and only go to the database on a miss (cache-aside).

AWSAzureGoogle Cloud

1. Application Load Balancer
2. EC2 app servers
3. ElastiCache for Valkey
4. Amazon RDS

Interview questions

Try answering out loud first, then open the model answer.

Walk me through cache-aside. What are its weak points?+

Model answer

On a read, the app checks the cache. On a hit it returns the value. On a miss it reads the database, writes the result into the cache with a TTL, and returns it. On a write, it updates the database and then deletes the cache key.

Weak points: the first read after every miss is slow; there's a small race where a reader can repopulate the cache with an old value right after a write; and if a hot key expires you can get a stampede. I'd mitigate with TTLs plus delete-on-write, request coalescing for hot keys, and jittered expiry.

How do you choose a TTL?+

Model answer

Ask how stale the data can be before a user would notice or be harmed. A product description can be minutes or hours old; stock levels, maybe seconds; a bank balance, not at all. Then balance against load: shorter TTLs mean more misses. I usually pair a moderate TTL with delete-on-write, so the TTL is only a safety net. And I add a small random jitter so keys created together don't expire together.

Your cache cluster goes down. What happens and how do you protect the database?+

Model answer

All reads fall through to the database at once, which can overload it and turn a cache outage into a full outage. Protections: run the cache with replicas across zones so one node failing doesn't empty it; use a circuit breaker or rate limit in front of the database; serve slightly stale data from an in-process cache for the hottest keys; and warm a new cache gradually instead of all at once.

When would you choose LFU over LRU?+

Model answer

When popularity is stable over time and there are occasional one-off scans. LRU would let a single large scan push out genuinely popular items, while LFU keeps them because they've been used many times. If popularity shifts quickly, as with trending content, LRU adapts faster. Many real systems use a mix, like Redis's approximated LFU with decay.

Quiz

Pick an answer to see whether it’s right and why.

1. 01Your cache has a 90% hit rate. Cache reads take 1 ms and database reads take 20 ms. Roughly what is the average read time, assuming a miss checks the cache first?

   About 1 msAbout 3 msAbout 11 msAbout 20 ms
2. 02You cache user profiles with cache-aside. A user changes their name. What should the app do so others don't keep seeing the old name?

   Nothing, the cache will notice the database changedUpdate the database, then delete (or update) that user's cache entryClear the entire cache
3. 03A job reads items A, B, C … J in order, over and over, with a cache that holds 4 items and uses LRU. What hit rate do you get?

   About 40%About 0%About 90%
4. 04Which write strategy risks losing recent writes if the cache crashes?

   Cache-asideWrite-throughWrite-back
5. 05A hot key expires and 5,000 requests hit the database at once. What is this, and what's one fix?

   A cold start: restart the cacheA cache stampede: let one request rebuild the value while others wait or get the old copyReplication lag: read from the leader
6. 06Which is the worst candidate for caching with a 10-minute TTL?

   A product descriptionA blog postAn account balance shown just before a payment
7. 07After a deploy, every server's in-memory cache is empty. What should you expect?

   A burst of database load until the caches warm upNo changeFaster responses

0 of 7 answered

Key terms

Cache

A small, fast store that keeps copies of data that is expensive to fetch, so repeat requests are quicker.

Cache hit

The requested data was found in the cache.

Cache miss

The requested data was not in the cache, so it had to be fetched from the slower source.

Hit rate

The share of requests answered from the cache. 90% means 9 in 10 requests never reach the database.

Cache-aside

The application checks the cache first, and on a miss reads the database and stores the result in the cache. Also called lazy loading.

Write-through

Every write goes to the cache and the database at the same time, so the cache is never stale.

Write-back

Writes go to the cache first and are saved to the database later in batches. Fast, but data can be lost if the cache fails.

TTL (time to live)

How long a cached item stays valid before it expires and must be fetched again.

Eviction

Removing an item from a full cache to make room for a new one.

LRU

Least recently used. An eviction policy that removes the item that hasn't been read for the longest time.

LFU

Least frequently used. An eviction policy that removes the item read the fewest times.

Cache invalidation

Removing or updating cached data when the source changes, so users don't see stale values.