Design a news feed

Think of it like this

Analogy

A newspaper delivery service. One option: when an article is published, a team immediately puts a copy in every subscriber's mailbox (fan-out on write). Opening your mailbox is instant, but a story with ten million subscribers takes a lot of work. The other option: subscribers come to the newsroom and the staff assemble a personal bundle on the spot from every writer they follow (fan-out on read). Publishing is instant, but every visit means work.

The problem it solves

Opening the app should show a fresh, personalized feed in well under a second. A user might follow hundreds or thousands of accounts. Gathering each one's recent posts, merging and ranking them on every app open is too slow at scale. Precomputing everything breaks down for accounts with millions of followers.

A real example

Example

Your feed when you open the app in the morning

Overnight, 300 of the people you follow posted. When each of them posted, a background job added the post's ID to the top of the feed lists of all their followers, yours included. That's fan-out on write, and it means your feed is already built when you open the app.

One account you follow has 40 million followers. Writing to 40 million lists for every post would be far too slow, so that account's posts aren't fanned out. When you open the app, its recent posts are fetched and merged into your list.

The app then loads the posts themselves from a cache, ranks them, and shows the first 20. Scrolling down asks for the next 20 using a cursor, so new posts arriving at the top don't push items you've already seen down into the next page.

How it works

1. Requirements

Functional: post text and photos; follow users; see a feed of posts from people you follow, newest or ranked first; infinite scroll.

Non-functional: feed loads in under about 300 ms; posts appear in followers' feeds within seconds (eventual consistency is fine); hundreds of millions of daily users.

2. Estimates

300 M daily users, each opens the feed 10 times (3 billion feed reads a day ≈ 35,000 per second) and 10% post once a day (30 M posts ≈ 350 per second). Reads outnumber writes 100 to 1, so optimize reads.

3. Data

• Posts: post_id, author_id, text, media keys, created_at, sharded by post_id or author.
• Follows: follower_id → followee_id and the reverse, sharded by user.
• Feeds: per user, a list of the latest few hundred post IDs, stored in a fast cache or key-value store.

4. Fan-out on write vs on read

Aspect	Fan-out on write (push)	Fan-out on read (pull)
On post	Insert post ID into every follower's feed	Store the post only
On feed read	Read one precomputed list	Fetch recent posts of everyone you follow and merge
Read speed	Very fast	Slower
Write cost	One write per follower	One write
Problem	Celebrities with millions of followers	Users who follow thousands of accounts

The hybrid most large networks use: push for normal accounts, pull for celebrities. When building your feed, read your precomputed list and merge in recent posts from the few celebrities you follow.

Architecture

Loading diagram…

> Initializing simulation kernel

> Allocating nodes

> Opening network links

• User posts to Post service
• Post service to Posts store
• Post service to Fan-out queue
• Fan-out queue to Fan-out workers
• Fan-out workers to Follower graph: who follows?
• Fan-out workers to Feed cache: push IDs
• Reader to Feed service
• Feed service to Feed cache

5. Reading the feed

1. Read the user's list of post IDs from the feed cache (one lookup).
2. Merge in recent posts from followed celebrities.
3. Fetch post bodies, authors and counts in batches from caches (“hydration”).
4. Rank (newest first, or a model scoring relevance), filter (blocked users, deleted posts) and return a page with a cursor for infinite scroll.

6. Details that matter

• Feed length: keep only the latest few hundred IDs per user; older items come from a slower path.
• Inactive users: don't fan out to users who haven't opened the app in a month; rebuild their feed on their next visit.
• Hot posts: like and comment counts for viral posts are a hotspot; use sharded counters and caching.
• Deletes and privacy: filter at read time, since a deleted post's ID may still be in many feeds.
• Media: images and videos are served from object storage through a CDN.

Trade-offs

What you gain

• Precomputed feeds make the most frequent operation, reading, very fast.
• The hybrid model handles both celebrities and heavy followers.
• Queue-based fan-out absorbs posting spikes.

What it costs you

• Lots of storage for duplicated post IDs.
• Feeds are eventually consistent; posts appear after a short delay.
• Ranking, filtering and deletions add complexity at read time.

When not to use it

• For small apps, simply query “recent posts from people I follow” with a good index; precomputed feeds are overkill.
• Don't fan out to every follower of a celebrity; use pull for them.

In the cloud

In-memory cachingEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS2

• Amazon ElastiCache (opens vendor docs)

  Managed Valkey, Redis OSS or Memcached, as a serverless cache or a cluster of nodes you size.

• Amazon DynamoDB Accelerator (DAX) (opens vendor docs)

  A cache that sits in front of DynamoDB and answers repeat reads in microseconds.

Azure1

• Azure Managed Redis (opens vendor docs)

  Microsoft's managed Redis service, the replacement for the retiring Azure Cache for Redis.

Google Cloud2

• Memorystore for Valkey (opens vendor docs)

  Managed Valkey (an open-source Redis fork) with automatic failover across zones.

• Memorystore for Redis Cluster (opens vendor docs)

  Managed, horizontally scalable Redis with a 99.99% availability SLA.

App servers check the cache first and only go to the database on a miss (cache-aside).

AWSAzureGoogle Cloud

1. Application Load Balancer
2. EC2 app servers
3. ElastiCache for Valkey
4. Amazon RDS

Interview questions

Try answering out loud first, then open the model answer.

Fan-out on write or on read? Explain your choice.+

Model answer

A hybrid. Most users have a modest number of followers, so pushing their post IDs into followers' feed caches at write time makes reads, which outnumber writes about 100 to 1, a single fast lookup. For accounts above a threshold (say a million followers), skip fan-out and pull their recent posts at read time, merging them with the precomputed feed. I'd also skip fan-out to inactive users and rebuild their feed on demand.

How would you add ranking instead of a purely chronological feed?+

Model answer

Gather candidates (the precomputed feed plus celebrity posts plus possibly recommended posts), fetch features for each (author affinity, engagement so far, recency, media type), score them with a model in a ranking service, and return the top results. Keep chronological as a fallback when ranking is slow or down. Cache ranked pages briefly per user, and log impressions and engagement to train the model.

How do you handle a post being deleted after it's been fanned out?+

Model answer

Mark it deleted in the posts store (a tombstone) and filter deleted or hidden posts during hydration at read time, since removing the ID from millions of cached feeds immediately is impractical. Optionally run a background job to clean IDs out of feeds lazily. Caches of the post itself are invalidated on delete so its content stops being served right away.

Quiz

Pick an answer to see whether it’s right and why.

1. 01With fan-out on write, what happens when a user with 50 million followers posts?

   One quick write50 million feed inserts, which is slow and expensiveNothing; followers must refresh
2. 02What's the main advantage of fan-out on write?

   Reading a feed is a fast, precomputed lookupPosts become visible faster to the posterIt uses less storage
3. 03Feeds are usually eventually consistent. What does that mean for users?

   Posts may appear in followers' feeds a few seconds after they're publishedPosts can be lostUsers see each other's private posts
4. 04How do most large feeds handle accounts with millions of followers?

   Fan out on write for everyoneA hybrid: fan out on write for normal accounts, and merge celebrity posts in at read timeDon't show their posts in feeds
5. 05What does the precomputed feed usually store?

   Post IDs, with the post content fetched and cached separatelyFull copies of every post, including imagesNothing
6. 06Why page through a feed with a cursor instead of page numbers?

   Cursors are shorterPage numbers aren't allowed in URLsNew posts keep arriving, so offsets shift and users would see repeats or miss posts

0 of 6 answered

Key terms

Fan-out

Delivering one item to many recipients, either when it's written (on write) or when it's read (on read).

Cache

A small, fast store that keeps copies of data that is expensive to fetch, so repeat requests are quicker.

Message queue

A buffer where producers leave messages and consumers pick them up later, so neither waits for the other.

Sharding

Splitting a dataset into pieces (shards) and storing each piece on a different machine.

Hotspot

One shard or server getting far more traffic than the rest.

Eventual consistency

Copies may disagree for a while, but if writes stop, they all end up the same.