Everyone has an opinion about "the algorithm." Nobody shows you how it actually works.
Feed design is one of those problems that looks deceptively simple — query posts from people you follow, sort by time, done. Then your service hits 10k users, your DB starts sweating, and your "chronological sort" suddenly feels like a liability. Scale it to millions, add personalization, and you’re staring at one of the hardest distributed systems problems in consumer tech.
This article rips apart the core mechanics of a social feed system: how posts move from writer to reader, how ranking actually works at scale, and where your caching layer fits. These are the exact decisions engineers at Instagram, Twitter/X, and TikTok agonize over — and the tradeoffs are messier than any blog post usually admits.
The Core Problem (It’s Not What You Think)
Your first instinct is "just query the DB." Let’s price that out.
User A follows 500 people. Each posts 3 times a day. A feed request needs to aggregate 1500 posts, filter them, rank them, and return 20 within 100ms. Now multiply that by 10 million active users hitting the app every morning.
A naive SELECT ... WHERE user_id IN (...) ORDER BY created_at DESC LIMIT 20 will destroy your database. Not eventually — fast. Even with indexes, the fan-in on read at scale is brutal.
This is the fundamental tension: you either do work at write time or at read time. Every feed architecture is just a point on that spectrum.
Push vs Pull: Choose Your Poison
These two models go by many names. The industry usually calls them fan-out on write (push) and fan-out on read (pull).
Fan-Out on Write (Push Model)
When Alice posts a photo, your system immediately fans out that post ID to every one of her followers’ feed inboxes. Each follower has a pre-computed feed stored in Redis or a similar fast store. When Bob opens the app, his feed is already assembled — you just read it.
Alice posts
│
▼
[Post Service] ──► [Kafka topic: new-posts]
│
▼
[Fan-out Worker]
│
┌─────────┼─────────┐
▼ ▼ ▼
Bob's feed Carol's feed Dave's feed
(Redis list) (Redis list) (Redis list)
Read latency is phenomenal. Bob’s GET feed:bob hits Redis and returns in sub-millisecond time.
The catch: Alice has 2 million followers. One post = 2 million Redis writes, triggered synchronously. This is the celebrity problem, and it will eat your fan-out worker fleet alive if you’re not careful.
Twitter called celebrities "high-fanout users" and spent years engineering around them. At Instagram’s scale, Kylie Jenner posting a selfie used to be an engineering event — not a joke.
Fan-Out on Read (Pull Model)
No pre-computation at write time. When Bob opens the app, your system queries posts from everyone Bob follows, merges and ranks them on the fly, and serves the result.
Bob opens feed
│
▼
[Feed Service]
│
├──► SELECT posts WHERE author_id IN Bob's following list
│ (potentially cached per-author)
│
└──► Merge + Rank + Return top-N
This model scales writes beautifully — one post, one write, done. But reads become expensive, especially as following lists grow. You also can’t pre-rank; every read is a fresh computation.
The Hybrid (What Everyone Actually Uses)
Instagram, Twitter, and most large platforms use a hybrid: push for regular users, pull for celebrities.
- Normal user with 300 followers → fan-out on write. Their posts go straight into follower feed caches.
- Celebrity with 5M followers → post is stored once. When fans open the feed, their pre-computed cache is merged with a pull of the top celebrity posts they follow.
This caps your worst-case fan-out while keeping read latency low for the common case. The complexity cost is real — you’re maintaining two code paths — but there’s no cleaner solution at scale.
Gotcha: The threshold between "normal" and "celebrity" is a moving target. A user with 10k followers today might go viral and hit 500k by tomorrow. Your fan-out worker needs circuit breakers and backpressure so a suddenly-popular account doesn’t become a DDoS on your own infrastructure.
The Storage Layer
Before we touch ranking, let’s talk about where feeds actually live.
Per-User Feed Cache in Redis
The canonical approach is a Redis sorted set per user, where the score is a timestamp (or ranking score) and the member is a post ID.
# Write: fan-out worker adds post to Bob's feed
ZADD feed:bob 1716700800 "post:abc123"
# Read: get top 20 posts
ZREVRANGE feed:bob 0 19
# Trim old posts (keep only latest 800)
ZREMRANGEBYRANK feed:bob 0 -801
Keep the feed cache shallow — 800 posts is plenty. Users almost never scroll past that. Deeper caches waste memory and make trimming harder.
Gotcha: Don’t store the full post payload in Redis. Store post IDs only. When you render the feed, do a batch MGET on post:{id} keys to hydrate the data. This way, if a post is edited or deleted, you update one place, not every feed cache it appears in.
Cold Start and Cache Miss
What happens when a new user’s feed cache doesn’t exist yet? Or it expired? You need a cache-miss handler that can reconstruct the feed from the DB. This path should be async where possible — serve a "loading" state, then push the assembled feed back via WebSocket or on the next poll.
Never let a cold-start path hit your main DB directly during peak traffic. Throttle it, queue it, and use a replica.
Ranking: The Part Nobody Explains
Chronological feeds feel fair but perform terribly for engagement. The moment you have more content than a user can consume, you need a ranking layer.
Signal Collection
Before you rank, you need signals. The core engagement signals every feed system tracks:
- Recency — how fresh is the post
- Affinity — how often does this user interact with this author (likes, comments, DMs, profile visits)
- Post engagement velocity — how fast is this post accumulating likes/comments/shares since it was posted
- Content type preference — does this user engage more with videos, carousels, or single images
- Session context — time of day, app version, network quality (yes, some systems serve lighter feeds on 3G)
The Scoring Function
In early-stage systems, a weighted linear score is good enough and explainable:
def score_post(post, user_affinity, engagement_rate):
recency_score = 1 / (1 + hours_since_posted(post))
affinity_score = user_affinity.get(post.author_id, 0)
virality_score = min(engagement_rate, 1.0)
return (
0.4 * recency_score +
0.35 * affinity_score +
0.25 * virality_score
)
This is dead simple and you can tune weights based on A/B tests without touching a model.
When to Add ML
Once you’re collecting interaction data at scale, a proper ranking model pays off. The standard approach is a two-stage ranking pipeline:
Stage 1 — Retrieval / Candidate Generation: Pull ~500 candidate posts from the user’s feed cache. Fast, dumb selection. Purpose is to limit the ranking set.
Stage 2 — Re-Ranking: Run candidates through a lightweight ML model (gradient boosted trees work well here, LightGBM is the industry default before you hit Google-scale). The model outputs a probability of engagement (P(like), P(comment), P(share)) and you compute a weighted expected value.
final_score = w1 * P(like) + w2 * P(comment) + w3 * P(share) - w4 * P(hide)
Note the P(hide) term — a negative signal. Optimizing only for positive engagement without penalizing "hide post" or "unfollow" events creates engagement loops that users eventually hate. Instagram learned this the hard way.
Gotcha: Your ranking model has to be fast. Sub-10ms for the re-ranking stage across 500 candidates. If your inference is slow, you’re introducing latency directly into every feed load. Run models in-process (not as a network call) and keep feature vectors precomputed where possible.
Caching Architecture in Full
Let me lay out the complete caching stack, because there are multiple layers and they serve different purposes.
User Request
│
▼
[CDN / Edge Cache] ← Static assets, media thumbnails
│
▼
[API Gateway + Rate Limiter]
│
▼
[Feed Service]
│
├──► [Redis L1: User Feed Cache] ← Sorted set of post IDs, TTL 24h
│ │ miss
│ ▼
│ [Feed Reconstruction Job] ← Async, from Postgres replica
│
├──► [Redis L2: Post Data Cache] ← Post payloads, keyed by post ID
│ │ miss
│ ▼
│ [Postgres (replica)] ← Source of truth
│
└──► [Redis L3: Author Affinity] ← Precomputed per (user, author) pair
TTL Strategy
- Feed sorted sets: 24-hour TTL with a background job that refreshes active users’ feeds every few hours.
- Post payloads: 1-hour TTL. Posts don’t change often; when they do, you invalidate explicitly.
- Affinity scores: Recalculate daily in a batch job. Don’t try to do this in real-time on every interaction — too expensive.
Cache Stampede Protection
When a popular user’s feed cache expires and 50k requests hit at once, you get a thundering herd. Fix this with probabilistic early expiration — the cache starts refreshing before it actually expires, based on a probability that increases as TTL shrinks.
In Redis, the common pattern is to store the TTL as part of the value and recompute if (ttl - time_left) / ttl > random().
Gotcha: Don’t use Redis EXPIREAT for this. Use your application layer to check and refresh. Letting Redis hard-expire popular keys under load is a silent killer.
The Write Path: From Post to Feed
Here’s the full write path from a post being created to it appearing in followers’ feeds:
1. User submits post
2. Post Service validates, stores in Postgres, returns post ID
3. Post Service publishes event to Kafka: { post_id, author_id, timestamp }
4. Fan-out Consumer reads from Kafka:
a. Fetches author's follower list (paginated from DB)
b. For each follower under threshold → ZADD to feed:{follower_id}
c. For celebrity accounts → skip; handle at read time
5. Post data is written to Redis post cache
6. Push notification sent (if follower has enabled them)
Kafka is the right tool for step 3-4. It decouples the post creation from the fan-out work, which can be slow (2M writes takes time). It also gives you replay capability if your fan-out worker crashes mid-job.
Production tip: Partition your Kafka topic by author_id. This ensures all posts from the same author go to the same partition and are processed in order — no race conditions where post B appears before post A in a follower’s feed because they landed on different partitions.
Gotchas and Production Realities
The delete problem. When a user deletes a post, it might exist in hundreds of thousands of feed caches. You can’t enumerate and purge them all synchronously. The practical solution: keep a deleted_posts set in Redis with a long TTL, and filter it at read time when hydrating the feed. Tombstoning, essentially.
Muted users and blocked users. Your feed cache doesn’t know about mute/block relationships — those are social graph edges that change independently of content. Filter at read time, after cache hydration. Never bake relationship state into the cache unless you want a massive invalidation problem.
Clock skew in distributed systems. Your fan-out workers run on multiple machines. If you’re using wall clock timestamps as Redis sorted set scores, a worker with a skewed clock will insert posts at the wrong position. Use a logical sequence from your Kafka offset or a centralized sequence generator for ordering, not wall clocks.
The "new user" ranking problem. A brand-new user follows 10 accounts. Your affinity model has zero data for them. Cold-start ranking falls back to global popularity signals — essentially showing them what’s trending among all users. This is fine for the first few sessions, but make sure your system detects when you have enough per-user data to switch to personalized ranking. Mixing signals from different regimes without flagging it quietly degrades quality.
Consistency vs. freshness tradeoff. Pre-computed feed caches are inherently stale. If someone you follow posts at 9:00 AM and you open the app at 9:01 AM, you might not see it for a few minutes. For most social apps, eventual consistency here is fine and users accept it. But if you’re building something where real-time freshness matters (news, sports scores), you need to either shorten your fan-out pipeline latency or add a real-time injection layer that bypasses the feed cache for time-sensitive content.
Scaling the Fan-Out Worker
Your fan-out worker is the bottleneck. Here’s how to scale it without losing your mind.
Run multiple consumer instances on your Kafka topic, partitioned by author hash range. Each worker handles a subset of authors.
For large accounts that do fan-out (just below your celebrity threshold, say 50k-500k followers), parallelize the follower list pagination. Split the follower list into chunks and write to Redis in parallel batches of 500.
async def fan_out_post(post_id, author_id):
followers = await get_followers_paginated(author_id)
# Chunk followers into batches of 500
for batch in chunks(followers, 500):
await asyncio.gather(*[
redis.zadd(f"feed:{fid}", {post_id: score})
for fid in batch
])
Use a dedicated Redis cluster for feed data, separate from your session and application cache. Feed data has very different access patterns and you don’t want a thundering herd on posts knocking out your session store.
A Note on Infinite Scroll and Pagination
Feeds are infinite. Your pagination strategy matters.
Cursor-based pagination on the sorted set score works well — the client sends back the score of the last item it saw, and you do ZREVRANGEBYSCORE feed:{uid} {last_score} -inf LIMIT 20. No offset drift when new posts are inserted at the top.
Gotcha: If you’re re-ranking posts, scores aren’t stable. A post’s score can change between page loads as new engagement data comes in. This causes posts to jump around or appear twice during pagination. Common fix: freeze the ranking for a given session. When a user opens the feed, snapshot the current ranked list into a session-scoped sorted set with a 30-minute TTL. Paginate from that snapshot, not the live ranked set.
Summary: The Decision Tree
Should you use push or pull?
- < 50k users: pull with aggressive query caching per author. Ship it, stop over-engineering.
- 50k–5M users: push (fan-out on write) with Redis sorted sets. Add celebrity threshold.
- > 5M users: hybrid model. Invest in a two-stage ranking pipeline. Separate Redis clusters per data type.
The ranking question:
- Start with a weighted linear score you can explain and tune.
- Add ML when you have 90 days of interaction data and an A/B testing framework.
- Never optimize for engagement alone — add negative signal terms from day one.
The caching question:
- Store post IDs in feed caches, not post payloads.
- Use tombstoning for deletes, filter mutes/blocks at read time.
- Protect against cache stampede with probabilistic early refresh.
None of this is magic. The complexity comes from edge cases — celebrity accounts, deleted content, cold starts, clock skew — not from the happy path. Design for those edges first, and the happy path takes care of itself.