---

1-Hour Interview Roadmap

Time	Topic
0 – 5 min	Requirements clarification
5 – 10 min	Capacity estimation
10 – 15 min	High-level architecture + core components
15 – 25 min	Photo/video upload pipeline
25 – 40 min	News feed — generation + fan-out strategy
40 – 48 min	Database design + caching
48 – 55 min	Search, notifications, scaling
55 – 60 min	Trade-offs + failure scenarios

---

What Are We Building?

A photo and video sharing social platform where users can:

• Upload photos and short videos
• Follow other users
• See a personalized home feed of posts from people they follow
• Like, comment, and share posts
• Search for users and hashtags
• Receive real-time notifications

Scale reference: Instagram has 2 billion+ monthly active users, 500 million daily active users, 100 million photos uploaded per day, and serves 4.8 billion photo views per day.

---

Step 1 — Requirements

Functional Requirements

#	Requirement
1	Users can register, log in, and manage their profile
2	Users can upload photos and short videos with captions
3	Users can follow / unfollow other users
4	Users see a personalized feed of posts from accounts they follow
5	Users can like and comment on posts
6	Users can search for other users and hashtags
7	Users receive notifications — likes, comments, new followers
8	Posts can have hashtags; hashtag pages show all related posts
9	Users can view any public profile and their posts

Non-Functional Requirements

#	Requirement
1	Feed generation latency < 200ms
2	Photo/video upload available 99.99% of the time
3	Eventual consistency acceptable for feed and counts (likes, followers)
4	Strong consistency for follows — user must see their own actions immediately
5	Read-heavy system — optimize for reads over writes
6	Horizontally scalable — support billions of users
7	Media served globally with low latency via CDN

Out of Scope

• Instagram Stories, Reels algorithm details
• Direct messaging (DMs)
• Shopping and advertising platform
• Content moderation pipeline

---

Step 2 — Capacity Estimation

Assumptions

Metric	Value
Daily Active Users (DAU)	500 million
Monthly Active Users (MAU)	2 billion
Photos uploaded per day	100 million
Average photo size (compressed)	200 KB
Average video size	5 MB
Video uploads (% of total)	20%
Average follows per user	500
Feed loads per user per day	10
Read-to-write ratio	~100:1

Upload Throughput

100 million uploads/day  (80M photos + 20M videos)
= ~1,160 uploads/second (sustained)
= ~3,500 uploads/second (peak)

Storage

Photos:  80M/day  × 200 KB  = 16 TB/day
Videos:  20M/day  × 5 MB    = 100 TB/day
Total:   ~116 TB/day

With 3× replication:  ~350 TB/day
5-year storage:       ~640 PB

Feed Read Traffic

500M DAU × 10 feed loads/day  = 5 billion feed requests/day
= ~58,000 feed reads/second (sustained)
= ~200,000 feed reads/second (peak)

Key Insight

Instagram is an extremely read-heavy system with an asymmetric write problem — a celebrity with 200 million followers creates a single post that must appear in 200 million feeds. The news feed fan-out strategy is the central design challenge.

---

Step 3 — High-Level Architecture

flowchart TD
    Client --> AG[API Gateway\nAuth + Rate Limiting]

    AG --> US[User Service]
    AG --> PS[Post Service]
    AG --> FS[Feed Service]
    AG --> SS[Search Service]
    AG --> NS[Notification Service]

    PS --> MQ[Kafka\nPost Events]
    PS --> MS[Media Service]
    MS --> S3[Object Storage\nS3]
    S3 --> CDN[CDN\nCloudFront]

    MQ --> FO[Fan-out Service]
    FO --> FC[(Feed Cache\nRedis)]
    FO --> DB[(Cassandra\nTimeline Store)]

    FS --> FC
    FC -->|miss| DB

    US --> UDB[(User DB\nPostgreSQL)]
    US --> GDB[(Graph DB\nCassandra\nFollow Graph)]

    NS --> KNS[Kafka\nNotification Events]
    KNS --> NW[Notification Worker]
    NW --> FCM[FCM / APNs]

Component Responsibilities

Component	Responsibility
API Gateway	JWT auth, rate limiting, request routing
User Service	Registration, profile management, follow/unfollow
Post Service	Create, edit, delete posts; trigger media upload and fan-out
Media Service	Accept uploads, transcode videos, generate thumbnails, store in S3
Fan-out Service	On new post — write to followers' feed caches or timelines
Feed Service	Assemble and return personalized feed from cache or DB
Search Service	User and hashtag search (Elasticsearch)
Notification Svc	Produce and deliver like/comment/follow notifications
CDN	Serve photos and videos globally with edge caching

---

Step 4 — Photo and Video Upload Pipeline

The Problem

Uploading 100 million photos/day means ~1,160 uploads/second at 200KB each. Videos are 25× larger. The upload path must be:

• Non-blocking (client should not wait for processing)
• Resilient (if transcoding fails, re-process; don't lose the upload)
• Globally fast (upload to nearest region, replicate asynchronously)

Upload Flow

sequenceDiagram
    participant Client
    participant PS as Post Service
    participant MS as Media Service
    participant S3
    participant KF as Kafka
    participant TC as Transcoder

    Client->>PS: Create post (metadata only)
    PS->>MS: Request pre-signed S3 URL
    MS-->>Client: Pre-signed upload URL
    Client->>S3: Upload file directly (bypass backend)
    S3-->>MS: Upload complete event
    MS->>KF: Publish "media.uploaded" event
    KF->>TC: Consume & transcode / generate thumbnails
    TC->>S3: Store multiple resolutions
    TC->>PS: Mark post as READY
    PS->>KF: Publish "post.created" event → triggers fan-out

Why Pre-signed URLs?

Raw media bytes never flow through your application servers:

• Upload goes straight to S3 — eliminates one network hop
• Your servers scale on requests/second, not on bandwidth
• Large video files (500MB Reels) would exhaust server memory if routed through backend

Media Processing Pipeline

flowchart LR
    S3[Raw Upload\nS3] --> TC[Transcoder\nFFmpeg workers]
    TC --> TH[Thumbnail\n150×150]
    TC --> SM[Small\n480px]
    TC --> MD[Medium\n1080px]
    TC --> LG[Original\n4K]
    TH --> CDN
    SM --> CDN
    MD --> CDN
    LG --> CDN

• Photos: generate 3 sizes (thumbnail, feed, full)
• Videos: transcode to HLS format for adaptive bitrate streaming
• All sizes stored in S3, served via CDN

---

Step 5 — News Feed

Why Feed Generation Is the Hard Problem

A user following 500 accounts expects to see a ranked feed of recent posts in under 200ms. At 500M DAU each loading 10 feeds/day, that is 5 billion feed reads/day. The central challenge is: how do you pre-compute or assemble this efficiently?

Two Approaches

Approach 1 — Fan-out on Write (Push Model)

When Alice posts, immediately write her post ID to the feed cache of every one of her followers.

flowchart TD
    Alice -->|New post| FS[Fan-out Service]
    FS -->|Lookup followers| GDB[Graph DB]
    FS -->|Write post_id| F1[Feed Cache\nBob]
    FS -->|Write post_id| F2[Feed Cache\nCarol]
    FS -->|Write post_id| F3[Feed Cache\nDave]
    F1 & F2 & F3 --> Redis[(Redis)]

Pros	Cons
Feed reads are instant (pre-built)	Write amplification — celebrity with 200M followers = 200M writes
Low feed latency	Wasted writes for inactive followers

Approach 2 — Fan-out on Read (Pull Model)

When Bob opens his feed, pull recent posts from each account he follows and merge/sort in real time.

flowchart TD
    Bob -->|Open feed| FS[Feed Service]
    FS -->|Fetch following list| GDB[Graph DB]
    FS -->|Fetch posts per account| PDB[Post DB]
    FS -->|Merge + rank| Feed[Ranked Feed]
    Feed --> Bob

Pros	Cons
No write amplification	Slow feed generation — N DB queries per feed load
Always fresh posts	Expensive for users following thousands of accounts

Recommended Hybrid Approach (Instagram's actual approach)

flowchart TD
    NewPost -->|Check follower count| FC{Celebrity?\n> 1M followers}
    FC -->|Regular user| FW[Fan-out on Write\nWrite to all follower caches]
    FC -->|Celebrity| FR[Fan-out on Read\nFetch at read time]

    Bob_feed[Bob opens feed] --> MG[Feed Merger]
    MG -->|Pre-built cache| FW2[Regular users' posts\nfrom Redis]
    MG -->|Fetch at read time| FR2[Celebrity posts\npull fresh]
    MG --> RankedFeed[Merge + Rank + Return]

• Regular users (< 1M followers): Fan-out on write → instant reads from Redis
• Celebrities (> 1M followers): Skip fan-out write → fetch their recent posts at read time and merge
• This eliminates the celebrity write amplification problem while keeping feed latency low

Feed Cache Structure (Redis)

Key:   feed:{user_id}
Value: Sorted Set — score = timestamp, member = post_id

GET feed:12345 → [post_99, post_97, post_95, post_88, ...]

• Each user's feed cache holds the most recent ~1,000 post IDs
• Feed service fetches post IDs from cache, then batch-fetches post details from Post DB or Post cache
• Feed is sorted and ranked (recency + engagement signals)

---

Step 6 — Database Design

Database Selection

Data	Database	Reason
Users & profiles	PostgreSQL	Structured, relational, strong consistency
Follow graph	Cassandra	Wide-row: (user_id, follower_id) — scales to billions
Posts metadata	Cassandra	Write-heavy, time-ordered, append-only
Feed timeline	Redis (cache) + Cassandra (persistence)	Fast reads; durable backup
Likes & comments	Cassandra	Very high write volume, time-ordered
Hashtags & search	Elasticsearch	Full-text search, inverted index
Media metadata	Cassandra	High volume, simple lookup by post_id

Post Table (Cassandra)

Table: posts

Partition key:  user_id       BIGINT
Clustering key: post_id       BIGINT (Snowflake — time-ordered, newest first)

Columns:
  caption        TEXT
  media_urls     LIST<TEXT>   (CDN URLs for each resolution)
  hashtags       SET<TEXT>
  like_count     COUNTER
  comment_count  COUNTER
  created_at     TIMESTAMP
  status         VARCHAR      (PROCESSING | READY | DELETED)

Query pattern:

• SELECT * FROM posts WHERE user_id = ? ORDER BY post_id DESC LIMIT 20 — profile page
• Post IDs fetched from feed cache, then batch-fetched here

Follow Graph (Cassandra)

Table: followers
  user_id      BIGINT   (who is being followed)
  follower_id  BIGINT   (who is following)
  created_at   TIMESTAMP
  PRIMARY KEY (user_id, follower_id)

Table: following
  user_id      BIGINT   (the follower)
  followed_id  BIGINT   (who they follow)
  created_at   TIMESTAMP
  PRIMARY KEY (user_id, followed_id)

Two tables (denormalized) — one for "who follows me" (fan-out), one for "who do I follow" (feed assembly).

Likes Table (Cassandra)

Table: likes

Partition key:  post_id      BIGINT
Clustering key: user_id      BIGINT
                liked_at     TIMESTAMP

PRIMARY KEY (post_id, user_id)

• Unique constraint: each (post_id, user_id) pair is written at most once
• To check if a user liked a post: point lookup — O(1)
• To count likes: Redis counter (approximate) or SELECT COUNT(*) (expensive)

User Table (PostgreSQL)

Table: users

  user_id        BIGINT   PRIMARY KEY  (Snowflake)
  username       VARCHAR  UNIQUE NOT NULL
  email          VARCHAR  UNIQUE NOT NULL
  display_name   VARCHAR
  bio            TEXT
  avatar_url     TEXT
  follower_count BIGINT   DEFAULT 0
  following_count BIGINT  DEFAULT 0
  post_count     BIGINT   DEFAULT 0
  is_private     BOOLEAN  DEFAULT false
  created_at     TIMESTAMP

---

Step 7 — Caching Strategy

Cache Layers

flowchart LR
    Client --> AG[API Gateway]
    AG --> LC[Local Cache\nCaffeine\nper pod]
    LC -->|miss| RC[(Redis Cluster)]
    RC -->|miss| DB[(Cassandra / PostgreSQL)]

Redis Cache Design

Cache	Key Pattern	TTL	Contents
Feed cache	feed:{user_id}	24 hrs	Sorted set of post IDs (newest first)
Post details	post:{post_id}	7 days	Full post object (caption, media URLs, counts)
User profile	user:{user_id}	1 hr	Profile data, counts
Like check	like:{post_id}:{user_id}	1 hr	Boolean — did user like this post?
Like count	likes:{post_id}	Eventual	Counter (INCR/DECR)
Follower count	followers:{user_id}	Eventual	Counter
Trending hashtags	trending:hashtags	1 hr	Sorted set by post count in last 24h
Celebrity recent posts	posts:celebrity:{user_id}	5 min	Last 20 post IDs for read-time fan-out merge

Like Count Strategy

Writing an exact like count to the database on every like is expensive at scale (millions of likes per minute on viral posts).

Solution:

1. On like: INCR likes:{post_id} in Redis — O(1), near-zero latency
2. Async worker periodically flushes Redis counters to Cassandra (every 30 seconds)
3. DB count is eventually consistent — acceptable for likes

---

Step 8 — Search

Search Requirements

• Search users by username or display name
• Search hashtags — find all posts with a given tag
• Autocomplete suggestions while typing

Why Elasticsearch?

Feature	Elasticsearch	SQL Database
Full-text search	Native inverted index — very fast	LIKE '%query%' — full scan
Autocomplete	search_as_you_type field type	Not supported natively
Fuzzy matching	Built-in — handles typos	Not supported
Horizontal scale	Sharding built-in	Difficult

Search Index Design

User index:
{
  "user_id": 12345,
  "username": "john_doe",
  "display_name": "John Doe",
  "follower_count": 4200,
  "avatar_url": "https://cdn.instagram.com/..."
}

Hashtag index:
{
  "hashtag": "travel",
  "post_count": 840000,
  "trending_score": 9.2
}

Search Flow

flowchart LR
    CLIENT["Client"]
    SS["Search Service"]
    ES["Elasticsearch"]
    RC["Redis Profile Cache"]

    CLIENT --> SS
    SS --> ES
    ES --> SS
    SS --> RC
    SS --> CLIENT

When a new user registers or changes their username → publish event to Kafka → Search indexer consumes and updates Elasticsearch.

---

Step 9 — Notifications

Notification Types

Event	Who Gets Notified
New like	Post owner
New comment	Post owner + tagged users
New follower	The user being followed
Comment reply	Original commenter
Post mention (@)	Mentioned user

Async Notification Pipeline

flowchart LR
    LikeSvc[Like Service] -->|like.created event| KF[Kafka]
    FollowSvc[Follow Service] -->|follow.created event| KF
    CommentSvc[Comment Service] -->|comment.created event| KF
    KF --> NW[Notification Worker]
    NW -->|Check user preferences| UP[User Prefs\nRedis]
    NW -->|Push if online| WS[WebSocket\nor SSE]
    NW -->|Push if offline| FCM[FCM / APNs]
    NW -->|Store| NDB[(Notifications DB\nCassandra)]

• Notifications are never written synchronously during the like/follow/comment action
• A Kafka event is published; the Notification Worker consumes it asynchronously
• If user is online: deliver via WebSocket or Server-Sent Events for instant in-app notification
• If offline: push notification via FCM/APNs

---

Step 10 — CDN and Media Delivery

Why CDN Is Critical

Instagram serves 4.8 billion photo views/day globally. Without CDN:

• Every photo request hits your origin S3 bucket in one region
• A user in Mumbai loading a photo uploaded in New York gets 200ms+ of network latency
• S3 egress costs are enormous at this scale

CDN Strategy

flowchart LR
    Client -->|GET photo| CDN[CDN Edge\nnear user]
    CDN -->|Cache hit| Photo[Serve from edge]
    CDN -->|Cache miss| S3[Origin S3]
    S3 --> CDN
    CDN --> Client

Media Type	CDN TTL	Strategy
Profile photos	7 days	Long TTL — rarely change
Post photos	30 days	Permanent — posts don't change after upload
Post videos	30 days	HLS segments cached at edge
Stories	24 hrs	Short TTL — expire automatically
Thumbnails	30 days	Permanent

Adaptive Bitrate Streaming for Video

Videos are transcoded to HLS (HTTP Live Streaming):

video_post.m3u8           (master playlist)
  ↓ 240p/400kbps
  ↓ 480p/800kbps
  ↓ 720p/2Mbps
  ↓ 1080p/4Mbps

Client automatically selects quality based on available bandwidth — mobile on 3G gets 240p, WiFi gets 1080p.

---

Step 11 — Scaling

Read Scaling (Feed)

flowchart TD
    LB[Load Balancer] --> F1[Feed Service Pod]
    LB --> F2[Feed Service Pod]
    LB --> F3[Feed Service Pod]
    F1 & F2 & F3 --> RC[(Redis Cluster\nFeed Caches)]
    RC -->|miss| CS[(Cassandra\nRead Replicas)]

• Feed Service is stateless — scale horizontally to any number of pods
• Redis Cluster handles billions of feed keys across shards
• Cassandra replication factor 3 — read from nearest replica

Write Scaling (Post Creation + Fan-out)

flowchart LR
    PS[Post Service\n× 50 pods] --> KF[Kafka\npost.created]
    KF --> FO1[Fan-out Worker]
    KF --> FO2[Fan-out Worker]
    KF --> FO3[Fan-out Worker]
    FO1 & FO2 & FO3 --> RC[(Redis\nFeed Caches)]

• Fan-out workers consume from Kafka in parallel
• Each worker handles a subset of partitions
• Add more workers to handle higher fan-out throughput

Database Scaling

Database	Scaling Strategy
PostgreSQL	Primary + read replicas; shard by user_id at large scale
Cassandra	Consistent hashing — add nodes, data rebalances automatically
Elasticsearch	Sharding built-in — index sharded by user_id or hashtag
Redis	Redis Cluster — slot-based sharding across nodes

---

Step 12 — Failure Scenarios

Failure	Impact	Mitigation
Redis (feed cache) down	Feed reads hit Cassandra directly	Redis Cluster with replicas; Cassandra serves as fallback timeline
Kafka down	Fan-out and notifications delayed	Kafka 3-broker cluster; producers buffer locally and retry
Fan-out service down	New posts don't appear in feeds immediately	Kafka retains events (7-day retention); workers catch up on restart
Media upload fails	Post created but no media	Pre-signed URL has 15-min expiry; client retries upload; post stays PROCESSING
Transcoding fails	Video not viewable	Dead letter queue; retry with backoff; fallback to original quality
Cassandra node down	Some shard unavailable	Replication factor 3; coordinator retries on available replicas
CDN cache eviction	Origin hit for popular media	Long TTL for photos/videos; S3 handles burst; CDN auto-repopulates
Search cluster down	Search unavailable	Graceful degradation — profile page still works; show cached results

---

Final Architecture

flowchart TD
    Mobile[Mobile / Web Clients] --> LB[Load Balancer]
    LB --> AG[API Gateway\nJWT + Rate Limiting]

    AG --> US[User Service]
    AG --> PS[Post Service]
    AG --> FS[Feed Service]
    AG --> SS[Search Service]
    AG --> NS[Notification Service]

    PS -->|pre-signed URL| MS[Media Service]
    MS --> S3[S3 / GCS\nObject Storage]
    S3 --> TC[Transcoder\nFFmpeg]
    TC --> S3
    S3 --> CDN[CDN\nCloudFront / Cloudflare]

    PS --> KF[Kafka Cluster]
    KF --> FO[Fan-out Workers × N]
    KF --> NW[Notification Workers]

    FO --> RC[(Redis Cluster\nFeed Cache)]
    FO --> CS[(Cassandra\nTimeline Store)]
    FS --> RC
    RC -->|miss| CS

    US --> UDB[(PostgreSQL\nUsers)]
    US --> GDB[(Cassandra\nFollow Graph)]
    NW --> NOFCM[FCM / APNs]
    NW --> NDB[(Cassandra\nNotifications)]
    SS --> ES[(Elasticsearch\nSearch Index)]

---

Technology Stack

Layer	Technology
API Gateway	Kong / AWS API Gateway / NGINX
Application services	Python / Go / Spring Boot
Message queue	Apache Kafka
Feed + presence cache	Redis Cluster
User data	PostgreSQL (primary + read replicas)
Posts, graph, likes	Apache Cassandra
Search	Elasticsearch
Object storage	Amazon S3 / Google Cloud Storage
Video transcoding	FFmpeg workers / AWS Elemental MediaConvert
CDN	AWS CloudFront / Cloudflare
Push notifications	FCM (Android) + APNs (iOS)
Monitoring	Prometheus + Grafana + ELK + Jaeger
Deployment	Kubernetes (multi-region)

---

Key Trade-Offs

Decision	Option A	Option B	Choice & Reason
Feed generation	Fan-out on write (push)	Fan-out on read (pull)	Hybrid — push for regular users, pull for celebrities; eliminates write amplification
Media routing	Through app servers	Pre-signed S3 + CDN	Pre-signed S3 — never route large files through backend
Like count storage	Exact DB count	Redis counter + async flush	Redis counter — exact counts not critical; latency is
Consistency (likes/follows)	Strong	Eventual	Eventual — a 1-second delay in follower count is acceptable
Video delivery	Direct MP4 download	HLS adaptive streaming	HLS — adapts to network conditions; CDN-friendly segments
Search backend	PostgreSQL LIKE queries	Elasticsearch	Elasticsearch — full-text, fuzzy, autocomplete not possible in SQL at this scale
Notification delivery	Synchronous	Async Kafka pipeline	Async — never block the like/comment action for notification delivery

---

Common Interview Mistakes

• ❌ Not addressing the celebrity / high-follower fan-out problem
• ❌ Using a single fan-out strategy (pure push or pure pull) without considering skew
• ❌ Routing media uploads through your application servers
• ❌ Storing like counts with DB row-level locks — will cause hotspot contention
• ❌ Not mentioning CDN — serving billions of photos from a single S3 region is not feasible
• ❌ No feed cache — assembling a feed from DB on every request cannot meet 200ms SLA
• ❌ Using SQL for the follow graph — joins at billion-scale are too slow
• ❌ Synchronous notifications on the write path — adds latency to user actions
• ❌ No video transcoding design — raw uploaded files cannot be streamed directly

---

Interview Questions

1. How does Instagram generate a personalized feed for 500M daily users?
2. What is the celebrity problem in feed fan-out, and how do you solve it?
3. Why are pre-signed S3 upload URLs better than routing through your backend?
4. How do you store and query the follow graph at scale?
5. Why is Cassandra chosen for posts and likes instead of PostgreSQL?
6. How do you keep like counts fast without overloading the database?
7. How does the CDN serve billions of photos with low latency globally?
8. How does adaptive bitrate (HLS) video streaming work?
9. How do you handle notifications without slowing down user actions?
10. How would you design the hashtag search feature?
11. How do you scale the fan-out service to handle 1,160 uploads/second?
12. What happens when Redis (feed cache) goes down?
13. How do you prevent duplicate likes from the same user?
14. How does Elasticsearch support autocomplete in the search bar?
15. How would you handle a viral post getting 10 million likes in 10 minutes?

---

Summary

Concern	Solution
Feed generation	Hybrid fan-out — push for regular users, pull for celebrities
Feed reads	Redis Sorted Set per user — O(1) feed fetch
Photo/video upload	Pre-signed S3 URLs → direct client upload
Video streaming	FFmpeg → HLS → CDN adaptive bitrate
Media delivery	CDN edge nodes — serve from nearest PoP globally
Like counts	Redis counters — async flush to Cassandra
Follow graph	Cassandra — two denormalized tables (followers + following)
Post storage	Cassandra — partitioned by user_id, clustered by post_id (time)
Search	Elasticsearch — inverted index for users and hashtags
Notifications	Async Kafka pipeline → FCM/APNs
Scale	Stateless services + Kafka + Redis + Cassandra horizontal scaling
High availability	Multi-region, Kafka replication, Cassandra RF=3, Redis Cluster

The core principle: Instagram is a read-heavy media platform. Optimize the feed read path with aggressive caching. Never route media through your servers. Handle fan-out asymmetry with the hybrid push-pull model.

---