Ranked News Feed: candidates, correctness and failure modes

Scope and workload model

The system accepts posts and returns a personalized cursor-paginated feed. It supports ordinary and high-fanout authors, ranking, deletion, audience changes, block and mute. Notifications, comments, media transcoding and ads are separate systems.

Capacity values are exercise inputs, not claims about Facebook, LinkedIn or X. Example: 100M daily active users × 20 feed opens/day = 2B feed reads/day, about 23.1K average reads/s; a 10× peak is about 231K/s. Ten million posts/day is about 116 average writes/s. The fanout workload is sum(active eligible followers) for push-tier authors; an average follower count hides the power-law tail.

Reliable write path

Post Service commits canonical post and outbox event in one database transaction. CDC Relay publishes only committed events. This repairs the DB→Kafka dual-write hole: a crash after the database commit cannot permanently suppress fanout, and an aborted post cannot create a valid event.

Delivery remains at least once. Every event has event_id, aggregate_id and version. Projector upserts Post KV by post_id and Recent Posts by (author_id, time bucket, created_at, post_id). Fanout upserts Durable Inbox by (viewer_id, post_id). Replay changes no logical result.

Подробный паттерн: Transactional Outbox. Это explore-страница и потому обычная Markdown-ссылка.

Hybrid candidate generation

Ordinary authors use fanout-on-write into viewer-partitioned durable inboxes. High-fanout authors are projected into an author/time index but skipped by mass push. On a feed read, Feed Service merges bounded pushed IDs with bounded recent IDs from followed high-fanout authors.

The threshold is measured from fanout backlog, active follower ratio, write amplification and acceptable freshness. It is not universally 10K followers. Tier changes use hysteresis and versioning to avoid duplicate/omitted candidates during transitions.

Redis holds only a capped hot copy of inbox IDs. The durable inbox supports cache rebuild. Capping to 500 candidates is a product choice; memory planning must include actual Redis encoding and allocator overhead, not just 500 × ID bytes.

Query-shaped storage

The previous design described storage partitioned by author and then attempted one batch GET of arbitrary post_ids. Those are different access patterns. The corrected design uses:

• Post KV by post_id for hydration.
• Recent Posts by author and time bucket for high-fanout pull.
• Durable Inbox by viewer_id and cursor for pushed candidates.

Cassandra documentation says performant queries supply partition keys and recommends query-driven tables. Fanout denormalization is deliberate; no table pretends to support an incompatible query.

The read pipeline is candidate generation → dedupe → batch hydration → current privacy/tombstone filter → ranking → cursor. A cursor includes stable sort/rank key, snapshot/version policy and post_id tie-breaker. OFFSET is avoided because new posts and re-ranking would create duplicates or skips.

Privacy is a serving invariant

Fanout-time filtering is only an optimization. A block, audience change or delete can happen after a candidate entered Redis. Every read rechecks the current post version and privacy policy before ranking and response. If the policy result is unknown, that item fails closed; a broad fail-open could expose private content.

Meta's engineering description of its video/feed delivery explicitly places privacy checks after entity loading and before response materialization. This source supports the ordering invariant, not the specific components chosen in this exercise.

Delete writes a new post version/tombstone and event. Projectors remove author-index entries and fanout workers best-effort clean inboxes, but stale IDs are harmless because Post KV tombstone and read-time privacy gate filter them.

Ranking and degradation

Ranker receives only a bounded visible set. It may use recency, affinity, predicted engagement, negative feedback and diversity constraints. A model score is not a durable ordering guarantee; cursor semantics must define how a session behaves when scores change.

On ranker timeout the service uses deterministic recency plus post_id tie-breaker and marks degraded=true. On Post KV failure it cannot safely hydrate and returns a bounded error/cached safe page. On Privacy failure uncertain items are omitted. Each dependency has a distinct policy; one global fail-open switch is unsafe.

Backpressure and replay

• Kafka partitions by author_id preserve per-author event order, not global order.
• Consumers checkpoint only after durable effects. Poison events go to a DLQ with alert, schema/version metadata and a replay path.
• Fanout uses bounded follower pages, leases and retry with jitter. Queue lag can trigger temporary pull-tier promotion.
• A duplicated event is expected. Natural keys and monotonic versions make every projection idempotent.
• A late older update cannot overwrite a newer privacy or delete version.

Связанные темы

Смотрите [CONCEPT]queues, [CONCEPT]change-data-capture, [CONCEPT]idempotency, [CONCEPT]caching-patterns, [CASE]social-graph, [CASE]recommendation-system и [CASE]twitter-system-design. Практический DLQ: Dead-Letter Queue.

Первичные источники

• Debezium official outbox event router: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html
• Apache Cassandra partition and query-driven modeling: https://cassandra.apache.org/doc/latest/cassandra/developing/cql/ddl.html
• Meta Engineering, TAO and read-dominated social graph workload: https://engineering.fb.com/2013/06/25/core-infra/tao-the-power-of-the-graph/
• Meta Engineering, News Feed ranking architecture: https://engineering.fb.com/2021/01/26/core-infra/news-feed-ranking/
• Meta Engineering, entity loading followed by privacy checks in feed serving: https://engineering.fb.com/2024/12/10/video-engineering/inside-facebooks-video-delivery-system/
• Redis sorted sets command semantics: https://redis.io/docs/latest/commands/zadd/

Источники подтверждают свойства технологий и опубликованные invariants. Масштабные числа, threshold и SLO этой страницы являются расчетными assumptions, а не описанием текущей private production architecture какой-либо компании.