Qdrant vector database: mutable segments, filtered ANN, and distributed shards

Qdrant stores points with vectors and payload in segments. Mutations first enter a write-ahead log and then segments; optimizers merge and rebuild segments and indexes in the background. HNSW is not an immutable structure that can never accept updates—its lifecycle is managed across mutable and optimized segments.

Approximate nearest-neighbor quality is empirical. HNSW parameters, quantization, filters, payload indexes, data distribution, and hardware change recall and latency. A production acceptance test needs a ground-truth set and measured recall/latency, not a guaranteed vendor number.

Модель и предпосылки

• Every node can accept a request and route it to the correct shard; shard placement and replication are distinct from collection metadata consensus.
• Vector indexes accelerate candidate search; payload indexes accelerate filtering and inform query planning. Filter-aware HNSW benefits from payload indexes created before ingestion or an explicit rebuild.
• Many small tenants usually share a collection using an indexed tenant payload; large tenants can use dedicated or tiered shards. Collection-per-tenant is reserved for a limited number needing strict isolation.

Проверяемые утверждения

• C1. Qdrant writes changes to a WAL before applying versioned updates to segments, enabling recovery after abnormal shutdown.
• C2. Indexes exist per segment and optimizers determine when indexing and segment rebuilds occur.
• C3. Payload indexes and vector indexes solve different parts of filtered ANN; filter-aware HNSW may need index rebuild after late payload-index creation.
• C4. Distributed Qdrant routes requests to shards; HNSW is rebuilt on a shard transfer rather than transferred as-is.
• C5. A shared collection with indexed tenant payload is preferred for many small tenants; dedicated/tiered shards trade overhead for isolation.
• C6. HNSW is an approximate graph index whose quality and speed depend on construction/search parameters and data; it does not provide a universal recall or latency guarantee.

Исполняемые сценарии

Upsert and segment lifecycle. A point is WAL-ordered, applied to a mutable segment, and later optimized or reindexed in the background.

Filtered ANN query. Payload selectivity and vector search cooperate; the planner may choose a full scan for a small candidate set.

Data replication versus metadata consensus. Shard data transfer and cluster topology coordination are related but distinct operations.

Tenant placement. Small tenants share indexed payload partitions; large tenants may be promoted to dedicated shards.

Recall and latency evaluation. Ground truth, filters, HNSW parameters, and quantization are tested together before rollout.

Ошибки проектирования

• Не называйте ANN точным поиском: recall нужно считать на актуальном corpus и фильтрах.
• Не создавайте payload index после массовой загрузки и не ожидайте автоматически filter-aware HNSW без rebuild.
• Не смешивайте consensus metadata и replication user data в одну гарантию.
• Не забывайте wait/ordering/consistency parameters: acknowledged API operation и видимость на каждой replica — разные договоры.

Границы гарантии

• Qdrant не создаёт embedding model correctness; смена модели требует versioning и controlled re-embedding.
• Quantization меняет memory/latency/recall trade-off и проверяется на вашем наборе.
• Tenant payload filtering — часть isolation design; authorization остаётся обязанностью API boundary.

Связанные материалы

Официальные и первичные источники

1. Qdrant: Storage and WAL versioning
2. Qdrant: Indexing and filterable HNSW
3. Qdrant: Distributed deployment
4. Qdrant: Multitenancy
5. Malkov and Yashunin, HNSW