System Design Cases
Embeddings Basics
Embeddings basics concept page: shows offline indexing pipeline (S3 docs -> chunker -> embedder API -> vector DB) and online query path (user -> API -> Redis cache -> query embedder -> vector DB -> LLM). Three scenarios: text-to-embedding model call, cosine similarity comparison (synonyms close, unrelated far), and batch embedding pipeline for 10K docs. Includes ADRs on embeddings vs BM25 keyword search and on dimension sizing (1536 vs 3072 with Matryoshka truncation).
Embeddings: model-scoped representations, not semantic truth
An embedding is a learned representation produced by one model and input contract. Distance is meaningful only under the model's documented preprocessing, similarity function, version, and evaluated task distribution.
Mental model
- Embeddings support efficient semantic comparison only within the learned representation and task setup. Describe embeddings as learned, model-scoped signals rather than coordinates of objective meaning.
- Cosine similarity is used by Sentence-BERT for evaluated sentence comparison; it is not a calibrated probability. Remove universal score bands and require task-specific labeled calibration.
- Provider embedding models and dimensions are versioned product contracts that can change. Pin model and preprocessing metadata and treat migration as re-indexing.
- Vector and embedding systems introduce access-control, poisoning, and sensitive-information risks. Put policy before encoding and apply authorization again at retrieval.
Guarantees and boundaries
- A stored vector is tied to one explicit schema and source version.
- Normalized vectors make cosine ranking equivalent to dot-product ranking only when both sides are normalized.
- Approximate search quality must be measured against labeled or exact-search ground truth.
Diagram scenarios
Versioned document encoding. The stored vector is meaningful only with its pinned preprocessing, model, dimension, and metric.
Similarity is an evaluated ranking signal. Cosine, dot product, and Euclidean distance are mathematical choices, not universal relevance thresholds.
Shadow re-embedding migration. A model change creates a new vector schema and index; it does not mutate old vectors in place.
Privacy, truncation, and drift failure path. Encoding can leak sensitive content, silently truncate evidence, or drift after a model change.
Architecture decision
Version the model, preprocessing, vector dimension, normalization, and distance metric as one schema. Build task-specific retrieval evaluations before choosing thresholds. Redact or reject sensitive inputs before an external embedding call, and re-embed into a new index before changing the schema.
Failure modes
- Silent truncation removes evidence while producing a syntactically valid vector.
- A mixed-version index can return meaningless rankings without throwing an error.
- Derived vectors can still expose sensitive attributes and must follow retention and access policy.
Operational checklist
- Pin model, tokenizer, input role, dimension, normalization, and metric.
- Keep source IDs and content versions for deletion and re-embedding.
- Evaluate thresholds and migrations on representative slices.
- Never log raw sensitive inputs merely to debug vectors.