What is a vector database, and what actually happens between a user's question and the returned documents in a RAG system?
Quick Answer
A vector database stores embeddings — dense numeric vectors capturing semantic meaning — and answers nearest-neighbor queries. RAG flow: the question is embedded by the same model that embedded the documents, the DB finds the k closest vectors via an ANN index, applies any metadata filters, and returns the associated chunks for the LLM's context.
Detailed Answer
Documents are chunked and each chunk run through an embedding model producing a vector (typically 384-3072 dimensions); vectors, chunk text, and metadata (source, tenant, timestamp, ACL) are stored together. At query time the question is embedded into the same vector space, and the database performs approximate nearest-neighbor (ANN) search — exact search is O(n) per query, so indexes like HNSW trade a little recall for orders-of-magnitude speed. Similarity is cosine/dot-product/L2 (must match how the embedding model was trained). Results usually flow through metadata filtering (tenant isolation, recency) and often a reranking stage (cross-encoder) before the top chunks enter the LLM prompt. The two invariants that break in production when violated: the same embedding model (and version) must be used at index and query time, and the distance metric must match the index configuration. Also know: 'vector database' spans dedicated engines (Qdrant, Weaviate, Milvus, Pinecone) and extensions to existing stores (pgvector, OpenSearch/Elasticsearch kNN, Redis) — the capability, not the product category, is the point.
Code Example
# same model both sides — non-negotiable
doc_vec = embed_model.encode(chunk) # at ingest
db.upsert(id, doc_vec, metadata={'tenant': t, 'src': url})
q_vec = embed_model.encode(question) # at query
hits = db.search(q_vec, top_k=8, filter={'tenant': t})Interview Tip
Say 'approximate' out loud and why (exact kNN is linear scan) — the recall/latency tradeoff is the core concept everything else in this domain hangs off.