What is hybrid search, and why do production RAG systems rarely rely on vector similarity alone?
Quick Answer
Hybrid search runs lexical retrieval (BM25) and vector retrieval in parallel and fuses results (typically reciprocal rank fusion), often followed by a cross-encoder reranker. Pure vector search misses exact identifiers, product codes, names, and rare terms — exactly what users search for — while BM25 misses paraphrases; each covers the other's blind spots.
Detailed Answer
Embeddings compress meaning, and that compression loses lexical precision: 'error QX-4471' and 'error QX-4472' embed nearly identically, version numbers and SKUs are noise to most embedding models, and out-of-vocabulary jargon lands in unhelpful regions of the space. BM25 nails those exact matches but fails on 'how do I make the pod stop restarting' vs a doc that says 'CrashLoopBackOff remediation'. Production pattern: run both retrievers, fuse with reciprocal rank fusion (score = sum of 1/(k + rank) across lists — rank-based, so no fragile score normalization across incomparable scoring scales), take a generous candidate pool (50-100), then apply a cross-encoder reranker that scores each (query, chunk) pair jointly for a final ordering — rerankers routinely add more quality than any index tuning. Many engines ship this natively (Weaviate/OpenSearch hybrid queries, Qdrant with sparse vectors, pgvector combined with Postgres full-text search in one SQL query). Cost note: rerankers add latency/compute per query, so budget them (rerank 50, not 1000). Evaluation note: hybrid's win shows up on identifier-heavy and long-tail queries — build your eval set from real production queries or you'll under-measure the benefit.
Code Example
-- Postgres: BM25-ish (ts_rank) + vector in one shot, RRF fusion
WITH lex AS (SELECT id, row_number() OVER (ORDER BY ts_rank(tsv, q) DESC) r
FROM chunks, websearch_to_tsquery(:query) q WHERE tsv @@ q LIMIT 50),
vec AS (SELECT id, row_number() OVER (ORDER BY embedding <=> :qvec) r
FROM chunks LIMIT 50)
SELECT id, sum(1.0/(60+r)) score FROM (SELECT * FROM lex UNION ALL SELECT * FROM vec) u
GROUP BY id ORDER BY score DESC LIMIT 10;Interview Tip
Give the concrete failure ('error code QX-4471 vs QX-4472 embed identically') and name RRF's virtue — rank fusion needs no score normalization. Mentioning the reranker as the biggest quality lever is a bonus point.