Mock scenario: users say your RAG assistant 'can't find' documents they know exist. Search latency is fine. Debug the retrieval quality.
Quick Answer
Work the ingestion-to-query pipeline: is the document actually indexed (ingestion gaps), does its chunking preserve the answer intact, does the embedding model match between index and query time, is ef_search/recall tuned too low, and are filters (tenant/ACL/recency) silently excluding it? Reproduce with the specific failing pair: fetch the doc's chunks directly, then test where similarity breaks.
Detailed Answer
This is a recall investigation, so make it concrete with one failing (question, known-answer-doc) pair: (1) Existence — query the DB by document ID: was it ever ingested? Ingestion pipelines fail partially all the time (parser errors on PDFs, size limits, sync lag from the source system). (2) Chunking — pull the doc's chunks and read them: does any chunk actually contain the answer coherently, or did a bad splitter shear it across boundaries / bury it in boilerplate? (3) Embedding-space sanity — embed the question and the best chunk directly and compute similarity; if it's low, the issue is semantic (chunk too long/noisy, or query vocabulary far from document vocabulary — consider query rewriting or hybrid search) rather than the index. (4) Index recall — compare ANN results against exact brute-force search on the same query; a gap means index tuning (raise ef_search, check index build params, look for degradation from heavy deletes). (5) Filters — rerun without metadata filters; ACL/tenant/date filters silently excluding the doc is extremely common. (6) Systemic checks — embedding model/version consistency between ingest and query paths, and distance-metric mismatch (cosine model queried with L2). Then institutionalize: a golden retrieval set with recall@k tracked over time, so this class of bug pages you instead of surprising users.
Code Example
# 1. is it there?
chunks = db.get(filter={'doc_id': doc})
# 3. direct similarity of question vs best chunk
print(cos(embed(q), embed(best_chunk_text)))
# 4. ANN vs exact
ann = db.search(embed(q), k=20)
exact = brute_force(embed(q), k=20) # small sample offline
print(recall(ann, exact))
# 5. filters off
db.search(embed(q), k=20, filter=None)Interview Tip
The ANN-vs-brute-force comparison is the professional's move — it cleanly splits 'index problem' from 'embedding/chunking problem' and most candidates never mention it.