Why does metadata filtering break naive vector search, and how do pre-filtering, post-filtering, and filter-aware indexes differ?
Quick Answer
ANN indexes are built over the whole vector space, not per filter value. Post-filtering (search then filter) returns too few results when the filter is selective; pre-filtering (filter then exact search) is correct but slow on large candidate sets; filter-aware traversal (Qdrant-style, filterable HNSW) applies conditions during graph walk — the reason filtered-search performance varies so wildly across engines.
Detailed Answer
The failure mode: you ask for top-10 with tenant_id=X. Post-filtering fetches top-k globally then drops non-matching — if tenant X owns 1% of vectors, top-100 global may contain zero of theirs, returning empty results despite relevant data existing ('k starvation'; oversampling k×100 is the crude fix and it wrecks latency). Pre-filtering selects matching IDs first then searches only those — exact and correct, but at large match counts it degenerates to brute force. Filter-aware indexes integrate predicates into traversal: the graph walk skips non-matching nodes while maintaining connectivity guarantees (Qdrant builds additional links for this; pgvector 0.8+ improved iterative scanning for filtered HNSW; engines differ enormously here — it was a documented reason teams migrated off Pinecone for filter-heavy workloads). Design responses worth naming: partition/collection-per-tenant when isolation is the dominant filter (turns filtering into routing — also better for compliance); composite strategies (coarse partition + fine filter); and testing recall under your real filters, because unfiltered benchmark recall says nothing about filtered behavior. Multi-tenancy adds the security angle: tenant filters must be enforced server-side (or by physical isolation), never assembled client-side.
Code Example
# k-starvation reproduction: selective filter + post-filtering
hits = db.search(qvec, top_k=10) # global top-10
mine = [h for h in hits if h.meta['tenant'] == 'X'] # often []
# qdrant: filter participates in traversal
client.search(collection='chunks', query_vector=qvec, limit=10,
query_filter=Filter(must=[FieldCondition(key='tenant', match=MatchValue(value='X'))]))Interview Tip
Demonstrate k-starvation with the 1%-tenant example — it makes the abstract pre/post distinction concrete and shows you've hit it. Then mention partition-per-tenant as the isolation-first alternative.