Mock scenario: your LangChain RAG service gives a correct answer in staging but nonsense for the same query in production. How do you debug it?
Quick Answer
Trace one production request end-to-end (LangSmith or callbacks): compare the retrieved chunks, the final rendered prompt, and model/config versions against staging. The divergence is almost always in retrieval inputs (different index contents, embedding model mismatch) or prompt assembly — not the LLM.
Detailed Answer
RAG bugs are pipeline bugs, so debug it as a pipeline: (1) Capture the full trace of the bad request — every step's inputs/outputs. Without tracing this incident is guesswork; with it, it's a diff. (2) Compare retrieval first: are the top-k chunks the same as staging? If prod retrieves garbage, check that the prod index was actually populated by the same ingestion job version, and — the classic — that the embedding model used at query time matches the one used at index time; a mismatch produces plausible-looking but semantically wrong neighbors. (3) Compare the rendered prompt: template version, context truncation (prod docs longer than staging's), and chat-template/formatting differences. (4) Only then the model: same model ID, temperature, max_tokens? (5) Check environment-shaped causes: prod-only metadata filters (multi-tenancy filters silently excluding all docs), stale caches, or a fallback chain having kicked in (a provider error silently routed to a weaker model — check fallback metrics). The meta-answer interviewers want: instrument chains with per-step tracing before you need it.
Code Example
# LANGSMITH_TRACING=true LANGSMITH_PROJECT=rag-prod
# then diff the two traces step by step
# Quick manual tap without LangSmith:
def tap(name):
return RunnableLambda(lambda x: (logger.info('%s: %r', name, x), x)[1])
chain = retriever | tap('retrieved') | prompt | tap('rendered') | modelInterview Tip
Structure beats speed here: retrieval -> prompt -> model -> environment, and say 'embedding model mismatch between index time and query time' explicitly — it's the single most common real cause.