LangChain

The value is the interfaces: a ChatModel abstraction over providers (swap OpenAI for Anthropic or a self-hosted vLLM endpoint without rewriting call sites), Runnables that compose with uniform invoke/stream/batch semantics, structured-output helpers, retriever/vector-store integrations, and callback hooks for tracing. That buys speed and portability. The costs are real too: an abstraction layer to debug through when something misbehaves, historical API churn (the 0.x line broke things repeatedly until v1.0 in late 2025 committed to stability), and the temptation to use framework machinery where 30 lines of direct SDK calls would be clearer. Wrong-tool cases: one-shot completions (call the provider SDK), maximal-control serving paths (token-level budget/retry logic), and teams that already built their own thin layer. Right-tool cases: RAG pipelines, multi-provider products, agents with tools, and anywhere you want LangSmith-style tracing without building it.

Everything in modern LangChain implements Runnable: prompts, models, parsers, retrievers, and arbitrary functions (RunnableLambda). Composition (prompt | model | parser) returns another Runnable whose sync/async/batch/streaming implementations are derived automatically — streaming propagates through the chain so tokens flow to the client even with a parser at the end, and batch gets provider-level concurrency. You also get declarative structure: RunnableParallel fans out, RunnableBranch routes, .with_retry() and .with_fallbacks() wrap any step, and with_structured_output binds schemas. Because chains are data structures rather than opaque code, callbacks/LangSmith can trace every step with inputs/outputs — the observability story is the strongest practical argument. The honest downside: stack traces route through the framework's invoke machinery, and inserting a debugger or log line between piped steps is less natural than in plain Python — the standard workaround is dropping in a RunnableLambda tap or relying on tracing.

ConversationBufferMemory and friends were fine for demos but structurally wrong for production: state lived in the Python process, so a pod restart wiped every conversation, horizontal scaling broke session affinity, and there was no per-user isolation story. The replacement pattern treats chat history as externally persisted state: LangGraph compiles your agent/chain into a graph with a checkpointer (e.g., PostgresSaver); each turn loads the thread's state, runs the graph, and persists the new state atomically. This gives restart-safety, multi-replica correctness, time-travel debugging over past states, and a clean place for retention policies. On top of raw history, long-term memory (facts that persist across threads) is handled separately — e.g., a store the agent reads/writes deliberately — which also forces you to confront fact-conflict problems (a user changing a stated preference must overwrite, not coexist with, the old fact). For interviews: in-process memory = demo; checkpointer-backed thread state = production.

The modern path binds a Pydantic model (or JSON schema) via with_structured_output: for providers with native structured output it uses function/tool-calling or JSON mode, so the model is constrained at the API level rather than asked nicely in a prompt, and the response parses into a typed object. This eliminates the classic 'Here's your JSON:' prefix breakage that plagued prompt-and-parse approaches. What can still go wrong: (1) semantically wrong but schema-valid data — validation must include business rules (Pydantic validators), not just types; (2) refusals or safety responses instead of the schema; (3) truncation when max_tokens clips a long array mid-object; (4) enum drift — the model inventing category values, mitigated by Literal types; (5) on self-hosted models without native JSON mode, you're back to parser + retry (OutputFixingParser-style loops or grammar-constrained decoding in vLLM). Production pattern: schema binding + strict Pydantic validators + bounded retry with the validation error fed back + metric/dead-letter on final failure rather than a 500.

RAG bugs are pipeline bugs, so debug it as a pipeline: (1) Capture the full trace of the bad request — every step's inputs/outputs. Without tracing this incident is guesswork; with it, it's a diff. (2) Compare retrieval first: are the top-k chunks the same as staging? If prod retrieves garbage, check that the prod index was actually populated by the same ingestion job version, and — the classic — that the embedding model used at query time matches the one used at index time; a mismatch produces plausible-looking but semantically wrong neighbors. (3) Compare the rendered prompt: template version, context truncation (prod docs longer than staging's), and chat-template/formatting differences. (4) Only then the model: same model ID, temperature, max_tokens? (5) Check environment-shaped causes: prod-only metadata filters (multi-tenancy filters silently excluding all docs), stale caches, or a fallback chain having kicked in (a provider error silently routed to a weaker model — check fallback metrics). The meta-answer interviewers want: instrument chains with per-step tracing before you need it.

Observability: LangChain's callback system emits events for every chain/LLM/tool step — feed them to LangSmith or an OpenTelemetry exporter so each request yields a trace with per-step latency, inputs/outputs, token counts, and errors. Tag traces with tenant/feature/version metadata so you can slice regressions. Metrics to dashboard: tokens in/out per request (p50/p99), cost per request (tokens x model price), time-to-first-token, retrieval latency vs generation latency, fallback activation rate, and structured-output validation failure rate — the last two are silent-degradation signals. Cost control: set max_tokens deliberately per route; cap conversation history length (summarize or window old turns); cache where legal (embedding calls, idempotent extractions); route by difficulty (cheap model default, expensive model on low-confidence or explicit escalation); enforce per-tenant daily budgets in middleware. And evaluation belongs in ops too: a golden-set eval run on every prompt/model change, because 'the model got worse' incidents need a regression artifact, not vibes.

The 0.0.x -> 0.3 era earned LangChain a reputation for churn: three memory API generations in ~18 months, AgentExecutor patterns superseded, and import paths reshuffled into langchain-core/langchain-community/provider packages. v1.0 drew a line: the core abstraction is Runnables + LangGraph for anything stateful/agentic, legacy chains live on in a compatibility package, and breaking changes now follow semver. A sane migration: (1) inventory — grep for deprecated imports (langchain.memory, langchain.agents.AgentExecutor, old chain classes) and rank call sites by traffic; (2) mechanical moves first — import-path updates, provider packages (langchain-openai etc.); (3) structural moves second — memory classes to LangGraph checkpointers, custom agent loops to create_react_agent or explicit graphs; (4) pin langchain-core/langchain/langgraph/provider packages as one tested set; (5) golden-trace regression — replay recorded production requests through the new stack and diff outputs/costs/latency before cutover. Budget the structural part like a real project; the import churn is the cheap half.