How do you make a LangChain application observable and cost-controlled in production?
Quick Answer
Per-step tracing (LangSmith or OpenTelemetry via callbacks), token/cost metrics per chain and per tenant, latency histograms split by step (retrieval vs LLM), and hard budgets: max_tokens, timeouts, retry caps, and per-user rate limits. Treat token spend like a cloud cost line item with alerts.
Detailed Answer
Observability: LangChain's callback system emits events for every chain/LLM/tool step — feed them to LangSmith or an OpenTelemetry exporter so each request yields a trace with per-step latency, inputs/outputs, token counts, and errors. Tag traces with tenant/feature/version metadata so you can slice regressions. Metrics to dashboard: tokens in/out per request (p50/p99), cost per request (tokens x model price), time-to-first-token, retrieval latency vs generation latency, fallback activation rate, and structured-output validation failure rate — the last two are silent-degradation signals. Cost control: set max_tokens deliberately per route; cap conversation history length (summarize or window old turns); cache where legal (embedding calls, idempotent extractions); route by difficulty (cheap model default, expensive model on low-confidence or explicit escalation); enforce per-tenant daily budgets in middleware. And evaluation belongs in ops too: a golden-set eval run on every prompt/model change, because 'the model got worse' incidents need a regression artifact, not vibes.
Code Example
# Cost/latency guardrails on any runnable
chain = (prompt | model.bind(max_tokens=800)
.with_retry(stop_after_attempt=2)
).with_config({'run_name': 'ticket-summarize',
'tags': ['prod', 'tenant:{tid}'],
'callbacks': [otel_handler]})
# Alert: daily token spend per tenant > budget; fallback_rate > 2%Interview Tip
Name fallback-activation rate and validation-failure rate as alertable metrics — they capture the LLM-specific failure mode of degrading silently while returning 200s.