Topic overview
Study offline evaluation (golden datasets, regression test suites), LLM-as-judge scoring, online evaluation (A/B tests, user feedback signals), and observability for multi-step LLM/agent systems — tracing a request across every model call, tool call, and retrieval step it touches.
Core concepts
Golden/labeled datasets as a fixed yardstick across model or prompt changes, LLM-as-judge for scoring outputs that are too open-ended for exact-match metrics, human-in-the-loop review for edge cases, spans/traces that link every step of a multi-hop chain back to one request, and alerting on regressions rather than only on hard failures.
Production monitoring
What to watch: latency, error and timeout rates, token usage and cost per request, tool-call failures, and quality signals such as user feedback, thumbs-down rates, and sampled evals on live traffic.
Drift: inputs and expectations change over time. Data drift means the incoming data looks different from what you tested on; concept drift means the right answer itself has changed. Both quietly lower quality, so compare live metrics against your baseline and alert on shifts.
Retrieval and prompt changes: for RAG systems, track retrieval hit rate and empty-result rate. Re-run your regression set whenever a prompt, model version, or index changes.
Feedback loop: log inputs and outputs (with privacy controls), review failures, add them to the eval set, and fix the cause. Monitoring is only useful if it feeds back into testing.
Why it matters
An agent or RAG pipeline can fail in a step nobody's watching — a bad retrieval, a tool call that silently returns the wrong result, a prompt change that quietly regresses one use case while improving another. Without evaluation and tracing, "it seems fine" is the only signal you have, and it's the wrong one.
Interview relevance
Expect questions on how you'd build a regression test suite for a prompt or model change, when LLM-as-judge is trustworthy versus when you need human review, and how you'd trace a bad output in a multi-step agent call back to the step that caused it.