📘 How do you evaluate an AI product in the wild?

What you’ll learn

1. Offline and Online EvaluationDistinguish offline and online evaluation regimes and apply correct data-splitting and continuous-evaluation discipline.AI product evaluation divides into offline evaluation on fixed datasets and online evaluation on live traffic, which serve complementary roles. Offline scores are fast and reproducible but can mislead due to distribution shift, proxy metrics, and feedback loops, so they screen rather than decide. Strict separation of validation and test sets prevents benchmark overfitting. Evaluation is a continuous lifecycle activity, with production data sampled back into refreshed eval sets, and every metric should mirror the decision it informs.
2. Building Eval Sets and Golden ReferencesConstruct representative, uncontaminated eval sets with reliable golden references and slice-based analysis.An eval set is a curated collection of inputs paired with golden references or judgment criteria, deliberately covering common cases, edge cases, and known failures rather than random samples. Golden-reference quality bounds the whole evaluation, so inter-annotator agreement (e.g., kappa) and clear rubrics are essential. Benchmark contamination, where test data leaks into training, inflates LLM scores and is countered with fresh, private, post-cutoff examples. Slicing the data by segment exposes failures hidden by aggregate scores.
3. Metrics for Classification and RankingSelect and interpret precision/recall/F1 for classification and MRR/nDCG for ranking, accounting for imbalance and position bias.Classification under class imbalance is evaluated with precision, recall, and their harmonic mean F1, plus threshold-free PR and ROC curves, because accuracy is misleading. Precision and recall trade off as the threshold moves, and the right balance depends on error costs. Ranking is judged by order: MRR rewards a high-placed first relevant item, while nDCG (Jarvelin and Kekalainen, 2002) handles graded relevance with a position discount. Offline ranking labels must be reconciled with position-biased online click signals.
4. Reference-Based Metrics and Their LimitsApply BLEU and ROUGE correctly while recognizing that n-gram overlap is a limited proxy for generation quality.BLEU (Papineni et al., 2002) scores n-gram precision with a brevity penalty for translation, while ROUGE (Lin, 2004) is a recall-oriented overlap family for summarization. Both measure surface lexical overlap, not meaning, so correct paraphrases score low and word-matching but poor outputs can score high, yielding weak correlation with human judgment on open-ended tasks. Embedding-based metrics like BERTScore compare meaning but still need references and ignore factuality. These metrics are best used as relative regression signals on constrained tasks.
5. Human Evaluation and LLM-as-a-JudgeRun reliable human evaluation and use LLM-as-a-judge with awareness of its biases and the need for validation.Human evaluation, especially pairwise preference comparison, remains the gold standard for open-ended quality because relative judgments are more consistent than absolute scores. LLM-as-a-judge scales evaluation cheaply and can approach human-human agreement, but it carries position, verbosity, and self-enhancement biases documented by Zheng et al. (2023). Before trusting a judge, teams must measure judge-human agreement on held-out data and apply mitigations such as order-swapping. A judge is a validated approximation of human judgment, not a substitute.
6. Hallucination, Faithfulness, and Online TestingMeasure hallucination and faithfulness as release gates and design valid A/B tests for online product metrics.Faithfulness measures whether an output is supported by its source context, distinct from world-factuality and fluency, and is the dominant trust risk in generative products. Groundedness is assessed claim-by-claim via entailment or LLM judges and often serves as a release gate. A/B testing uses randomization to support causal claims about a change, requiring significance testing and adequate sample size. Pitfalls include peeking, novelty and long-term effects, and the need for guardrail metrics; online and offline evaluation reinforce each other.
7. Production Evaluation Simulation and MasteryIntegrate calibration, fairness, regression testing, cost, and latency into a holistic ship/no-ship evaluation scorecard.Production readiness depends on more than task quality. Calibration aligns confidence with accuracy (via reliability diagrams and ECE), enabling safe deferral, while fairness evaluation uses sliced metrics and documented criteria like equalized odds. Eval suites act as regression tests gating prompt and model changes, with prompts, models, and data versioned together. Latency (reported as p95/p99 tails) and cost per request are first-class product metrics. A scorecard combining these signals supports an explicit, threshold-based ship decision, exercised in this lesson's simulation.

Questions this course answers

Grounded in trusted sources

• Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly.
• NIST (2023). AI Risk Management Framework (AI RMF 1.0).
• Artstein, R. & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics.
• Jarvelin, K. & Kekalainen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM TOIS.
• Papineni, K. et al. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL.
• Lin, C-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. ACL Workshop.

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses