LLM Evaluation
Evaluating LLM systems is fundamentally different from traditional ML. This chapter covers metrics, methodologies, and practical approaches for measuring quality in production. It is about evaluating your system; for how to read public model benchmarks like MMLU, SWE-bench, and Arena Elo, see Benchmarks and Leaderboards.
LLM Observability
Observability for LLM systems requires adapting the three pillars of logs, metrics, and traces for the unique characteristics of AI applications.
Benchmarks and Leaderboards
Public benchmarks are how the field talks about model capability: MMLU, SWE-bench, GPQA, Arena Elo. They are useful for orientation and dangerous for decisions. This chapter teaches you what each major benchmark measures, which ones still separate frontier models and which have saturated, and how to read a benchmark claim critically so you are not fooled by a number.