Evaluation

Evaluation is about measuring a model or a pipeline honestly: the statistics, the datasets and the traps that make a number look better than it is. Both articles here ship a standard-library script and their per-item data, so every figure can be recomputed.

Bootstrap CIs for eval deltas starts from a question every team meets, whether a gap of about four points between two systems on 500 items is real, and answers it with a paired bootstrap, McNemar's test and a coverage check over 1,000 replays in which the true gap is known. Is your test set in the training data? puts the contamination rules published with GPT-3, PaLM, Llama 2 and GPT-4 into one script and measures them on GSM8K: a 13-word overlap caught 69.1 to 95.6 percent of edited copies of a question but only 0.8 percent of rewordings.

Start with the bootstrap piece; it gives you what you need to read any benchmark gap, including the ones in our paper readings. Two tutorials are evaluation work as well, the golden test set and LLM-as-judge done carefully; they sit under Tutorials because each one builds something. Next in this section: why accuracy is the wrong metric for most LLM tasks, worked through a confusion matrix, and later red-team evaluation sets.