Evaluation as a first-class artifact
How we version, run and compare evaluations across thousands of model checkpoints.
Evaluations are often treated as scripts that live next to the real work. We treat them as artifacts in their own right: versioned, reviewed, and reproducible.
This post describes the infrastructure that lets us compare thousands of checkpoints on hundreds of evaluations every day.