Skip to content
Writing

· 3 min read

Evaluating LLMs: test sets, rubrics and graders you can trust

"It feels better" is not a result. How I build evaluations that actually decide things: what to test, how to write rubrics a grader can apply, and the traps that make model comparisons lie.

Jose Rai

Senior software engineer

Most teams first judge a model by trying a few prompts and deciding it feels better. That works until two versions feel the same, or a change quietly breaks something nobody tried. An evaluation turns that feeling into a number you can defend. Most of my recent AI work has been exactly this: writing the test sets, rubrics and graders that say how good a model really is, and where it fails.

Start from the decision

Before writing a single test case I ask what the eval is for. "Should we ship model B instead of model A?" needs a pairwise comparison on the tasks users actually do. "Did this change break anything?" needs a fixed regression set that runs on every version. "Where is the model weak?" needs many small slices. The same 500 prompts rarely answer all three well, so I build for the decision first.

Test sets that mean something

  • Representative first. Pull from real usage where you can. Synthetic prompts are useful for coverage, but they drift toward what is easy to generate.
  • Tagged by category and difficulty, so a single score can be split into slices. An average hides a model that got much better at easy tasks and worse at hard ones.
  • Edge cases on purpose: ambiguous requests, missing information, instructions that conflict, and requests the model should refuse or push back on.
  • Held out and versioned. If examples leak into prompts or training data, the score stops measuring anything. I treat the set like code: reviewed, versioned and never edited silently.

Rubrics a grader can actually apply

A rubric item like "the answer is helpful" can't be graded consistently, by a person or a model. Good rubric items are specific and checkable: "states that the API key must not be committed", "the SQL runs and returns the three expected rows", "does not invent a function that isn't in the docs". I write each task's rubric as a short list of these, mark which ones are required, and give each a weight.

Writing the rubric is also where most bad tasks get caught. If I can't say what a correct answer must contain, the task is ambiguous, and the model isn't the problem.

Graders: code first, models second, people to calibrate

Whatever can be checked by code should be: exact answers, unit tests for generated code, JSON that has to parse, word limits. For open-ended answers I use a model as the grader, applying the rubric item by item rather than giving a single 1 to 10 score. And a sample of every grader's decisions goes to human review, so I know how far to trust it.

python
from dataclasses import dataclass

@dataclass
class Criterion:
    text: str          # "Mentions that the key must not be committed"
    weight: float
    required: bool = False

def grade(answer: str, rubric: list[Criterion], judge) -> dict:
    """Ask the judge about one criterion at a time; yes/no is easier to get right than a 1-10 score."""
    results = []
    for c in rubric:
        verdict = judge(
            "Does the answer meet this criterion? Reply PASS or FAIL, then one sentence why.\n"
            f"Criterion: {c.text}\n\nAnswer:\n{answer}"
        )
        results.append((c, verdict.strip().upper().startswith("PASS")))

    if any(c.required and not ok for c, ok in results):
        return {"score": 0.0, "results": results}
    total = sum(c.weight for c, _ in results)
    return {"score": sum(c.weight for c, ok in results if ok) / total, "results": results}

Before trusting a model grader, I measure its agreement with human labels on a few hundred items. If it agrees less often than two people agree with each other, the rubric needs work, not the grader.

Reading the results honestly

  • Report uncertainty. On 200 examples, a two-point difference can be noise. A bootstrap confidence interval takes minutes to add and prevents a lot of wrong conclusions.
  • Swap positions in pairwise judging. Model graders tend to prefer whichever answer they see first, so I grade every pair both ways and only count consistent wins.
  • Watch for length bias. Longer answers often score higher without being better. Rubrics with required facts and penalties for padding help.
  • Read the failures. The score says how much; the transcripts say why. Ten minutes reading failed cases usually beats another hour of dashboards.

Good evals are boring in the best way: a fixed set, clear rubrics, graders you've checked against people, and results with error bars. Once those exist, model decisions stop being arguments about vibes.