A support team wants to compare two generated answers about a delayed export. One answer confidently tells the reader to start the job again. The other asks the reader to check whether a worker is still running. The first sounds decisive. If the runbook says that retrying a live job can produce duplicate exports, the second is safer and more useful. A language model used as a judge can miss that distinction unless the evaluation gives it the same evidence and rules that a human reviewer would use.

Write a test case someone else can review

Record four things for each example. Save the original question, the exact reference passage available at the time, the candidate answer and a human label for the behaviour you care about. Give the case a stable ID and keep the reference version with it. If the runbook changes next week, you can still explain why a verdict from this week was correct.

For the export example, use a short fictional runbook. When a job is running, wait for the next status update. When it has failed and no output file exists, allow one retry. When the state is unknown, ask for the job ID so the state can be checked. A reviewer can use those three rules without knowing which model produced either answer.

Include a straightforward case and a few close calls. An answer can reach the right conclusion while citing an invented condition. It can answer safely but omit the one step the operator needs next. A short reply can cover the question completely. Add a case with missing information and decide what a justified request for clarification looks like. Each of these examples gives you a distinct failure to inspect after a run.

Give programmatic checks the things they can prove

Validate the answer's structure before asking a model for an opinion. If your application requires a fixed set of fields, validate the schema in code. For a closed set of states such as running, failed and unknown, compare the returned state against the known label. Those checks are repeatable and can tell you exactly which requirement failed.

Use the judge for the remaining questions. Did the reply answer what the operator asked? Did its recommendation follow from the reference? Ask for a separate decision for each question and a short note that points to the relevant sentence in the answer or runbook. Read a sample of those notes. An apparently correct score with an invented justification is a useful warning about how the judge arrived at it.

Swap the answers and keep disagreements visible

For a paired comparison, show the judge the same two answers twice. Reverse their positions on the second run, then translate both verdicts back to the stable answer IDs. If the judge prefers the same answer in both orders, you have a stable vote. If the preference changes, record it as unresolved and send that case for human review. Keep human-labelled ties separate from these unstable votes so you can tell reasonable ambiguity from an order effect.

Track the number of reviewed cases, agreement with human labels and order reversals. Report them for each kind of case as well as for the whole set. A judge that handles ordinary cases well can still fail on the ambiguous states where your application most needs help. Keep a few cases aside while tuning the rubric, so your final check measures examples the prompt was never adjusted to pass.

Keep enough context to explain a changed score

Version the rubric, test set and exact judge model together. Rerun the same cases when you change the judge or the rubric. Look at any cases where the decision changed before you update a release threshold. When a case keeps producing contradictory decisions, let a named reviewer resolve it and record what evidence made the difference. This gives the team a way to improve both the judge and the instructions it uses.

For the wider comparison of pointwise and pairwise judging, see the original guide. How to Use LLM-as-a-Judge (Ailoot)