LLM as a Judge
By Edward Grundy, Bay Information Systems. Published , updated .
Most outputs from a language model have no single correct answer. A summary can be accurate in many wordings, and a support reply can be fluent and still miss the customer’s question. Exact-match scoring fails on outputs like these, and overlap metrics such as BLEU and ROUGE count words shared with a reference answer, which says little about whether a summary is accurate.
An LLM judge is the usual answer: a second model reads the output and grades it. We cover what a judge is, why teams use one, how to build one, and how to tell whether its verdicts can be trusted. A judge is a measuring instrument, and its readings mean something only once it has been checked against a known standard. (The wider question of how to evaluate a pipeline, with datasets and metrics by task type, is in How to Evaluate an AI Pipeline.)
What is an LLM judge?
An LLM judge is a language model prompted to evaluate another model’s output. It receives the original input, the output, a written criterion, and sometimes a reference answer or the context the output was meant to draw on. It returns a verdict, usually with a short explanation.
Judges come in three forms.
Pointwise. The judge grades one output against the criterion, as pass or fail or on a scale. This suits release testing, where every output in a test set needs a verdict.
Pairwise. The judge sees two outputs for the same input and picks the better one, or calls a tie. This suits comparing two prompts or two models. Many pairwise verdicts can be combined into a ranking (Chatbot Arena does this with a Bradley-Terry model, the statistics behind Elo ratings).
Reference-based. The judge compares the output with a known correct answer and checks whether they agree on the facts. This is the closest of the three to exact match, with the judge tolerating differences in wording.
The term comes from Zheng et al. (2023), who tested GPT-4 as a judge on the MT-Bench and Chatbot Arena datasets. Its verdicts agreed with human preferences more than 80% of the time, about the rate at which human raters agreed with each other. The same paper catalogued the judge’s systematic errors, which we return to below.
Why use one?
Human review does not scale to the rate at which a system changes. A test set of 500 outputs, reviewed by a domain expert at a minute each, takes over a day per release; a judge grades the same set in minutes and runs on every prompt change. The cost is inference (500 outputs scored on four criteria is 2,000 judge calls per release, a line item covered in the cost of machine learning).
A judge applies the standard written into its prompt. That standard comes from people who know the task, and the judge’s job is to apply it to more outputs than they could read (Task Specification is the Primary Artefact makes the longer case for labels as the definition of correct).
Cheaper checks come first wherever they work. A unit test, a JSON schema, a regular expression or an exact match on an extracted field is faster, free and always right about what it checks. A judge is for the criteria code cannot check: whether a claim is supported, whether an answer addresses the question, whether a reply would satisfy the customer.
How do we build one?
We can build a judge in five steps, each of which narrows what the judge is asked to do.
Start from failures. Read fifty to a hundred real outputs and note what goes wrong. Each recurring failure (an invented figure, a missed question, a wrong tone) becomes a criterion. A judge written from a generic list of qualities such as “helpfulness” measures something nobody has defined.
One criterion per judge, with a binary verdict. A narrow pass or fail question (“does every figure in the answer appear in the source document?”) is easier for a model to apply and for a person to check. Asked for a score from 1 to 10, judges tend to cluster in a narrow band, and the difference between a 6 and a 7 has no stable meaning. RAGAS follows the same principle, computing each metric with its own judge call (RAG evaluation describes how).
Reasoning first, then the verdict, in a fixed format. A model generates text left to right, so reasoning written after a verdict can only justify it. Ask for the analysis first and the verdict last, in a structure that parses (formatting for LLMs covers output formats).
Give it what it needs. The judge sees the input, the output, the source context for a retrieval system, a reference answer where one exists, and a few labelled examples of passes and failures. A judge asked whether a claim is supported, without the source, can only guess.
Fix the instrument. Set the temperature to zero or close to it, and pin the judge model’s version. A new judge model is a new instrument and needs checking again.
A judge prompt built this way looks like this:
You are grading an answer from a customer support assistant.
Criterion: every factual claim in the ANSWER is supported by the SOURCE.
A claim with no support in the SOURCE is a failure, even if it is true.
SOURCE:
{source}
QUESTION:
{question}
ANSWER:
{answer}
Labelled examples:
{examples}
First list each factual claim in the ANSWER and whether the SOURCE
supports it. Then give the verdict.
Respond in JSON:
{"claims": [{"claim": "...", "supported": true}], "verdict": "pass"}
(Where a scale is unavoidable, the G-Eval method reads the probability the model assigns to each score token and takes the weighted average, which gives a finer reading than the single sampled token. It needs an API that returns token log probabilities.)
Three biases are well documented and have standard controls.
Position bias. In pairwise judging, models favour one position, often the first. Run every comparison twice with the order swapped, and count a verdict only when both runs agree; otherwise record a tie.
Length bias. Judges favour longer answers. Where length is not part of the criterion, say so in the prompt, and check the effect by comparing verdicts against output length on the labelled set. (AlpacaEval’s length-controlled scores adjust for this statistically.)
Self-preference. Judges rate outputs from their own model family more highly. Use a judge from a different family to the system it grades.
How do we know it’s working?
A judge’s verdicts are measurements, and the standard to measure them against is a set of human labels. We can turn that into a procedure.
Label a set. A domain expert labels a hundred or more outputs as pass or fail on the judge’s criterion, with a one-line reason for each failure. The set needs enough failures to test the judge on them: a set that is 95% passes cannot show whether the judge catches the other 5%.
Split it. Use part of the set to develop the prompt and hold the rest back as a test set, scored once at the end. A prompt tuned until it agrees with every label it has seen will look better than it is (the same discipline as cross-validation).
Measure two rates. The true positive rate is the share of expert passes the judge also passes. The true negative rate is the share of expert failures the judge also fails. Report both, since a judge that passes everything scores 95% accuracy on a set that is 95% passes and catches nothing. Cohen’s kappa summarises agreement corrected for chance:
kappa = (p_o - p_e) / (1 - p_e)
Here p_o is the observed agreement and p_e is the agreement expected if both raters labelled at random with their own pass rates. (On Landis and Koch’s widely used scale, 0.61 to 0.80 counts as substantial agreement; the bands are a convention.)
Check the humans too. Have a second expert label part of the set. If two experts agree with each other at a kappa of 0.5, the criterion is ambiguous, and no judge can be held to a higher standard than the people who wrote it. Rewriting the criterion comes before tuning the prompt.
Correct the pass rate. A judge with a known true positive rate (TPR) and true negative rate (TNR) misreports a system’s pass rate in a predictable direction, and the observed rate can be corrected:
true_rate = (observed_rate + TNR - 1) / (TPR + TNR - 1)
A lenient judge with a TPR of 0.98 and a TNR of 0.60 that reports 90% passes describes a system passing about 86%. (This is the Rogan-Gladen estimator from diagnostic testing. It needs TPR + TNR above 1, and its uncertainty grows as either rate falls.)
The calculations take a few lines of Python:
def judge_agreement(human, judge):
"""human, judge: lists of booleans (True = pass), in the same order."""
pairs = list(zip(human, judge))
tp = sum(h and j for h, j in pairs)
tn = sum(not h and not j for h, j in pairs)
fp = sum(j and not h for h, j in pairs)
fn = sum(h and not j for h, j in pairs)
n = len(pairs)
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)
p_o = (tp + tn) / n
p_human = (tp + fn) / n
p_judge = (tp + fp) / n
p_e = p_human * p_judge + (1 - p_human) * (1 - p_judge)
kappa = (p_o - p_e) / (1 - p_e)
return tpr, tnr, kappa
def corrected_pass_rate(observed, tpr, tnr):
"""Estimate the true pass rate from the judge's observed rate."""
return (observed + tnr - 1) / (tpr + tnr - 1)
Keep checking. Re-run the judge against the labelled set whenever its prompt, its model or the system it grades changes. Sample a few production outputs each week for human review, and add every case where the expert and the judge disagree to the labelled set. Over months the set becomes the most precise statement of the criterion the team has.
A judge with measured true positive and true negative rates is an instrument with known error, and its pass rate can be corrected and reported within those bounds. Without them, its pass rate is a number nobody can interpret. The check matters more again when a judge’s scores are used to train a model, because a model under training searches for exactly the cases where the judge is wrong.