Tutorials · 2026-09-15 · 1,933 words · 9 min read

LLM-as-judge done carefully: position bias, length, and agreement

A pairwise judge harness run in both orders over a 48-item golden set, with swap tests, a padding probe, and Cohen's kappa. Code and data included.

evaluation llm-as-judge position bias python

An LLM judge is a classifier, and a classifier needs a test set, a swap test, and an agreement number before anyone trusts it. This article builds that harness for pairwise judging and runs it over the 48-item parcel-bot golden set from the previous tutorial, where every pair of answers already has a verdict written as assertions. One thing up front: I did not call a hosted model for this piece. The harness takes a prompt and returns text, and the five judges I ran through it are deterministic rules, so every number below reproduces exactly and none is a claim about any model. What the numbers show is how the protocol behaves, and which reported biases can appear with no model at all. The listing is code/llm-as-judge-position-bias.py, the 1,440 judge calls are in datasets/llm-as-judge-position-bias.csv, and testing a real model means adding one function.

What the papers measured

Zheng et al. (arXiv 2306.05685v4), the MT-Bench paper, run the swap test in Table 2: two similar answers per question, each judged twice with the order reversed. With the default prompt GPT-4 was consistent in 65.0% of cases, GPT-3.5 in 46.2%, and Claude-v1 in 23.8%, favouring the first answer 75.0% of the time; the authors note the test "is challenging because the answers are very similar". Table 3 is a length attack that prepends a rewritten copy of a numbered list and counts how often the padded answer wins: 91.3% for Claude-v1 and GPT-3.5, 8.7% for GPT-4, on 23 answers. Their fix, in section 3.4, is to "only declare a win when an answer is preferred in both orders", and call it a tie otherwise.

Wang et al. (arXiv 2305.17926v2) found that with ChatGPT as evaluator, swapping the order changed the verdict on 66 of 80 questions; GPT-4 changed it on 37 of 80. Their Figure 2 shows the conflicts concentrate where the two scores are within a point, while gaps of three or more are "relatively stable". Shi et al. (arXiv 2406.07791v9), across 15 judges, conclude that position bias "is not due to random chance", "varies significantly across judges and tasks", and is "strongly affected by the quality gap between solutions". Saito et al. (arXiv 2310.10076v1) define a signed length score (equation 6): the error rate when the better answer is shorter minus the error rate when it is longer. Their Table 1 gives GPT-4 0.328 and GPT-3.5 0.428.

The harness

The prompt is Figure 5 of the MT-Bench paper, verbatim apart from straight apostrophes, ending with "[[A]]", "[[B]]", or "[[C]]" for a tie. An optional reference-answer block is my addition. Parsing takes the last verdict token and refuses to guess:

VERDICT_RE = re.compile(r"\[\[([ABC])\]\]")


def parse_verdict(text: str) -> str:
    """Last [[A]]/[[B]]/[[C]] in the output wins; anything else is an error, never a guess."""
    found = VERDICT_RE.findall(text)
    return found[-1] if found else "error"

Every pair is judged twice. Candidates carry identities, x and y, so "A" maps back to whichever was shown first in that call, and the both-orders verdict is the section 3.4 rule:

raw1 = judge(build_prompt(p["question"], p["x"], p["y"], p["reference"]))
raw2 = judge(build_prompt(p["question"], p["y"], p["x"], p["reference"]))
v1, v2 = parse_verdict(raw1), parse_verdict(raw2)
m1 = {"A": "x", "B": "y", "C": "tie"}.get(v1, "error")
m2 = {"A": "y", "B": "x", "C": "tie"}.get(v2, "error")
cons = m1 if (m1 == m2 and m1 != "error") else "tie"

Each pair is also classed as Shi et al. do: consistent, primacy-preferred, or recency-preferred. The cost doubles, which is the price of knowing the verdict is about the answers.

Five judges I can run

Longer answer wins counts words, a pure length control. F1 1-10, tie to first turns token F1 against the reference into a 1-10 score and compares with >=, so equal scores go to assistant A, the tie-breaking bug I see most in hand-rolled graders. F1 1-3, tie to first is the same on a three-point scale, and F1 1-3, tie declared returns "[[C]]" on equal scores. Reference-free rubric gives a point each for mentioning the parcel id asked about, including a date, naming a status, and using 20 words or fewer.

The main probe pairs bot A against bot B on all 48 items. From the golden verdicts, bot B is better on 12 pairs, bot A on 6, and both pass on 30, scored as a tie. Bot B's reply is longer in all 48 pairs, 17.5 words on average against 9.6.

Where position bias comes from

A deterministic comparator cannot change its mind when answers swap places unless the scores are equal. The identical-answer probe shows it: shown the same passing reply twice, the 1-10 tie-to-first judge named a winner in 96 of 96 single calls, always the copy on top. The both-orders rule turned all of them into ties.

On the real pairs the 1-10 scale never produced an equal score, so that judge was 48/48 consistent. The three-point version was 35/48, with 13 primacy-preferred pairs and no recency. All 13 conflicts had a score gap of zero; none of the 35 pairs with a gap of one changed winner. That is the shape of Wang et al.'s Figure 2, produced here by rounding alone, so read it as a warning about mechanism rather than evidence about any model. Eight of the 13 are unknown-parcel items, where bot A invents a date (token F1 0.261) and bot B correctly refuses (0.378); on three points both round to 2. On the 18 pairs with a winner, one call with bot A shown first gets 10 right and one call with bot B first gets 18 right.

Scale size does not predict this:

The five-point scale is clean because 4 x 0.261 rounds to 1 and 4 x 0.378 to 2; on four and seven points they share a bin. A scale that is clean on one test set can manufacture conflicts on the next. The both-orders column does not recover the lost pairs either. It reports them as ties, which is correct for a judge that cannot separate the answers. The swap adds no information; it stops you reading position as preference.

Length: the padding probe

The padding probe appends "I hope that helps, and if there is anything else I can do for you today, just let me know." to a passing reply. Forty-four padded replies still pass their assertions, so the right verdict is a tie; four are exact readbacks the padding breaks. The length judge preferred the padded copy in 48 of 48 pairs, the repetitive-list attack at full strength. The F1 judges went the other way, preferring the original in 100% (1-10) and 91.7% (1-3) of pairs, because extra tokens cut precision against the reference. That flips the sign without removing the bias: they matched the golden verdict on only 4 and 8 of 48 padding pairs.

Saito et al.'s score, counting a judge's tie as an error (my adaptation, since their judge output is binary), gives +1.00 for the length judge, 0.00 for F1 1-10, -0.67 for both three-point judges, and -0.33 for the rubric. The groups are 6 pairs where the shorter answer is right and 12 where the longer is, so these are coarse; the per-pair CSV is the plot Saito et al. recommend reading alongside the score.

Agreement with the golden verdict

Consistency is not correctness: four judges are perfectly swap-consistent and range from 2/18 to 18/18, which matches Zheng et al.'s caution that "high consistency may not imply high accuracy". The rubric has the best three-way agreement and the worst decisive score, because it says "tie" on many pairs where both bots pass; it rewarded bot A's invented date on all 8 unknown-parcel pairs and its address leak on all 4 third-party pairs. The two columns answer different questions. Zheng et al. report both, as setup S1 (ties included) and S2 (ties dropped), with random baselines of 33% and 50%. The 1-10 F1 judge's 30 misses are pairs where both bots pass and it preferred bot A's reference-shaped wording: a fair answer to "which is better", a wrong one to "which is correct". Decide which question your judge answers before scoring it.

Kappa corrects raw agreement for chance, defined in the scikit-learn documentation as (p_o - p_e) / (1 - p_e). The listing's version matches cohen_kappa_score to within 5.55e-17. For scale, Wang et al. report their three human annotators averaged 0.54 against the majority vote, and GPT-4 with the plain prompt 0.24.

Agreement between judges is not validity

The two three-point judges agree with each other at kappa 1.00, because a tie-to-first verdict and a declared tie collapse to the same both-orders label, and each agrees with the golden verdict at 0.06. The 1-10 and 1-3 judges agree at 0.45 while measuring the same F1. The rubric scores -0.24 against the 1-10 judge. The length judge scores 0.00 against everything, despite 25% raw agreement with the golden verdict, because a rater that always says B carries nothing beyond its base rate. Judges that share a blind spot agree well, so the anchor has to be a labelled set, not a second judge.

What to do when the judge is a model

Run every pair in both orders and keep the conservative verdict. Log consistency and primacy and recency counts per task, since Shi et al. found the direction can flip between benchmarks for the same model. Add identical and padded pairs to the test set; they cost a few dozen calls. Give the judge a tie option and watch how often it uses it. Score it against a golden set with a decisive-pair column, a three-way column, and kappa. Keep reference answers in the prompt where they exist: in the MT-Bench paper, a reference-guided prompt cut math-grading failures from 14 of 20 to 3 of 20.

What this does not show

No language model was run, so nothing here estimates any model's biases; the judges are also deterministic, where Shi et al. sampled at temperature 1. Eighteen decisive pairs is a small set, and the golden verdicts are my assertions from the previous article, not a panel of human raters. Self-preference needs a model judging its own outputs, and Zheng et al. say their own data "cannot determine whether the models exhibit a self-enhancement bias".

datasets/llm-as-judge-position-bias.csv holds one row per judge call.

_README: - probe: main (bot A against bot B), identical (a passing reply against itself) or padding (a passing reply against itself plus one filler sentence) - item_id, intent: the golden-set item and its intent stratum - judge: longer, f1_10_first, f1_3_first, f1_3_tie or rubric - order: x_first or y_first, which candidate was shown as assistant A; in main, x is bot A and y is bot B; in padding, y is the padded copy - verdict, maps_to: the parsed token (A, B, C) and the candidate it refers to (x, y, tie) - both_orders, position: the conservative verdict for the pair and its consistency class (consistent, primacy, recency) - gold: the correct verdict from the golden assertions (x, y, tie) - x_words, y_words, judge_output: candidate lengths and the judge's full text