LLM-as-a-judge position bias, measured on 4,796 GPT-4 swaps

An LLM judge is a classifier, and a classifier needs a test set, a swap test, and an agreement number before anyone trusts it. This article builds that harness for pairwise judging and runs it over the 48-item parcel-bot golden set from the previous tutorial, where every pair of answers already has a verdict written as assertions. One thing up front: we did not call a hosted model for this piece. The harness takes a prompt and returns text, and the five judges we ran through it are deterministic rules, so every number from them reproduces exactly and none is a claim about any model. What those numbers show is how the protocol behaves, and which reported biases can appear with no model at all. The listing is code/llm-as-judge-position-bias.py, the 1,440 judge calls are in datasets/llm-as-judge-position-bias.csv, and testing a real model means adding one function.

Updated October 27, 2026 with a real judge. We still called no model. The MT-Bench authors published the verdicts behind their leaderboard, in which GPT-4 judged 4,800 pairs of answers once in each order, and we re-scored all of them with the metrics below. GPT-4 kept its verdict when the answers swapped places in 83.7% of the 4,796 usable pairs. Its lean toward the first or the second answer nearly cancels overall but flips between the first and second turn of a conversation and between task types, consistency climbs from 75% to 96% as the two answers grow further apart in quality, and on the items where it flipped, human experts disagreed with each other far more than elsewhere. Those results start at the section headed "Update, October 27, 2026".

What the papers measured

Zheng et al. (arXiv 2306.05685v4), the MT-Bench paper, run the swap test in Table 2: two similar answers per question, each judged twice with the order reversed. With the default prompt GPT-4 was consistent in 65.0% of cases, GPT-3.5 in 46.2%, and Claude-v1 in 23.8%, favouring the first answer 75.0% of the time; the authors note the test "is challenging because the answers are very similar". Table 3 is a length attack that prepends a rewritten copy of a numbered list and counts how often the padded answer wins: 91.3% for Claude-v1 and GPT-3.5, 8.7% for GPT-4, on 23 answers. Their fix, in section 3.4, is to "only declare a win when an answer is preferred in both orders", and call it a tie otherwise.

Wang et al. (arXiv 2305.17926v2) found that with ChatGPT as evaluator, swapping the order changed the verdict on 66 of 80 questions; GPT-4 changed it on 37 of 80. Their Figure 2 shows the conflicts concentrate where the two scores are within a point, while gaps of three or more are "relatively stable". Shi et al. (arXiv 2406.07791v9), across 15 judges, conclude that position bias "is not due to random chance", "varies significantly across judges and tasks", and is "strongly affected by the quality gap between solutions". Their section 2.2 scores a judge three ways: repetition stability, the share of repeated identical calls that return the most common answer; position consistency, the share of swapped pairs that keep the same winner; and preference fairness, a signed score from -1 (always the answer shown first) through 0 to +1 (always the one shown second). In their Table 2, GPT-4 (the gpt-4-0613 snapshot) judging MT-Bench answers against a vicuna-13b-v1.3 baseline at temperature 1, with no tie option, scores 0.97 stability, 0.82 ± 0.15 consistency and +0.02 fairness. Saito et al. (arXiv 2310.10076v1) define a signed length score (equation 6): the error rate when the better answer is shorter minus the error rate when it is longer. Their Table 1 gives GPT-4 0.328 and GPT-3.5 0.428.

The harness

The prompt is Figure 5 of the MT-Bench paper, verbatim apart from straight apostrophes, ending with "[[A]]", "[[B]]", or "[[C]]" for a tie. An optional reference-answer block is our addition. Parsing takes the last verdict token and refuses to guess:

VERDICT_RE = re.compile(r"\[\[([ABC])\]\]")


def parse_verdict(text: str) -> str:
    """Last [[A]]/[[B]]/[[C]] in the output wins; anything else is an error, never a guess."""
    found = VERDICT_RE.findall(text)
    return found[-1] if found else "error"

FastChat, the code that produced the MT-Bench judgments, parses differently: it looks for "[[A]]" anywhere in the output, then "[[B]]", then "[[C]]". Across the 9,600 GPT-4 outputs it published, the two rules disagree once, on a turn-two judgment that ends "Final verdict: [[A]] for the first question and [[C]] for the second question." FastChat reads that as A and the rule above reads it as a tie. Neither is right: the judge answered a question it was not asked, and a parser can only decide which wrong answer to record.

Every pair is judged twice. Candidates carry identities, x and y, so "A" maps back to whichever was shown first in that call, and the both-orders verdict is the section 3.4 rule:

raw1 = judge(build_prompt(p["question"], p["x"], p["y"], p["reference"]))
raw2 = judge(build_prompt(p["question"], p["y"], p["x"], p["reference"]))
v1, v2 = parse_verdict(raw1), parse_verdict(raw2)
m1 = {"A": "x", "B": "y", "C": "tie"}.get(v1, "error")
m2 = {"A": "y", "B": "x", "C": "tie"}.get(v2, "error")
cons = m1 if (m1 == m2 and m1 != "error") else "tie"

Each pair is also classed as Shi et al. do: consistent, primacy-preferred, or recency-preferred. The cost doubles, which is the price of knowing the verdict is about the answers.

Five judges we can run

Longer answer wins counts words, a pure length control. F1 1-10, tie to first turns token F1 against the reference into a 1-10 score and compares with >=, so equal scores go to assistant A, the tie-breaking bug we see most in hand-rolled graders. F1 1-3, tie to first is the same on a three-point scale, and F1 1-3, tie declared returns "[[C]]" on equal scores. Reference-free rubric gives a point each for mentioning the parcel id asked about, including a date, naming a status, and using 20 words or fewer.

The main probe pairs bot A against bot B on all 48 items. From the golden verdicts, bot B is better on 12 pairs, bot A on 6, and both pass on 30, scored as a tie. Bot B's reply is longer in all 48 pairs, 17.5 words on average against 9.6.

Where position bias comes from

A deterministic comparator cannot change its mind when answers swap places unless the scores are equal. The identical-answer probe shows it: shown the same passing reply twice, the 1-10 tie-to-first judge named a winner in 96 of 96 single calls, always the copy on top. The both-orders rule turned all of them into ties.

On the real pairs the 1-10 scale never produced an equal score, so that judge was 48/48 consistent. The three-point version was 35/48, with 13 primacy-preferred pairs and no recency. All 13 conflicts had a score gap of zero; none of the 35 pairs with a gap of one changed winner. That is the shape of Wang et al.'s Figure 2, produced here by rounding alone, so read it as a warning about mechanism rather than evidence about any model. Eight of the 13 are unknown-parcel items, where bot A invents a date (token F1 0.261) and bot B correctly refuses (0.378); on three points both round to 2. On the 18 pairs with a winner, one call with bot A shown first gets 10 right and one call with bot B first gets 18 right.

Scale size does not predict this:

The five-point scale is clean because 4 x 0.261 rounds to 1 and 4 x 0.378 to 2; on four and seven points they share a bin. A scale that is clean on one test set can manufacture conflicts on the next. The both-orders column does not recover the lost pairs either. It reports them as ties, which is correct for a judge that cannot separate the answers. The swap adds no information; it stops you reading position as preference.

Length: the padding probe

The padding probe appends "I hope that helps, and if there is anything else I can do for you today, just let me know." to a passing reply. Forty-four padded replies still pass their assertions, so the right verdict is a tie; four are exact readbacks the padding breaks. The length judge preferred the padded copy in 48 of 48 pairs, the repetitive-list attack at full strength. The F1 judges went the other way, preferring the original in 100% (1-10) and 91.7% (1-3) of pairs, because extra tokens cut precision against the reference. That flips the sign without removing the bias: they matched the golden verdict on only 4 and 8 of 48 padding pairs.

Saito et al.'s score, counting a judge's tie as an error (our adaptation, since their judge output is binary), gives +1.00 for the length judge, 0.00 for F1 1-10, -0.67 for both three-point judges, and -0.33 for the rubric. The groups are 6 pairs where the shorter answer is right and 12 where the longer is, so these are coarse; the per-pair CSV is the plot Saito et al. recommend reading alongside the score.

Agreement with the golden verdict

Consistency is not correctness: four judges are perfectly swap-consistent and range from 2/18 to 18/18, which matches Zheng et al.'s caution that "high consistency may not imply high accuracy". The rubric has the best three-way agreement and the worst decisive score, because it says "tie" on many pairs where both bots pass; it rewarded bot A's invented date on all 8 unknown-parcel pairs and its address leak on all 4 third-party pairs. The two columns answer different questions. Zheng et al. report both, as setup S1 (ties included) and S2 (ties dropped), with random baselines of 33% and 50%. The 1-10 F1 judge's 30 misses are pairs where both bots pass and it preferred bot A's reference-shaped wording: a fair answer to "which is better", a wrong one to "which is correct". Decide which question your judge answers before scoring it.

Kappa corrects raw agreement for chance, defined in the scikit-learn documentation as (p_o - p_e) / (1 - p_e). The listing's version matches cohen_kappa_score to within 5.55e-17. For scale, Wang et al. report their three human annotators averaged 0.54 against the majority vote, and GPT-4 with the plain prompt 0.24.

Agreement between judges is not validity

The two three-point judges agree with each other at kappa 1.00, because a tie-to-first verdict and a declared tie collapse to the same both-orders label, and each agrees with the golden verdict at 0.06. The 1-10 and 1-3 judges agree at 0.45 while measuring the same F1. The rubric scores -0.24 against the 1-10 judge. The length judge scores 0.00 against everything, despite 25% raw agreement with the golden verdict, because a rater that always says B carries nothing beyond its base rate. Judges that share a blind spot agree well, so the anchor has to be a labelled set, not a second judge.

Update, October 27, 2026: GPT-4's own swap tests

The MT-Bench authors published every GPT-4 pairwise judgment behind their leaderboard: 30 chat models, each against gpt-3.5-turbo's answer, on all 80 questions and both turns of each conversation. FastChat's common.py in the same release shows how each record was made: GPT-4 at temperature 0, called once with the model's answer as Assistant A and once with the baseline's answer first, both outputs kept. The timestamps run from June 15 to June 23, 2023, and the judge is named only as "gpt-4". The prompts allow a tie and tell the judge in so many words to "Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision." Four of the 4,800 records contain an output with no verdict token, which leaves 4,796.

We classed each pair as Shi et al.'s Figure 1(b) does, from the letters chosen in the two calls. The same winner both ways, or a tie both ways, is consistent. A pair that favours whichever answer sits in the first slot (A then A, a tie then A, A then a tie) is primacy-preferred, and the mirror image is recency-preferred. Shi et al.'s fairness score scales (recency count x recency share) minus (primacy count x primacy share) between its lowest and highest possible values. That difference reduces to recency minus primacy, and with the extremes taken as every pair primacy-preferred or every pair recency-preferred, which is how the paper defines -1 and +1, the score is (recency - primacy) / pairs. That reduction is our reading of their equation 3.

GPT-4 kept its verdict on 4,014 of 4,796 pairs, 83.7%. Averaged over the 240 model-and-category cells, as Shi et al. average, it is 0.837 ± 0.106, close to their 0.82 ± 0.15 for gpt-4-0613 with a different baseline, a different temperature and no tie option. It is well above the 65.0% in the MT-Bench paper's own Table 2, which tested pairs of near-identical answers built to be hard. Of the 782 flips, 425 were primacy-preferred and 357 recency-preferred, a fairness score of (357 - 425) / 4,796 = -0.014: on the face of it, nearly fair.

That small number is two large leans cancelling:

On the first turn GPT-4 favoured the first answer it read; on the second turn, judging the follow-up question with the whole conversation in the prompt, it favoured the second. Roleplay shows it most sharply: 43 primacy and 7 recency flips on turn one, 7 and 18 on turn two. The tasks split too. Reasoning, math and coding, the three categories judged with a reference answer in the prompt, lean recency, 133 flips to 215; the other five, judged without one, lean primacy, 292 to 142. In this data the prompt and the task type always change together, so the file cannot say which one moves the lean. What it does say is that a correction tuned on one slice ("this judge prefers the first answer") would push the other slices the wrong way. Shi et al. report the same instability one level up: GPT-4's fairness is +0.02 on MT-Bench and -0.13 on DevBench in their Table 2.

The listing for this section is code/llm-as-judge-mt-bench-gpt4.py. It downloads the files at pinned revisions, checks their SHA-256 hashes and prints every number here.

Consistency follows the gap between the answers

The same release holds GPT-4's single-answer grades: every answer scored from 1 to 10 on its own, in separate calls that never show the other answer. The gap between the two grades is a measure of how far apart the answers are that is not built from the pairwise verdicts. Grades are whole numbers except 38 half points among 5,440 grades, and 4 grades of -1 mark failed calls, which removes 4 pairs.

619 of the 782 flips, 79.2%, came from pairs graded less than two points apart. This is the pattern in Wang et al.'s Figure 2, conflicts bunched where the scores are close, measured here on a real judge at scale, and the same pattern as the local three-point judge earlier, whose flips all sat at a score gap of zero. Shi et al.'s own gap measure, an overall win rate per model and category, gives the same shape on this data, from 74.5% consistent in the closest tenth to 94.3% in the widest. That measure is built from the very verdicts being tested, though: a cell with many flips has its win rate pulled toward one half by construction. The single-grade gap avoids that loop but comes from the same model, so it can share the judge's blind spots.

Where GPT-4 flips, the experts split too

A second release, the CC BY 4.0 dataset mt_bench_human_judgments, holds GPT-4's verdicts on all 15 pairs of six models (GPT-4, Claude-v1, GPT-3.5-turbo, Vicuna-13B, Alpaca-13B and LLaMA-13B) over the 80 questions and both turns. That is 2,400 items, each already reduced to one verdict, with flips marked "tie (inconsistent)". It also holds 3,355 human votes on 1,814 of those items: 2,668 from experts, whom the dataset card describes as mostly graduate students with expertise in each question's topic, and 687 from the paper's authors.

First, a method check. Counted our way, the GPT-4-versus-human cells of the MT-Bench paper's Table 5 come out exactly: on turn one, 66% agreement over 1,343 votes with ties counted (setup S1) and 85% over 859 without them (S2); on turn two, 66% over 1,325 and 85% over 864. The expert-versus-expert cells match too: 63% over 721 and 81% over 479, then 67% over 707 and 82% over 474. Only the authors' column differs, by one vote on turn one (346 against the table's 345).

GPT-4 flipped on 380 of the 2,400 items, 15.8%. Splitting the expert votes by what GPT-4 did:

The items where GPT-4 flipped are the items where experts disagree with each other most: 52.9% agreement against 67.9% where GPT-4 picked a stable winner, and 69.7% against 83.0% once ties are set aside. Position bias is landing on genuinely hard calls, which matches what Shi et al. found among LLM judges, here seen with human raters. Yet the experts called those items a tie only 35.2% of the time, so the conservative rule's "tie" matches about a third of expert votes where it fires. Dropping the flipped items instead lifts GPT-4's agreement from 66.0% to 71.5% on turn one and from 65.7% to 71.8% on turn two, but only by marking the judge on the easier items. The flip rate belongs next to the agreement number, never folded into it.

What to do when the judge is a model

Run every pair in both orders and keep the conservative verdict. Log consistency and primacy and recency counts per task and per turn, never one pooled figure: GPT-4's pooled fairness of -0.014 hid a first-turn lean toward the first answer and a second-turn lean toward the second, and Shi et al. found the direction can flip between benchmarks for the same model. Bucket the flips by an outside measure of how close the answers are, such as single-answer grades, and expect most of them in the closest buckets; four in five of GPT-4's were there. Add identical and padded pairs to the test set; they cost a few dozen calls. Give the judge a tie option and watch how often it uses it. Score it against a golden set with a decisive-pair column, a three-way column, and kappa, and report the flip rate beside them rather than dropping the flipped items. Keep reference answers in the prompt where they exist: in the MT-Bench paper, a reference-guided prompt cut math-grading failures from 14 of 20 to 3 of 20.

What this does not show

We ran no language model. The five local judges estimate no model's biases, and they are deterministic, where Shi et al. sampled at temperature 1. The GPT-4 sections re-score outputs published in June 2023 by a judge named only "gpt-4", at temperature 0, against one baseline answer, with a tie option and with prompts that ask it to avoid position bias; they describe that judge on that benchmark, not later GPT-4 versions, other models or other prompts. The single-answer grades come from the same model as the pairwise verdicts, the experts rated only six 2023-era models, and the turn and task patterns are counts from one run of each call, so a repeat at temperature 0 could still move a few of them. Eighteen decisive pairs is a small set, and the golden verdicts are our assertions from the previous article, not a panel of human raters. Self-preference needs a model judging its own outputs, and Zheng et al. say their own data "cannot determine whether the models exhibit a self-enhancement bias".

datasets/llm-as-judge-position-bias.csv holds one row per judge call.

_README:

• probe: main (bot A against bot B), identical (a passing reply against itself) or padding (a passing reply against itself plus one filler sentence)
• item_id, intent: the golden-set item and its intent stratum
• judge: longer, f1_10_first, f1_3_first, f1_3_tie or rubric
• order: x_first or y_first, which candidate was shown as assistant A; in main, x is bot A and y is bot B; in padding, y is the padded copy
• verdict, maps_to: the parsed token (A, B, C) and the candidate it refers to (x, y, tie)
• both_orders, position: the conservative verdict for the pair and its consistency class (consistent, primacy, recency)
• gold: the correct verdict from the golden assertions (x, y, tie)
• x_words, y_words, judge_output: candidate lengths and the judge's full text

The GPT-4 sections ship two more files. The Space that hosts GPT-4's pairwise and single-answer outputs states no data licence, so we do not redistribute its rows: datasets/llm-as-judge-mt-bench-gpt4-swaps.csv holds only our counts, one row per model, category and turn, 480 rows, and the listing downloads the source file to rebuild it. datasets/llm-as-judge-mt-bench-gpt4-experts.csv has one row per item of the CC BY 4.0 dataset, 2,400 rows, with vote counts and no conversation text; it is derived from LMSYS's mt_bench_human_judgments (Zheng et al., arXiv 2306.05685) under that licence.

_README (swaps):

• model, baseline, category, turn, prompt: the judged model, always against gpt-3.5-turbo; the MT-Bench category; turn 1 or 2; pair-v2 or pair-math-v1 (the reference-guided prompt)
• pairs: records in the cell (10 questions); consistent, primacy, recency: the cell's pairs by Shi et al.'s classes, with errors counted in n_error only
• n_AB to n_BC: the letter chosen with the model shown first, then with the baseline shown first (C = tie); n_AB means the model won both calls and n_BA the baseline won both

_README (experts):

• question_id, category, turn: the MT-Bench item
• model_x, model_y: the two models, in alphabetical order
• gpt4_pair: GPT-4's combined verdict, x, y, tie or tie (inconsistent) for a flip
• expert_x, expert_y, expert_tie, author_x, author_y, author_tie: vote counts from the expert raters and from the paper's authors