Your AI Judge Can Pick the Same Winner Twice and Still Be Wrong

A team has hundreds of generated answers to compare. It asks another language model to choose the better response, counts the wins, and announces an improvement. The process is convenient. The harder question is whether the judge rewarded the property the team actually needed.

A fluent response may contain the wrong number. A longer answer may sound more helpful while making an unsupported promise. A judge can prefer one answer because of its location in the prompt, and it can also prefer the same incorrect answer in both locations.

LLM-based judging can assist evaluation, but its output needs a defined role. Treat it as a measurement process with assumptions, not an independent certificate of truth. This guide, checked on 25 October 2026, uses established research and an original audit worksheet; it does not present a new model ranking.

Decide what the judge should measure

“Better” can mean factual correctness, instruction following, clarity, completeness or a user's stylistic preference. Those properties can disagree. Write the priority rule before showing the judge the candidates.

For a source-grounded business answer, the rule might require every numerical claim to agree with the provided source, then assess whether the response answers the question. Style becomes relevant only after those requirements are met.

For creative writing, preference may legitimately be the target. That still does not allow preference results to be relabeled as factual accuracy. The intended construct should appear in the evaluation report, not only in the prompt hidden behind the dashboard.

Read the research at its actual scope

Zheng and colleagues' MT-Bench and Chatbot Arena paper, first submitted in 2023, examines LLM judging and limitations including position and verbosity biases. Its reported agreement with human preferences belongs to its tasks, models and study design; it is not a guarantee about today's application.

Shi and colleagues' position-bias study, in its 2025 revision, explores how judging behavior varies across models, candidates and tasks. The lesson for an application team is to test its own judging setup. Neither paper validates the original worksheet below or establishes a universal correction factor for bias.

This qualitative research summary follows the primary abstracts and metadata. No new effect-size claim or replication result is asserted here.

Create a case with a checkable answer

Consider an original fictional source note: “Period revenue was 100 units and operating costs were 70 units. No forecast was supplied.” The question asks for the difference and whether next period's profit is known.

Candidate A says the difference is 30 and the source supplies no forecast. Candidate B gives an elegant paragraph saying the difference is 40 and predicts strong profit growth. Under the stated rubric, A is preferable regardless of tone.

The term “difference” avoids silently importing accounting assumptions about taxes, interest or other costs. A benchmark should make its question as precise as its expected answer.

Swap the order and map back to identity

Present A then B, and separately B then A. Keep the question, source and rubric unchanged. Map the chosen label back to the underlying answer, rather than comparing a bare “first” or “second” label.

If both presentations select A, the result is consistent for that pair. If one selects A and the other B, mark it inconsistent and retain both rationales. Do not quietly discard the inconvenient run.

Order swapping is a diagnostic. It does not establish that the two runs are independent, that every source of bias is removed, or that the winner is correct. It also doubles the number of comparisons, which belongs in the evaluation's cost record.

Keep consistency separate from correctness

Suppose an original audit contains 12 answer pairs. Nine receive the same underlying winner in both orders; three do not. Order consistency is nine divided by 12, or 75%.

Now suppose independent reference checks accept the first-order selection as correct in eight of the 12 cases. That separately defined accuracy is eight divided by 12, approximately 66.7%. It uses a different rule and should be labeled as such.

These counts illustrate the distinction, not statistical evidence of a model's general performance. Preserve the pair-level cross-tabulation if you want to investigate which consistent decisions were wrong.

Ground the judgment in evidence

Ask the judge to identify the claim and source passage supporting its decision. Then check whether the cited evidence exists and supports the conclusion. A rationale can be fluent without being faithful.

For numerical tasks, use a deterministic calculation where possible. In the fictional source note, subtraction supplies stronger evidence about the numerical difference than another model's confidence language. For software tasks, meaningful executable checks can verify behavior that a prose judge might overlook.

Some answers require expert interpretation rather than a single reference value. Record disagreements and adjudication rules. Do not manufacture a definitive ground truth merely because the evaluation tool requires one field.

Freeze the evaluation setup

Record the judge model and version, prompt, rubric, generation settings, candidate outputs, source material and order mapping. Changing any of these can change the measurement.

Use a small development set to refine the rubric, then keep a separate evaluation set for the reported comparison. If the same cases repeatedly drive prompt changes, their later scores no longer provide an untouched check of the revised process.

When a model version changes, compare behavior on preserved cases before treating the new scores as continuous with the old series. The measurement system is part of the experiment.

Route ambiguous cases to a defined review

A judge can return a tie, uncertainty or an unsupported rationale. Define how those outcomes enter the denominator. Counting only decisive wins can make a fragile system appear more successful.

An original review policy might send order-inconsistent and evidence-mismatched pairs for human assessment. Also inspect a sample of consistent pairs, because consistency can conceal shared errors. The sample design should fit the consequence and purpose of the evaluation.

Do not claim a human-reviewed result if a reviewer merely accepted the judge's label without checking the source. Review requires an actual decision and enough information to make it.

Publish a result someone else can interpret

Report the task population, sample count, rubric, handling of ties, order-consistency result and reference-check procedure. Separate preference, correctness and operational usefulness rather than combining them under one unqualified score.

The practical aim is not to abandon model judges. It is to use them where their behavior has been examined and to retain stronger evidence for claims that can be checked directly. A repeatable preference is one observation; a supported answer is another.

Sources and further reading

• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023 research, checked 25 October 2026.
• Shi et al.: Judging the Judges, position-bias study, revised 2025, checked 25 October 2026.
• Related: independent evaluation access and when AI should abstain.