Do AI judges favour their own model? Two of five do, by up to 19 points
When GPT-5.6 Sol judges a duel involving GPT-5.6 Sol, its own texts collect 88% of the points; under the other judges they collect 69%. Claude Opus 5 shows the same lean, smaller. The two Claude Fable judges do the opposite and mark their own model down.
Update, 1 October 2026. The article below uses data as of 19 September 2026. The judges active since then also judge duels involving their own model, so we ran the same calculation on the English data of 1 October. GPT-6 Sol gives its own model 94.4% of the points over 436 self-judged duels, against 86.0% under the other judges (1,045 duels): a gap of 8 points, smaller than GPT-5.6 Sol’s 19. Claude Opus 5.5 gives its own model 99.2% over 606 duels, against 97.6% under the other judges (868 duels); its model wins almost every duel whoever judges, which leaves little room for a bonus. Muse Spark 1.3 now gives its own model 75.6% over 291 duels, against 74.2% under the other judges (1,828 duels), a difference too small to mean much.
Every judge on this benchmark sees two anonymous texts and picks the better one. Anonymous for the judge, that is. We know who wrote each text, so we can ask a question the judge cannot: when one of the two texts comes from the judge’s own model, does the verdict change?
For two judges it does. The chart compares, for each judge, the share of points its own model earns when that judge decides with the share the same model earns under all the other judges. A win counts one point, a tie half a point. Data: every published English duel as of 19 September 2026, about 64,000 verdicts.
GPT-5.6 Sol gives its own texts 19 extra points, Claude Opus 5 gives its own 11
Each judge's own model, under its own verdicts and under everyone else's
Share of points the model collects (a win counts 1, a tie ½) when its own model is the judge, and when the other judges decide.
- Under its own judge
- Under the other judges
View as a data table
| Judge | Under its own judge | Under the other judges | Own-judge bonus | n |
|---|---|---|---|---|
| GPT-5.6 Sol | 88.1 % | 69.3 % | +18.8 | 1,013 |
| Claude Opus 5 | 94.1 % | 82.7 % | +11.4 | 806 |
| Muse Spark 1.3 | 82.6 % | 73.8 % | +8.8 | 172 |
| Claude Fable 5 | 88.1 % | 89.8 % | −1.7 | 905 |
| Claude Fable 5.1 | 84.4 % | 88.0 % | −3.6 | 189 |
GPT-5.6 Sol is the clearest case. Its own texts collect 88.1% of the points when Sol is the judge and 69.3% when anyone else is, over about a thousand self-judged duels. Claude Opus 5 leans the same way with a smaller gap: 94.1% against 82.7%. Muse Spark 1.3 follows at a distance, 82.6% against 73.8%, on a sample of 172 duels that is too small to be sure.
The same pattern appears in the French, German and Spanish editions, with Sol’s bonus between 15 and 20 points every time. It is not an artefact of one language.
Claude Fable 5 and Claude Fable 5.1 mark their own model down
The two Fable judges go the other way. Under their own verdicts their texts earn fewer points than under the other judges: 88.1% against 89.8% for Fable 5, 84.4% against 88.0% for Fable 5.1. A one- to four-point discount, again in all four languages. Whatever the reason, they are not flattering themselves.
Why the bonus matters: two judges supply 85% of all verdicts
A bonus from a judge with a small share of the verdicts would wash out in the pooled ranking. Here it cannot, because Claude Opus 5 and GPT-5.6 Sol supply 85% of everything. Around 800 Opus-on-Opus and 1,000 Sol-on-Sol duels enter the Bradley–Terry fit with everyone else’s, and they pull.
The effect is visible in the per-judge tables: GPT-5.6 Sol places its own model second when it judges alone, while the pooled ranking has it ninth. Claude Opus 5 is first or second everywhere, so its bonus changes little at the top but still inflates its score.
Why we do not call it bias yet
Two caveats stop us short of a verdict. First, the duels are not the same: a judge’s own model meets a different mix of opponents under that judge than under the others, and the mix changes the expected share. Sol’s gap is large enough that opponent mix is unlikely to explain all of it, but it can explain part. Second, “the other judges” is not a neutral instrument: it is mostly Opus 5 when we look at Sol, and mostly Sol when we look at Opus 5. If one is severe with the other, that severity lands on the same row.
The clean test is a controlled sample, the same pairs judged by every judge, and a ranking that excludes self-judged duels. Neither is published yet; the update at the top of this page gives the same measurement for the current judges.
Conclusion
Self-preference is real for two judges and absent for two others. Because Claude Opus 5 and GPT-5.6 Sol supply 85% of all verdicts, their bonus can lift their own models by a few places in the main ranking. A ranking computed without self-judged duels would show exactly how much moves; it is not published yet.
