The two main AI judges pick the same winner nine times out of ten. The disagreements are elsewhere
Claude Opus 5 and GPT-5.6 Sol, who supply 85% of the verdicts, side with the same model on 89% of the 2,700 model pairs they have both judged. Claude Opus 4.8 agrees with them only two times in three, and the judges do not agree at all on when to call a tie: GPT-5.6 Sol does it for one duel in 200, Claude Fable 5.1 for one in six.
Update, 1 October 2026. The figures in this article come from data as of 19 September 2026. Since 23 September, Claude Opus 5.5 and GPT-6 Sol have replaced Claude Opus 5 and GPT-5.6 Sol as the main judges; Muse Spark 1.3 has been judging since 8 September. The verdicts of the former judges stay in the ranking. The agreement rates below have not been recomputed for the new judges.
The ranking pools verdicts from six AI judges. That only makes sense if they mostly agree. Do they?
We measured it the way a reader would: for every pair of models that two judges have both seen at least four times, does each judge give the same model the majority of points? Pairs that end exactly level under either judge are left out. English edition, data as of 19 September 2026.
Claude Opus 5 and GPT-5.6 Sol agree on 89% of 2,700 shared model pairs
How often two judges pick the same winner
Share of shared model pairs where both judges give the majority of points to the same model. Small print: number of shared pairs.
View as a data table
| Judges | % | pairs |
|---|---|---|
| Claude Opus 5 × GPT-5.6 Sol | 89.1 % | 2,738 |
| Claude Opus 5 × Claude Fable 5.1 | 91.3 % | 69 |
| GPT-5.6 Sol × Claude Fable 5.1 | 76.8 % | 69 |
| Claude Opus 5 × Claude Fable 5 | 75.5 % | 49 |
| GPT-5.6 Sol × Claude Fable 5 | 86.0 % | 50 |
| Claude Opus 5 × Claude Opus 4.8 | 72.4 % | 87 |
| GPT-5.6 Sol × Claude Opus 4.8 | 66.3 % | 83 |
| Claude Opus 5 × Muse Spark 1.3 | 93.5 % | 108 |
| GPT-5.6 Sol × Muse Spark 1.3 | 75.7 % | 136 |
The two big judges have judged 2,738 model pairs in common. They side with the same model in 89.1% of them. In French, German and Spanish the figure is 91 to 93%.
Two judges built by different companies, trained on different data, reaching the same verdict on nine pairs out of ten: that is the strongest evidence this benchmark has that its ranking measures something real about writing rather than the taste of one model. The tenth pair is where the interesting questions live, and we will publish those disagreements as a list.
Claude Opus 4.8, the retired judge, agrees with the current ones only two times in three
Claude Opus 4.8, the previous generation of Opus, agrees with Claude Opus 5 on 72% of shared pairs and with GPT-5.6 Sol on 66%. Its verdicts look like an earlier taste, which is why it is now retired, with a small share of the pool. Its 843 English verdicts stay in the ranking; they no longer grow.
Claude Fable 5.1 sides with Claude Opus 5 far more often than with GPT-5.6 Sol
On the pairs it shares with each, Claude Fable 5.1 agrees with Claude Opus 5 91% of the time and with GPT-5.6 Sol 77% of the time. Muse Spark 1.3 shows the same tilt, 94% against 76%. A family resemblance between two Claude judges is not surprising, but a benchmark that pools their verdicts should keep an eye on it, and this one does: the counts behind those rates are 69 and 136 pairs, so read them as indications until the controlled sample is in.
On ties, the judges do not agree at all: GPT-5.6 Sol calls one in 200, Claude Fable 5.1 one in six
How often each judge calls a tie
Share of verdicts that end level. Small print: verdicts judged.
View as a data table
| Judge | % | verdicts |
|---|---|---|
| Claude Fable 5.1 | 15.6 % | 1,741 |
| Claude Fable 5 | 10.2 % | 1,084 |
| Muse Spark 1.3 | 9.7 % | 6,325 |
| Claude Opus 5 | 5.4 % | 28,800 |
| Claude Opus 4.8 | 3.7 % | 843 |
| GPT-5.6 Sol | 0.5 % | 25,553 |
GPT-5.6 Sol almost always picks a winner: 0.5% of its English verdicts are ties. Claude Fable 5.1 calls a tie for 15.6% of its duels, one in six. Neither is wrong. A tie is a legitimate answer when two texts are equally good, and English duels between top models often are. But it means that a “win” under Sol is a weaker statement than a “win” under Fable 5.1.
This is one reason the main ranking uses a Bradley–Terry fit over verdicts, in which a tie counts half a win for each side. The judges give no score, only a choice, so a judge’s tie rate changes how much each of its verdicts moves the fit, not the direction.
What we will do with it
Three things follow. The list of pairs on which the two main judges disagree will be published. A controlled sample, the same pairs judged by every active judge, is being built so that the agreement rates rest on identical duels. And the per-judge tables stay on the results page, so that anyone who trusts one judge more than the others can read the ranking that judge alone would produce.
Conclusion
Two judges built by different companies reach the same verdict on nine model pairs out of ten. That is the strongest evidence this benchmark has that its ranking measures writing rather than one model's taste. The remaining tenth, the retired judge's older taste and the very different tie rates are exactly what the pooled Bradley–Terry fit is there to absorb.
