The four judge validations among this card's parents put disagreement between an automatic support judge and human raters at 2 percent (98 of 100 verdicts, AI Overviews), 11.3 percent (88.7 percent agreement with a three-doctor consensus, where doctors agreed with each other at 86.1 percent) and 14.9 to 22.4 percent (ALCE's NLI metric, accuracy 85.1 percent for recall and 77.6 for precision); on an adversarial benchmark the best of eight judges reaches F1 0.750 on factual support. These are decision-level disagreements of these judges on their own material. The rate-level gaps the parents print are smaller: 0.0 to 3.3 points between human and automatic system scores on ELI5 (six printed pairs), and 2.0 points (40.4 against 42.4 percent fully supported responses) in SourceCheckup.

Each parent contributes one validated judge on its own material; the step only converts agreement into disagreement (100 minus the printed agreement) and places them side by side. The doctors' own 86.1 percent agreement is carried along because it bounds what agreement with humans can mean. The F1 anchor is kept separate because its material is synthetic and adversarial and its metric is not an agreement rate. Hidden premise: validation samples of 100 to 400 items are representative of the full runs.

A larger validation (a thousand or more items) of one of these judges on its own benchmark shows disagreement well outside 2 to 22 percent, or shows that disagreement concentrates on one system so that rankings change. Query: per-system breakdown of judge-human disagreement in any of the four papers' released data.

The numbers come from papers, so they are measured. Too coarse: most of them are scored by an automatic judge, and where such a judge was validated its agreement with humans was between 78 and 98 percent of decisions.