Annotation Agreement: Why the Same Agreement Rate Can Produce Different Kappa
· Annotation Quality
An original two-annotator contingency-table worksheet for observed agreement, class marginals, expected agreement and unweighted Cohen kappa.
Two annotators can agree on the same percentage of items while producing different Cohen kappa scores. Kappa adjusts observed agreement using the annotators' class marginals; neither agreement nor kappa establishes accuracy against an independently reviewed reference.
This guide is for annotation buyers reviewing a pilot with two annotators assigning one nominal class per item. It concerns quality evidence, not company hiring, pay, job availability or a vendor ranking. Data Annotation Companies is independent and is not affiliated with scikit-learn.
What calculation does the primary source define?
The official scikit-learn Cohen kappa documentation defines kappa as (observed agreement minus expected agreement) divided by (one minus expected agreement). Expected agreement uses the empirical label proportions of each annotator. The worksheet below uses two nominal classes, no sample weights and no ordinal weighting.
Keep the original independent labels before adjudication. Comparing one annotator's labels with labels already reconciled from the same work changes the question. A pilot's declared sampling and metric basis should be retained with the table.
How can identical agreement produce different scores?
Illustrative example. Each invented pilot contains 100 items labelled by two annotators as class A or class B. All items have exactly one label from each annotator. The two pilots are different synthetic samples, not real vendor results.
| Pair of labels | Balanced pilot count | Skewed pilot count |
|---|---|---|
| Both A | 41 | 81 |
| First A, second B | 9 | 9 |
| First B, second A | 9 | 9 |
| Both B | 41 | 1 |
| Total | 100 | 100 |
In the balanced pilot, agreement is (41 + 41) / 100 = 0.82. Both annotators assign A to 50 items and B to 50. Expected agreement is 0.5 x 0.5 + 0.5 x 0.5 = 0.50. Kappa is (0.82 - 0.50) / (1 - 0.50) = 0.64.
In the skewed pilot, agreement is (81 + 1) / 100 = 0.82. Both annotators assign A to 90 items and B to 10. Expected agreement is 0.9 x 0.9 + 0.1 x 0.1 = 0.82. Kappa is (0.82 - 0.82) / (1 - 0.82) = 0.
The same 82 percent agreement therefore produces 0.64 and zero in these examples. Zero does not mean the annotators agreed on no items. It means observed agreement equals the expected agreement calculated from these marginals. No universal acceptance threshold is supplied.
Which evidence should accompany a reported score?
| Review field | Evidence to retain |
|---|---|
| Sampling | Population, selected item IDs and declared sample method |
| Independent labels | Original labels from both annotators before adjudication |
| Task basis | Nominal classes, ontology revision and missing-label treatment |
| Counts | Complete contingency table, total and class marginals |
| Metric | Unweighted or weighted definition and any sample weighting |
| Disposition | Disagreement review, independently supported reference and acceptance decision |
What should not be inferred from kappa?
Agreement can be high when both annotators make the same mistake. Accuracy needs a separately justified reference and evaluation basis. These two samples also cannot establish which annotator or vendor is better because their class distributions differ.
For constant-label cases, the expected-agreement denominator can become zero. Preserve an undefined metric as undefined under the declared method rather than replace it with a convenient success score. Missing labels, multiple annotators, multilabel outputs and ordinal weighting need an appropriate separately specified evaluation design.
Frequently asked questions
Does zero kappa mean zero agreement?
No. The skewed example has 82 percent observed agreement and zero kappa.
Is annotator agreement ground-truth accuracy?
No. Agreement and accuracy against a supported independent reference are different questions.
Can a universal kappa threshold be taken from this table?
No. Agree the sampling, metric and acceptance basis for the actual task.
Use the pilot acceptance register to keep metric evidence separate from acceptance and rework. Browse the company directory for vendor research and ask potential providers to explain their actual pilot methodology; this guide supplies no quality ranking.