Macro vs Weighted F1: Check Class Balance in an Annotation Pilot
· Annotation Quality
A worked annotation-quality worksheet showing how a high support-weighted score can coexist with zero F1 for a minority class.
Macro F1 gives each included class equal weight, while weighted F1 weights each class score by its reference-label support. A high weighted score can therefore coexist with poor performance on a small class; report class scores and counts beside the average.
This guide is for annotation buyers comparing submitted labels with a separately justified reference. It does not describe company hiring, pay or job availability, and it supplies no vendor ranking. Data Annotation Companies is independent and is not affiliated with scikit-learn.
Which metric definition is being used?
The official scikit-learn F1 documentation defines class F1 as 2 x true positives divided by (2 x true positives + false positives + false negatives). Macro is the unweighted mean of class F1 scores. Weighted averaging uses support, meaning the number of true instances for each class.
Declare the class set and averaging method before reviewing the pilot. Here, support means the count in the independently supported reference labels, not the submitted-label count. Both classes are included, each item has one nominal class and all items have equal weight.
What can a support-weighted average conceal?
Illustrative example. An invented 100-item pilot has 90 reference-A items and 10 reference-B items. The submitted annotation labels every item A. The reference is assumed separately justified for this arithmetic example; the table does not establish how any real ground truth was created.
| Reference class | Submitted A | Submitted B | Support |
|---|---|---|---|
| A | 90 | 0 | 90 |
| B | 10 | 0 | 10 |
For class A, true positives are 90, false positives are 10 and false negatives are zero. F1 is 180 / 190 = 18 / 19, approximately 0.9474. For class B, true positives are zero, false positives are zero and false negatives are 10. F1 is zero because its denominator is 10, not zero.
Macro F1 is ((18 / 19) + 0) / 2 = 9 / 19, approximately 0.4737. Weighted F1 is (90 x (18 / 19) + 10 x 0) / 100 = 81 / 95, approximately 0.8526. The underlying class-B failure remains visible only when the class results are retained.
Accuracy against this reference is 90 / 100 = 0.90. It is a different metric from either F1 average. These synthetic values demonstrate arithmetic, not an acceptance threshold or a prediction of production quality.
Which fields should a buyer request?
| Review field | Evidence to retain |
|---|---|
| Reference basis | Independent review method, ontology revision and provenance |
| Sampling | Population, selected item IDs and class distribution |
| Counts | Confusion matrix, per-class support and missing-label treatment |
| Metric settings | Included classes, averaging, weights and undefined-score policy |
| Class results | Per-class F1 beside the aggregate and relevant error examples |
| Acceptance | Task-specific agreed criteria and reviewer disposition |
What changes when a class is absent?
If a class is absent from both reference and submitted labels, its F1 denominator is zero. The documented zero_division setting can affect its treatment and the resulting average. Preserve that policy and the included labels rather than silently drop a class or change an undefined score to improve the result.
The class-B score in the example is not that undefined case: reference-B items exist and were all missed. Macro averaging also does not repair a poor reference or an unrepresentative sample. Do not compare two vendor averages without reconciling class sets, sampling and metric settings.
Frequently asked questions
Does weighted F1 give every class equal weight?
No. It weights class F1 by reference support under the stated definition.
Is the zero class-B F1 undefined in this example?
No. There are 10 false negatives, so the F1 denominator is nonzero.
Can this example define an acceptance threshold?
No. Agree the metric and acceptance basis for the actual annotation task.
Keep the metric evidence with the pilot acceptance register. The kappa and agreement example addresses a different question: agreement between annotators rather than these predictions against a reference. Browse the company directory for vendor research and ask providers for their actual evaluation method.