annotation quality

Read 3 articles about annotation quality.

• Annotation Dataset Splits: Keep Related Assets Out of Both Train and Test

  A delivery can have unique file IDs and still mix related source material across evaluation splits. Check the approved grouping basis.

• Macro vs Weighted F1: Check Class Balance in an Annotation Pilot

  A worked annotation-quality worksheet showing how a high support-weighted score can coexist with zero F1 for a minority class.

• Annotation Agreement: Why the Same Agreement Rate Can Produce Different Kappa

  An original two-annotator contingency-table worksheet for observed agreement, class marginals, expected agreement and unweighted Cohen kappa.