Annotation Pilot Acceptance Register: Denominators, Unresolved Items and Rework
· Data Annotation
A buyer worksheet for fixing annotation pilot acceptance rules before delivery and retaining denominators, unresolved items and rework versions.
Agree the annotation pilot's acceptance rules before delivery, then preserve the original results and any rework as separate versions. A quality percentage is not interpretable without its denominator, reference basis and treatment of unresolved items.
This guide is for buyers evaluating annotation services. It is not a work opportunity, hiring announcement or annotator application guide. The original acceptance register below helps compare a delivered pilot with the rules the buyer and provider actually agreed.
What should be frozen before the pilot starts?
Record the task definition, dataset version, labeling guideline revision and required output format. Specify which items are in the acceptance sample and how missing, invalid, ambiguous and abstained items will be handled. Assign an authorized owner to decide unresolved cases rather than allowing the calculation to make that decision silently.
AWS responsible AI guidance on annotation quality discusses annotator training, agreement checks, known-correct examples and feedback on difficult cases. Those practices support defining the review evidence. AWS does not supply the acceptance thresholds or the original register on this page. Data Annotation Companies is independent and is not affiliated with AWS.
Which fields make the acceptance result reproducible?
Create the register before inspecting the outcome. Changing the denominator after seeing the score can make two deliveries appear comparable when they were judged under different rules.
| Field | Agreed definition | Evidence retained |
|---|---|---|
| Dataset and sample | Version, selected items and selection method | Item IDs under the approved access controls |
| Unit of assessment | Item, label, span or another defined unit | Calculation uses the same unit throughout |
| Reference basis | Approved reference or adjudication process | Reference version and decision owner |
| Metric definition | Numerator, denominator and allowed exclusions | Counts beside the percentage |
| Unresolved items | Ambiguous, missing or abstained treatment | Separate issue count and disposition |
| Acceptance rule | Buyer-approved threshold or decision method | Agreement recorded before results are reviewed |
| Rework | Version and affected item set | Original result and revised result kept separately |
How can denominator choices change the reported result?
Illustrative example. A synthetic pilot contains 100 single-label items. Against the approved reference, 86 match, 10 differ and 4 remain unresolved. If the agreed measure includes every delivered item, the observed match result is 86 / 100 = 86%, with four unresolved items still reported. If a separately agreed measure excludes those four pending items, it is 86 / 96, approximately 89.6%. These are different measures, not interchangeable descriptions of the same acceptance result.
The example sets no recommended threshold and gives no provider performance claim. Retain the counts and the pre-agreed treatment. Do not select the larger percentage after seeing the result or count unresolved items as correct without a disposition.
For multi-label or span tasks, define the unit and matching rule explicitly. The single-label arithmetic above cannot simply be reused for a task where one item contains several decisions.
Why should agreement and reference matching be separate?
Annotator agreement describes consistency between judgments. Matching an approved reference describes a different comparison. Two annotators can make the same incorrect judgment. Record both measures where relevant and state how the reference was established.
Keep the reference review independent of the result being assessed where the agreed process requires it. Record adjudicated changes with their reasons. Do not silently modify a reference label to make a provider's original output appear correct.
How should rework be accepted?
Preserve delivery version 1, its metric definitions, counts and unresolved issues. Record which items were reworked and whether the guideline changed. If the rules or sample change, explain why the revised score cannot be treated as an unchanged repeat of the original assessment.
Close the pilot with the authorized decision, remaining limitations and the version approved for use. A passed pilot under a defined sample is not a guarantee that every production item will meet the same result. Keep production monitoring and future acceptance arrangements explicit.
Use the annotation services buyer page to explore providers, then ask each candidate to respond to the same pilot scope and acceptance register. Verify current capabilities directly with the provider; this worksheet does not rank companies or certify their quality.
Frequently asked questions
Is annotator agreement the same as accuracy?
No. Agreement compares judgments with each other; reference matching uses a separately defined reference. State which measure is reported.
Can unresolved items be removed after the score is known?
Use the pre-agreed rule and report the unresolved count. A changed rule needs a recorded decision and a clearly different calculation.
Should a revised delivery overwrite the original pilot score?
No. Retain the original and revised versions with their counts, definitions and rework history.
Does this page recommend a universal acceptance threshold?
No. The buyer and provider must define requirements suitable for the task and intended use before assessing the pilot.