Annotation Dataset Splits: Keep Related Assets Out of Both Train and Test
· Annotation Quality
A delivery can have unique file IDs and still mix related source material across evaluation splits. Check the approved grouping basis.
Unique annotation asset IDs do not prove that training and test sets are independent. Declare the relevant source group and check that related assets stay within the approved split boundary before treating a delivery manifest as evaluation ready.
This guide is for annotation buyers coordinating dataset handover. It does not describe job opportunities, hiring status or pay, and it provides no vendor ranking. Data Annotation Companies is independent and is not affiliated with scikit-learn.
What does a group-based split actually control?
The official scikit-learn GroupShuffleSplit documentation describes splitting using supplied group labels. Its train_size and test_size parameters refer to groups, not sample counts. It also warns that test sets from separate random repetitions can overlap with each other.
The buyer must define what related means for the evaluation objective. A group might be a source video, original document or collection session. A software splitter cannot discover a missing relationship or guarantee that the chosen grouping captures every leakage route. Group isolation and class stratification are separate requirements.
How can unique frames still cross the split boundary?
Illustrative example. A fictional annotation delivery contains four distinct frames from video A, four from video B and four from video C. All 12 frame IDs are unique. The approved demonstration policy treats each video as one group and requires a held-out video for testing. This is a small arithmetic example, not a recommended evaluation sample size.
| Source group | Frame IDs | Row-based assignment | Group-based assignment |
|---|---|---|---|
| Video A | A1, A2, A3, A4 | A1 to A3 train; A4 test | All train |
| Video B | B1, B2, B3, B4 | B1 to B3 train; B4 test | All train |
| Video C | C1, C2, C3, C4 | C1 to C3 train; C4 test | All test |
The row-based assignment has nine training frames and three test frames, but all three video groups appear on both sides. Under the stated policy it fails the group-overlap check despite unique frame IDs. The group-based assignment has eight training frames and four test frames, with zero video groups shared between those two splits.
Holding out one of three equal-size groups gives one third of these frames, not an arbitrary 25 percent frame target. Unequal group sizes can produce a different sample fraction. Do not move individual frames between splits just to make the percentages look tidy while breaking the agreed group boundary.
Which handover fields make the split reproducible?
| Manifest field | Purpose | Review observation |
|---|---|---|
| Asset ID | Identify each expected annotated item | Missing, duplicate or unexpected IDs |
| Source group ID | Link related assets under the approved policy | Missing group or conflicting mappings |
| Split and version | Record the actual train, validation or test assignment | Each group appears only where policy permits |
| Parent or derivative link | Trace crops, frames and other derived items | Derivatives preserve the intended boundary |
| Counts | Report both group and sample totals | Actual fractions are disclosed separately |
| Assignment basis | Record method, configuration and saved assignment | Another reviewer can reproduce the intended split |
| Disposition | Preserve buyer approval and unresolved exceptions | A manifest check is not a model-performance guarantee |
What does a passed group check leave unresolved?
A zero intersection between the declared train and test group IDs checks only that declared grouping. Related videos, duplicate source documents or other relationships may require a broader grouping decision. The model-evaluation owner must also decide whether time, collection location or another separation is necessary.
Review class coverage, task definitions and annotation quality independently. This worksheet does not establish a representative test population, a reliable reference label or a production acceptance threshold. Keep the held-out assignment controlled and follow the evaluation owner's access policy so test material does not silently become training material.
Frequently asked questions
Do unique filenames prevent evaluation leakage?
No. Distinct files can come from the same related source group.
Does a group fraction equal the same fraction of samples?
Not necessarily. Groups can contain different numbers of samples.
Does zero declared group overlap prove model quality?
No. It verifies one specified boundary, leaving sampling, annotation and evaluation checks separate.
Keep source asset IDs and derivative links in the controlled delivery manifest, then add this explicit grouping and assignment basis. For buyer research, browse the company directory and ask providers for their actual handover and quality methods. Preserve the agreed checks in the pilot acceptance register.