Why Can Line-Ending Changes Break NER Text Offsets?
· Data Quality
A text delivery can contain valid offsets that extract the wrong entity after line-ending conversion. Keep the exact annotated text and offset basis together.
Changing line endings after text annotation can shift named entity recognition offsets. Preserve the exact text version that the offsets refer to, and validate that each stored span still extracts the intended text before accepting a delivery.
This guide is for annotation buyers and delivery teams. It is not a work opportunity, hiring announcement or worker earnings claim. Data Annotation Companies is independent and is not affiliated with HumanSignal or Label Studio.
What does the tool documentation say?
The official Label Studio Text tag documentation, checked on 23 October 2026, states that its editor counts a carriage-return plus line-feed sequence as two symbols. It warns that line-ending conversion matters when using offsets to slice the text for span NER labeling.
The documentation discusses normalizing before labeling or preserving the original line endings during reading. Those are controlled workflow choices. Normalizing an already annotated source without reconciling its offsets is a different operation and can invalidate the span references.
How can one removed character change the selected entity?
Illustrative example. Use the fictional ASCII text A followed by carriage return, line feed and pump, represented as A\r\npump. With zero-based indexing and an exclusive end, pump occupies [3, 7). If the line ending is normalized first, the new text A\npump places pump at [2, 6).
Applying the old slice [3, 7) to that shorter text yields ump, not pump. A file can retain valid numeric start and end fields while pointing at the wrong text. This example is deliberately ASCII; it does not establish one universal indexing convention for Unicode, JavaScript, Python or every annotation exporter.
| Text version | Span check |
|---|---|
| Original CRLF text | [3, 7) extracts pump |
| Normalized LF text | [2, 6) extracts pump |
| Old offsets on LF text | [3, 7) extracts ump |
What should the acceptance record preserve?
Agree the text and offset contract before delivery. Validate representative records with the actual receiving toolchain, including boundary cases and non-ASCII examples appropriate to the dataset.
| Record | Evidence to retain |
|---|---|
| Source | Exact file or text version and identity |
| Processing | Approved transformations and their timing |
| Line endings | CRLF, LF or other actual sequence |
| Offset basis | Index unit, origin and end convention |
| Decoded span | Actual extracted text and expected entity |
| Disposition | Reviewer, mismatch and approved correction |
Keep the received export and original text before creating a processed derivative. A checksum can identify a version, but it does not explain the offset convention or prove that the entities are correct. Link transformation history and span checks to that version.
What should happen when the source and offsets disagree?
Record the mismatch and ask the delivery owner which text version was annotated. Reconcile it against the agreed contract before an approved correction. Do not silently subtract one from every offset: multiple line breaks, earlier transformations and different indexing units can require different treatment.
A corrected span extraction is only one acceptance check. Entity class, boundary policy and review provenance still matter. The NER labeling overview covers the wider task; the export provenance review addresses saved-result status separately.
Can line endings be normalized safely?
Yes, through an agreed process that keeps the annotated text and offsets consistent. Do not assume a post-labeling text change preserves existing offsets.
Do valid JSON numbers prove the span is right?
No. Extract the actual text using the receiving system's documented convention.
Does the ASCII example settle every Unicode offset question?
No. Verify the actual indexing contract and appropriate non-ASCII boundary cases with the tools used.
Ask suppliers for a reproducible text handoff
Use the company directory to research providers, then request the proposed text version, transformations, offset convention and reviewed sample. A directory listing does not independently prove an export meets your acceptance criteria.