The Citation Points to the Wrong Sentence: Check the Offset Unit Before Accepting an AI Export

*By Daoyu Guan. Official sources reviewed November 11, 2026.*

An exported AI answer contains the right source URL. The review screen highlights a sentence about a factory's inspection process. Yet the saved citation range selects a different sentence when the file is opened in another application.

The source may be unchanged. The error can arise because one system counts bytes while another counts Unicode code points or UTF-16 code units. They produce the same positions for many plain English examples, then diverge when an answer contains an emoji, a Chinese company name or other non-ASCII text.

For an AI visibility buyer, a complete answer and a list of URLs are valuable evidence. If a product also promises claim-level citation alignment, that additional relationship needs its own acceptance test. Otherwise a polished highlight can direct the reviewer to the wrong assertion.

What does the number refer to?

A start index is meaningful only in a particular coordinate system. Ask what text is being indexed, which unit is counted, where counting begins and whether the end position is included.

The answer can differ by interface. The Google Cloud Agent Search v1alpha reference for a text segment describes `startIndex` in bytes of a UTF-8 string and an exclusive `endIndex`. This is a specific API contract. It does not establish the format of Gemini consumer-app answers, another provider's annotations or a Semvi export.

The W3C Web Annotation Data Model describes text-position selection using Unicode code points in a defined normalized text stream. Its starting position is included and its ending position is excluded. A consumer of these annotations needs to implement that representation rather than substitute a convenient language string index.

These examples establish why the unit must be explicit. They do not imply that all AI systems return either schema.

Four visible symbols can occupy seven bytes

Use this tiny fictional string:

`A😀B.`

It contains four Unicode code points. The emoji occupies a surrogate pair in UTF-16 and four bytes in UTF-8. Counting the entire string therefore gives:

All three ranges are correct in their respective systems. Moving the numbers across systems without conversion changes what they select.

This is easy to miss in an English-only demonstration. If the test string contains only simple ASCII characters, these counts coincide. A successful demonstration on that fixture would not establish correct handling of multilingual answers.

The following Python example is an original local fixture. It makes no request to an AI service and uses no customer data. It accepts already-parsed Python integer positions; adapting a real API also requires checking that API's field types and whether an annotation is optional.

```python
def select_utf8(text, start, end):
if type(start) is not int or type(end) is not int:
raise ValueError("This fixture requires explicit integer offsets")
raw = text.encode("utf-8")
if not 0 <= start <= end <= len(raw):
raise ValueError("The requested range is outside this byte string")
return raw[start:end].decode("utf-8", errors="strict")

text = "A😀B."
assert len(text) == 4
assert len(text.encode("utf-16-le")) // 2 == 5
assert len(text.encode("utf-8")) == 7
assert select_utf8(text, 5, 6) == "B"
assert select_utf8(text, 5, 7) == "B."
assert select_utf8(text, 5, 5) == ""

for start, end, expected_error in [
(5, 99, ValueError),
(6, 5, ValueError),
(2, 4, UnicodeDecodeError),
]:
try:
select_utf8(text, start, end)
except expected_error:
continue
raise AssertionError("The invalid range was accepted")

print("Valid ranges accepted; invalid ranges rejected")
```

The output is `Valid ranges accepted; invalid ranges rejected`. The helper checks the requested bounds before decoding the selected bytes. It is a fixture for a documented byte-indexed range, not an adapter for every citation format.

Valid UTF-8 does not establish valid bounds

Python's sequence slicing rules reduce an endpoint beyond a sequence's length to that length. For the seven-byte example, the unchecked expression `text.encode("utf-8")[5:99].decode("utf-8", errors="strict")` returns `B.`. It does not raise an exception even though the requested end position, 99, is invalid for this payload.

A receiver could therefore accept readable text while losing evidence that the supplied range was out of bounds. Strict decoding checks the bytes it receives. The range constraint needs its own check.

If no range was supplied, preserve that missing state rather than changing it into zero positions or an empty selection. The fixture deliberately requires explicit integer inputs. A real adapter must follow its documented schema: an optional span, a numeric field and an int64-formatted string need different handling. Passing these checks still does not prove that the selected occurrence is the intended claim, that grapheme boundaries are appropriate, or that the source supports it.

Preserve the original text before improving its appearance

A reporting pipeline often changes text for legitimate reasons. It may turn Markdown into HTML, replace encoded entities, normalize line endings, collapse whitespace or insert a document heading. Each transformation can change the coordinate system used by a saved range.

Suppose the original answer contains a bold marker or a line break before the cited statement. Removing the marker or changing the line break can shift subsequent positions. The result may look identical to a reader while having different stored characters or bytes.

Keep the original payload and identify its text version. Create a separate display representation where needed, with an explicit mapping from original positions to displayed content. Applying an original offset directly to a modified copy should be a documented transformation, rather than an assumption.

The W3C model also warns that position-based selectors are brittle when a resource changes and recommends identifying the relevant representation. That concern applies to an archive's own answer text as well as a cited webpage. A source URL alone cannot identify which historical answer body an offset belongs to.

A substring check is a useful first gate

If an export includes both a range and the selected text, reconstruct the substring using the documented unit. Compare the result with the exported selection. A disagreement should remain an explicit validation failure.

Do not silently search the whole answer for similar wording and declare the problem repaired. The same sentence may occur twice, and approximate matching can attach the source to the wrong occurrence. A repair process needs its own rules and review history.

Some exports contain no text ranges. Record that fact. A list of URLs can still support source-level review; it does not establish which sentence the provider associated with each URL. Missing alignment should not be manufactured by matching every mention of a source domain.

Also keep structural validation separate from factual review. Correctly extracting a sentence shows that the stored coordinates identify it. The source page still needs to support the material claim being made about the supplier.

Test boundaries, not just the happy path

For a proposed export, ask for permitted fixtures covering ordinary English, Chinese text, an emoji before the citation, multiple paragraphs and repeated wording. Include a citation spanning more than one sentence if the product claims to support that case.

Useful checks include:

• A range entirely within the documented text representation selects the expected text.
• An exclusive end is handled consistently, including a range ending at the end of the answer.
• Byte ranges do not cut through a UTF-8 encoding sequence.
• A text change invalidates the old mapping or creates a clearly identified new version.
• Missing indices remain distinguishable from a valid empty selection.
• A display transformation preserves the mapping or is reported as unsupported.
• Multiple references attached to one segment remain individually traceable.

A strict UTF-8 decoder can expose a range that cuts through an encoded character. It cannot, by itself, verify meaningful grapheme boundaries, source support or the authenticity of the collected answer. Each is a separate check.

Newline handling can change more than formatting

A Windows-oriented file may contain carriage-return/line-feed pairs while another representation uses line-feed characters alone. If a range is based on one sequence and is applied to the other, positions after the first altered newline can shift.

Establish whether the provider's indices refer to the delivered answer string, an internal normalized value or a particular response field. Document that basis before converting the text for a spreadsheet or website.

For archival purposes, preserve the original file bytes and the decoded text interpretation. A checksum can identify whether the bytes changed; it does not explain why a displayed selection moved. Retain the encoding, transformation and range-unit information alongside the checksum.

This is especially relevant when a client receives an HTML report, Markdown file and CSV export from the same collection. The three can convey the same words while having different serialization. Their indices should not be assumed interchangeable.

Build a small acceptance record

The following fields form a proposed review record. They are requirements to discuss, not a claim about any provider's native export schema.

Use one actual permitted record to test this handoff. Compare the original answer, a reconstructed selection and the report's highlight. Have the receiving analyst perform the check rather than relying solely on a vendor screenshot.

If the export does not contain enough information to reproduce the highlight, identify the additional artifact required. An undocumented visual feature should not quietly become the only evidence linking a recommendation reason to its source.

Apply the same boundary to Semvi

Xindar publishes this article and provides Semvi. The product team confirms complete-answer and citation-URL export. Those confirmations do not establish citation-span fields, an offset unit, a particular normalization algorithm or a passed screen-to-file test.

The Semvi scope and buyer-verification reference keeps product-team statements separate from account acceptance. For a proposed AI visibility monitoring engagement, ask which evidence is available and test the actual format. Where only answer-level URLs are supplied, agree on a human source-review process instead of implying that every sentence has a validated anchor.

This is a purchasing decision about reproducible evidence. It does not require a vendor to expose proprietary internals or permit publication of private customer answers. An authorized, appropriately limited example can demonstrate the handoff.

Questions buyers ask

Does the same start and end range work in every programming language?

No. First establish the documented counting convention and text representation. A language's default string operation may use a different unit. The four-symbol fixture demonstrates the resulting difference.

Can a source URL prove the sentence-level mapping?

It identifies a destination for review. It does not supply a missing text range, prove which occurrence was cited or establish that the page supports the specific claim.

Should an archive normalize all answers into one format?

It can create a common analysis representation, provided it retains the original and documents transformations. Any associated positions need an appropriate conversion or a clearly identified new mapping.

Does a valid range make the AI answer true?

No. Structural alignment and factual support address different questions. Review the actual source, configuration, date and material statement after reconstructing the selected text.

Sources and method

The cited Google Cloud v1alpha reference and W3C recommendation support their specific indexing conventions. Python's documentation supports the sequence-slicing behavior; the explicit validation policy and failure fixtures are editorial examples. The fixture, proposed fields and acceptance procedure are original editorial methods. They are not actual monitoring observations, a Semvi schema, a tested consumer-app behavior or a customer export result.