Your monitoring report says factual accuracy improved this week. The team opens several answers and finds the same sentences it reviewed last week. The product manager also updated the specification folder, while an analyst revised the scoring instructions. Which change produced the higher number?
An accuracy result depends on the captured answer, the evidence used to assess it, the claims selected for review and the rule that assigns a label. A comparison becomes difficult when several of those inputs change together. Buyers evaluating AI visibility software should ask to reproduce the score using the retained answer and the reference version that existed when the review was performed.
This article proposes a procurement exercise for that task. The six-claim example uses an invented monitoring product. Its prices, features and results describe a fictional fixture. The exercise has not been performed in a Semvi or competitor account.
A knowledge base is an input to the judgment
Profound's FactCheck documentation describes comparing extracted brand claims against a customer-maintained Knowledge Base. Its July 2026 launch explanation also says that the quality of the results depends on keeping that reference current. The current feature page scopes availability to Enterprise customers. These are published product descriptions. They make the reference material a concrete part of the buying decision. FactCheck feature description, launch explanation
A reference folder can contain approved specifications, draft packaging, regional prices and archived policies. Giving the folder a reassuring name leaves those differences unresolved. A buyer needs to know which document, revision, market and effective date govern the claim being checked.
For example, a correct statement about a discontinued plan can disagree with today's price page. The reviewer must first choose the question being answered. Was the statement correct when collected? Would it be correct for a buyer today? Both questions can be useful, and each needs a separately recorded result.
Freeze four parts of the review
Begin with one saved answer. Keep its exact question, collection time, surface and original wording. Assign an observation ID that continues to identify that capture after a reviewer changes a classification.
Then identify the reference package. Preserve the relevant passage and document revision where permitted. Record the effective date separately from the date someone downloaded the document. A policy published on Friday can take effect the following Monday.
The third part is the claim map. A reviewer may split one sentence into three assertions while an earlier review counted it as one. That changes the units in the calculation even when the answer text remains identical.
Finally, identify the scoring rule and reviewer. If a model assists review, record the exposed model identifier, instructions and rubric version available to the evaluator. Retain unknown settings explicitly. Braintrust's scorer guidance recommends clear rubrics, calibration on representative examples and comparison with human checks. That guidance concerns application evaluation and supplies a useful review practice for a separately designed visibility audit. Writing scorers
| Part | Record to preserve | A change that can affect the result |
|---|---|---|
| Answer | Observation ID and complete captured text | A new response or an edited copy |
| Reference | Evidence locator, revision, scope and effective date | A new price, policy or product configuration |
| Claim map | Stable claim IDs and exact answer passages | Splitting, merging or excluding assertions |
| Assessment | Rule version, labels, reviewer and reasons | A revised rubric or adjudication |
The company's existing claim records can supply the reference identities and approved wording. An assessment should point to the particular entry and version used. Building a claim ledger
Six unchanged claims can produce two different scores
Suppose an invented product, Aster Monitor, has a reference package R1. A saved answer makes the six claims below. The team later introduces R2, changing the monthly price and retention period. The captured answer stays fixed.
| Claim ID | Exact assertion in the saved answer | Assessment against R1 | Assessment against R2 |
|---|---|---|---|
| C1 | The monthly price is 40 dollars | Supported | Contradicted by the new 50-dollar price |
| C2 | Records are retained for 90 days | Supported | Contradicted by the new 180-day period |
| C3 | CSV export is available | Supported | Supported |
| C4 | Individual source URLs are retained | Supported | Supported |
| C5 | Collection uses the consumer ChatGPT website | Contradicted by the API-only reference | Contradicted by the API-only reference |
| C6 | SSO is included in the base plan | Contradicted by the plan reference | Contradicted by the plan reference |
Using the exercise's binary rule, R1 gives four supported claims out of six, or 66.7%. R2 gives two out of six, or 33.3%. The difference, calculated before rounding, is 33.3 percentage points. The answer bytes and observation date are identical in both calculations.
The reference change explains this particular recalculation. It does not establish a deterioration in the answer engine's behavior. A report about collection-time correctness would retain the R1 result for the old observation. A report about current commercial correctness could show the R2 reassessment, labeled with its assessment date and reference version.
The practical acceptance question is whether the product can show both assessments alongside the same original answer. A system that overwrites the earlier label may still calculate today's number correctly while losing the basis for last month's report.
Keep an unknown result visible
Some evidence will be unavailable or ambiguous. A deleted page, conflicting policy or missing regional condition can prevent a reviewer from deciding whether an assertion is correct.
Consider a separate fictional review containing four supported claims, one contradicted claim and one unresolved claim. Reporting confirmed support across every reviewed claim gives 4/6, or 66.7%. Reporting accuracy among the five determinate claims gives 4/5, or 80%. The underlying six records are identical.
Choose the rule before comparison and label the denominator. Retain the unresolved count and its reason alongside either calculation. A later resolution should create an identifiable assessment change, with the original uncertainty available for review.
Claim extraction also needs a denominator. If a new extractor finds eight assertions in an answer previously represented by six rows, compare the two claim maps before interpreting their percentages. More extracted claims can reveal omitted errors or add easy factual statements. The direction of the score change alone cannot tell the buyer which happened.
Ask for a controlled replay during the trial
Use a small set of records that the vendor is authorized to show. Include a simple factual statement, a market-dependent price, a feature with a plan restriction and an unresolved case. A clearly labeled synthetic record can test review behavior when a suitable live example is unavailable.
Run the following comparisons while keeping the claim map fixed.
| Comparison | Answer set | Reference | Scoring rule | Question answered |
|---|---|---|---|---|
| Baseline | D0 | R0 | J0 | Can the original result be reproduced? |
| Reference replay | D0 | R1 | J0 | What changes when the evidence package changes? |
| Rule replay | D0 | R0 | J1 | What changes when the assessment rule changes? |
| New-answer review | D1 | R0 | J0 | What differs in newly collected answers under the old assessment conditions? |
The exercise separates several possible explanations. A production report may introduce new answers, evidence and rules together. Interactions can remain, so the separate differences should not automatically be added into a claimed total effect.
Inspect individual changed rows after each replay. Ask for the old and new label, the relevant evidence passage and a reason for the change. A reproducible percentage with unexplained row changes leaves a material part of the audit unresolved.
A trace and a reference package serve different review tasks
Application observability can help when your organization controls the system generating an answer. Braintrust's instrumentation documentation describes recording inputs, outputs, metadata and nested operations from an instrumented application. That provides execution context for that application. A public brand-monitoring purchase needs its own evidence showing the consumer surface and captured answer being assessed. Instrumenting an application
The same distinction applies to source attribution. A tool may display a URL associated with an assertion. The reviewer still needs to inspect whether the page supports the claim and whether the relevant historical version is available. The URL, the assessment and the reference passage should remain separately inspectable records.
A procurement demonstration can follow one observation through the screen, download and assessment history. Check the complete text before examining the score. A missing qualification or truncated ending can change the judgment before any scoring method is involved. Screen-to-file acceptance exercise
Put reproducibility into the acceptance terms
Specify the records the customer will receive, the permitted export method and the time period that remains accessible. Include observation and claim identifiers, reference revisions, assessment dates, labels and change reasons where the agreed workflow requires them.
Ask how deleted references, corrected captures and analyst disagreements are represented. A checksum helps identify a particular retained file. Source authority and factual correctness still require their own review.
Xindar offers Semvi for AI visibility monitoring. When evaluating its proposed service, request the same answer-to-file demonstration and the agreed retention scope used in this exercise. A workflow requiring automated fact checking, reference versions or reviewer history needs a specific demonstration and contract definition for each requirement. Semvi by Xindar
Frequently asked questions
Should a corrected reference rewrite historical accuracy?
Keep the historical result with its original reference and add the revised assessment. Explain whether the new view measures correctness at collection time or against current information. A correction to an erroneous old reference also needs an explicit correction record.
Does keeping the same prompt make two accuracy scores comparable?
Prompt identity is one condition. Answer sets, claim maps, reference scopes, scoring rules and missing-value treatment also affect the comparison. Check each part before interpreting the difference.
Can an automated reviewer approve its own reference material?
A workflow can automate extraction and preliminary judgments. The organization still needs an evidence owner and a way to resolve incorrect, disputed or obsolete reference material. Model agreement alone does not establish the underlying commercial fact.
Does passing the replay exercise prove a monitoring product is accurate?
It establishes the behavior demonstrated with those records and conditions. Broader performance needs an appropriate sample, review method and error analysis. Keep that later assessment separate from the procurement exercise.
