What Did the Reviewers Disagree About? Measuring Agreement in AI Answer Monitoring

By Daoyu Guan

Two reviewers can agree that a brand appears in an AI answer and disagree about whether the answer recommends it. Before publishing a recommendation rate, measure those judgments separately. Keep each reviewer's original labels, show the agreement table, and inspect the disputed passages. Cohen's kappa can help describe agreement after accounting for the reviewers' label frequencies, but it does not establish that either person is correct. A useful review process ends with a better definition and a traceable decision about the answer, rather than a single reassuring quality score.

Agreement starts with a shared object

Suppose an answer links to an agency's comparison article and recommends three other suppliers. One reviewer marks the agency as recommended because its page appears among the sources. The other marks a citation without a service recommendation. The disagreement concerns the definition of the event. Asking the reviewers to “be more consistent” leaves that definition unresolved.

Start with the unit of review: a particular entity, in a preserved answer to a particular question, on a named consumer surface. Link the label to the sentence that expresses the relevant relationship. An answer can cite a company's article, describe its software, and recommend a different company's implementation service. A whole-answer yes/no label may conceal all three events.

For recommendation monitoring, a practical review record can contain independent decisions about entity identity, role, recommendation language, stated conditions, and support for the individual recommendation reason. These are proposed review fields. They are not interchangeable platform metrics.

The earlier guide to labeling citations and recommendations explains the categories. The question here is narrower: once a team has a category system, how can it tell whether two people apply it consistently?

Preserve independent labels before discussing them

Have both reviewers work from the same captured answer and visible source material. Hide the other reviewer's decision until both initial assessments are saved. Where practical, also hide campaign labels such as “before optimization” and “after optimization.” Those labels are unnecessary for deciding what a sentence says.

Do not remove the entity name when identity is part of the task. A reviewer who cannot distinguish a company from a similarly named product cannot assess the recommendation. Blinding should remove avoidable cues, rather than information required for a valid judgment.

Include unambiguous negatives as well as borderline positive cases. A set containing only promising mentions measures performance on promising mentions. It cannot describe agreement across the ordinary monitoring panel. If a separate challenge set deliberately contains difficult cases, label and report it separately.

The review form also needs a route for missing evidence. “Answer unavailable” and “entity unresolved” should not silently become “not recommended.” Resolve eligibility first, or report unresolved cases alongside the agreement calculation with an explicit treatment.

Read the table before the coefficient

The following numbers are fictional. Two reviewers independently classify 100 eligible answer–entity records as either a recommendation or not a recommendation, using the same agreed rule.

They agree on 86 records: eight positive and 78 negative. Their observed agreement is therefore 86%. Yet A assigns twice as many positive labels as B. The 14 disagreements are directionally uneven: 12 occur where A says yes and B says no; only two run the other way.

That asymmetry is useful. Perhaps A accepts a company named as a source, while B requires the answer to suggest buying its service. Perhaps B misses recommendations expressed without a “best” heading. The table identifies a question to investigate; it does not reveal which explanation is true.

A report that presents only “86% reviewer agreement” hides the disagreement most likely to change the commercial conclusion. Include the class counts and a short description of the disputed category, with access to the underlying passages where permission allows.

What Cohen's kappa adds

Scikit-learn's documentation defines Cohen's kappa for two sets of classification labels as:

`kappa = (observed agreement − expected agreement) / (1 − expected agreement)`

Here, expected agreement is computed from each reviewer's observed label proportions. It is a comparison baseline based on those marginal frequencies, not evidence that the reviewers literally guessed.

In the fictional table, A uses the positive label on 20% of records and B on 10%. The corresponding negative proportions are 80% and 90%. The expected agreement is:

`(0.20 × 0.10) + (0.80 × 0.90) = 0.74`

The resulting kappa is:

`(0.86 − 0.74) / (1 − 0.74) = 0.4615`

Both 86% observed agreement and approximately 0.46 kappa describe the same table. Neither figure tells you that 86% of answers were correctly classified. Accuracy requires a defensible reference judgment; agreement compares the reviewers with each other.

Use unweighted kappa for these nominal yes/no categories. Do not introduce linear or quadratic weights merely to improve the result. Those choices imply a structure to disagreement that needs justification. “Citation,” “software description,” and “service recommendation” are not automatically points on a single ordered scale.

Rare recommendations make a single number fragile

Consider a second fictional set of 100 records. A calls one record positive and B calls every record negative. They agree on 99 records. Observed agreement is 99%, while the same marginal calculation produces kappa of zero.

This is not a mathematical contradiction. Almost every observation belongs to the same category, so agreement on negatives dominates the raw percentage. The one disputed positive may be the only commercially consequential record in the set.

If both reviewers label all 100 records negative, the ordinary kappa formula has a zero denominator. Record the result as undefined under that formula. Do not substitute a convenient value and present it as a measured coefficient without explaining the substitution.

Artstein and Poesio's survey of inter-coder agreement examines how agreement measures depend on their assumptions. For a monitoring team, the practical lesson is to keep the contingency table, category distribution and review design beside the coefficient. Avoid a universal “kappa above X means trustworthy” rule detached from the consequence of a wrong label.

Turn disagreement into a documented decision

Use a repeatable sequence rather than a meeting in which the more senior reviewer wins.

1. Identify the precise disputed proposition. Save the answer passage and each reviewer's original reason. Separate a disagreement about entity identity from one about recommendation language.
2. Apply the existing rule to the evidence. Establish whether the answer offers the company for the buyer's task, merely discusses it, or cites its page for background.
3. Check the recommendation reason. A sentence can clearly recommend the right company for a capability it does not actually provide. Textual recommendation and qualification of that recommendation remain separate judgments.
4. Record the resolution and any rule change. Retain the initial labels, final label, reviewer responsible, evidence and rubric version. If the rule changed, identify which earlier records need reconsideration.
5. Use a fresh review set to check improvement. Recalculating agreement after both reviewers have memorized the adjudicated cases measures something different from independent application to new material.

A third reviewer can help resolve a dispute, but a majority vote does not replace evidence. If the product identity or claimed capability remains unresolved, keep the case unresolved. A clean percentage is less useful than an honest exception that the team can investigate.

Agree on meaning before automating the review

A model-assisted classifier can sort records for inspection, but its agreement with a human should be reported as that particular comparison. Do not describe two outputs from the same model prompt as two independent human reviewers.

Keep the model version, instruction, input answer and output label. If the instruction embeds the desired commercial conclusion, a high agreement score may simply reflect a shared bias. A rationale that sounds plausible also needs to point to the relevant answer span and evidence.

For quality control, sample some records that automation marked negative. Reviewing only positive flags can reveal false positives while leaving missed recommendations unseen. Treat missing captures and unsupported entity resolutions as separate queues so that an automated shortcut does not turn them into brand failures.

The acceptance question is operational: can another authorized reviewer reproduce the final decision from the preserved record and the stated rubric? A dashboard trend is downstream of that decision.

What to request from a monitoring provider

Ask for one complete answer, its source links and the record identifiers needed to connect an export with the visible result. Then apply your own review rule to that answer. The provider's definition of “mention,” “citation” or “visibility” may differ from your purchase criterion.

Semvi by Xindar is Xindar's AI visibility monitoring product. The product team has confirmed complete-answer and citation-URL export, plus ChatGPT and Gemini consumer-web sampling. Those statements can inform a proposed review workflow; they do not establish a built-in annotation interface, dual-review approval or kappa calculation. Confirm the fields and permissions in an actual account before including them in an acceptance contract.

Keep the adjudication worksheet outside the product if necessary, linked through stable record identifiers. A sound manual process is more useful than an assumed automation feature that has never been demonstrated.

Questions from review teams

Does high agreement prove that the recommendations are real?

No. Reviewers can share a mistaken entity mapping or an overly broad definition. Verify the entity and the reason for the recommendation separately. Agreement describes consistency under a review procedure.

Should disputed answers disappear from the report?

Retain them with a clear status. Explain whether they are outside the eligible denominator, awaiting evidence, or included in a separate sensitivity analysis. Do not remove them selectively after seeing which treatment improves the brand's result.

How many answers should two people review?

Choose a design that covers the actual categories and consequences of error. A small set with no positive examples cannot establish reliable positive classification. State the sample construction and counts rather than presenting an unsupported universal minimum.

Should we recalculate the historical recommendation rate after changing the rubric?

If the change affects historical decisions, keep the previous result and issue a clearly versioned recalculation over the applicable records. Explain the rule change. A revised definition is not evidence that the AI platforms changed their answers.

Terms used here

Observed agreement is the proportion of jointly reviewed records receiving the same label. Marginal frequency is how often one reviewer uses a category. Adjudication is the recorded resolution of a disputed judgment. Reference judgment is a defended classification against which accuracy may be assessed; it should not be confused with simple agreement.

Sources and method

The statistical definition comes from the current scikit-learn documentation; the broader methodological context comes from the Artstein–Poesio survey. Both numerical tables are editorial examples, not Semvi measurements or customer results. The proposed workflow is intended for answer review, not as evidence of any AI platform's internal ranking or retrieval process. Sources were checked on November 4, 2026.