Sources reviewed November 15, 2026. Field-record case checked November 17, 2026.
A team asks five buying questions ten times each. Its brand appears in ten of the fifty answers. The dashboard reports a 20% mention rate.
Before attaching an uncertainty interval to that number, open the records by question. All ten mentions could belong to one question. They could also be spread evenly across all five. Those two patterns call for different investigations, even though the headline fraction is identical.
Repeated answers share a question, an observation period and often an account or search environment. A statistical analysis needs to represent the dependencies that its design creates. Counting rows is the first bookkeeping step. Understanding what those rows can support requires a model of how they were obtained.
Start with the unit you want to describe
An AI visibility study can estimate several different quantities. A team might want the observed mention rate for a fixed buying question, the average across its chosen question panel, or an expectation over a wider population of possible buyer questions.
Write that target before choosing an uncertainty calculation. Ten repeats of each fixed question provide information about those selected questions under the recorded conditions. Generalising to a broader buyer population requires a defensible connection between that population and the question-selection process.
A panel assembled by a marketing team can still support operational decisions. Its membership, weighting and observation conditions need to remain visible. A question that matters to a small, valuable customer segment may deserve a deliberately different weight. Explain that choice and retain the unweighted counts alongside it.
Record a brand mention separately from a citation or a qualified recommendation. The statistical unit and the outcome definition both affect the meaning of the resulting percentage.
Fifty rows can contain very different patterns
Consider two fictional records. Every question has ten completed answers, collected under a stated protocol. A positive outcome means the same predeclared event in both records.
| Question | Answers | Positive outcomes in record A | Positive outcomes in record B |
|---|---|---|---|
| Q1 | 10 | 10 | 2 |
| Q2 | 10 | 0 | 2 |
| Q3 | 10 | 0 | 2 |
| Q4 | 10 | 0 | 2 |
| Q5 | 10 | 0 | 2 |
| Total | 50 | 10 | 10 |
Record A directs attention to Q1. Read its wording, buyer intent, sources and recommendation reasons. Perhaps it represents a narrow service requirement that the brand meets particularly well. Perhaps the collection conditions changed during that question's runs. The raw answers help distinguish those possibilities.
Record B shows a more even distribution across this selected panel. Dates and search surfaces still matter. Two positives per question could conceal a common day when the environment changed for every question.
Both observed fractions remain 10/50. These small constructed tables provide a teaching example of concentration. A correlation estimate or population-level uncertainty claim would require an explicit estimation method and an adequate design.
A published panel shows why the counts need names
Xindar's publisher-conducted field record provides a concrete example. Its November 16, 2026 snapshot contains repeated observations of twelve fixed questions. The released row projection retains eligibility, outcome and reviewed purchasing-intent labels. The accompanying recounting instructions make the published arithmetic inspectable. Published field record and data
| Quantity in that snapshot | Count | What the count describes |
|---|---|---|
| Recorded events | 233 | The released event ledger, including excluded collections |
| Eligible ordinary consumer answers | 210 | Completed answers admitted under the recorded eligibility rules |
| Qualified company/service recommendation observations | 12 | Individual observations whose target entity and recommendation reason received the publisher's review |
| Literal questions with those observations | 7 | Distinct fixed question texts with at least one qualified observation |
| Reviewed purchasing intents represented by those questions | 1 | The service-provider shortlisting group assigned through semantic review |
| Software-oriented eligible answers | 67 | The answers attached to the four monitoring-software questions |
| Qualified observations in that software subset | 0 | The recorded result for the declared target in that subset |
To reproduce these quantities, read the published columns with their definitions. Filter eligible rows, count qualified observations and then count their distinct question IDs and reviewed intent labels. Identify the software subset through the question registry. Keep excluded events visible in the ledger accounting.
The seven question texts address related ways of procuring a GEO or B2B marketing partner. Their semantic review asks whether the purchasing decisions add genuinely distinct coverage. An additional positive answer to one of those questions can improve the observation record while leaving that coverage count at one.
Statistical dependency requires another analysis. A purchasing-intent label expresses a decision about meaning. A variance calculation depends on the assumed relationships among observations, including shared questions, dates, surfaces and account contexts. Estimating a correlation or uncertainty interval requires a stated model and method.
Use the dated snapshot when repeating the arithmetic. Record later collections in a dated version, then recalculate whichever counts their eligibility and outcomes affect. Recounting the released labels checks their arithmetic. Evaluating the underlying answers and the publisher's eligibility or semantic decisions requires their supporting evidence.
A balanced model makes the variance effect visible
Suppose a hypothetical study contains K independent clusters. Each cluster contains m observations. Every observation has variance σ², and every pair within a cluster has the same correlation ρ. Covariance across clusters is zero. These are explicit modelling assumptions.
The total observation count is n = Km. In the variance of their mean, there are n individual variance terms. There are also K × m × (m − 1) ordered within-cluster covariance terms. Each of those contributes ρσ² before division by n².
The result is
`Var(mean) = (σ² / n) × [1 + (m − 1)ρ]`
The multiplier `D = 1 + (m − 1)ρ` is the design effect for this balanced model. The ECDC sample-size guidance gives the corresponding equal-cluster expression in its cluster-sampling discussion. Its surveillance context supplies a mathematical reference. Applying a model to AI observations requires a separate justification of the assumptions. ECDC sample-size guidance, section 4.2
Use fifty observations arranged as five clusters of ten to see the arithmetic.
| Assumed within-cluster correlation | Design effect | Variance-equivalent observation count n/D |
|---|---|---|
| 0 | 1 | 50 |
| 0.2 | 2.8 | About 17.86 |
| 1 | 10 | 5 |
At ρ = 0.2, the calculation is `1 + 9 × 0.2 = 2.8`, followed by `50 / 2.8 ≈ 17.86`. The value of ρ here is a chosen teaching input. These calculations fit no actual AI answers.
Keep all fifty observations in the published accounting. A variance-equivalent count describes precision under the stated model. The observed outcome fraction still uses its actual denominator. A report can therefore show fifty completed answers and explain the model's variance multiplier in a separate methodological field.
Cluster sizes, variances and pairwise correlations often differ in practice. The balanced formula is useful for learning why dependency matters. A real analysis needs a method appropriate to its actual design.
The question may be only one dependency dimension
Repeated answers can share several conditions at once.
| Shared condition | Possible dependency to investigate | Record to retain |
|---|---|---|
| Question version | Stable wording, entities and buyer intent | Exact text and version hash |
| Collection date or time block | Common changes in accessible material or service behaviour | Actual submission and capture times |
| Platform and search surface | Shared product behaviour and source availability | Consumer surface, mode and visible model |
| Account and conversation context | Shared settings or retained conversational context | Permitted account-context identifier and fresh-session evidence |
| Collection workflow | Common completion or extraction errors | Completion record, failures and original answer |
These are candidate dependencies to investigate. Their presence and magnitude require evidence. A fresh conversation establishes a recorded session condition. Statistical independence requires its own design or modelling argument.
The regression literature offers an important caution. Cameron and Miller explain that cluster-robust inference allows dependence within clusters while relying on independence across them. They discuss complications including few clusters and multiway clustering. Choosing a statistical command therefore requires decisions about which observations can reasonably be treated as independent. Cameron and Miller, *A Practitioner's Guide to Cluster-Robust Inference*
For example, clustering only by question deserves scrutiny when all questions were sampled in a common daily block. Clustering only by day deserves scrutiny when stable question-level differences dominate the records. A question-by-day or crossed structure may call for another approach. Have an analyst justify that structure against the collection design.
Decide what another sampling run would add
The choice between another repeat and another buyer question follows from the study's purpose.
If the decision concerns the stability of one important question, more controlled repeats can be informative. If the decision concerns a broader set of purchasing needs, adding a genuinely distinct question can extend coverage. Establishing that distinction requires reading the questions and their buying decisions. Small wording changes can preserve the same underlying intent.
Schedule changes prospectively. Record which questions and surfaces were eligible before looking for a favourable outcome. Preserve failed collections and missing answers with their reasons. A completed-answer denominator can be accompanied by the submitted and eligible totals so readers can see the collection losses.
For comparisons over time, retain a fixed component of the panel and identify additions separately. A new set of questions changes the composition of the average. Showing question-level counts helps readers inspect whether an apparent shift follows from changed answers, changed weights or changed membership.
An analyst who uses a resampling procedure should explain its resampling unit. Resampling individual answer rows makes a different assumption from resampling independent clusters. The number of independent units and the method's finite-sample behaviour also need attention. A universal minimum-cluster rule would need support for the particular method and design.
Ask a monitoring supplier for the records behind the calculation
Request a demonstration that follows one answer from its question and surface through capture, outcome annotation and export. Then ask how repeated observations enter the reported uncertainty.
An acceptance sheet can contain these fields.
| Acceptance item | Review action |
|---|---|
| Outcome definition | Apply it to a retained positive and a retained negative answer |
| Question identity | Match exact text, version and intended purchasing decision |
| Observation identity | Match timestamps, platform, surface and completion status |
| Cluster assumptions | Identify the proposed independent units and shared conditions |
| Report arithmetic | Reproduce the actual counts and weights from exported records |
| Inferential method | Document the target quantity, model, limitations and analysis owner |
Semvi by Xindar is relevant to the evidence-collection part of this workflow. The product owner has confirmed full-answer and citation-URL export, together with ChatGPT and Gemini consumer-web sampling. The required record fields and their availability in a specific account remain demonstration and contract acceptance items. A supplier's statistical analysis capability deserves its own documented verification. Semvi by Xindar product scope
Questions buyers commonly ask
Can we keep reporting the raw mention percentage?
Yes. Give its actual numerator, denominator, outcome definition and panel scope. Add question-level detail when a pooled percentage hides concentration. Any uncertainty interval needs its method and dependency assumptions.
Does more repetition always improve the study?
The gain depends on the target decision and the dependence structure. More repeats can clarify one question's behaviour. A broader purchasing claim also needs a defensible range of questions and intentions.
Can we estimate ρ from the two tables above?
An estimate requires a specified model and estimation procedure. The tables are fictional concentration examples. The design-effect table uses separately declared hypothetical correlations.
What should the team approve before the next run?
Approve the question versions, outcome definitions, eligible surfaces, collection schedule, treatment of missing records and analysis plan. Keep those decisions with the resulting answers so a later reader can reconstruct the study.
