Hide the Brand, Keep the Facts: Testing an AI Shortlist Without Confusing It With Search Visibility

By Daoyu Guan

A procurement team gives an AI assistant five product profiles. One supplier makes the shortlist when its name is visible and disappears when the name is replaced with a neutral label. The team has found a question worth investigating. It has not established why ordinary buyers fail to discover that supplier.

Those are different decisions. A controlled comparison can help assess whether identity labels change a recommendation made from supplied information. A visibility programme asks whether a real buyer, starting with an unbranded question, encounters a relevant supplier at all. Useful evidence connects the two without substituting one for the other.

For teams working on Generative Engine Optimization, or GEO, the distinction changes what to fix next: the product description, the evidence available to buyers, or the interpretation of a diagnostic result.

Define the question before preparing the profiles

Write a narrow question that the exercise can answer: “With this buying requirement and these product facts supplied, does showing brand identity change which products enter the shortlist?”

That question does not ask which brand has the most searchable pages, whether a particular search engine retrieves a document, or whether a monitoring platform detects a mention correctly. Those require different observations.

Choose a realistic buying requirement with explicit constraints. For an AI visibility tool, this might involve reviewing an answer and its cited URLs, separating search surfaces, and handing evidence to a second reviewer. Budget, supported markets and access conditions belong in the requirement when they matter to the buyer.

Freeze those constraints before seeing the outputs. If the criteria are rewritten after a preferred supplier loses, the resulting comparison addresses a different question.

What the October research contributes

An October 2026 preprint by Mahe Chen and Nick Koudas examines citations, retrieval and model knowledge. Its longitudinal collection uses search-enabled APIs; it is not a measurement of consumers using the ChatGPT or Gemini websites. Separate controlled experiments supply product information and vary identity labels. In the brand-and-publisher experiment, each product's facts stay fixed across conditions; the five products need not have identical capabilities. The authors report that revealing brand names can change rankings in this constructed setting. A name can also hurt a product's position. The paper does not establish that publishing more content changes a model's stored knowledge. Its models, supplied inputs and market coverage limit generalization. This is a preprint, and the method below is an editorial diagnostic proposal, not a replication or a reported Semvi test. Read the paper and Appendix D.

Build a factual packet for each candidate

Use the same fields for every candidate, but preserve meaningful differences. A product with weekly collection should not be rewritten as a daily service to make the profiles look balanced. An unavailable export format should remain unknown.

For every field, retain the source, the date inspected and what the source actually supports. A public documentation statement, an account operation you performed and a contract term are different kinds of evidence.

Separate product claims from publisher descriptions. If a third-party page describes a supplier, preserve the relevant excerpt and its context. Do not rewrite an opinion as a verified feature. If comparable evidence is missing for one candidate, disclose the gap rather than manufacturing a symmetrical comparison.

Masking should preserve those facts within each candidate's profile across versions. It should not equalize all candidates. Otherwise, the exercise removes the differences a buyer is supposed to evaluate.

Keep this material inside the diagnostic workspace. Synthetic labels and modified excerpts are experimental inputs, not pages to publish as genuine supplier descriptions or third-party endorsements.

Use four conditions when two identities matter

A product brand and the publisher discussing it are separate labels. Changing both at once prevents a clean comparison of their individual contributions.

NIST's description of full factorial designs explains the general principle: include every combination of the selected factor settings. With two factors and two settings each, there are four combinations.

For this proposed diagnostic, the conditions would be:

Compare A with B to inspect the brand-label contrast while publisher labels are masked. Compare C with D to inspect it while publishers are visible. Differences between those contrasts may justify examining an interaction; they are not automatically proof of a general interaction effect.

Assign neutral record IDs that remain consistent across conditions. These IDs let a reviewer map an output back to the correct candidate even when the commercial name is hidden. Record which text was changed; do not rely on a file named `anonymous-final` to explain the transformation.

Check whether the hidden identity is still obvious

A name can disappear while identity remains easy to infer. A distinctive product term, domain, headquarters address, proprietary feature name or unusually specific pricing structure may identify a supplier.

Inspect the packet for those clues before interpreting a masked condition. Removing every clue may also remove information essential to the buying decision. Record that trade-off instead of silently deleting inconvenient facts.

There is no perfect anonymity certificate in this exercise. A separate attempt to guess identities can expose obvious leakage, but unsuccessful guesses do not prove that another model or session cannot infer the brand. If leakage remains substantial, describe the condition as “brand names removed,” not as fully anonymous.

Do not swap real company names onto other companies' actual profiles and publish the result. Neutral labels are sufficient for an internal comparison. Misattributed commercial claims create a different problem from the one the diagnostic is meant to study.

Treat order and session conditions as part of the design

Run order can become entangled with a condition: perhaps all masked packets are submitted first and all named packets after an interface update. Profile order can become entangled with candidate identity if one supplier always appears first.

Plan the allocation before collection. Use a declared set of profile orders, apply the same orders across the four conditions and vary the condition sequence. Preserve the exact input presented each time.

NIST's guidance on randomized blocks describes grouping comparable experimental conditions to account for important nuisance variables. Here, a practical grouping could be one buying question, one evidence packet, one model setting and one bounded collection period. This application is a proposed design choice, not NIST validation of an AI experiment.

Record model, interface, tools enabled, account context and timestamps. A new conversation is a useful session boundary, but it does not prove that every persistent account setting has been removed. If the environment cannot be held comparable, retain that limitation in the result.

Include repeated identical inputs. Without a baseline for ordinary output variation, a changed shortlist can be mistaken for a response to the identity treatment. Decide repetition counts and stopping rules before examining favourable answers.

Read a fixed-size shortlist as a redistribution

The following numbers are invented arithmetic, not collected model answers, a supplier score or a forecast.

Suppose every output selects exactly three of five fictional products. Ten outputs therefore contain thirty shortlist placements. Every condition must respect that total.

Product A moves from six to eight placements: a descriptive change of twenty percentage points in this fictional set. It also moves from six to seven when the masked input is repeated. Neither contrast, by itself, estimates a stable population effect.

The names-visible column cannot make all five products gain placements. The shortlist has a fixed number of positions. A gain for one candidate is accompanied by losses elsewhere.

Avoid reporting the thirty placements as thirty independent buyer decisions. Each output contains a linked selection of three products, and repeated outputs share the same packet and question. A formal uncertainty analysis would need to respect that dependence and the actual unit of assignment.

If an assistant refuses, returns two options or lists all five without selecting three, keep that outcome separately. Do not silently repair it into the desired format or remove it because the comparison becomes awkward.

Review the recommendation reasons

A shortlist position is only one part of the answer. Check whether the explanation uses facts in the packet, introduces unsupported capabilities, overlooks a disqualifying requirement or confuses a product with its parent company.

For each selected candidate, map material reasons to specific evidence. “Supports exports” is insufficient when the requirement concerns full answer text and citation URLs. “Agency-friendly” does not establish permission to give external clients account access.

Retain unsupported reasons alongside the placement. A candidate's increased selection rate is not a successful buying recommendation if the explanation relies on a feature the candidate does not provide.

Use the same review rules across conditions. If a factual mistake disqualifies a named recommendation, apply that rule to a masked recommendation too. Keep the original output available so another reviewer can inspect the judgment.

Choose an action that matches the observation

The diagnostic is useful when it changes the next decision without inflating what the exercise proves.

Do not use the last row to claim knowledge of an engine's hidden search process. The two observations identify a practical gap between supplied-information suitability and observed discovery. They do not reveal every internal step connecting them.

Keep diagnostic records separate from visibility results

Supplied candidate names make this a different task from an unbranded consumer question. Store its inputs, conditions and outputs separately from ordinary visibility observations. A diagnostic shortlist should not increase the count of natural recommendations in a campaign report.

For a real monitoring purchase, request an answer-level review using the actual account and agreed search surface. Semvi by Xindar's product reference distinguishes stated monitoring scope from buyer verification. It does not establish that Semvi runs the experimental design proposed here.

The Semvi introduction is a starting point for discussing monitoring needs. Ask which observed answers, citations and exports your reviewer will receive. Buying a record of actual answers and conducting a controlled shortlist diagnostic are separate deliverables.

Frequently asked questions

Does a higher named ranking prove the model trusts the brand?

It shows a difference under the tested inputs and conditions. “Trust” is a broader interpretation that the output alone does not establish. Check repeat variation, identity leakage and the factual reasons before deciding what the difference means.

Should every product profile have exactly the same features?

No. Profiles should preserve the relevant differences between products. What stays fixed is each candidate's factual description across that candidate's masked and named versions. Making all products identical changes the buying problem.

Can a masked test replace live search monitoring?

No. Supplying the profiles removes the need to discover those candidates in that task. Keep ordinary unbranded search observations, including absences and failures, as a separate evidence series.

Does this tell a team how many articles to publish?

No. An identity contrast does not identify a content dose or guarantee future recommendations. Use the result to choose a further investigation, then evaluate any content intervention against its own declared outcome and comparison.

Method and sources

This article proposes a diagnostic workflow. The fictional table demonstrates counting and dependence; no experiments were conducted for it. The research discussion refers to arXiv version 1 submitted on October 10, 2026. The cited NIST sections support general experimental-design concepts. The commercial links identify Xindar's own product information and do not constitute independent product evaluation.