Before You Buy an AI Visibility Tracker, Run This Two-Week Test

A tracker says your brand disappeared from an AI answer overnight. Did your visibility collapse, or did you ask a system that gives different answers to the same question? Before you pay to monitor AI visibility, measure how much your own answers vary. Otherwise, a monthly dashboard may give ordinary variation the appearance of a marketing emergency.

Our earlier piece, The Real Cost of In-House AI Visibility Monitoring: The Tool Is the Cheap Part, addressed the cost of doing this work in-house. I’m taking a second look because cost is not the first question. If you do not know how often your brand appears when you repeat the same buyer questions, you cannot tell whether a tracker’s movement means anything. Spend two weeks establishing that baseline by hand. Then judge the software.

The question: is the signal steadier than the noise?

This is a prospective test, not a claim that your brand’s answers will behave like anyone else’s. Over fourteen days, you will submit the same commercial buyer prompts to ChatGPT, Gemini, Perplexity, and Google AI Overviews. You will record your brand, its citations, and a few competitors each time. You will also repeat a small, fixed subset of prompts within each session to see whether answers change even before the next observation day.

My expectation: some prompts will produce dependable brand mentions, while others will swing enough that a single dashboard snapshot is hard to interpret. The mechanism is straightforward. These interfaces can assemble answers from changing sources and generate different wording or recommendations across runs. A mention count is useful only if you know what it does under repeat observation.

The decision at the end is not whether AI visibility matters. It is whether a vendor can measure yours well enough to justify its fee.

Setup: test the interfaces your buyers actually use

You do not need a scraper, an API key, or an enterprise demo. Open the consumer interfaces: ChatGPT with Search enabled, Gemini, Perplexity, and standard Google Search for AI Overviews. Record when Google does not show an AI Overview; an absent Overview is not an answer that omitted your brand.

Do not substitute a raw API call and assume you have measured the same experience. Surfer’s comparison of APIs and web interfaces reported brand overlap as low as 24% to 30%. In Paprik AI’s test queries, raw APIs surfaced an 84% smaller brand universe than consumer chat interfaces. Those are findings from their tests, not a correction factor you can apply to yours. They are reason enough to keep the surfaces separate.

Use a dedicated private window or clean browser profile. Keep account sign-in, cookies, extensions, location settings, and Search mode consistent across sessions. Start a fresh conversation for each prompt rather than letting earlier questions shape later answers. Record your market and any interface setting you cannot hold constant. A controlled run changes the day, not your testing conditions.

Write buyer prompts, not vanity prompts

Build 15 to 25 specific questions that resemble purchase research. A query such as “what does [Brand Name] do?” will usually tell you less than a question about choosing a service for a particular problem. If you need phrasing ideas, use our prompt sheet of 60 buyer questions. For this run, cover four intents:

• Category exploration: Buyers want options without naming a brand, as in “what are the best cloud-based dispatch systems for HVAC contractors?”
• Direct comparison: Buyers weigh named alternatives, as in “Tool A vs Tool B for mid-market manufacturing.”
• Problem-based search: Buyers describe a job to be done, as in “how to reduce inventory shrinkage without hiring warehouse security.”
• Brand validation: Buyers check a company before buying, as in “is [Brand Name] reliable for enterprise billing?”

Choose prompts you can reuse verbatim. Keep branded validation separate when you interpret the results: appearing in an answer that already names you is not the same achievement as being recommended unprompted.

Choose competitors as controls

Add two or three direct competitors to the same sheet and score them against the same answers. Pick a dominant market leader and one or two peers. You are not testing whether they have better marketing. You are checking whether changes affect only your brand or the whole group.

If every brand disappears from an answer, do not call it evidence that your site broke. If your brand repeatedly vanishes while peers remain, investigate. Neither pattern proves a cause by itself. Controls give you context, not a diagnosis.

Run schedule: five observation days, with a repeat check

Run the full prompt set on Day 1, Day 4, Day 7, Day 11, and Day 14. Use the same time window in your target market, the same browser setup, and identical prompt text, down to punctuation. Do not improve a question halfway through because its first answer looked odd. That turns a repeat test into a new test.

Within each observation session, rerun the same small subset of prompts in fresh conversations. Choose that subset before Day 1 and keep it fixed across engines and days. This gives you two views of variation: what changes between observation days and what changes when you ask again under nearly the same conditions. Keep both observations. Do not average away an inconvenient answer.

This is a manageable manual sample, not a simulation of every buyer query. Resist the urge to search every morning and then skim the results. Consistent observations beat a large pile of casually collected screenshots.

Record the answer, not just a mention count

Make one row for each prompt, engine, and observation. Label the row with its day, identify same-session repeats, and retain the answer or a screenshot so you can check your scoring later. For every brand you are tracking, fill these six fields:

1. Mention: Does the answer explicitly name the brand?
2. Position: Where does it appear among recommended brands, if there is a list?
3. Citation type: Does the answer link to the brand’s own domain, to a third party, or to nothing clickable?
4. Cited URL: Which specific page appears in a source link or citation, if any?
5. Recommendation context: Is the brand endorsed, mentioned with reservations, or described with inaccurate pricing or capabilities?
6. Competitor presence: Which control brands appear, and where are they placed relative to yours?

Keep a named mention and a link to your domain as separate observations. A brand can appear without a clickable source, or an answer can cite a third-party page about it. Those differences matter when you decide what a tracker must capture.

They also matter across engines. In a study of 356,019 citations across 100 metro markets, Cheers reported different source patterns: Gemini cited a business’s direct website in 80.7% of its citations, while ChatGPT cited the direct domain 48.8% of the time and third-party directories or review aggregators 44.7% of the time. Do not assume those proportions describe your category. Use your cited-URL column to find out where your answers lead.

Mark a missing AI Overview as unavailable, not as a zero. Likewise, if an interface fails or a response does not finish, note the failure rather than quietly scoring the brand absent. A clean sheet is less valuable than an honest one.

Read the results without promoting variation to a crisis

For each prompt on each engine, count the observation days when your brand appears. Then compare those day-to-day results with your same-session repeats. If the answer changes when you rerun it immediately, a change days later needs particularly cautious interpretation. If repeats agree but the observation days diverge, look at the saved answers, cited URLs, and competitor controls before deciding what shifted.

Published research gives a reason to expect movement, but it cannot supply your baseline. In a 28-day study of more than 70,000 responses to 1,300 commercial prompts, Glen Allsopp at Detailed reported that the exact brand order matched on two random days just 0.3% of the time for ChatGPT and 1.1% for Google AI Overviews. That measures exact order, a stricter standard than whether your brand appears at all. Do not turn it into a predicted absence rate for your business.

Similarly, MaxAEO’s analysis of more than 890,000 AI answers found that its median brand recommendation list stayed unchanged for five days, with shorter and longer spans reported for Perplexity and AI Overviews. The useful lesson is not “ignore every change.” It is check whether a change exceeds the variation your own prompts already show.

Sort prompts into working buckets

Use these buckets to decide what deserves a closer look. They are descriptions of your five observations, not grades of your site’s authority:

• Consistent presence: four or five observation days. Check whether citations and claims remain accurate. Do not rebuild a page merely because a recommendation moves within the list.
• Intermittent presence: one to three observation days. Compare same-session repeats, cited sources, and competitor results. This is where a tracker needs enough sampling to keep routine changes from becoming false alarms.
• No observed presence: none of the five observation days. Check whether the prompt genuinely fits your offer, whether competitors appear, and whether relevant sources can describe your business. You have found a gap in this sample, not proved that a crawler is blocked.

Accuracy gets its own check. A recommendation that repeatedly gives buyers the wrong price or describes a capability you do not have is not a win just because the mention column says yes. Save the answer and its cited source. That is a problem you can investigate; a headline share-of-voice percentage may conceal it.

Put the vendor’s dashboard through the same test

Now bring your sheet to a sales conversation. A clean trendline is not enough. Ask the vendor to explain what its number represents and whether its measurement conditions resemble the interfaces you tested. I would ask:

• Does the product measure consumer interfaces, raw APIs, or a mix? Does it label them separately?
• How does it handle repeat samples before calling a missing mention a drop?
• Can you inspect the answer, its cited URLs, and the difference between a brand mention and a direct-domain citation?
• How does it record prompts for which no AI Overview appears?
• Can you see the exact prompts, location, and other settings behind a reported change?

Those questions are more useful than asking whether the dashboard has an “AI visibility score.” As Discovered Labs argues about share-of-voice reporting, a shift without a way to bound ordinary variation can invite the wrong conclusion. Use your log to press for a concrete explanation. For a broader contract discussion, see our open letter to the owner about to buy an AI visibility dashboard.

Buy measurement you can interrogate, not a percentage that merely looks precise.

Decide whether to keep testing, automate, or fix the underlying work

A spreadsheet is not a permanent operating model. It is a way to learn what an automated one must do. If your important prompts show steady presence and you mainly need to watch for changes, a tracker that preserves answers, sources, and repeat-sampling context may earn its place. If the answers vary widely even within a session, demand stronger sampling before paying for alerts.

Building an internal script is not automatically the cheaper escape. A raw-API tracker may observe something different from the consumer interface you just tested. A browser-based workaround brings its own maintenance work. The DIY tracker discussion here is worth reading before assigning the project to a developer. Automating a mismatched measurement does not make it accurate.

If your answers reveal persistent omissions, thin citations, or inaccurate descriptions, another chart will document the problem, not repair it. Inspect the pages and third-party sources the engines actually use. Decide whether your site needs clearer direct answers, whether important external profiles need correction, or whether paid search should cover commercial queries while you work on organic visibility. Do the work that changes what buyers can find.

That is the distinction behind groas: continuous execution across paid and organic search, with autonomous systems doing the ongoing work and a named human strategist setting direction and owning accountability. I would rather put effort into that work than pay an agency to review a volatile chart once a month and call the meeting optimisation.

What fourteen days can—and cannot—settle

Five observation days will not capture every phrasing a buyer might use, every location, or a gradual change over months. A same-session repeat helps expose immediate variation; it does not tell you why an answer changed. Keep those limits attached to your conclusions.

But the test can change your next purchase. If identical prompts routinely produce different brand answers, do not accept a tracker’s single-sample drop as a diagnosis. Ask for its sampling method or keep your money. If your brand is repeatedly absent while relevant competitors appear, stop shopping for prettier reporting and investigate the sources and content behind that gap. If presence is steady and citations are useful, look for monitoring that can tell you when that pattern genuinely changes.

Run the prompts. Keep the repeats. Fill the six fields. At the end of Day 14, let the behaviour of your own answers set the standard for any number a vendor wants to sell you.