Last updated September 2026.
Most LLM visibility dashboards report a single number: a share-of-voice score, a mention count, a citation total. Marketers copy that number into a slide, then move on. Almost nobody checks it against what ChatGPT or Perplexity actually shows a real person that same week.
That gap matters more than most buyers assume. A dashboard pulling from an unauthenticated API call can return something different than the same prompt typed into a live, logged-out browser session, and the engines themselves are not perfectly consistent from one run to the next either. A tool’s accuracy claim is only as good as your ability to check it. This protocol is the check: a seven-day, hand-run audit any team can run against any tool on the roster, using nothing but a spreadsheet and two browser tabs.
You do not need engineering time or an enterprise support contract to run it. A marketer, an SEO lead, or an ops person can carry the whole thing on top of a normal week, spending roughly 30 to 45 minutes a day. The output is not a vague impression that the tool “feels off.” It is a specific, defensible number you can hand to whoever owns the renewal decision.
The 7-day protocol
Run each day’s task in order. Nothing here needs a developer or an enterprise support contract, only two logged-out browser sessions and a place to log what you see.
Day 1: Build the panel and freeze the wording
Pick 25 prompts that span how people actually ask about your category. A workable mix looks like this:
- Five branded prompts, questions that name your company directly.
- Ten category or “best X” prompts, the kind a buyer types before they know your name.
- Five direct comparison prompts, “X versus Y” style, against named competitors.
- Five problem-aware, long-tail prompts that describe a need without naming a product category at all.
Write down the exact wording and pick one calendar week as your test window. Do not edit a single prompt once the week starts.
Twenty-five is a size most plans can already absorb. Temso’s Starter tier runs 50 prompts across three models, Peec AI’s Starter tier runs 50 prompts across three chosen models, and Profound’s Starter tier covers ChatGPT only before Growth opens the full engine set. If your plan caps prompts lower, such as Otterly.AI’s 15-prompt Lite tier, shrink your panel to match and say so in your writeup.
Day 2: Hand-run the panel through ChatGPT and Perplexity, logged out
Use a logged-out or private browser session for both engines, or ChatGPT’s Temporary Chat mode, so memory and personalization do not skew what you see. Run every prompt at roughly the same time of day, since some answers change with how current a topic is.
For each of the 25 prompts, log four columns in a spreadsheet:
- The prompt, exactly as written.
- Mention: yes or no.
- Source URLs cited, if any appeared.
- Position: where your brand landed in a ranked or comparative answer.
Save the raw answer text alongside those columns. You will need it again on day 6.
Day 3: Pull your tool’s numbers for the identical week
Add the same 25 prompts to your tool’s panel, or confirm it already runs equivalent wording, early enough that its scheduled refresh covers the same week as your hand run. Export or screenshot what the dashboard shows for each prompt: mention status, cited sources, and any sentiment or position score it assigns.
Match the wording exactly, down to punctuation. A tool that rewrites “best project management software” into a slightly different internal query is no longer testing the same question you asked by hand, and any gap you find later will be impossible to attribute to accuracy versus prompt drift.
Day 4: Rerun the panel and calculate your noise floor
Twenty-four hours after day 2, rerun the same 25 prompts through both engines under the same logged-out conditions and the same time of day. Count how many prompts flipped: mentioned in run one but not run two, or the reverse. Divide that count by your total number of answer-checks to get your noise floor, the variance built into the engines themselves, before your tool ever touches the data.
This step is the one most audits skip, and skipping it is what makes a tool look worse than it actually is. Without a noise floor, every mismatch between the dashboard and your hand run looks like a tool error. With one, you know how much of that mismatch was never the tool’s to control.
Say four of your 25 ChatGPT answers flip and two of your 25 Perplexity answers flip. That is six flips across 50 answer-checks, a 12% noise floor. Anything your tool gets “wrong” inside that 12% is the engines shifting on their own, not the tool’s error.
Day 5: Compare the dashboard to your hand run and isolate the tool’s error
Take the majority result between your two hand-run passes as that prompt’s ground truth for the week. Compare it, prompt by prompt, to what the dashboard reported on day 3. Divide the number of prompts where they agree by 25 to get your match rate.
Continuing the example: your tool agrees with your hand-run majority on 20 of 25 prompts, an 80% match rate. That leaves a 20% mismatch. Subtract your 12% noise floor, and the tool is responsible for roughly 8 percentage points of that gap, error that traces back to its own collection method rather than to the engines moving underneath it.
Day 6: Check citation-level accuracy, not just mention-level
A tool can pass the mention question, did the brand show up, and still misreport which page the engine actually cited. That distinction matters more to a content team than a simple yes or no. For every prompt where either you or the dashboard logged a source, compare the exact URLs. Divide the number of prompts with a matching URL by the number of prompts where a citation appeared at all, to get a second, citation-level match rate.
Say a citation appeared in 15 of your 25 prompts. Your dashboard names the correct URL on 11 of those 15, a 73% citation-level match rate, noticeably lower than the 80% mention-level match rate from day 5. That pattern, a tool that clears mentions but stumbles on exact sources, is common enough that it deserves its own line in your writeup rather than getting averaged away.
Day 7: Grade the tool and write the one-page summary
Apply the rubric below to your match rate. Write up five numbers any budget owner can read in under a minute: your panel size, your noise floor, your match rate, your citation-level match rate, and the resulting grade. Then decide: keep the tool, escalate what you found to its support team, or start comparing the rest of the roster.
Why your dashboard and your own eyes disagree: the UI-vs-API gap
Most of the mismatch you will find traces back to one structural fact. The answer a live person sees in a browser and the answer a vendor’s system pulls through a model API are not always drawn from the same conditions. Call this the UI-vs-API gap, and it is worth understanding before you blame a tool for a number the underlying engines never agreed on in the first place.
| Condition | What your hand run sees (browser UI) | What many dashboards pull (model API) |
|---|---|---|
| Memory and personalization | Off only if you log out or use a private or temporary session; on by default otherwise | Off by default; each call starts with no history |
| Live web browsing | On by default for Perplexity; ChatGPT triggers it when a prompt needs current information | Depends on the endpoint; some calls skip browsing entirely unless the vendor wires it in |
| Model version | Whatever the consumer product ships that week, which can change without notice | Pinned to a specific model version until the vendor updates it |
| Run-to-run consistency | Lower; the same prompt can return a different answer minutes apart | Higher for a single endpoint, but still not identical across repeated calls |
| What it actually tells you | What a real customer actually encounters | A controlled, repeatable read, not necessarily the customer experience |
Neither side is wrong. A dashboard sourced from an API gives you a cleaner week-over-week trend line. Your hand run gives you what a prospect actually sees. This protocol is not about picking a winner between them. It is about knowing how far apart they sit for your brand, so a widening gap tells you something changed instead of leaving you to guess.
Grade the tool: A to F, with the pass line marked
Apply your match rate from day 5 to the bands below. This audit’s pass line sits at B-: a 75% match rate or higher, with your noise floor already subtracted. That line is deliberately below a perfect score, because no dashboard should match a live, personalized browser session exactly. It is set well above a coin flip, because a tool that agrees with reality on three prompts out of four is doing real work, and one that does not is asking you to act on a guess.
| Grade | Match rate (noise-floor adjusted) | What it means |
|---|---|---|
| A | 95% or higher | Trust the dashboard as your primary source. |
| A- | 90% to 94% | Trust it; spot-check the outliers by hand. |
| B+ | 85% to 89% | Reliable for trend reporting; confirm any single number before you cite it externally. |
| B | 80% to 84% | Usable, but hand-check anything that drives a content or budget decision. |
| B- | 75% to 79% | Passes. Hand-check the prompts that matter most before you act on them. |
| C+ | 65% to 74% | Fails. Treat every number as directional only, never as a citable fact. |
| C | 55% to 64% | Fails. The gap is too wide to explain by engine noise alone. |
| D | 40% to 54% | Fails badly. Open a support ticket and ask the vendor to explain the gap before you renew. |
| F | Below 40% | Fails. The tool is not measuring what it claims to measure. |
Run the same bands against your day 6 citation-level match rate too. A tool that scores an A on mentions but a C on exact source URLs is still worth flagging. It can tell you that you got cited without reliably telling you where, and “where” is the number a content team actually needs to act on.
Where the roster differs on collection depth
The tools worth running this protocol against most often collect their baseline data in different ways, and none of that is a verdict on accuracy by itself. It only tells you what you are actually testing before you start day 1.
Profound leans on enterprise sampling depth. Its Growth tier runs up to 100 prompts a day across the full engine set, starting at $399 per month, and maps citations down to individual source URLs. Peec AI runs on a daily refresh cadence, with its Starter tier covering 50 prompts across three chosen models from €85 per month, so a citation swing shows up inside a day rather than a week. Temso pairs a smaller all-in-one prompt panel, 50 prompts on Starter at $89 per month across three models, with content drafting, site audits, and bot-traffic reporting inside the same account. Otterly.AI keeps the lightest entry point of the group, 15 prompts on its $29-per-month Lite plan, built for a quick first pass rather than a full ongoing program.
None of that ranks one collection method above another. A high-depth tool that fails this audit is a bigger problem than a lightweight one that fails it, since you are paying more to trust its numbers. Read the gap you measure on day 5 next to the collection depth you are actually paying for, not in isolation.
Run it before you renew
Run this protocol once, on whatever tool you already pay for, before your next renewal date. Put the finished worksheet, panel, noise floor, match rate, citation match rate, and grade, somewhere your whole team can find it, then put the audit itself on a recurring quarterly calendar entry so it survives past this one renewal cycle.
A tool that clears the B- line has earned another quarter of budget on the strength of a real number, not a sales deck. One that does not gives you the leverage to push the vendor for an answer, or the evidence to start comparing the rest of the field at /rankings/llm-visibility-tools. Our own scoring criteria, built on the same kind of scored test run, are laid out in full at /methodology.