Last updated October 2026.
A vendor demo shows you its best day. A sales call shows you a slide with someone else’s brand on it. Neither one tells you what the tool will report about your own brand next Tuesday, when nobody from the vendor is in the room.
The only way to know if an LLM visibility tool is accurate is to test it yourself, against a fixed panel of prompts, over enough days to see how its numbers move. This kit lays out that test: a full week, a frozen prompt list, and a scoresheet you can print and fill in by hand.
None of it requires a research team. It requires a spreadsheet, a free trial or a low-cost plan, and about 20 minutes a day for seven days.
What “accurate” actually means here
Every LLM visibility tool reports a number: a mention rate, a citation count, a share-of-voice score. None of those numbers mean anything on their own. They only mean something next to a ground-truth run, the same prompt, asked by hand, in the real chat interface, logged the moment it comes back.
A ground-truth run is not a fancy term for a spot check. It is the manual, unautomated answer you get when you open ChatGPT, Perplexity, Google AI Overviews, Gemini, or Microsoft Copilot yourself and type the exact prompt the tool claims to be tracking. Whatever the tool reports gets checked against that.
Accuracy, in this kit, is the gap between what a tool says happened and what you can independently confirm happened. Close that gap and a tool is doing its job. Widen it and the dashboard is decoration.
The 7-day protocol, day by day
| Day | What you do | What it produces |
|---|---|---|
| 1 | Freeze your 20-prompt panel and start the tool’s trial or lowest paid plan | A frozen prompt list and the tool’s first automated report |
| 2 | Run all 20 prompts by hand in the real chat interfaces | Ground-truth run 1 |
| 3 | Pull the tool’s report again, same panel, no changes | Automated data point 2 |
| 4 | Run the manual check again, same 20 prompts | Ground-truth run 2 |
| 5 | Pull the tool’s report a third time | Automated data point 3, closing the variance triangle |
| 6 | Set the tool’s live dashboard, and its API if it has one, next to a fresh manual run | UI-vs-API divergence check |
| 7 | Score every criterion against your logs | A filled-in A-F scoresheet |
Day 1: freeze the panel and start the trial
Building the panel is the one step you cannot rush. See “Build the 20-prompt panel” below for the exact mix. Once the list is frozen, sign up for the tool’s trial or its lowest paid plan and load every prompt into it exactly as written.
Days 2 and 4: your own ground-truth runs
Twice this week, you play the part no tool can automate: a person typing the same 20 prompts into the real product and reading what comes back. Full instructions sit in “Run the manual ground-truth check” below.
Days 3 and 5: let the tool run itself
Pull the tool’s own report for the identical panel on Day 3 and again on Day 5, changing nothing else: same account, same panel, same time of day if the tool lets you control that. Those two pulls, plus Day 1’s first report, give you three data points spread across the week, the minimum needed to see a real pattern instead of one lucky or unlucky day.
Log the exact figure the tool reports for the metric you care about most, whether that is a mention rate, a share-of-voice score, or a citation count, and screenshot the dashboard each time. A vendor’s numbers can shift retroactively if the tool recalculates an earlier day’s report, and a screenshot is the only way to catch that later.
Day 7: score it
Pull together your Day 1, 3, and 5 automated logs and your Day 2 and 4 manual logs. Work through the scoresheet below one criterion at a time, and cite the specific run that earned each grade. Don’t average from memory. If the evidence for a criterion is thin, mark it “not verified” instead of guessing at a grade.
Build the 20-prompt panel
Twenty prompts is small enough for one person to run by hand twice in a week, and large enough to show a pattern instead of a coincidence. Split them into four even bands of five:
- Discovery prompts, the broad “best X for Y” questions a buyer types before they know your brand exists.
- Comparison prompts, “X vs Y” questions that put your brand next to a named competitor.
- Decision prompts, “is X worth it” and pricing questions asked right before a purchase.
- Branded prompts, questions that name your brand directly, to check whether the tool catches the mentions it should find easily.
Source the wording from real searches or real sales-call questions, not a guess at what someone might type. Once your list hits 20 prompts, freeze it. Do not swap one out mid-week because the tool handled it badly. That single rule is what turns this into a test instead of a demo.
Run the manual ground-truth check
For each of the 20 prompts, open the real product, whichever engine the prompt targets, and type the prompt exactly as written. Log three things for every run: whether your brand appeared, which sources the answer cited, and the exact time you ran it.
Do this on Day 2 and again on Day 4. The gap between those two manual runs is not noise to throw out. It is your baseline. AI answers shift on their own, prompt to prompt and day to day, so before you can judge whether a tool’s variance is normal, you need to see what normal looks like without any tool involved at all.
The variance calculation, worked through
A number on its own does not tell you if 7 points is high or low. Compare it against your own Day 2 and Day 4 ground-truth spread. If your manual runs swung by 18 points on their own, a tool showing 7 points of variance is smoothing something out, whether that is caching, averaging, or a sample size too small to catch the real swing. If your manual runs barely moved and the tool’s number swings wildly instead, the tool’s own collection method is likely adding noise the underlying model never had.
Neither pattern makes a tool wrong by default. It tells you what kind of number you are looking at, and what to ask the vendor about next.
The UI-vs-API divergence check
Some tools read a brand’s standing straight from a model’s API. Others read it from the same consumer interface a real customer would open. The two methods can return different numbers for the identical prompt on the identical day, because a consumer app layers in citations, formatting, and sometimes different default settings that a raw API call does not carry.
Temso, for one, documents that it draws its readings from real chat interfaces rather than engine APIs, a method built to mirror what a person actually sees rather than a raw model response. Otterly.AI publishes a customer-facing API, so if that is the tool you are testing, pull the same prompt through its API on Day 6 and set that payload next to what the dashboard shows for the identical prompt.
If the tool you are testing does not expose an API, run the divergence check the other direction: open the tool’s live dashboard next to a fresh manual run in the real chat interface, same prompt, same hour. A dashboard number that does not match either source under any condition is a bigger flag than a small percentage-point gap.
Refresh cadence is the other variable worth watching on Day 6. Peec AI documents daily prompt-level refreshes across its engine set, while a lower tier on some tools samples less often or gates certain engines behind an add-on. A tool that refreshes weekly cannot show real day-over-day variance no matter how the math is done, so confirm the cadence before you blame the tool for a flat line.
The printable A-F scoresheet
Print this table, or copy it into your own sheet, and fill in a grade for each row using your Day 1 through Day 7 logs.
| Criterion | What earns an A | What earns an F | Your grade |
|---|---|---|---|
| Citation accuracy | Every cited source in the tool’s report matches a source you found by hand | Cited sources are fabricated, broken, or never appear in your manual runs | |
| Coverage completeness | The tool returns real data for every engine it claims to track | The tool claims an engine it never actually returns data for | |
| Day-over-day stability | The tool’s variance tracks your own ground-truth spread within a few points | The tool’s variance has no relationship to your manual baseline | |
| UI-vs-API consistency | The dashboard number matches a live check on the same prompt and day | The dashboard number never matches either a live UI or API check | |
| Refresh reliability | Data updates on exactly the schedule the vendor states | Data sits unchanged past its stated refresh window | |
| Methodology disclosure | The vendor’s own documentation explains how it samples and collects answers | The vendor cannot or will not explain its collection method |
Average the six letter grades, and weight citation accuracy and coverage completeness the heaviest if your use case depends on either one. That average is your overall grade for the tool, built entirely from your own numbers.
What this kit can’t tell you
A single week can also land on an unusually quiet or unusually noisy stretch for reasons that have nothing to do with the tool: a model update mid-week, a slow news day for your category, a prompt that happens to sit on the edge of two answer types. Run the kit twice, a month apart, if a single grade surprises you.
This kit also cannot verify a vendor’s enterprise-only claims. If API access, sample counts, or engine coverage sit behind a sales call rather than a self-serve trial, grade what you can actually see and mark the rest “not verified” rather than guessing at it.
Put the kit to work
Building your own panel and logging seven days of runs by hand takes real effort, and that effort is the point. It is the only way to know if a tool’s number matches reality before you commit a budget to it.
If you would rather start from a result someone else already ran, the graded comparison across every tool this lab covers is live on the full LLM visibility tool ranking, built on the same kind of ground-truth checking this kit walks you through. The measurement protocol behind that board explains the grading scale in full.