The Shortlist Benchmark: 100 AI Answers, Four Prompt Types

The shortlist benchmark is our recurring study of what AI engines say at the point buyers decide. Not the category question that gets asked once by people who are researching, but the four questions asked by someone who already knows your name and is narrowing to one vendor. The terms are defined in the AI visibility glossary if any of this is new.

This is the first full run of the shortlist benchmark. Thirty distinct prompts, four engines, 100 runs, captured across September 2026. It sits underneath everything else we publish on AI visibility.

The headline finding is structural rather than a single number. Each of the four prompt types draws on a completely different set of sources. Comparison answers lean on vendor pages. Pricing answers lean on SEO blogs. Migration answers lean on support documentation. Category answers, the ones everybody measures, lean on listicles, forums and video.

Which means one blended AI visibility score is averaging four different problems, with four different fixes, into a figure that tells you which of them to work on: none.

What we ran

Four separate runs, one per prompt type, across September 2026.

Comparison. Five brand pairs through ChatGPT, Claude and Google AI Mode. 15 runs. Published here.

Alternatives. Five “best alternative to” prompts through ChatGPT as a control. 5 runs.

Pricing. Twelve prompts across four brands through four engines. 48 runs. Published here.

Migration. Eight prompts across two brand pairs through four engines. 32 runs, 31 captured. Published here.

The brands were Ahrefs, Semrush, HubSpot, Salesforce, Zapier, Make, Notion, Confluence, Klaviyo, Mailchimp and LymLyt. We included ourselves because a study that only tests other people is easy to dismiss.

Every run used the free consumer tier, default model, logged out where possible, one prompt per fresh conversation. The full procedure is written up as a repeatable audit so you can run it in your own category.

The four prompt types come from the shortlist prompts framework. If the term is new, start there.

The headline: four prompt types, four source profiles

This is the finding that matters, and it only appears when you run all four types in one quarter.

Category prompts draw on listicles and review platforms. “Best CRM for startups” returns the same set of roundups everyone else is competing in, which matches what we found on G2 and Capterra, in our wider citation study, and what Ahrefs found across 26,283 cited URLs.

Comparison prompts draw on vendor pages, when they cite at all. Every Google AI Mode answer that cited anything included at least one vendor-owned page, including both vendors’ own comparison pages in one case.

Pricing prompts draw on SEO blogs. Google AI Mode’s Ahrefs answer cited six of them and not the vendor. Perplexity was the only engine to cite the pricing page directly.

Migration prompts draw on support documentation. help.ahrefs.com, knowledge.hubspot.com, help.salesforce.com, semrush.com/kb. Vendor marketing pages appeared once in 31 answers.

Four questions, asked by the same buyer in the same week, answered from four different corners of the internet. That split is the practical consequence of how search behaviour has changed, and it is a different discipline from ranking, which is the SEO versus AI search distinction. The work required to move each one is different, and nothing on a standard dashboard tells you which one is broken.

Silence is not evenly distributed

The most useful number in the benchmark is how often an engine cites nothing at all, because it tells you whether publishing can move the answer this quarter.

It varies sharply by prompt type.

Comparison prompts were the quietest. Ten of fifteen answers cited no source whatsoever. Claude cited nothing across all five. ChatGPT cited on one of five.

Migration prompts were quieter than expected. Ten of thirty-one answers had no sources, with Claude accounting for eight of them.

Pricing prompts were the most sourced. Almost every answer carried citations, which makes sense: pricing is the most documented thing on a software site and the most written-about by third parties.

So the same brand can be highly influenceable on pricing and almost unreachable on comparison, in the same week, with the same content budget. Where silence is high, no page you publish gets consulted, and what decides the answer is whatever the model absorbed over years. That is closer to why sites fail to show up in ChatGPT than to a ranking problem, and it usually needs earned mentions rather than more publishing.

That is a different problem with a different timeline, and it is invisible in a single score.

The engines are not interchangeable

Running all four types across four engines exposed behaviour that a single-engine audit would miss entirely.

Claude cited nothing on comparison or migration prompts. Zero across five, zero across eight. It cited on pricing, and there it leaned on TechRadar and an aggregator rather than the vendor. It also could not find our company at all on a pricing prompt, while three other engines returned our exact tiers from our public pricing page.

Google AI Mode was the most thorough and the most third-party dependent. It gave annual effective rates, seat costs and difficulty ratings nobody asked for, built on SEO blogs and implementation-partner content.

ChatGPT was the most verbose and the most affected by account memory. In our pricing run it answered generic questions with advice tailored to a business it remembered, which is why logged-out runs are not optional.

The engines differ in retrieval, citation habits and freshness weighting, which we broke down across ChatGPT, Claude, Gemini, Perplexity and Grok.

Perplexity cited the vendor most consistently, which is consistent with what we found about getting cited there. It was the only engine to cite the Ahrefs pricing page directly, and the only one to carry sources on both phrasings of the migration question.

Averaging four engines that behave this differently produces a number that describes none of them. Report per engine, or the reporting is fiction, which is the wider argument in why your AI visibility score measures the wrong stage.

Rewording the question changes everything

The single most surprising result in the quarter came from the migration run.

“How do I migrate from Ahrefs to Semrush” produced citations from three engines. “Is it hard to migrate from Ahrefs to Semrush” produced citations from one. Same on the CRM pair: the how-to form drew on documentation and seven partner blogs, the is-it-hard form drew on nothing in two of four engines.

Same buyer, same decision, same week. One phrasing retrieves. The other answers from memory.

No prompt set catches this, because a prompt set holds one phrasing per question. If yours uses the how-to form you are measuring a query your documentation can win. If it uses the other, you are measuring one nothing you publish will touch.

Track both phrasings of every prompt that matters. It costs nothing and it separates the questions you can influence from the ones you cannot.

Who intercepts, and where

Third parties do not appear evenly. They show up at specific stages, and who they are changes with the question.

At comparison, a rival product intercepts. SE Ranking ranks first for “Ahrefs vs Semrush” and its comparison closes by recommending SE Ranking. Whalesync, Knack and Softr all hold results for “Zapier vs Make”. We covered this in the vs keywords teardown.

At alternatives, the shortlist is a duopoly. Every alternatives prompt returned the competitor from our original pair as the top pick. Five for five. Both question types funnel a buyer to the same two names.

At pricing, SEO blogs intercept, and one engine actively coached the buyer on how to spend less, then named a cheaper rival in the same answer.

At migration, service firms intercept. Google AI Mode’s CRM answers were assembled from implementation partners: Ateko, GetOnCRM, Grazitti, Hexaview, Axon Garside, Bright Digital. Companies paid to move you.

Notably, nothing intercepted at migration in the product sense. No engine suggested staying put or named a third product. Once a buyer reaches that question, the engines stop selling and start assisting.

What to publish for each type

The practical output of the benchmark is that each prompt type is answered by a different page.

Comparison is answered by a vs page for each rival that matters, written as an assessment rather than a pitch. Pages that concede nothing get treated as promotional by readers and by engines.

Alternatives is answered by an alternatives page covering five to eight options honestly, including the ones that are not you.

Pricing is answered by a pricing page with the numbers in text, the billing basis stated beside every figure, and a visible date, which is the same structure and crawlability argument in its simplest form. Every engine that got our prices right was reading text.

Migration is answered by a switching guide, in both directions, and by your knowledge base. Customers who already switched are your strongest case study material for it. Support documentation is doing marketing work here whether or not marketing knows it, which is generative engine optimisation applied to the part of the site nobody in marketing owns.

Most B2B SaaS companies have the first two and are missing the last two. That is where the cheapest wins in this benchmark are, and the comparison pages pillar covers how to build the first two properly.

What we got wrong

Two hypotheses, written down before the runs, both wrong in useful ways.

We predicted AI engines would quote wrong prices. More than half, we guessed. All four engines returned the correct tier ladder for a well-documented tool. The errors were in billing basis, plan names and allowances, not the headline number.

We predicted migration answers would come from forums. They came from vendor support documentation. Half right, and the half we got wrong is the more actionable half, because a knowledge base is something you control.

There were method failures too. Four pricing rows were recorded under the wrong engine and had to be re-run. One migration row was never captured. ChatGPT’s pricing run was contaminated by account memory and is reported as directional only.

All of it is in the individual studies. A benchmark that only ever confirms its own thesis is the thing this cluster criticises other people for.

Running it yourself

Thirty prompts, four engines, one day per quarter.

Six comparison, four alternatives, nine pricing, four migration, two category controls, plus both phrasings of anything that matters. Logged out, one prompt per conversation, whole set in one sitting, same person every time.

Log six columns: prompt, engine and date, sources cited, whether you appear, what the answer said about you, and the action it implies. The AI visibility checklist covers the on-site half that this benchmark does not measure, and the brief generator turns the gaps into something a writer can start on. The full method covers scoring, and a tracking tool takes over past fifteen prompts a run.

Judge on the trend across three runs. Answers move week to week, and explaining a single-run movement to a board is how teams lose credibility.

What happens next

This benchmark runs quarterly. Same prompt types, same engines, expanded brand set.

The second run is more valuable than the first, because it can show movement. Any figure here is a snapshot until then.

We will publish whatever comes back, including results that contradict anything above.

FAQ

What is the shortlist benchmark? A recurring study of what AI engines say at the evaluation stage, across four prompt types: comparison, alternatives, pricing and migration. It measures whether you win, not whether you are known.

We appear everywhere and still get no demos. Does this help? Partly. It will tell you which evaluation questions you lose. If you are present and winning those and still not converting, that is a content and offer problem rather than a visibility one, and traffic alone has never been the measure, as HubSpot found at scale.

How is this different from an AI visibility score? A score tells you whether you appear across a prompt set you chose, usually weighted toward category questions. This measures the four questions buyers ask when they are deciding, and reports them separately because they behave differently.

Is 100 runs enough to draw conclusions from? It is enough to show that the four prompt types behave differently, which is the structural finding. It is not enough to rank vendors or claim industry averages, and we do not.

Do keyword tools show these queries? Barely. Most evaluation prompts report near-zero volume in a keyword research tool, which is a reporting floor rather than evidence nobody asks.

Can we get the raw data? The individual studies carry the counts, sources and method. Email us if you want the full logs.

Category visibility tells you whether buyers can find you. The shortlist benchmark tells you what they hear once they already have.

That is the job we do. See how we approach AI visibility and how we write comparison pages, look at our work, check pricing.