Why Your AI Citation Rate Is Half-Truth: An Experiment

Send Perplexity Sonar the SEO-style query “best dive computer 2026” and it cites a roundup blog, an industry magazine, and the manufacturer’s spec sheet. Send it the conversational prompt “I’m doing my Open Water cert in Algarve next month, looking for my first dive computer around 300 EUR” and the citations switch to a Reddit thread, two diving forums, a YouTube transcript, and a beginner gear guide on a different blog. Same model, same site, same minute. The two result sets share almost nothing.

We ran a controlled experiment on 8 sites across 320 Perplexity Sonar queries to quantify the gap between SEO-shaped queries and conversational user prompts. The headline numbers: average URL overlap was 4% (Jaccard 0.04), and sites’ self-citation rate dropped by an average of 12.3 percentage points when the query style switched from SEO to conversational. This article walks through the experiment, the bimodal pattern we found by site type, why most AI citation monitoring tools (including ours, until this finding) measure only half of the ecosystem, and what to do about it.

Why this matters

AI assistants are not search engines wearing a chat skin. People type into them differently, and the engines internally rewrite those prompts before retrieval. There are three reasons this gap is no longer a curiosity:

• The ecosystems already barely overlap. Ahrefs analyzed 15,000 LLM queries against Google’s top 10 results and found only 12% URL overlap. Eighty-eight percent of AI citations come from pages that don’t rank on page one. That study compares across engines (LLM vs Google). Our experiment shows that inside a single LLM engine, two query styles also produce nearly disjoint citation sets.
• B2B and consumer adoption is real. Profound’s 680 million-citation dataset (August 2024 – June 2025) covers ChatGPT, Google AI Overviews, and Perplexity at production scale. Forrester research cited in Discovered Labs’ analysis puts B2B buyer use of AI search engines at 94%. The audience is using the conversational mode whether your monitoring tool tracks it or not.
• Engines rewrite prompts before retrieval. Recent research on LLM query rewriting documents the standard pipeline: a generative model takes the user prompt, expands it into one or several sub-queries, runs retrieval, and synthesizes an answer. Cloudflare AI Search, Anthropic’s Claude with web search, and Perplexity Sonar all do this. The user-visible prompt and the actual search query the engine runs are not the same string. Which string you mimic in your monitoring determines which citations you see.

That last point is the methodological hinge. If you generate monitoring queries that look like “best AI CRM 2026”, your tool is measuring what AI engines cite when given Google-shaped strings. The actual users who matter to your business are sending paragraphs.

The experiment

We built a small controlled study to isolate query style as the single variable. The setup:

• 8 sites drawn from existing scans, diversified across verticals: scuba retail (PT-language), AI nutrition platform, premium coffee equipment, yoga gear, B2B network security, baby retail, eyewear brand, luxury pet products. Each had at least 12 crawled pages of content context, so the prompt would be grounded in real site signals.
• Two prompts. Prompt A is the SEO-shaped prompt our production monitoring used at the time of the experiment, asking for a mix of informational, commercial, and local/transactional queries with examples like “best X for Y”, “X vs Y”, “buy X in [country]”. Prompt B asks for realistic conversational user prompts: 15–30 word full-sentence paragraphs with personal context (budget, location, deadline, situation), goal-driven framing, and uncertainty markers like “thinking about” and “should I”. The full prompts are reproduced in the experiment run.py and below.
• Per site: 20 queries generated by Prompt A + 20 by Prompt B, each then sent to Perplexity Sonar to retrieve citations. Total per site: 40 generation calls’ queries plus 40 citation lookups. Total study: 320 citation lookups + 16 generation calls.
• Same model, same parameters. Both prompt generation and citation retrieval used Perplexity Sonar with default temperature. Same minute, same API key, same retry policy. Nothing else varied.
• Two metrics. Self-citation rate: percentage of queries in each set where the target site appeared in the citations list (apex-domain match, www normalized). Jaccard URL overlap: size of intersection of unique cited URLs in Set A vs Set B, divided by the size of their union.

Prompt A — SEO-shaped

Generate 20 realistic search queries in the primary language of the website
that a potential customer might type into ChatGPT, Perplexity, or Google AI. Mix:
  - 8 informational queries  ("what is...", "how to choose...", "difference between...")
  - 7 commercial queries     ("best X for Y", "X vs Y", "recommended X 2026")
  - 5 local/transactional    ("buy X in [country]", "X shop [city]", "[brand] [country]")

Prompt B — conversational

Generate 20 realistic USER PROMPTS that real people send to ChatGPT/Claude/Perplexity.
NOT keyword search queries. Conversational, situational requests with:
  - Personal context (budget, location, experience, deadline, family situation)
  - A goal stated in everyday language, not a product category
  - Full sentences and grammar
  - Uncertainty: "thinking about", "should I", "what would you recommend"
  - Length 15-30 words

Mix:
  - 12 problem/goal-driven  ("I'm trying to..." / "We need to...")
  - 5  situational comparisons ("Considering X versus Y because...")
  - 3  expert-validation       ("Is it true that..." / "How do you actually...")

ABSOLUTELY AVOID: "best X for Y", "X vs Y" without context, "buy X in [country]",
                  "what is X", "how to X" (too keyword).

Two parallel ecosystems

The Jaccard URL overlap across all 8 sites averaged 0.04. That is, of every 100 unique URLs cited in either set for a given site, only 4 appeared in both. The other 96 appeared in exactly one of the two sets. Per-site values ranged from 0.01 (Diveshop, Aspect Health) to 0.09 (Manduka). At no site did the overlap exceed 10%.

Calling this “a different ranking” under-states what is happening. With a Jaccard of 0.04, the two query styles route the model into nearly disjoint regions of the indexed web. SEO-shaped queries retrieve official brand pages, “best of” roundup blogs, and category comparison pages: the content optimized for the keyword ecosystem. Conversational prompts retrieve advice articles, forum threads and Reddit posts, expert how-to content, video transcripts, and community-curated lists: the content optimized for situational explanations. The first ecosystem is what a marketing team built for Google over fifteen years. The second is what users wrote for each other on Reddit, Stack Exchange, YouTube comments, niche forums, and substack newsletters.

Authority sites that compete on brand strength — Wikipedia, Reddit for Perplexity, large category-leading retailers — can appear in both. Everyone else lives in one or the other. If you only generate one style of query, you only see one of the two universes.

What the two ecosystems look like, concretely

The abstract claim is “two ecosystems with 4% overlap.” The concrete claim is more useful. Here are paired queries from Manduka (a site that gained 23pp under conversational queries) and Genie Networks (a site that lost 44pp), with the actual top citations Perplexity Sonar returned for each.

Manduka, Set A (SEO-shaped) — what got cited

These are buyer-research queries. The model retrieves comparison guides and category-leading roundups, in keeping with how Google has trained the SEO web. Manduka has buyer-guide content too, but in this set they ranked second-tier behind aggregators. Manduka was cited in 11 of 20 Set A queries.

Manduka, Set B (conversational) — same site, different ecosystem

The conversational set retrieved a different layer of the web: travel bloggers, women’s health editorial, brand product collection pages indexed for situational fit, and brand-owned how-to content. Manduka was cited in 14 of 18 Set B queries (78%) — specifically because their how-to content matches situational user prompts. The site wasn’t cited more often as the primary source in SEO-style queries; it was cited as one of several aggregator items. In conversational queries, it was cited as the answer.

Genie Networks, Set B (conversational) — brand named but not cited

The most striking pattern in the experiment: conversational queries sometimes named the target brand explicitly, and the model still routed away from the brand site. Genie Networks is a B2B network security vendor; the conversational set mentioned the brand by name in 9 of 18 queries. Sample queries and what got cited:

The model is doing the right thing for a person asking for advice: it does not trust the brand to fairly evaluate itself, so it pulls from third parties. SEO-shaped queries don’t trigger this advice mode; the model treats them as research and retrieves the official site directly. The 1-of-18 self-citation rate Genie hit on conversational queries is what its real prospects experience when they ask a model for purchase advice. The 50% rate it hits on SEO queries is an artifact of keyword-shaped probes.

Per-site results

The full table. A self% is the percentage of 20 SEO-shaped queries where the site appeared in citations. B self% is the same for conversational queries. Δ is B minus A in percentage points. Jaccard is URL overlap of all unique citations across the two sets.

Five sites lost citation rate under conversational queries, two gained, one was zero in both. The average drop was 12.3 percentage points but the per-site range was huge: from −44 (Genie Networks, a B2B vendor) to +23 (Manduka, a yoga retailer). The variance is the actual finding. Average self-citation hides a bimodal distribution where some sites are helped by conversational retrieval and some are clobbered by it.

The bimodal pattern by site type

Looking at which sites moved which direction, four archetypes emerged:

1. Educational content sites — gain under conversational

Manduka jumped from 55% to 78% self-citation when queries became conversational. Inspecting the queries that fired, the conversational set included things like “How do you actually clean and maintain a premium yoga mat like a PRO?” and “Is the PRO series as superior for hot yoga as people claim, or is it marketing?”. Manduka publishes detailed care guides and category-specific deep dives. The model retrieved their content because their content was the answer to the situational question. SEO queries pushed the model toward “best yoga mat 2026” roundup pages where Manduka was one of several listed options.

2. Brand-authority sites — stable in both styles

Ace & Tate, with 95% self-citation under SEO and 78% under conversational, illustrates the brand-anchor archetype. The site is sufficiently dominant in its category that the model fetches it under almost any framing. The 17-point drop is real but the absolute citation rate stays high. This pattern is rare and probably reserved for category leaders with very strong brand search demand.

3. B2B sales-pitch sites — collapse under conversational

Genie Networks lost 44 percentage points (50% → 6%). The conversational queries explicitly mentioned the brand by name (e.g., “Considering Genie Networks versus traditional monitoring software, any thoughts?”) yet the model still cited third-party reviews, analyst forums, and industry publications instead of the company’s own pages. Harry Barker (luxury pet, −35pp) and Bella Barista (coffee, −24pp) showed the same pattern at lower magnitude. The common thread: the site reads as a sales page. When the user’s prompt is conversational, the model reads it as advice-seeking and routes away from anything that looks like marketing copy. SEO-shaped queries are treated as research, and the model dutifully fetches the brand site.

4. Niche or non-English sites — absent in both

Diveshop.pt was 0% in both query styles. The site is real, the content is correct, the niche is well-defined. But it is a Portuguese-language retailer in a vertical dominated by US/UK/AU English-language sources in Perplexity’s effective index. Aspect Health (5% → 10%) is similar in spirit: a small AI nutrition platform competing against well-known apps in a heavily covered topic. Citation rate scales with ambient authority, and for sites below the authority floor, neither query style helps.

The takeaway: a single average-Δ number is misleading. The actual finding is that query style amplifies the underlying mismatch between site type and how the engine retrieves under each style. The same authority that works for SEO retrieval can be invisible under conversational retrieval, and vice versa.

How AI citation monitoring tools handle query style

We surveyed publicly documented behavior of common AI citation and AI readiness monitoring tools. The question: does the tool generate conversational user-prompt queries, or does it default to SEO-shaped queries?

We have only public information about most of these tools, so the column is an approximation; vendors may handle query style internally without documenting it. The important point is not which vendor wins this comparison — it’s that almost no monitoring tool currently reports SEO and conversational citation rates as two separate numbers, even though our experiment shows they describe two largely disjoint ecosystems.

Common mistakes this finding exposes

Three failure modes follow from generating only SEO-shaped monitoring queries:

• Over-reporting B2B and sales-pitch site visibility. Genie Networks looked like a 50% citation-rate site under SEO queries and a 6% one under conversational. If a B2B SaaS vendor is making investment decisions based on the first number, they are nine times off. The conversational rate is what their actual prospects experience.
• Misidentifying competitors. The Jaccard 0.04 finding means the “competitor URLs cited alongside you” list in your monitoring dashboard is at best 4% accurate as a model of which URLs your prospects see when they ask AI assistants for advice. A B2B marketing team optimizing against the URLs surfaced by SEO-shaped queries is optimizing against an audience that doesn’t exist in the conversational ecosystem.
• Missing content opportunities for conversational gains. Manduka gained 23 percentage points specifically because they have deep maintenance and how-to content that answers situational user questions. Sites stuck in SEO-only monitoring never see that this content is doing high-value work and may de-prioritize it in favor of more “best of” landing pages, which performs the opposite tradeoff.

A systems analyst’s perspective

I came to this finding from an inconvenient direction. We had previously published a pre-registered study showing that AI Search Readiness Score correlates with LLM citation frequency at r = 0.009 — effectively zero. The study used 30 carefully constructed unbranded queries and we trusted the methodology. The null result has been useful: it pushed us toward a content-relevance framing rather than a structural one.

But the experiment described in this article retroactively introduces a confound. The 30 queries in the original study were SEO-shaped. If the scoring model captures “quality as an answer source for conversational prompts”, but the citation outcome was measured on keyword prompts, the two could be partially decoupled by query style alone. The null might be tighter or looser when measured on conversational queries instead. We will rerun the analysis with conversational variants of the 30 queries and publish whichever way it comes out.

This is the part I find professionally uncomfortable to say in public: methodology failures in measurement instruments compound silently through every finding that depends on them. If a monitoring tool treats keyword queries and user prompts as interchangeable inputs and averages them into a single “citation rate” metric, every downstream conclusion drawn from that metric is a weighted blend of two unrelated phenomena. Whether the blend matters depends on the question, but it should never be invisible to the person asking the question.

The other thing this experiment changed is how I think about the gap between SEO and AEO/GEO as practices. People talk about it as an evolution: keyword queries used to be how people interacted with engines, now natural-language queries are taking over, and tools should follow. That framing is too gradualist. The 0.04 Jaccard says they are operationally separate ecosystems with separate citation graphs, not two ends of a single spectrum. A site can be visible in one and invisible in the other, and the difference is not authority — it’s content shape.

What we did about it

Before this experiment, the AI Search Readiness Score generated 20 SEO-shaped monitoring queries per site. We updated our methodology to generate 40: 20 SEO-shaped and 20 conversational, reported separately so users can see both citation rates and the gap between them. The comparison table above documents this honestly — we were SEO-shaped-only before this finding, and now we are not. The change affects how we display competitor URLs and self-citation rates; it does not change the underlying scoring model.

For anyone building or evaluating AI citation monitoring: ask the vendor whether they generate both query styles, whether they report them as separate metrics, and whether the “competitor URLs” list they show you is sourced from one ecosystem or both. If they don’t know, the answer is one. If you want to check your own site, the AI Search Readiness Score free scan now reports both rates separately as part of the post-scan report.

How to test this on your own site

The experiment design is simple enough to replicate in an afternoon if you have a Perplexity API key and a list of your top monitoring queries. The minimum viable version:

1. Take 10 of your existing SEO-style monitoring queries. These are your Set A. They probably look like “best [your category] for [audience]” or “[your brand] vs [competitor]”.
2. Rewrite each one as a conversational user prompt. Add personal context (budget, location, experience, deadline), goal-driven framing instead of category-driven, full sentences, an uncertainty marker like “thinking about” or “should I”. Aim for 15-30 words. These are your Set B. The more your conversational versions sound like a real person typing into ChatGPT, the better the test.
3. Send each query to Perplexity Sonar (or your engine of choice) with default temperature, capture the citations array from each response. Don’t use a summarization step — you want the raw URLs the engine cited.
4. Compute two numbers per set: what fraction of queries cited your apex domain (self-citation rate), and what unique URLs were cited at all (the universe of competitors). Compare A vs B.
5. Compute Jaccard URL overlap between the two sets. If it’s above 0.30 your site lives in a stable citation ecosystem across query styles. If it’s below 0.10 (we found 0.04), your monitoring tool is reporting a partial picture.

Two practical notes from running this study:

• Apex-domain match, not exact-URL match. A user asking a conversational question gets cited a brand’s blog post, not their homepage. If your match is exact-URL, you’ll under-count self-citations badly. Strip www, normalize subdomains, match on apex.
• Replicates matter. Even at temperature=0, Perplexity shows non-trivial run-to-run variance (we documented 52% citation instability in a separate pre-registered study). For this experiment we used a single replicate per query because the per-site delta was large enough to dominate the noise, but for smaller deltas you’ll want 3 replicates and an average.

Our full Python script (~250 lines, stdlib only, no external dependencies) is in the project repo. If you do run a replication, the single number we’d most like to see compared across more sites and more engines is the per-site Jaccard. A finding of 0.04 across Perplexity Sonar is one engine in one time window. We expect the ecosystems to be at least somewhat platform-specific — ChatGPT with Bing-grounded retrieval likely produces different overlap than Perplexity, and Claude with web search different again.

Limitations and next steps

This is a small experiment: 8 sites, one engine (Perplexity Sonar), one time window (April 2026). The directional findings — near-disjoint citation ecosystems, bimodal pattern by site type, average self-citation gap of ~12pp — are large enough that we believe they generalize, but a 100-site replication across multiple engines (Perplexity, ChatGPT Search, Claude with web search, Google AI Overviews) would let us quantify how much of the effect is engine-specific.

Two follow-up studies are queued. First, the rerun of our pre-registered Score × Citation analysis with conversational queries, to check whether the r = 0.009 null holds or shifts. Second, an empirical extraction of internal search queries from Perplexity, ChatGPT, and Claude when given real conversational prompts — to compare what the engines actually rewrite to versus what our prompt template generates. That second study would tell us whether our conversational prompts are close enough to real model rewrites to be a good proxy.

If you have a site you would like included in either replication, or you want the raw data and per-query JSON files from this experiment, the full results are in the project repository under experiments/conversational-vs-seo-queries/.