We Ran a 28-Day AI Skills Manifest Experiment. The Data Says: It Doesn't Work.

In early 2026, a new idea emerged in the AI search optimization space: if you publish machine-readable files describing your site's content and capabilities, AI crawlers will shift from scattered broad crawling to targeted, efficient consumption. The concept draws on llms.txt (a proposed standard for describing your site to LLMs), plus a “skills manifest” (a JSON file listing API endpoints that bots can call directly).

LightSite AI, a managed GEO service, published claims that deploying these files produced dramatic results: ChatGPT traffic increased 3x, Q&A-focused visits increased 5x, and path diversity dropped from 51.6% to 30% — suggesting bots shifted from crawling random pages to consuming structured APIs.

Their methodology had gaps: no baseline measurement before deployment, no control group, and results observed only inside their proprietary proxy infrastructure. We decided to replicate the experiment on our own production site, with proper before/after measurement.

Experiment Design: 28 Days, Two Phases

What We Deployed in Phase 2

• llms.txt and llms-full.txt — machine-readable site descriptions following the proposed standard
• skills.json — a manifest listing 4 “skills” (search_articles, get_article, search_faq, get_methodology) with API endpoint definitions
• /api/ai/* endpoints — structured JSON APIs for blog search, FAQ aggregation, article metadata, and scoring methodology
• Discovery signals — <link> tags in HTML head, reference in robots.txt, cross-references between files
• robots.txt update — explicit Allow: /api/ai/ before the existing Disallow: /api
• Canary tokens — unique strings embedded in llms.txt and skills.json to detect if content appeared in LLM outputs

All Phase 2 components were deployed simultaneously on Day 14. This means we cannot attribute effects to individual components — a trade-off we accepted for practical experiment design.

Raw Data

Phase 1: Baseline (March 10–23)

Phase 2: AI Layer Active (March 24 – April 7)

AI Layer Endpoint Usage (Phase 2 only)

What the Data Shows

ChatGPT-User: The Dominant Bot Was Unaffected

ChatGPT-User accounted for 54% of all bot traffic across both phases (606 of 1,119 total hits). It visited exactly 9 unique paths in both phases — the same pages. Path diversity held steady at 3.1% → 2.8%.

This bot serves live ChatGPT user queries: it fetches specific pages when users ask about topics we cover. It does this regardless of whether we provide a skills manifest. LightSite claimed a 3x ChatGPT traffic increase. We observed +10%, within normal variation.

Path Diversity: No Consistent Pattern

LightSite's central claim was that path diversity decreased after deployment (from scattered crawling to targeted API consumption). Our data shows no consistent direction:

Three bots increased diversity, two decreased, one stayed flat. OAI-SearchBot's drop is an artifact — it collapsed to visiting a single path, not a sign of targeted API usage.

Other Bot Changes

ClaudeBot appeared only in Phase 2 (84 hits, 53 unique paths). Tempting to attribute to the AI layer, but ClaudeBot did not fetch llms.txt, skills.json, or any /api/ai/ endpoint. Most likely, Anthropic's crawler added our domain to its index during this period.

GPTBot decreased 64% (50 → 18 hits) — the opposite of what the skills manifest hypothesis predicts. Meta-AI disappeared entirely. Both are likely natural crawl scheduling variation.

PerplexityBot was absent throughout both phases — despite Perplexity being an active AI search engine. This may indicate our site is not yet in their crawl queue, or they use a different user-agent string.

Head-to-Head: Our Data vs. LightSite AI Claims

Why the Results Differ

Three structural differences likely explain the gap:

1. No baseline in LightSite's experiment. They deployed the skills manifest and observed traffic, but had no pre-deployment comparison period. Any pre-existing bot traffic gets attributed to the manifest.
2. Proxy infrastructure confounds. LightSite's system works through their proxy layer, which intercepts and routes bot requests. A proxy can redirect bots to API endpoints by design — that is not bots choosing to use the manifest, it is the proxy routing them there.
3. No standard for skills.json. There is no W3C or IETF standard for skills manifests. No browser vendor or AI lab has committed to reading them. Without adoption by the actual AI systems, publishing a manifest is writing a letter that nobody checks their mailbox for.

Limitations

• Single site (N=1). Results may not generalize to sites with different traffic volumes, niches, or domain authority.
• No control group. We compare the same site before/after. External factors cannot be isolated.
• Simultaneous deployment. All Phase 2 components launched at once. We cannot attribute effects to individual components.
• 14 days per phase. Bot crawl cycles may operate on longer timescales. Some effects might take months to materialize.
• Observational, not causal. Even with a change, we could not prove causation without a multi-site controlled experiment.

What This Means for Your AI Search Strategy

1. Our data does not support investing in skills manifests. No bot consumed skills.json or used our content APIs. Until a major AI provider confirms support for this format, the engineering effort is not justified.
2. llms.txt is low-cost, low-evidence. It takes 15 minutes to deploy and Google's crawler did fetch it. But there is no evidence it changes citation behavior. File under “cannot hurt, might help eventually.”
3. Content relevance remains the only proven signal. Our previous research showed content relevance is 62x more predictive of AI citations than structural optimization. This experiment reinforces that finding: bots come for your content, not your metadata files.
4. Focus on what bots actually do. ChatGPT-User visits specific pages when real users ask relevant questions. The way to get more AI search traffic is to have more pages that answer more queries well — not to publish manifests describing your existing pages.