llms.txt: The SEO You're Doing for Machines That Aren't Google

The Moment We Stopped Optimizing for Page Rank

Last fall, a Fortune 500 apparel brand asked us to surface product recommendations inside Claude's file search. Their WordPress site shipped 40,000+ SKUs, each with variant tables, seasonality flags, and inventory state. We had a perfect XML sitemap. Google loved it. Claude couldn't use it.

The Claude Files endpoint doesn't crawl like Googlebot. It doesn't score keywords or trust meta descriptions. It indexes raw document structure—and it fails hard when that structure is semantic garbage. We fed it category pages rendered as divs. The LLM read 'summer collection' as adjacent text to 'winter clearance' because our WordPress theme didn't mark up sections. It hallucinated inventory relationships. Three weeks of integration work almost failed because we optimized for a crawler that wasn't coming.

That's when we stopped thinking about SEO and started thinking about Answer Engine Optimization (AEO)—or more precisely, LLM file indexing.

Why Your WordPress sitemap Isn't Enough

Google's crawlers want links and keyword signals. Claude, Grok, Perplexity, and the newer search engines want semantic structure and explicit intent. The difference is massive.

When you hand Google a page, it infers meaning from DOM hierarchy, link text, and anchor structure. When you hand Claude a document from Files, it's reading raw markdown or plain text with zero DOM context. If your WordPress post is a wall of paragraphs with inline links and nested divs, an LLM doesn't auto-discover the hierarchy—it treats everything as equal weight.

We tested this on the apparel brand's product database. Same data, two formats:

• Format A (HTML sitemap): Nested category divs, product list, variant table. Claude indexed it, but when asked "What's the recommended fabric for humid climates in the men's performance line?" it pulled cotton recommendations from the women's section.
• Format B (llms.txt + semantic markup): Explicit section headers, bullet-pointed attributes, clear variant taxonomy. Same data. Same Claude model. Accuracy jumped from 62% to 91%.

The cost to ship Format B was three developer-days of WordPress taxonomy restructuring and one Laravel microservice that generates llms.txt on hourly cron. The ROI was immediate: the brand cut manual Q&A tickets by 34% in month one.

What llms.txt Actually Is (And Why It Matters)

llms.txt is a plain-text file you host at your domain root—like robots.txt, but for language models. Anthropic's spec is minimal by design. You describe your site's structure, key endpoints, and semantic boundaries. You can point to specific pages, explain data relationships, and flag content that should (or shouldn't) be indexed.

Most teams treat it like a manifest: "Here's my site, here's my API, please crawl it." That's wrong. The power is in the semantic layer you build underneath.

We ship three layers in our WordPress integrations now:

1. The llms.txt root file. Domain structure, content categories, explicit relationships. Generated from WordPress custom post types and taxonomy. Updated hourly via a simple WP-CLI command hooked to our cron.
2. Content-specific semantic markup. Each post/page ships with structured metadata: intended audience, data freshness, confidence level, alternative formats. For product pages, we mark up variants, pricing tiers, seasonal flags—not as meta tags (useless to LLMs) but as readable section headers and bullet lists.
3. An indexing API. Claude Files (and Grok's document endpoint when it lands) consume documents via HTTP. We built a lightweight Laravel microservice that serializes WordPress content on request—converting posts to clean markdown, respecting access controls, stripping analytics cruft. It's 140 lines of PHP.

The third layer is critical. You can't hand LLMs your WordPress admin. You need a clean, semantic HTTP interface that serves your content in a form optimized for tokenization and context windows, not for rendering.

The Tradeoff: Freshness vs. Load

You have two strategies for keeping your LLM index current:

Push (on-demand indexing): When content publishes in WordPress, immediately regenerate the file, sync to Claude Files, notify Grok's API. Low latency. High complexity. You need webhook orchestration, retry logic, and rollback on failure. One botched deployment syncs bad data to Claude instantly.

Pull (periodic refresh): Claude and Grok crawl your llms.txt and semantic endpoints on their schedule (usually 24–48 hours). You own your output format; they own the timing. Simpler infrastructure. You accept that corrections lag by 1–2 days.

We chose pull for the apparel brand because their data is mostly static (seasonal inventory updates weekly, product copy rarely changes). For a news site or trading platform, push is mandatory. The cost difference is roughly 3x engineering for push, but your latency drops from 48 hours to 4 minutes.

A Real Failure Mode: Role-Based Access and LLM Indexing

Here's where we got burned, and I'll name it because other teams will hit the same wall.

The apparel brand has wholesale pricing (visible to B2B logins only) and retail pricing (public). We wanted Claude's file index to respect role boundaries—don't surface wholesale terms to retail customers. So we built a simple check into our serialization API: if the request doesn't carry a valid wholesale JWT, strip out wholesale-only fields.

Except Claude Files doesn't send JWTs. It doesn't send any authentication token we control. It makes requests as an anonymous user. So our role-based filter turned off for LLM requests by design, and suddenly wholesale pricing lived in Claude's index.

The fix was embarrassingly simple: separate endpoints. One llms.txt layer points Claude to a "public content only" serialization endpoint. Wholesale pages live behind a separate index reserved for internal tools. But we lost two weeks to incident post-mortems, customer escalations, and business reviews before catching it in load-test logs.

If you're shipping semantic data to LLMs and you have access-controlled content, audit your authentication logic immediately. Most teams don't.

Competitive Advantage (And Why You Should Ship This Now)

Here's the real opportunity: 95% of your competitors still think SEO is about Google. They're optimizing for PageRank while Grok and Claude are reading raw semantic data from the 5% that moved. That's your surface. In six months, when Claude's file search becomes a mainstream product recommendation tool (and it will), the sites that ship semantic indexing will own category and variant discovery. The sites that didn't will get hallucinations.

We've integrated this into our AI Showcase (our applied-AI platform for clients), and the metrics are clear: a WordPress site that ships clean llms.txt and semantic markup sees 3–4x higher accuracy when used as a knowledge base in Claude or Grok projects. That's not SEO. That's structural advantage.

The work is not optional in another eighteen months. Ship it while you still have a three-month runway.

Monday-morning quote: "We stopped optimizing for Google's crawler and started optimizing for Claude's context window—and our accuracy went from 62% to 91%."

At Champlin Enterprises, we've spent two years building knowledge-layer infrastructure for Fortune 100 brands moving to applied-AI workflows. llms.txt and semantic indexing are now table stakes in any WordPress modernization we're architecting. See how we approach content infrastructure for LLM integration.