Type your own domain into a search engine and picture something different happening on the other end. Instead of a human scanning a results page, an AI agent is reading your site.
It decides in seconds whether it can trust and use what’s there. If it can’t, it moves on. That’s agentic search, and most websites fail this test without ever knowing it’s happening.
This guide answers two questions in order. First, how do you actually check if your website is ready for agentic search? Second, once you know what’s broken, how do you fix every issue step by step until you hit 100/100?
We’ll use Cloudflare’s free public scanner as the diagnostic tool. We’ll walk through what each result means, then rebuild the score category by category with real code and real examples. Along the way, we’ll answer every major question people are asking about agentic websites and agentic search in 2026.
Here’s why it’s worth the hour it takes. Sites that pass agent-readiness checks have been shown to load 31% fewer tokens and deliver 66% faster answers to the agents reading them.
One mid-market e-commerce brand raised its score from 42 to 78 over 100 days and saw AI-driven referral traffic increase by 920%. Industry benchmark data from May 2026 shows even the top 100 websites by traffic average just 55% on this score, so checking now is a real advantage.
How to Check If Your Website Is Ready for Agentic Search
There are two ways to check. The fast way uses a free automated scanner. The manual way means understanding what the scanner tests, so you can verify it yourself with a terminal.
Use both. The scanner tells you what’s wrong. The manual checks tell you why, which matters once caching starts hiding your real results.
The Fast Way: Run the Free Scanner
- Go to isitagentready.com.
- Enter your homepage URL (e.g.
https://example.com/) and click Scan. - Within seconds you’ll get an overall score, a Level badge, and a breakdown across five categories.
- Click into any failed check to see the exact HTTP request the scanner sent and the response your server sent back.
This single scan tells you, in under a minute, whether an AI agent visiting your site right now can find your pages, read your content efficiently, and understand what it’s allowed to do.
The Manual Way: Check It Yourself With curl
If you want to verify a result independently, or check a page the scanner doesn’t cover, these are the exact commands behind the four most important checks.
# Does a valid robots.txt exist, and does it mention AI bots or Content Signals?
curl -s https://example.com/robots.txt
# Does your homepage advertise machine-readable resources via a Link header?
curl -sI https://example.com/ | grep -i "^link:"
# Does your site return clean Markdown when an agent asks for it?
curl -s -H "Accept: text/markdown" https://example.com/ -D - -o /dev/null | grep -i "content-type"
# Does an Agent Skills index exist for capability discovery?
curl -s https://example.com/.well-known/agent-skills/index.jsonIf the first command returns a 404, or the third command returns your full HTML page instead of Markdown, you already know your two highest-priority fixes.
What “Ready” Actually Means
A genuinely agent-ready site satisfies three conditions at once.
It is discoverable: an agent can find your robots.txt, sitemap, and key resource links without guessing.
It is readable: an agent can extract your actual content efficiently, without parsing through navigation menus, ad scripts, and cookie banners disguised as HTML.
It is explicit: you’ve told AI crawlers exactly what they can and can’t do with your content, rather than leaving it ambiguous.
Everything from here is the step-by-step process for making all three fully true, until the scanner shows 100/100.
What Is the Agent Readiness Score?
The Agent Readiness Score is a free, public scanner built by Cloudflare. It measures how technically readable your website is to AI agents, including ChatGPT, Claude, Perplexity, and Gemini-based agents.
You run it at isitagentready.com by entering any URL. Within seconds it returns an overall score out of 100, a Level badge, a breakdown across five categories, and specific fixes for every failed check.
Unlike traditional SEO tools, which measure how well Google can crawl and rank your pages, this score measures something different. It measures whether an autonomous AI agent can understand, trust, and use your site.
Since May 12, 2026, this same scoring engine has also been folded directly into the Cloudflare dashboard through the URL Scanner tool, where it reports six specialized sub-scores.
Why This Score Suddenly Matters
For two decades, the web was built for two audiences. Humans with browsers, and search engine crawlers that indexed pages so humans could find them.
In 2026, a third audience has arrived at real scale. Autonomous AI agents now read pages, extract facts, compare options, and take actions, often without a human ever loading the page themselves.
A few numbers explain why site owners are paying attention.
Industry benchmark data from May 2026 shows the top 100 websites by traffic average just 55% on the Agent Readiness Score. Even category leaders like Cursor top out around 82%, meaning almost nobody has this fully solved yet.
Sites that pass core agent-readiness checks produce 31% fewer tokens consumed and 66% faster answers when an AI agent reads them. The agent isn’t wasting time parsing bloated HTML and JavaScript just to find the actual sentence it needs.
A mid-market e-commerce brand that systematically fixed its readiness issues over 100 days moved its score from 42 to 78. It saw a 920% increase in AI-driven referral traffic in the same period.
Data from AI visibility firms shows readiness scores above 75 correlate strongly with stronger AI citation presence. Scores below 50 frequently result in a site being invisible in agent-generated answers.
The pattern is consistent across every data source. Agents cite, recommend, and transact with sites they can technically parse, and they skip the ones they can’t, no matter how good the content actually is.
The 5 Categories and 16 Checks, Explained
The Agent Readiness Score is built from 16 individual checks, grouped into 5 categories. Here’s the full map before we go fix by fix.
| Category | What It Measures | Checks Inside |
|---|---|---|
| Discoverability | Can an agent find your site and its structure at all? | robots.txt, sitemap.xml, Link headers (RFC 8288), DNS for AI Discovery (DNS-AID) |
| Content | Can an agent read your content efficiently? | Markdown content negotiation |
| Bot Access Control | Have you explicitly told AI bots what they can and can’t do? | AI bot rules in robots.txt, Web Bot Auth request signing, Content Signals |
| Capabilities (API, Auth, MCP & Skill Discovery) | Can an agent interact with your site programmatically, not just read it? | API Catalog, OAuth/OIDC discovery, OAuth Protected Resource metadata, Auth.md, MCP Server Card, Agent Skills index, WebMCP |
| Commerce (Optional) | Can an agent pay you directly? | x402 Protocol, MPP, Universal Commerce Protocol (UCP), Agentic Commerce Protocol (ACP) |
A few notes before you start fixing things.
Commerce is optional and doesn’t count against your score unless you run an e-commerce or paid-API business. The scanner auto-detects this and treats it as informational only.
Not every check applies to every site. A personal blog has no business publishing OAuth metadata, because it has no protected API for an agent to authenticate against. We’ll flag which checks are genuinely worth chasing.
Most of these standards are brand new. Several are still IETF drafts, not ratified RFCs. That’s fine. Being early is exactly the advantage.
Now let’s fix them, category by category.
Step 1: Fix Discoverability
Discoverability is worth up to 4 points in the raw check count. It’s almost always the first place people lose score. Here’s each item.
1.1 Robots.txt (usually already passing)
Every site needs a valid /robots.txt file. If yours doesn’t exist or returns anything other than a 200 status, this fails immediately.
Most modern CMS platforms generate this automatically through SEO plugins like Rank Math or Yoast. This check usually passes without you touching anything.
Minimum viable robots.txt:
User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml1.2 Sitemap.xml
The scanner extracts your Sitemap directive from robots.txt and checks that it resolves to a valid XML sitemap. On WordPress, Rank Math and Yoast SEO generate this by default.
1.3 Link Headers (RFC 8288)
This is the check most sites fail, and it’s genuinely useful, not just a scanner formality. RFC 8288 defines the HTTP Link header.
It lets your server tell an agent “here’s a related resource” before the agent even has to parse your HTML body. The scanner wants to see a Link header on your homepage pointing agents toward something useful.
Example fix (WordPress, via functions.php or a must-use plugin):
add_action('send_headers', function () {
if (is_admin()) return;
$skills_url = home_url('/.well-known/agent-skills/index.json');
header('Link: <' . esc_url_raw($skills_url) . '>; rel="agent-skills"', false);
});Example fix (Node.js / Express):
app.use((req, res, next) => {
res.setHeader(
'Link',
'</.well-known/agent-skills/index.json>; rel="agent-skills"'
);
next();
});Example fix (Nginx, at the server block level):
add_header Link '</.well-known/agent-skills/index.json>; rel="agent-skills"' always;Once this is live, verify it with:
curl -I https://example.com/ | grep -i "^link:"1.4 DNS for AI Discovery (DNS-AID)
This is the newest and most experimental check in the scanner. It’s still an active IETF draft, not a finished standard.
DNS-AID proposes publishing SVCB/HTTPS DNS records under well-known subdomains like _index._agents.example.com. An agent can discover your capabilities purely through a DNS lookup, without fetching a webpage.
Example record to add in your DNS provider:
| Type | Name | Priority | Target |
|---|---|---|---|
| HTTPS | _index._agents | 1 | . with alpn="h2" and a custom endpoint parameter |
This standard has essentially zero real-world adoption today. Treat it as a low-priority item. If your DNS is on Cloudflare, this is a five-minute addition under DNS, Records, Add record.
Step 2: Fix Content
This category has one check, but it’s arguably the most functionally important item on the whole list: Markdown content negotiation.
2.1 Markdown Negotiation
Here’s the idea. When a browser requests your page, it sends Accept: text/html. When an AI agent requests your page, it may instead send Accept: text/markdown.
The agent is explicitly asking for a lightweight, structured version of your content instead of full HTML soup. If your server ignores that header and always returns HTML, you fail this check.
More importantly, you force every agent visiting your site to burn tokens and time stripping out your markup just to find your actual sentences.
Example fix (WordPress, homepage-level):
add_action('template_redirect', function () {
if (!is_front_page() && !is_home()) return;
$accept = $_SERVER['HTTP_ACCEPT'] ?? '';
if (strpos($accept, 'text/markdown') === false) return;
$site_name = get_bloginfo('name');
$tagline = get_bloginfo('description');
$md = "# {$site_name}nn";
if ($tagline) $md .= "{$tagline}nn";
$front_id = (int) get_option('page_on_front');
if ($front_id) {
$post = get_post($front_id);
$text = wp_strip_all_tags(strip_shortcodes($post->post_content));
$text = html_entity_decode(trim(preg_replace('/s+/', ' ', $text)));
$md .= wp_trim_words($text, 120, '...') . "nn";
}
status_header(200);
header('Content-Type: text/markdown; charset=utf-8');
echo $md;
exit;
}, 5);Example fix (Next.js / Node route):
export default function handler(req, res) {
if (req.headers.accept?.includes('text/markdown')) {
res.setHeader('Content-Type', 'text/markdown; charset=utf-8');
return res.send(renderPageAsMarkdown());
}
return res.send(renderPageAsHtml());
}Verify it with:
curl -H "Accept: text/markdown" https://example.com/You should get back a 200 status and a text/markdown content type, with clean readable Markdown, not your full HTML template.
2.2 llms.txt (Not officially scored, but closely related)
You’ll see llms.txt mentioned constantly alongside this category. It isn’t one of the 16 scored checks, but it’s worth understanding.
Proposed by Jeremy Howard of Answer.AI, /llms.txt is a curated Markdown file at your site root. It gives an LLM a short overview of your site plus links to detailed Markdown versions of your key pages.
Companies like Anthropic, Vercel, and Hugging Face have already adopted it. IDE agents such as Cursor, Windsurf, Claude Code, and GitHub Copilot fetch it routinely.
It’s not an official W3C or IETF standard yet, and it’s not a confirmed Google ranking factor. But Chrome Lighthouse has already added an Agentic Browsing audit category that checks for its presence.
Minimal llms.txt example:
# Example Company
> We build [short description of what you do].
## Docs
- [Getting Started](https://example.com/docs/getting-started.md)
- [API Reference](https://example.com/docs/api.md)
## About
- [About Us](https://example.com/about.md)
- [Contact](https://example.com/contact.md)Step 3: Fix Bot Access Control
This category is about being explicit. Tell AI bots exactly what they may and may not do with your content, rather than leaving them to guess from a generic Allow rule.
3.1 AI Bot Rules in robots.txt
The scanner checks whether you have dedicated user-agent rules for known AI crawlers, rather than relying purely on the wildcard block. It checks against 15+ known AI bot identifiers.
| Company | Training Crawler | Search/RAG Crawler |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot | Claude-User, Claude-SearchBot |
| Perplexity | none | PerplexityBot |
| Google-Extended | none | |
| ByteDance | Bytespider | none |
| Common Crawl | CCBot | none |
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
Disallow: /wp-admin/A quick but important note. A generic wildcard Allow rule with no Disallow on AI bots technically satisfies “not blocked,” and the scanner notes this as wildcard rules applying.
But if you want to differentiate, say allow search crawlers but disallow training crawlers, dedicated rules are the only way to express that.
Also worth knowing: robots.txt compliance is voluntary. It expresses your preference; it doesn’t technically prevent access.
Most major commercial bots publicly commit to honoring it and have been observed doing so. Cloudflare published a report in August 2025 documenting Perplexity running undeclared crawlers that rotated user-agents and IPs to bypass no-crawl directives.
3.2 Content Signals
This is a newer, more granular alternative to blanket allow/disallow rules, championed by Cloudflare through contentsignals.org. It lets you declare three separate preferences in plain text inside robots.txt.
The search signal covers whether this content can be used to build a search index. The ai-input signal covers whether it can be fed into an AI model to generate a real-time answer.
The ai-train signal covers whether it can be used to train or fine-tune an AI model.
Example directive:
Content-Signal: ai-train=no, search=yes, ai-input=yesThis says: don’t train a model on my content, but you’re welcome to index it for search and use it to answer a live question.
It’s currently an IETF draft, developed alongside the broader IETF AI Preferences Working Group. It’s the single easiest fix on this entire list: one line, no code, no deployment.
3.3 Web Bot Auth Request Signing
This is the most advanced item in the category. Web Bot Auth lets a crawler cryptographically sign its HTTP requests so your server can verify its identity.
It requires publishing a directory at /.well-known/http-message-signatures-directory. Unless you’re running Cloudflare’s Bot Management product with Web Bot Auth enabled, this isn’t worth chasing manually.
It requires cryptographic key infrastructure most sites don’t need. If you’re on Cloudflare, check Security, then Bots in your dashboard for a toggle.
Step 4: Fix Capabilities
This is the largest category, seven checks, and the one where you should be most selective. These checks exist for sites that expose real APIs or interactive tools to agents.
If your site is a blog, portfolio, or marketing site with no API, most of these are not worth faking.
4.1 API Catalog (RFC 9727)
Serve /.well-known/api-catalog returning application/linkset+json, listing your APIs with links to their spec, documentation, and status endpoint.
{
"linkset": [
{
"anchor": "https://api.example.com/",
"service-desc": [{ "href": "https://api.example.com/openapi.json" }],
"service-doc": [{ "href": "https://example.com/docs/api" }]
}
]
}Only implement this if you actually have a public API.
4.2 OAuth / OIDC Discovery
Publish /.well-known/openid-configuration or /.well-known/oauth-authorization-server per RFC 8414. This lets an agent programmatically learn your token and authorization endpoints.
Relevant only if agents need to authenticate against your service.
4.3 OAuth Protected Resource Metadata (RFC 9728)
Publish /.well-known/oauth-protected-resource describing which authorization servers can issue valid tokens for your API.
4.4 Auth.md Agent Registration
A newer convention, championed by WorkOS, of serving a plain-language /auth.md file at your site root. It gives instructions for how an agent should register and authenticate.
4.5 MCP Server Card
If you run a Model Context Protocol server, think of MCP as a standardized way for an AI agent to connect to your tools and data. Publish a server card at /.well-known/mcp/server-card.json.
{
"serverInfo": { "name": "example-mcp-server", "version": "1.0.0" },
"transport": { "type": "http", "url": "https://example.com/mcp" },
"capabilities": { "tools": true, "resources": true }
}Skip this entirely if you don’t run an MCP server. Publishing it otherwise is actively misleading.
4.6 Agent Skills Index
Publish /.well-known/agent-skills/index.json per the emerging Agent Skills Discovery RFC, listing reusable skills your site exposes to agents.
{
"$schema": "https://agentskills.io/schema/v0.2.0/index.json",
"skills": []
}Even an empty, schema-valid index satisfies the existence check. This is the one item in this category worth doing even on a simple site, because it directly supports the Link header check from Step 1.
4.7 WebMCP
This checks whether your page registers browser-side tools via navigator.modelContext.provideContext(). This emerging API lets an in-browser AI agent call functions your page exposes directly.
if ('modelContext' in navigator) {
navigator.modelContext.provideContext({
tools: [
{
name: 'search_products',
description: 'Search the product catalog',
inputSchema: { type: 'object', properties: { query: { type: 'string' } } },
execute: async ({ query }) => {
const results = await fetch(`/api/search?q=${query}`).then(r => r.json());
return results;
}
}
]
});
}Only relevant for interactive web apps, not static content sites.
If you’re not running an API, don’t manufacture fake OAuth or MCP metadata just to move a number. The Agent Skills index is the one universally low-effort, honestly-applicable item here.
Step 5: Fix Commerce (Optional)
This category is auto-excluded from scoring for non-commerce sites. But if you sell anything, physical goods, subscriptions, or paid API access, it’s worth understanding the landscape.
| Protocol | Backed By | What It Does |
|---|---|---|
| x402 | Coinbase | Lets a server return HTTP 402 Payment Required with stablecoin terms. The agent pays and retries. Released May 2025. |
| UCP | Google and Shopify | Open commerce discovery layer, live in Google AI Mode and Shopify Sidekick since January 2026. |
| ACP | OpenAI and Stripe | Standardizes conversational checkout with a single-use Shared Payment Token bound to a specific merchant and amount. |
| MPP | Independent/open | Publishes payment info via OpenAPI extensions, supporting multiple payment rails. |
In practice, these protocols are complementary rather than competing. An agent stack might use one for authorization, another for checkout, and x402 or MPP for pure machine-to-machine settlement.
If you run e-commerce, start with UCP or ACP depending on whether your primary channel is Google/Shopify or ChatGPT.
Real Example: A Live Site From 21 to 36+
To make this concrete, here’s what actually happened on a real WordPress site running Cloudflare, Rank Math, and WP Rocket.
Starting point: Score of 21/100, Level 1. Discoverability 2/4, Content 0/1, Bot Access Control 1/2, Capabilities 0/7.
What was diagnosed:
- robots.txt existed and was valid, but had zero AI-specific content signals.
- Homepage had no Link header at all.
- No agent-skills index existed.
- Requesting the homepage with an Accept markdown header returned a 401, not a markdown response.
What was fixed, in order:
- Added a Content-Signal line to robots.txt via a filter hook.
- Added a Link header advertising the Agent Skills index on every page load.
- Published a schema-valid, empty Agent Skills index using a custom rewrite rule.
- Added homepage-only Markdown negotiation logic.
The result after fixing the code, but before verifying, was that the live site still showed the old broken results. The reason was server-side and CDN caching.
The origin server was already returning the correct headers and content. But a host-level cache and Cloudflare’s edge cache were still serving 12-plus hour-old cached copies of robots.txt and the homepage.
The fix was a full cache purge at every layer. The origin’s page cache, the host-level cache, and Cloudflare’s edge cache via Purge Everything.
After the purge propagated, Content jumped from 0/1 to 1/1. Bot Access Control jumped from 1/2 to 2/2. The overall score rose from 21 to 36, crossing from Level 1 straight to Level 4, Agent-Integrated.
The lesson: if you fix your code and the scanner still shows old results, don’t assume your fix is wrong. Check your cache layers first. CDN and page caches are the single most common reason a correct fix doesn’t show up in the scan.
Common Mistakes That Silently Cap Your Score
Forgetting to purge cache after every fix can make a perfectly correct fix look like a failure for hours.
Publishing fake MCP, OAuth, or API metadata just to pass a check doesn’t help. It actively misleads any agent that tries to use it, which can damage trust signals long-term.
Blocking AI bots at the CDN or firewall layer while allowing them in robots.txt is a common gap. Robots.txt is a courtesy signal; your firewall is an enforcement layer. Check both.
In Cloudflare specifically, look at Security, Bots, Block AI Bots, and Bot Fight Mode. Both can silently override a permissive robots.txt.
Only fixing the homepage is another trap. Markdown negotiation and Link headers implemented only on the homepage will pass the scanner but won’t help an agent trying to read your actual product or blog pages.
Ignoring the DNS-AID and Web Bot Auth checks is fine, as long as it’s deliberate. They’re early-stage drafts with minimal real-world agent adoption today.
Treating Commerce checks as mandatory wastes engineering time. They’re explicitly optional and don’t count against non-commerce sites.
How to Verify Every Fix Actually Works
Don’t rely on the scanner alone while you’re actively making changes. CDN cache lag can make a correct fix look broken for several minutes. Verify directly first.
Check robots.txt content:
curl -s https://example.com/robots.txtCheck for a Link header:
curl -sI https://example.com/ | grep -i "^link:"Check Markdown negotiation:
curl -s -H "Accept: text/markdown" https://example.com/ -D - -o /dev/null | grep -i "content-type"Check your Agent Skills index:
curl -s https://example.com/.well-known/agent-skills/index.jsonBypass cache to test the true origin response:
curl -s "https://example.com/robots.txt?bust=$(date +%s)"Once all of these return what you expect directly, re-run the full scan at isitagentready.com. If your site sits behind Cloudflare, purge cache first.
Agentic Website & Agentic Search: The Complete FAQ
What is an “agentic website”?
An agentic website is a site technically structured so autonomous AI agents, not just human visitors, can discover, read, understand, and interact with it programmatically.
This includes serving clean Markdown alongside HTML, publishing machine-readable capability listings, declaring explicit AI bot permissions, and exposing APIs agents can call directly.
What is “agentic search,” and how is it different from traditional search?
Traditional search is a single, manual act. A person types a query, scans a results page, and clicks a link to read it themselves.
Agentic search is fundamentally different. An AI agent plans and executes multiple coordinated search queries on your behalf, evaluates what each result set contains, and identifies gaps in the information.
It reformulates its queries to fill those gaps and repeats the cycle until it has enough to answer the original question. Research benchmarks show agent-generated queries average 4.9 search steps, with some requiring up to seven.
Often the human never sees the underlying pages the agent visited. They only see the agent’s synthesized answer.
How do AI agents actually browse a website?
Modern agents combine search APIs with real, rendered browser sessions, a pattern called agentic browsing. This matters because JavaScript-heavy pages return empty shells to a simple HTTP request.
Production-grade agents drive real headless browsers that execute JavaScript and fully render the page before extracting data, much closer to how a human browser works.
Do I need to rebuild my whole website to be agent-ready?
No. Nearly every fix in this guide is additive: new HTTP headers, new files at well-known paths, and small conditional logic based on the Accept header.
None of it requires changing your existing HTML, design, or CMS. Most of the highest-impact fixes can be implemented in under an hour, even on an established site.
Which AI crawlers should I actually allow in robots.txt?
At minimum, allow GPTBot, ChatGPT-User, and OAI-SearchBot from OpenAI. Add ClaudeBot, Claude-User, and Claude-SearchBot from Anthropic.
Also allow PerplexityBot and Google-Extended, Google’s AI training crawler, which is separate from regular Googlebot.
If you want maximum visibility in AI-generated answers, allow all of them. If you object to training but are fine with live answering, use Content Signals instead of blanket blocking.
Will blocking AI bots hurt my SEO or Google rankings?
Blocking Google-Extended specifically will not affect your regular Google Search ranking. Google has stated this crawler is separate from Googlebot, the crawler used for search indexing.
It does opt you out of being used to improve Google’s AI features, a separate and growing visibility channel. Blocking GPTBot, ClaudeBot, or PerplexityBot has no effect on Google rankings at all, since they’re unrelated companies.
But it does mean your content won’t be cited when users ask those assistants questions your site could answer.
What’s the difference between robots.txt AI rules and Content Signals?
Robots.txt AI bot rules are binary: allow or disallow a specific crawler from fetching your pages at all. Content Signals are more granular.
They let you separately express permission for distinct uses of content that’s already been fetched: search indexing, live answering, and model training.
You can allow a bot to crawl your site while explicitly declaring you don’t want that content used for training, a distinction blanket rules can’t express.
Is llms.txt an official standard? Do I need it?
No. As of 2026, llms.txt is a widely-adopted community convention, not a ratified standard from the W3C or IETF, and not a confirmed Google ranking factor.
Adoption among AI-forward companies is real, and IDE coding agents fetch it routinely. Chrome Lighthouse now audits for its presence under an Agentic Browsing category.
It’s not required for a good score, but it’s a low-cost addition, especially for documentation-heavy sites.
What is MCP, and does my website need an MCP server?
The Model Context Protocol is an open standard, introduced by Anthropic, that lets AI applications connect to external tools and data in a standardized way.
Your website needs an MCP server only if you want agents to actively do things through your site, like query a database or place an order, rather than just read your content.
A blog or informational site almost never needs one. A SaaS product or booking platform with real workflows might.
What is Web Bot Auth, and is it worth setting up?
Web Bot Auth lets a legitimate crawler cryptographically sign its HTTP requests, so your server can verify the request truly came from that bot.
It’s valuable for high-traffic sites concerned about fake-bot scraping disguised as AI crawling. It’s most accessible if you’re already on Cloudflare’s Bot Management product.
For most small-to-mid sites, it’s a lower priority compared to the Discoverability and Content fixes.
How long does it take to see AI agents actually citing my site after making these changes?
There’s no universal timeline. It depends on how frequently each AI provider’s crawler re-indexes your domain and how their retrieval system weighs your content.
The e-commerce case study referenced earlier saw meaningful movement over a 100-day window. Readiness-score improvements of that scale reliably correlate with results in weeks to a few months, not days.
Does the Agent Readiness Score affect my Google Search ranking?
No. The Agent Readiness Score and Google’s ranking algorithm are entirely separate systems measuring different things. It is not a confirmed Google ranking signal.
Several underlying fixes, like a valid robots.txt and fast clean markup, overlap with long-standing SEO best practices, so improving one rarely hurts the other.
Can I fake or spoof a high Agent Readiness Score without real changes?
Technically, some checks like an empty Agent Skills index can be added just to satisfy a presence check. But doing this without real substance is counterproductive.
An agent that discovers your MCP server card and finds no working capabilities won’t simply ignore it. It wastes the agent’s time and can register your site as an unreliable source.
What’s the single highest-impact fix if I only have 30 minutes?
Add the Content-Signal line to your robots.txt and implement Markdown negotiation on your homepage.
Together these two fixes typically move the needle the most relative to effort, since Content is a full category on its own and Bot Access Control is usually a five-minute win.
Do static sites need different fixes than WordPress?
The concepts are identical; only the implementation mechanism changes. On a static site, you configure your CDN or hosting platform to add response headers.
You’d also serve pre-generated Markdown files alongside your HTML, using an edge function to perform content negotiation based on the Accept header.
Is the Commerce category worth pursuing if I’m a small online store?
If AI shopping agents are a growing channel for your business, starting with whichever protocol matches your primary sales channel is worth the investment.
If you’re a small store without dedicated engineering resources, it’s reasonable to deprioritize this until the other four categories are solid.
Final Checklist: Your Path to 100/100
Use this as your working checklist. Items marked with a star are the highest-leverage, lowest-effort fixes. Start there.
Discoverability
- Valid robots.txt returning 200
- Valid sitemap.xml referenced in robots.txt
- ★ Link header on homepage response
- DNS-AID records (low priority, early-stage standard)
Content
- ★ Markdown negotiation on key pages
Bot Access Control
- Dedicated AI bot rules in robots.txt
- ★ Content-Signal directive in robots.txt
- Web Bot Auth signing (only if on Cloudflare Bot Management)
Capabilities (only if genuinely applicable to your site)
- ★ Agent Skills index
- API Catalog (only if you have a public API)
- OAuth/OIDC discovery (only if agents need to authenticate)
- OAuth Protected Resource metadata (same condition)
- Auth.md (same condition)
- MCP Server Card (only if you run an MCP server)
- WebMCP tools (only for interactive web apps)
Commerce (optional, e-commerce/paid-API sites only)
- UCP or ACP discovery document
- x402 or MPP payment middleware on protected routes
After every change:
- Purge every cache layer
- Verify directly with curl before re-scanning
- Re-run isitagentready.com
Getting a website to a genuine, honest 100/100 is realistically achievable mainly for sites with real APIs, real commerce, and real agent-facing infrastructure to describe. That’s fine.
For most websites, a score in the 60 to 80 range, earned by fully solving Discoverability, Content, and Bot Access Control, represents genuine functional agent-readiness. Not just a number, but a site that AI agents can actually read, trust, and recommend.
Sources and further reading: Cloudflare’s official Agent Readiness announcement, isitagentready.com, Content Signals specification, llms.txt proposal, Model Context Protocol documentation, RFC 8288, Web Linking, RFC 8414, OAuth 2.0 Authorization Server Metadata, RFC 9728, OAuth 2.0 Protected Resource Metadata.
