AI search readiness
Know the day ChatGPT can't read a client's site
Check every client's site each night for what keeps ChatGPT, Claude and Perplexity out, and hear when that changes.
- Free for one client
- No card
- Read-only access
The numbers behind it
- robots.txt for OAI-SearchBot
- Disallow: / (User-agent: oai-searchbot)
- robots.txt for ChatGPT-User
- Disallow: / (User-agent: chatgpt-user)
- robots.txt for Claude-SearchBot
- Disallow: / (User-agent: claude-searchbot)
- robots.txt for Claude-User
- Disallow: / (User-agent: claude-user)
Score
75
/ 100trailheadcoffee.example · checked today
| Assistant | Can quote it in answers | Can learn from it |
|---|---|---|
| ChatGPT | Blocked (OAI-SearchBot, ChatGPT-User) | Blocked (GPTBot) |
| Claude | Blocked (Claude-SearchBot, Claude-User) | Blocked (ClaudeBot) |
| Perplexity | Allowed | |
| Google AI Overviews & Gemini | Allowed | Allowed |
| Microsoft Copilot | Allowed | |
| Apple Intelligence | Allowed | Allowed |
| Meta AI | Allowed | |
| Common Crawl | Allowed |
AI assistants can read the page17/30
robots.txt blocks ChatGPT, Claude (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User), so they can't quote or link this page.
How to fix: Remove the Disallow rules for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User in robots.txt.
Page text is in the HTML20/20
540 words readable without running JavaScript.
Allowed in search and AI answers15/15
No noindex or nosnippet on the page.
Structured data (schema.org)10/15
Found CafeOrCoffeeShop, PostalAddress. Nothing says who's behind it (Organization, LocalBusiness, Product…).
How to fix: Add Organization or LocalBusiness JSON-LD with the business's name, logo and contact details.
Title and description5/5
“Trailhead Coffee – roastery and café in Baner, Pune”
Sitemap5/5
Crawlers can find every page from the sitemap.
llms.txt0/5
No /llms.txt: an optional Markdown summary of the site for AI tools.
How to fix: Publish /llms.txt: the site's name, what it offers, and links to its key pages in Markdown.
AI models can learn from the page3/5
Blocked for ChatGPT, Claude. A fair choice, but those models learn less about the business, so they mention it less.
Blocking training crawlers is a fair choice and costs little here; blocking the answer crawlers keeps the page out of AI answers altogether.
Every page the demo agency watches, checked today
- northwinddental.exampleNorthwind Dental
AI assistants can read and quote it
95/100 - lumenskincare.exampleLumen Skincare
AI assistants can read and quote it
100/100 - trailheadcoffee.exampleTrailhead Coffee
robots.txt blocks ChatGPT, Claude
75/100 - trailheadcoffee.example/shopTrailhead Coffee
robots.txt blocks ChatGPT, Claude
80/100
Sample data. Want to check one page now? Try the free AI search checker.
The check, before and after
Making sure one client page can be read and quoted by AI assistants.
Time you get back
2 h 20 min
Each time you check, if you look after 10 clients.
For one client
Our estimate for one page per client, checked by hand. Most people check far less often than every plugin update or deploy, which is how blocks go unnoticed for months.
By hand
15 min- 01Open robots.txt and read it for each AI crawler: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest5 min
- 02View the page source to see what's there before JavaScript runs3 min
- 03Look for noindex in the page and its headers2 min
- 04Run the structured data through a validator3 min
- 05Check for a sitemap and llms.txt2 min
With MetricDeck
1 min- 01Nothing to run: every page is checked each night, and an alert says when an answer crawler is blockedAutomatic
- 02Read the score and the fixes on Site health when you want the detail1 min
What you get
A score for every page
Eight checks worth 100 points, each with what it found and how to fix it.
Assistant by assistant
Whether ChatGPT, Claude, Perplexity, Google's AI Overviews, Copilot and Apple may quote the page and learn from it, with the robots.txt rule that blocks one.
An alert when it breaks
When an answer crawler is blocked (critical if it's Googlebot), robots.txt answers with an error, or the page says noindex. Blocking training crawlers is your call and never alerts.
What crawlers really see
The words in the HTML before JavaScript runs, the title and description, and the structured data types on the page.
Every page you watch
The website plus the extra pages you add: 2 on Starter, 5 on Pro and 10 on Agency.
In the summary and the report
The AI summary reads it with everything else, and the client report has an AI search readiness section for each page.
Why you can trust the result
No AI in the check: it's rules applied to what a crawler gets, so the same page always gives the same answer.
robots.txt is read the way RFC 9309 says, for each crawler, on the page's own path.
Training crawlers are your choice: blocking them costs a few points and never raises an alert.
It's the same check as the free AI search checker and the Site health tab, so they always agree.
FAQ
Questions
- How is this different from the free AI search checker?
- It's the same check and the same report. The free checker looks at one page, once, when you ask. MetricDeck checks every page you watch for every client each night, tells you when an assistant gets shut out, and puts the result in the client's report.
- Does blocking GPTBot keep a site out of ChatGPT?
- No. GPTBot only collects pages to train OpenAI's models. ChatGPT's search, and the pages it opens while answering, use OAI-SearchBot and ChatGPT-User; those are the ones that decide whether a site can be quoted.
- Why would a site block AI assistants without meaning to?
- Usually a "block AI" list copied into robots.txt that names the answer crawlers along with the training ones, a rule left over from a staging site, or a security plugin's setting. Pages that build their text in the browser are the other common gap: the crawlers behind ChatGPT, Claude and Perplexity don't run JavaScript.
- Does it catch a firewall that turns AI crawlers away?
- Not yet. MetricDeck reads robots.txt and fetches the page as its own checker, so a firewall rule that refuses only AI crawlers won't show up. robots.txt, the page's text, noindex and the rest are what it checks.
- How often is each page checked?
- With every full sync: each night, and whenever you press Sync now. Not hourly, since robots.txt and a page's content don't change by the hour.
- Which pages are checked?
- The project's website, plus the extra pages you add on its Site health tab: 2 on Starter, 5 on Pro and 10 on Agency.
More of what MetricDeck does
Attention page
Every client's Search Console, Analytics, Ads and website checked overnight: open alerts worst first, and each client's week against the one before.Alerts
Alerts when ad spend jumps, conversions stop, traffic or rankings drop, tracking breaks or a site goes down, each with its numbers and an AI explanation.AI summary
A summary of each client after every sync: what needs fixing, what went well and what to do next. Code works out every number; the AI only explains them.Client reports
Pick the sections, edit the AI summary, and send a PDF or a link with a password: Search Console, Analytics, Ads and site health in one client report.Search Console reporting
Clicks, impressions, CTR and position for every client, the pages and queries that lost clicks, and an alert when organic traffic drops.Google Analytics reporting
Sessions and key events by channel, landing page, device and country, visits from ChatGPT and Perplexity counted apart, and alerts on drops.Google Ads wasted spend
See the Google Ads search terms that spend without converting, the ones that don't match what you sell, and copy a negative keyword list. Read-only.ChatGPT ads reporting
Every client's ChatGPT ads from the OpenAI Ads API: spend, clicks, conversions, lifetime budgets running out and rejected ads, beside Google Ads.Site health monitoring
Hourly uptime checks confirmed before a site is called down, SSL and domain expiry, and mobile and desktop speed for every client's site, with alerts.
Watch every client's site for AI search
Add a project and its website is checked straight away, AI search readiness included. Free for one site.
Free for one client · No card · Read-only access