How to let ChatGPT, Claude and Perplexity read your website

More and more people ask ChatGPT, Claude, Perplexity or Google's AI Overviews before they search, or instead of it. To be the answer, an assistant first has to fetch your page and read it. A surprising number of sites stop that from happening without meaning to: a robots.txt rule copied from somewhere, a firewall setting, or a page whose text only appears after JavaScript runs.

Here's how assistants read the web, the mistakes that keep sites out of their answers, and how to check your own site in about a minute.

Two kinds of AI crawler

Most AI companies run more than one crawler, and robots.txt can treat each one differently:

• Answer crawlers fetch pages to show, quote and link them in an answer. Some build a search index ahead of time; others fetch a page the moment someone asks about it. Block these and the assistant can't cite you.
• Training crawlers collect pages to train future models. Blocking them is a reasonable choice: it doesn't stop an assistant quoting you today, though the models may know less about your business over time.

Which is which, for the assistants people use most:

Assistant	Fetches pages for answers	Collects pages for training
ChatGPT	OAI-SearchBot, ChatGPT-User	GPTBot
Claude	Claude-SearchBot, Claude-User	ClaudeBot
Perplexity	PerplexityBot, Perplexity-User	–
Google AI Overviews & Gemini	Googlebot	Google-Extended
Microsoft Copilot	bingbot	–
Apple Intelligence	Applebot	Applebot-Extended
Meta AI	–	meta-externalagent
Common Crawl	–	CCBot

Two things trip people up:

• Blocking GPTBot doesn't keep you out of ChatGPT. GPTBot is OpenAI's training crawler. ChatGPT's search uses OAI-SearchBot, and the pages it opens while answering come through ChatGPT-User.
• Google-Extended isn't about AI Overviews. AI Overviews and AI Mode are built from what Googlebot crawls for Search. Google-Extended only decides whether your pages are used for Gemini models. Blocking Googlebot to stay out of AI Overviews takes you out of Google altogether.

How sites shut assistants out by mistake

1. A robots.txt that blocks the answer crawlers

The most common case is a "block AI" list copied into robots.txt that includes the answer crawlers along with the training ones. Another is a blanket rule left over from a staging site:

User-agent: *
Disallow: /

Rules are read per crawler and per path, so a site can be open on the home page and closed on /pricing or /services, often the pages you most want quoted.

2. A firewall that says no when robots.txt says yes

Some CDNs and hosting firewalls, Cloudflare among them, offer a one-click setting to block AI crawlers, and some newer setups switch it on by default. When it's on, robots.txt can invite an assistant in while the firewall turns it away. If you use one, check its bot settings as well as robots.txt.

3. Text that only appears after JavaScript runs

The crawlers behind ChatGPT, Claude and Perplexity read the HTML your server sends and don't run JavaScript. A page that builds its text in the browser, as many single-page apps do, looks almost empty to them, so there's nothing to quote. Server-side rendering or pre-rendering fixes it.

4. A noindex or nosnippet left in place

noindex keeps a page out of search, and the assistants built on search leave it out too. nosnippet lets the page be found but not quoted: Google won't use it in AI Overviews, and Bing won't in Copilot. Both are easy to leave behind after a redesign.

A robots.txt that lets them in

Anything robots.txt doesn't disallow is allowed, so most sites need no change at all. If yours does block crawlers, naming the answer crawlers like this lets assistants read and quote you while your other rules stay as they are:

# Let AI assistants find, read and quote the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
Allow: /

Sitemap: https://example.com/sitemap.xml

Keep answers, skip training

If you'd rather models didn't train on your pages but still want to appear in answers, block only the training crawlers:

# Keep training crawlers out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# Everyone else, the answer crawlers included, may read the site
User-agent: *
Allow: /

Help them understand the page

Being readable is the first step; being understood is the next:

• Structured data. JSON-LD that says who's behind the site (Organization or LocalBusiness) and what the page is (Product, Service, Article, FAQPage) saves an assistant guessing.
• A clear title and description. They're the page's summary for any reader, person or model.
• A sitemap, listed in robots.txt, so crawlers find the rest of the site.
• An llms.txt at the root of the site: a short Markdown summary with links to your key pages. It's a newer, optional convention, and not every assistant reads it yet.

Check your site in a minute

Our free AI search checker reads any page the way these crawlers do: robots.txt for each assistant on that page's own path, the text that's there before JavaScript runs, noindex and nosnippet, structured data, the sitemap and llms.txt. It gives a score out of 100 and says how to fix each problem. No sign-up, and no AI involved: the same page always gets the same answer.

Sites change, though. A new robots.txt rule or a redesign can shut assistants out overnight. MetricDeck runs the same check on every site you look after each day, beside uptime, speed, SSL and your Search Console, Analytics and Google Ads numbers, and tells you when something breaks. It's free for one site.