robots.txt and AI crawlers

robots.txt is a small text file at the root of your site that tells crawlers which parts they may visit. One wrong line, Disallow: /, can hide your whole site from search and AI tools.

A safe example file

This file lets all crawlers in, keeps them out of private areas and points to your sitemap. Change the paths to match your site.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Common AI and search crawlers

Companies can change how their crawlers work, so check their own documentation too.

CrawlerRun byWhat it does
GooglebotGoogleCrawls pages for Google Search.
BingbotMicrosoftCrawls pages for Bing.
OAI-SearchBotOpenAIFinds pages for ChatGPT search results.
GPTBotOpenAICollects content that may be used to train AI models.
ChatGPT-UserOpenAIOpens a page when a person asks ChatGPT to look at it.
PerplexityBotPerplexityFinds pages for Perplexity answers.
ClaudeBotAnthropicCollects content that may be used to train Claude models.
Google-ExtendedGoogleA control that decides whether content Google crawls may be used for Gemini. It is not a separate crawler.

The Disallow: / mistake

The line Disallow: / under User-agent: * blocks every crawler from every page. It is often left in by mistake after a site is built on a test address.

If your site is not appearing anywhere, check this file first. Remove the line, or change it to a narrower path.

Should you allow AI crawlers?

  • If you want AI tools to know and name your brand, allowing them is sensible.
  • If you prefer to keep content out of AI training, you can block the training crawlers and still allow search crawlers. This is your choice.
  • Remember that robots.txt is a request. Well-behaved crawlers follow it, but it is not a lock.

Other things that can block crawlers

  • A firewall or bot-protection setting that returns an error to crawlers.
  • A noindex tag or X-Robots-Tag header.
  • Pages behind a login.

Frequently asked questions

Where does robots.txt go?
At the root of your site, for example https://example.com/robots.txt. It will not work in a sub-folder.
Does robots.txt hide a page from search?
Not reliably. It stops crawling, but a blocked page can still appear if other pages link to it. Use noindex to keep a page out of results.
Will allowing AI crawlers guarantee a mention?
No. It only lets them read your pages. Whether a tool names you is up to the tool.
⚡ Hosted for free with WebotApp Cloud ↗