robots.txt and AI crawlers
robots.txt is a small text file at the root of your site that tells crawlers which parts they may visit. One wrong line, Disallow: /, can hide your whole site from search and AI tools.
A safe example file
This file lets all crawlers in, keeps them out of private areas and points to your sitemap. Change the paths to match your site.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlCommon AI and search crawlers
Companies can change how their crawlers work, so check their own documentation too.
| Crawler | Run by | What it does |
|---|---|---|
| Googlebot | Crawls pages for Google Search. | |
| Bingbot | Microsoft | Crawls pages for Bing. |
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search results. |
| GPTBot | OpenAI | Collects content that may be used to train AI models. |
| ChatGPT-User | OpenAI | Opens a page when a person asks ChatGPT to look at it. |
| PerplexityBot | Perplexity | Finds pages for Perplexity answers. |
| ClaudeBot | Anthropic | Collects content that may be used to train Claude models. |
| Google-Extended | A control that decides whether content Google crawls may be used for Gemini. It is not a separate crawler. |
The Disallow: / mistake
The line Disallow: / under User-agent: * blocks every crawler from every page. It is often left in by mistake after a site is built on a test address.
If your site is not appearing anywhere, check this file first. Remove the line, or change it to a narrower path.
Should you allow AI crawlers?
- If you want AI tools to know and name your brand, allowing them is sensible.
- If you prefer to keep content out of AI training, you can block the training crawlers and still allow search crawlers. This is your choice.
- Remember that robots.txt is a request. Well-behaved crawlers follow it, but it is not a lock.
Other things that can block crawlers
- A firewall or bot-protection setting that returns an error to crawlers.
- A noindex tag or X-Robots-Tag header.
- Pages behind a login.
Frequently asked questions
Where does robots.txt go?
At the root of your site, for example https://example.com/robots.txt. It will not work in a sub-folder.
Does robots.txt hide a page from search?
Not reliably. It stops crawling, but a blocked page can still appear if other pages link to it. Use noindex to keep a page out of results.
Will allowing AI crawlers guarantee a mention?
No. It only lets them read your pages. Whether a tool names you is up to the tool.