Technical setup
Technical setup makes sure AI tools and search engines can reach, read and understand your pages. If a crawler is blocked by mistake, nothing else you do will be seen.
Let the right crawlers in
We check your robots.txt file, your sitemap and any firewall or bot-protection settings. A single line such as Disallow: / can hide a whole site. Some security tools also block crawlers without telling you.
What the main crawlers do
Each company can change how its crawlers work. We check their own documentation before we advise you.
| Crawler | Run by | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search results. |
| GPTBot | OpenAI | Collects content that may be used to train AI models. |
| ChatGPT-User | OpenAI | Opens a page when a person asks ChatGPT to look at it. |
| PerplexityBot | Perplexity | Finds pages to use in Perplexity answers. |
| ClaudeBot | Anthropic | Collects content that may be used to train Claude models. |
| Google-Extended | A robots.txt control that decides whether content Google crawls may be used for Gemini. It is not a separate crawler. |
Structured data
Structured data is code that describes a page to machines. We add the types that fit your site, such as Organization, LocalBusiness, Service, Product, Article and FAQPage. The markup always matches the text a visitor can see.
Speed and clean pages
Fast, clean pages are easier to crawl and nicer to use. We flag heavy scripts, broken links and missing titles or descriptions.
Honest limits
- robots.txt is a request, not a lock. Well-behaved crawlers follow it. Others may not.
- Letting a crawler in does not guarantee that your page is shown or quoted.
- Google says its AI features use the same systems as Search, so good basic SEO still matters.