Technical setup

Technical setup makes sure AI tools and search engines can reach, read and understand your pages. If a crawler is blocked by mistake, nothing else you do will be seen.

Let the right crawlers in

We check your robots.txt file, your sitemap and any firewall or bot-protection settings. A single line such as Disallow: / can hide a whole site. Some security tools also block crawlers without telling you.

What the main crawlers do

Each company can change how its crawlers work. We check their own documentation before we advise you.

CrawlerRun byWhat it does
OAI-SearchBotOpenAIFinds pages to show in ChatGPT search results.
GPTBotOpenAICollects content that may be used to train AI models.
ChatGPT-UserOpenAIOpens a page when a person asks ChatGPT to look at it.
PerplexityBotPerplexityFinds pages to use in Perplexity answers.
ClaudeBotAnthropicCollects content that may be used to train Claude models.
Google-ExtendedGoogleA robots.txt control that decides whether content Google crawls may be used for Gemini. It is not a separate crawler.

Structured data

Structured data is code that describes a page to machines. We add the types that fit your site, such as Organization, LocalBusiness, Service, Product, Article and FAQPage. The markup always matches the text a visitor can see.

Speed and clean pages

Fast, clean pages are easier to crawl and nicer to use. We flag heavy scripts, broken links and missing titles or descriptions.

Honest limits

  • robots.txt is a request, not a lock. Well-behaved crawlers follow it. Others may not.
  • Letting a crawler in does not guarantee that your page is shown or quoted.
  • Google says its AI features use the same systems as Search, so good basic SEO still matters.

Frequently asked questions

Should I allow AI training crawlers?
It is your choice. If you want AI tools to know your brand, allowing them can help. If you prefer to keep content out of training, we set that up and explain the trade-off.
Can you fix this on my site?
We give you exact changes, and where we have access we apply them. We do not change things without your approval.
⚡ Hosted for free with WebotApp Cloud ↗