Your robots.txt file tells crawlers which parts of your site they may access. With the rise of AI search, it now also decides whether AI assistants can read — and cite — your content.
Types of AI crawlers
- Search/retrieval bots fetch pages to answer user questions in real time and often cite sources (for example OAI-SearchBot, PerplexityBot, Claude-SearchBot).
- User-triggered agents fetch a page when a user asks an assistant to (for example ChatGPT-User, Claude-User).
- Training crawlers collect content for model training (for example GPTBot, ClaudeBot, CCBot, Google-Extended as a control token).
The trade-off
Blocking all AI bots protects content from use in training but can reduce your visibility in AI answers. For most service businesses that want to be recommended, allowing search and retrieval bots is the better default. Publishers with premium content may choose differently.
An example configuration
``` User-agent: * Allow: / Disallow: /admin
User-agent: OAI-SearchBot Allow: /
User-agent: PerplexityBot Allow: /
Sitemap: https://example.com/sitemap.xml ```
Common mistakes
- Blocking CSS and JavaScript files that pages need to render
- Using robots.txt to hide sensitive pages (use authentication instead)
- Forgetting the sitemap reference
- Accidentally blocking the whole site after a staging launch
FAQs
Does robots.txt stop content from being indexed?
It stops crawling, not necessarily indexing of URLs linked elsewhere. Use noindex for pages that shouldn't appear in search.
Do all bots obey robots.txt?
Reputable crawlers do; malicious bots may not.
How do I test robots.txt?
Use Search Console's robots.txt report and check server logs for bot activity.
