What is robots.txt?
robots.txt is a text file at the root of a website that tells crawlers, including search engines and AI crawlers, which paths they may fetch.
robots.txt is a plain text file served at /robots.txt that tells automated crawlers which parts of a site they may fetch. Rules are grouped by User-agent.
User-agent: *
Disallow: /admin/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Things to know
- robots.txt controls crawling, not indexing. Use
noindexto keep a page out of results. - When several rules match, the most specific (longest) rule wins; for equal length,
Allowwins. - It is a request, not access control. Never use it to hide sensitive URLs, because the file itself is public.
- Each AI company uses its own user agent, so blocking one does not block the others.
Related
- AI crawler access checker
Free tool that reads your robots.txt and shows which AI crawlers are allowed or blocked, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot.
- Noindex and indexability checker
Free indexability checker. Find accidental noindex tags and X-Robots-Tag headers, robots.txt rules that block Google, missing sitemaps, off-site canonicals and broken pages.
- What does noindex mean?
Noindex is a directive in a robots meta tag or X-Robots-Tag header that tells search engines not to show a page in search results.