The essentials
- Definition: robots.txt is a text file at a site's root telling crawlers which URLs they may and may not visit.
- The limit: it blocks crawling, not indexing; a blocked page can stay indexed with no content. To remove a page from the index, use noindex.
- Syntax: User-agent, Disallow, Allow, Sitemap; the rules are read per crawler and per URL prefix.
- AI in 2026: GPTBot, ClaudeBot, PerplexityBot and Google-Extended identify themselves here; blocking them removes the site from assistants' answers.
The aim of robots.txt is a text file placed at a site's root (https://www.exemple.fr/robots.txt) telling crawlers which parts of the site they are allowed to visit. It is the first file a crawler consults before visiting a page. It belongs to the robots exclusion protocol, standardised in 2022 (RFC 9309) and respected by Google, Bing and most AI crawlers, but it carries no binding force: a malicious crawler can ignore it.
What robots.txt does, and what it doesn't
It controls crawling, not indexing. A URL blocked by Disallow isn't visited, but it can be indexed if links point to it: Google then shows a result with no description. To remove a page from the index, you have to leave it crawlable and put a noindex tag on it; blocking a noindex page in robots.txt is precisely what stops Google reading that tag. Nor does robots.txt protect confidential content: on the contrary, it says where it is. The meta robots tags and the response codes remain the tools of indexing.
The syntax
| Directive | Role | The full destination URL |
|---|---|---|
User-agent | The crawler the following rules apply to; * for all of them | User-agent: Googlebot |
Disallow | A URL prefix forbidden to crawling | Disallow: /admin/ |
Allow | An exception to a Disallow | Allow: /admin/public/ |
Sitemap | The location of the XML sitemap | Sitemap: https://www.exemple.fr/sitemap.xml |
* and $ | Wildcard and end-of-URL in paths | Disallow: /*?sessionid= |
The rules are case-sensitive, apply per User-agent group (the most specific group wins) and, where Allow and Disallow conflict, the longest rule wins. An empty or absent file allows everything. The Crawl-delay directive is ignored by Google. The XML sitemap is declared here so it can be discovered without submission.
An example for a small business site
User-agent: *
Disallow: /admin/
Disallow: /panier/
Disallow: /*?tri=
Allow: /
Sitemap: https://www.exemple.fr/sitemap.xmlThis file leaves everything crawlable except the admin area, the basket and the parameter-based sorting URLs, which produce duplicate content, and it declares the sitemap. CSS and JavaScript files must never be blocked: Google needs them to render the page the way a visitor sees it.
The generative AI crawlers
| Crawler | Publisher | Use |
|---|---|---|
| GPTBot, OAI-SearchBot | OpenAI | ChatGPT training and search |
| ClaudeBot | Anthropic | Claude training and search |
| PerplexityBot | Perplexity | Search and citations |
| Google-Extended | Gemini training; affects neither search nor the AI Overviews | |
| Bingbot | Microsoft | Bing search and Copilot |
Blocking these crawlers removes the site from AI assistants' answers, which in 2026 account for a growing share of how businesses are discovered. For a small business seeking visibility, the rule is to allow them; media companies that sell their content make a different choice. The question is developed in indexing by AI.
The frequent mistakes
- A Disallow: / left in after go-live: the staging file blocks the whole site in production; the leading cause of sudden deindexing.
- Blocking a page to deindex it: it stays indexed with no description; use noindex.
- Blocking CSS, JavaScript or images: Google can no longer render the page and judges it badly.
- Blocking the AI crawlers by default: lost visibility in assistants with no benefit.
- A file in another location: only
/robots.txtat the domain root is read; a subdomain has its own.
The robots.txt report in Search Console shows the version Google read, any syntax errors, and lets you test a URL.
How GreenRed helps
Rather than juggling several tools, GreenRed's Site health module brings these metrics together in a single dashboard, compares them over time and tells you which actions come first. You can try it free, with no card, from the Pricing.
Frequently asked questions
What is the robots.txt file for?
To tell crawlers which parts of the site they may visit and which they should ignore, and to declare the sitemap's location. It controls crawling, not indexing, and offers no protection whatsoever.
Can you deindex a page with robots.txt?
No. A page blocked by Disallow can stay indexed if links point to it, shown without a description. To remove it from the index, leave it crawlable and add a meta robots noindex tag, or return a 410.
Should you block AI crawlers such as GPTBot?
For a business seeking visibility, no: blocking them removes the site from ChatGPT, Claude or Perplexity answers, which account for a growing share of discovery in 2026. Google-Extended concerns only Gemini's training and doesn't affect the AI Overviews.
How do I check my robots.txt is correct?
Open the domain's /robots.txt URL, confirm there is no global Disallow: /, then use the robots.txt report in Search Console, which shows the version Google read, flags syntax errors and lets you test a specific URL.