Key Takeaways
- Two mechanisms: training the models (GPTBot, ClaudeBot, Google-Extended, CCBot) and real-time search at the time of the response (OAI-SearchBot, PerplexityBot, Bing and Google indexes).
- The rule that matters: Blocking the training does not prevent the citation; blocking the search deletes it.
- Prerequisites: Be indexed and rank well on Google and Bing; have content available in HTML without JavaScript; and have fast-loading pages.
- Additional Information: llms.txt file, structured data, up-to-date sitemap; verification via server logs.
To be cited by an AI assistant, your site must first be read by it. Contrary to popular belief, AI assistants do not have their own comprehensive web index: they combine what the model learned during training with real-time searches in existing indexes (Bing for ChatGPT and Copilot, Google for Gemini, and a proprietary, enhanced index for Perplexity). This article describes crawlers, indexes, robots.txt settings, rendering conditions, and how to verify that your site is being crawled properly. This is the technical foundation of the GEO.
Training and Research: Two Distinct Mechanisms
| Training | Search at the time of response (retrieval) | |
|---|---|---|
| When | Periodic bulk data collection, integrated into the model during its update | For each question, the assistant consults an index and reads a few pages |
| Benefits for You | The model “knows” your brand, with a delay of several months | Your pages are cited with links in the answer |
| Robots | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), CCBot (Common Crawl), Applebot-Extended, Meta-ExternalAgent | OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User, Googlebot (AI previews, Gemini), Bingbot (Copilot, ChatGPT) |
| If you block | The model learns less about you; little effect on citations | You're removing the answers that include sources |
Google-Extended only controls the use of content for training Gemini; the AI-powered previews in Google Search use Googlebot and follow the standard rules (noindex, nosnippet, max-snippet). Details on the sources used by each assistant can be found in Sources of Generative AI.
Configure robots.txt
Recommendation for a company that wants to be indexed: Allow all AI search bots, and make a separate decision regarding training bots. Minimal example:
| Directive | Effect | Recommendation |
|---|---|---|
User-agent: OAI-SearchBot then Allow: / | ChatGPT can include your pages in its search results | Allow |
User-agent: PerplexityBot then Allow: / | Perplexity indexes and cites | Allow |
User-agent: Googlebot / Bingbot | Classic Search, AI Insights, Copilot, ChatGPT via Bing | Allow (already the case) |
User-agent: GPTBot, ClaudeBot, Google-Extended, CCBot | Model Training | Choose one: Allow access so that the models can learn about your brand, or block access to protect high-value content (paid knowledge base) |
Tag nosnippet or max-snippet | Limits the snippet displayed by Google, including AI previews | Do not use on pages that require citations |
Many websites reflexively blocked all AI bots in 2023–2024—including GPTBot—and copied lists that included search bots. Check your file: a small business that sells services has no reason to prevent ChatGPT from recommending it. The rules for robots.txt are detailed in Meta robots and robots.txt.
Reading Requirements
- To be indexed and ranked: Assistants draw from the top results on Bing and Google; a page that isn't indexed or appears on page 5 isn't read. SEO remains a prerequisite; see the Search Console Indexing Report. Also check your indexing status in Bing (Bing Webmaster Tools), which is often overlooked even though ChatGPT and Copilot rely on it.
- Content in the HTML: Most AI bots do not execute JavaScript. Content loaded on the client side is invisible to them. Use server-side rendering or static pages for key pages.
- Speed and availability: AI search bots have short time limits; slow pages or servers that return a 5xx error are ignored.
- No wall: paywall, required login, a banner that hides the content in the HTML, overly aggressive anti-bot measures (some firewalls block PerplexityBot).
- Clear structure: Hierarchical headings, standalone paragraphs, HTML tables, structured data; see Structuring Your Pages for AI.
The llms.txt file
To be introduced in 2024, llms.txt is a text file located at the root of the site that introduces the company and lists its important pages with a title and description, written in Markdown, to make it easier for crawlers to read. No major search engine has confirmed that it uses this for search, but it costs nothing and serves as a clean table of contents. GreenRed publishes one, which is regenerated every time the resources are updated: www.greenred.fr/llms.txt. Think of it as a supplement to the sitemap, not as a primary tool.
Check to see if AI systems can read your site
- Server logs: Filter for the user agents GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, and Google-Extended; their presence and frequency indicate what is being read. Visits from ChatGPT-User and Perplexity-User correspond to content views triggered by user questions.
- Hands-on test: Ask ChatGPT (with search) and Perplexity, “What does [your company] offer?” and “Summarize the page at [URL]”; an accurate and recent response proves you’ve read it.
- Referral traffic: Sessions from chatgpt.com, perplexity.ai, and gemini.google.com in GA4; see Identify AI traffic in GA4.
- Bing Webmaster Tools: indexed pages and errors, which often differ from those in Search Console.
- GEO Tracking Tool: hits per query and per search engine.
How GreenRed Can Help You
Instead of juggling multiple tools, GreenRed's GEO tracking brings these metrics together in a single dashboard, compares them over time, and shows you which actions to prioritize. You can try it for free—no credit card required—on the page Rates.
Frequently Asked Questions
Should we block GPTBot?
GPTBot is used to train OpenAI models, not for real-time searches in ChatGPT, which uses OAI-SearchBot and the Bing index. Blocking it limits what the model learns about you without removing the citations. A company that wants to be known and recommended is better off allowing it; a publisher of paid content may block it.
How does ChatGPT find the pages it cites?
When web search is enabled, ChatGPT reformulates the question into search queries, queries an index (Bing, supplemented by its own OAI-SearchBot), reads the first few relevant pages, and drafts the answer while citing its sources. Being indexed and well-ranked in Bing is therefore just as important as being indexed and well-ranked in Google.
Is the llms.txt file required or useful?
It’s neither required nor proven: no major search engine has confirmed that it uses this for search. It’s inexpensive to produce and provides a readable summary of your important pages. Publish it as a supplement to your sitemap, without expecting it to have a direct impact on your backlinks.
Can AI read my JavaScript website?
Often a problem: Most AI bots read HTML without executing JavaScript. If your content only appears after scripts are executed (client-side rendering frameworks), it’s invisible to them. Convert key pages to server-side or static rendering, and check them using the “summarize this page” test in Perplexity.