7/7training tokens allow /
6/6retrieval tokens allow /
32/33UA probes reached (no challenge)
10content pages fetched
Training vs retrieval
Blocking a training crawler (GPTBot, ClaudeBot, CCBot, Google-Extended) is not the same
as blocking a retrieval agent (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot).
Sites that paste a blanket “block all AI bots” file often disappear from answer-engine
citations while still leaking training data — or the reverse. GeoPack splits the two.
This is a policy observation, not a promise that anyone will cite you.
robots.txt matrix — training
Evaluated for path /. Longest matching group wins; * is fallback only (RFC 9309-style), not first-match.
| Token | Operator | Job | / | Rule source | Note |
|---|---|---|---|---|---|
GPTBot | OpenAI | training | allow / | explicit | Training crawl for OpenAI foundation models. |
ClaudeBot | Anthropic | training | allow / | via * | Anthropic training crawler. |
anthropic-ai | Anthropic | training | allow / | explicit | Legacy Anthropic robots.txt token. Still appears on many sites. |
Google-Extended control token — no HTTP crawler | training | allow / | explicit | robots.txt control token for Gemini / AI training use of already-crawled content. Googlebot is the HTTP crawler. No Google-Extended user-agent is sent. | |
Applebot-Extended control token — no HTTP crawler | Apple | training | allow / | via * | robots.txt control token for Apple Intelligence training. Applebot is the HTTP crawler. No Applebot-Extended user-agent is sent. |
Bytespider | ByteDance | training | allow / | via * | ByteDance training crawler. Public reports say it often ignores robots.txt. |
CCBot | Common Crawl | training-corpus | allow / | explicit | Open web corpus. Many labs train on Common Crawl snapshots. |
robots.txt matrix — retrieval / user-fetch
| Token | Operator | Job | / | Rule source | Note |
|---|---|---|---|---|---|
OAI-SearchBot | OpenAI | search | allow / | via * | Indexes pages for ChatGPT search results. Independent of GPTBot. |
ChatGPT-User | OpenAI | user-fetch | allow / | explicit | On-demand fetch when a ChatGPT user asks about a page. May ignore robots.txt. |
PerplexityBot | Perplexity | search | allow / | explicit | Search indexer for Perplexity answers. Not a foundation-model trainer. |
Google-CloudVertexBot | grounding | allow / | via * | Fetches pages to ground Vertex AI / Gemini Enterprise agents. | |
Amazonbot | Amazon | search-and-assistant | allow / | via * | Amazon search / assistant crawler. |
Googlebot | search | allow / | via * | Classic Google Search crawler. Shown for context, not an AI-training bot. |
Context tokens
| Token | Operator | / | Reason |
|---|---|---|---|
* | default | allow / | inherited from * |
robots.txt: https://www.cloudflare.com/robots.txt → HTTP 200.
Content-Signal: ai-train=yes, search=yes, ai-input=yes.
Live user-agent probes
Homepage plus a few important URLs, fetched as each crawler UA. Allowed even when robots.txt would block that bot — the question is whether the site (WAF/CDN) blocks them. Gentle: few URLs, timeouts, sleep between requests. Not a full crawl.
Reached without challenge: 32 · Challenge-like body/headers: 0 · Error or other 4xx/5xx: 1
| URL | UA token | Purpose | Status | Server | WAF hints | Time |
|---|---|---|---|---|---|---|
https://www.cloudflare.com/ | GPTBot | training | 200 | cloudflare | cf-ray=a2c35f36abfc23a3-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 386 ms |
https://www.cloudflare.com/ | OAI-SearchBot | retrieval | 200 | cloudflare | cf-ray=a2c35f3f79dcc3af-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 502 ms |
https://www.cloudflare.com/ | ChatGPT-User | retrieval | 200 | cloudflare | cf-ray=a2c35f48ef04b338-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 408 ms |
https://www.cloudflare.com/ | ClaudeBot | training | 200 | cloudflare | cf-ray=a2c35f51a986ff1a-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 499 ms |
https://www.cloudflare.com/ | anthropic-ai | training | 200 | cloudflare | cf-ray=a2c35f5b0c085ed8-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 547 ms |
https://www.cloudflare.com/ | PerplexityBot | retrieval | 200 | cloudflare | cf-ray=a2c35f64bf64dd76-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 532 ms |
https://www.cloudflare.com/ | Google-CloudVertexBot | retrieval | 200 | cloudflare | cf-ray=a2c35f6e4f46e102-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 493 ms |
https://www.cloudflare.com/ | Bytespider | training | error SSLError: HTTPSConnectionPool(host='www.cloudflare.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1029)'))) | — | 12 ms | |
https://www.cloudflare.com/ | CCBot | training | 200 | cloudflare | cf-ray=a2c35f7df90968f1-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 486 ms |
https://www.cloudflare.com/ | Amazonbot | retrieval | 200 | cloudflare | cf-ray=a2c35f874a7787cd-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 425 ms |
https://www.cloudflare.com/ | Googlebot | retrieval | 200 | cloudflare | cf-ray=a2c35f9029f1ef34-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 518 ms |
https://www.cloudflare.com/products | GPTBot | training | 200 | cloudflare | cf-ray=a2c35f99c91a5ef1-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 469 ms |
https://www.cloudflare.com/products | OAI-SearchBot | retrieval | 200 | cloudflare | cf-ray=a2c35fa2ed871561-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 62 ms |
https://www.cloudflare.com/products | ChatGPT-User | retrieval | 200 | cloudflare | cf-ray=a2c35fa99ca3de1d-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 66 ms |
https://www.cloudflare.com/products | ClaudeBot | training | 200 | cloudflare | cf-ray=a2c35fb039cff3b4-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 64 ms |
https://www.cloudflare.com/products | anthropic-ai | training | 200 | cloudflare | cf-ray=a2c35fb6edd89b30-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 61 ms |
https://www.cloudflare.com/products | PerplexityBot | retrieval | 200 | cloudflare | cf-ray=a2c35fbd8d8eabbf-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 59 ms |
https://www.cloudflare.com/products | Google-CloudVertexBot | retrieval | 200 | cloudflare | cf-ray=a2c35fc42fc3ff13-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 61 ms |
https://www.cloudflare.com/products | Bytespider | training | 200 | cloudflare | cf-ray=a2c35fcac9dab024-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 64 ms |
https://www.cloudflare.com/products | CCBot | training | 200 | cloudflare | cf-ray=a2c35fd179b92095-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 63 ms |
https://www.cloudflare.com/products | Amazonbot | retrieval | 200 | cloudflare | cf-ray=a2c35fd81b826c24-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 67 ms |
https://www.cloudflare.com/products | Googlebot | retrieval | 200 | cloudflare | cf-ray=a2c35fdeba104518-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 151 ms |
https://www.cloudflare.com/products/ | GPTBot | training | 200 | cloudflare | cf-ray=a2c35fe5ddd68969-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 50 ms |
https://www.cloudflare.com/products/ | OAI-SearchBot | retrieval | 200 | cloudflare | cf-ray=a2c35fec7dde9976-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 53 ms |
https://www.cloudflare.com/products/ | ChatGPT-User | retrieval | 200 | cloudflare | cf-ray=a2c35ff30a6f68f1-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 52 ms |
https://www.cloudflare.com/products/ | ClaudeBot | training | 200 | cloudflare | cf-ray=a2c35ff999339d6b-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 48 ms |
https://www.cloudflare.com/products/ | anthropic-ai | training | 200 | cloudflare | cf-ray=a2c360002b87ef34-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 63 ms |
https://www.cloudflare.com/products/ | PerplexityBot | retrieval | 200 | cloudflare | cf-ray=a2c36006ccfd1f9b-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 49 ms |
https://www.cloudflare.com/products/ | Google-CloudVertexBot | retrieval | 200 | cloudflare | cf-ray=a2c3600d5a306c24-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 49 ms |
https://www.cloudflare.com/products/ | Bytespider | training | 200 | cloudflare | cf-ray=a2c36013ea97b7b9-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 48 ms |
https://www.cloudflare.com/products/ | CCBot | training | 200 | cloudflare | cf-ray=a2c3601a783b5f03-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 52 ms |
https://www.cloudflare.com/products/ | Amazonbot | retrieval | 200 | cloudflare | cf-ray=a2c360210c6c1571-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 147 ms |
https://www.cloudflare.com/products/ | Googlebot | retrieval | 200 | cloudflare | cf-ray=a2c360283bd5ad66-PDX, server=cloudflare, nel={"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} | 51 ms |
Sitemap (capped)
Sitemaps used: https://www.cloudflare.com/sitemap.xml. Listed 10 HTML URLs (cap 10).
https://www.cloudflare.com/https://www.cloudflare.com/planshttps://www.cloudflare.com/productshttps://www.cloudflare.com/bloghttps://www.cloudflare.com/case-studieshttps://www.cloudflare.com/learninghttps://www.cloudflare.com/careershttps://www.cloudflare.com/careers/jobshttps://www.cloudflare.com/partnershttps://www.cloudflare.com/partners/cloud-and-platform/oracle
Pages fetched for titles / H1 / JSON-LD
Our own crawl uses the GeoPack user-agent and respects robots.txt. Cap is small on purpose.
| URL | Title | H1 | JSON-LD types |
|---|---|---|---|
https://www.cloudflare.com/ | Cloudflare: Build for the agent era | Everything we learned from powering 20% of the Internet—yours by default | Organization, SearchAction, WebPage, WebSite |
https://www.cloudflare.com/plans/ | Pricing | Scale predictably | Organization, SearchAction, WebPage, WebSite |
https://www.cloudflare.com/products/ | Products | Cloudflare | Build without boundaries | Organization, SearchAction, WebPage, WebSite |
https://blog.cloudflare.com/ | Cloudflare Blog | Cloudflare Blog | WebSite |
https://www.cloudflare.com/case-studies/ | Customer Case Studies | Cloudflare | Organization, SearchAction, WebPage, WebSite | |
https://www.cloudflare.com/learning/ | Just a moment... | — | |
https://www.cloudflare.com/careers/ | Cloudflare Careers | Cloudflare | Build without boundaries | Organization, SearchAction, WebPage, WebSite |
https://www.cloudflare.com/careers/jobs/ | Careers at Cloudflare â Open Positions | Cloudflare | Help Us Build a Better Internet | Organization, SearchAction, WebPage, WebSite |
https://www.cloudflare.com/partners/ | Partners | Cloudflare | Build without boundaries | Organization, SearchAction, WebPage, WebSite |
https://www.cloudflare.com/partners/cloud-and-platform/oracle/ | Oracle Cloud Infrastructure Partner | Cloudflare | Build without boundaries | Organization, SearchAction, WebPage, WebSite |
Schema + sitemap gaps
Types observed: Organization, SearchAction, WebPage, WebSite
- Blog/news-like URLs seen, but no Article JSON-LD in this crawl.
- Product/pricing-like URLs seen, but no Product or SoftwareApplication JSON-LD.
- No BreadcrumbList JSON-LD on crawled pages (minor).
Common types not seen:
BreadcrumbList— common type not observed on crawled pagesArticle— common type not observed on crawled pagesFAQPage— not observed; optional unless an FAQ page existsProduct— common type not observed on crawled pagesSoftwareApplication— common type not observed on crawled pagesLocalBusiness— not observed; expected-missing for a global/SaaS homepagePerson— not observed; optional unless author/about pages were crawled
Schema presence is a technical observation from a tiny crawl. It is not a ranking, rich-result, or citation prediction.
Drafted llms.txt
/llms.txt → HTTP 200; /llms-full.txt → HTTP 200
See llms.txt and llms-full.txt in this pack. Every generated link is marked
DRAFTED and needs a human edit before you publish it. Publishing the file does not
opt you out of training and does not change Google Search.
How to use this pack
- Read the training vs retrieval split. Decide a policy (example: block training, allow retrieval — or the reverse).
- Compare robots.txt rules to live probe status. A robots
Allowplus a WAF challenge means the file is lying. - Edit the drafted
llms.txt. Do not ship DRAFTED lines as-is. - Treat schema gaps as a checklist for your developer, not as SEO score.