AI-crawler readiness pack

Training vs retrieval

robots.txt matrix — training

Evaluated for path /. Longest matching group wins; * is fallback only (RFC 9309-style), not first-match.

robots.txt matrix — retrieval / user-fetch

Context tokens

robots.txt: https://www.nytimes.com/robots.txt → HTTP 200. Content-Signal: none.

Live user-agent probes

Homepage plus a few important URLs, fetched as each crawler UA. Allowed even when robots.txt would block that bot — the question is whether the site (WAF/CDN) blocks them. Gentle: few URLs, timeouts, sleep between requests. Not a full crawl.

Reached without challenge: 6 · Challenge-like body/headers: 0 · Error or other 4xx/5xx: 27

Sitemap (capped)

Sitemaps used: https://www.nytimes.com/sitemaps/new/news.xml.gz. Listed 12 HTML URLs (cap 12).

• https://www.nytimes.com/2026/08/15/us/ugliest-dog-contest-winner-2026.html
• https://www.nytimes.com/live/2026/08/16/us/trump-news
• https://www.nytimes.com/2026/08/16/world/middleeast/kushner-hamas-talks.html
• https://www.nytimes.com/athletic/live-blogs/cameroon-vs-malawi-live-updates-wafcon-final-score-result/RNn1oZ4IbCZY/
• https://www.nytimes.com/athletic/live-blogs/arsenal-vs-manchester-city-live-updates-community-shield-score-result/UZhVEw0YArDF/
• https://www.nytimes.com/live/2026/08/16/weather/lala-tropical-storm-hurricane-hawaii
• https://www.nytimes.com/athletic/7517530/2026/08/16/bears-ozzy-trapilo-activated-knee-injury-recovery/
• https://www.nytimes.com/athletic/7516846/2026/08/16/how-to-watch-fever-vs-dream-tv-channel-and-streaming-options-for-august-16/
• https://www.nytimes.com/athletic/7517061/2026/08/16/how-to-watch-fire-vs-mercury-tv-channel-and-streaming-options-for-august-16/
• https://www.nytimes.com/athletic/7516849/2026/08/16/how-to-watch-sky-vs-storm-tv-channel-and-streaming-options-for-august-16/
• https://www.nytimes.com/athletic/7514863/2026/08/15/how-to-watch-the-mens-western-southern-open-streaming-options-for-the-round-of-64/
• https://www.nytimes.com/2026/08/16/us/politics/uss-lincoln-iran-admiral.html

Pages fetched for titles / H1 / JSON-LD

Our own crawl uses the GeoPack user-agent and respects robots.txt. Cap is small on purpose.

Schema + sitemap gaps

Types observed: BlogPosting, DefinedTerm, EntryPoint, ImageObject, ItemList, ListItem, LiveBlogPosting, NewsArticle, NewsMediaOrganization, Organization, Person, Place, PostalAddress, SearchAction, SportsEvent, WebPageElement, WebSite

• No BreadcrumbList JSON-LD on crawled pages (minor).

Common types not seen:

• WebPage — common type not observed on crawled pages
• BreadcrumbList — common type not observed on crawled pages
• Article — not observed; no blog/news URLs in this tiny crawl — may be expected
• FAQPage — not observed; optional unless an FAQ page exists
• Product — not observed; no product/pricing URLs in this tiny crawl — may be expected
• SoftwareApplication — not observed; relevant for software vendors if a product page exists
• LocalBusiness — not observed; expected-missing for a global/SaaS homepage

Schema presence is a technical observation from a tiny crawl. It is not a ranking, rich-result, or citation prediction.

Drafted llms.txt

/llms.txt → HTTP 404; /llms-full.txt → HTTP 404

See llms.txt and llms-full.txt in this pack. Every generated link is marked DRAFTED and needs a human edit before you publish it. Publishing the file does not opt you out of training and does not change Google Search.

How to use this pack

1. Read the training vs retrieval split. Decide a policy (example: block training, allow retrieval — or the reverse).
2. Compare robots.txt rules to live probe status. A robots Allow plus a WAF challenge means the file is lying.
3. Edit the drafted llms.txt. Do not ship DRAFTED lines as-is.
4. Treat schema gaps as a checklist for your developer, not as SEO score.