The New York Times
nytimes.com
Category: Content & News · ranked #2 in category
Scanned 6/7/2026, 3:18:42 PM
How to access The New York Times as an agent
- ○Respect explicit crawler block
robots.txt disallows known AI crawlers. Do not crawl with those user-agents. Look for a partner / data-licensing path before automating.
- ○Check for an undocumented developer API
No OpenAPI at the standard paths. Look for /developer, /developers, /docs, /api for hand-authored documentation.
https://nytimes.com/developers - ○Find the developer signup
No standard OAuth discovery - most sites still have a developer-portal signup. Find the OAuth client registration page on their site.
- ○Publish your own agent identity
Generate an Ed25519 keypair, host a public key directory, sign your HTTP requests (Web Bot Auth), apply to Cloudflare's bots & agents directory if you cross their edge.
https://developers.cloudflare.com/bots/concepts/bot/verified-bots/ - ●Respect robots.txt and Content Signals
Even when you are an approved agent, honor the site's policy preferences. Cache them; refresh weekly.
What we found
| Signal | Status | Evidence | Points |
|---|---|---|---|
| licensed_commercial_access | found Commercial agent-access mechanism detected: Licensing page (https://nytimes.com/licensing) mentions: content licens, nyt licens. Sites with explicit licensing pathways are the most agent-prepared - they have both access and disintermediation handled. | +30 | |
| robots_allows | found 60 UA rules · sitemap: yes | nytimes.com/robots.txt | +4 |
| blocks_automation | found Explicit Disallow for: anthropic-ai, applebot-extended, bytespider, ccbot, claudebot, claude-web, diffbot, google-extended, gptbot, oai-searchbot, perplexitybot | nytimes.com/robots.txt | -30 |
Show 13 signals not present
- · llms_txt (+5 if found)
- · llms_full_txt (+3 if found)
- · agents_txt (+6 if found)
- · agents_json (+15 if found)
- · a2a_agent_card (+25 if found)
- · agents_md (+4 if found)
- · openapi (+18 if found)
- · oauth_discovery (+12 if found)
- · content_signals (+5 if found)
- · ai_plugin_json_legacy (+0 if found)
- · mcp_registry (+25 if found)
- · in_llms_txt_hub (+0 if found)
- · in_agent_friendly_directory (+4 if found)
If you own this site
Concrete moves that would raise The New York Times's AgentRank:
- →Publish /llms.txt - a curated markdown index of your useful docs for LLMs.
- →Publish /agents.txt declaring which agent protocols you support.
- →If you operate an agent, publish /.well-known/agent-card.json so others can discover it.
- →Publish /AGENTS.md with human-readable agent instructions.
- →Publish your API as OpenAPI 3.x at a discoverable location.
- →Expose OAuth/OIDC discovery at /.well-known/oauth-authorization-server.