Blocking the Wrong Crawlers Costs You AI Visibility; IAB Formalizes Citation Tiers
New IAB measurement standards split AI citation tracking into 'directional' and 'decision-grade' data while Anthropic's three-crawler architecture means most robots.txt files are configured for the wrong threat. Both stories demand an immediate audit of your site's crawler permissions.
This Week's Signals
MCP July 2026 Spec Goes Stateless With Close to Half a Billion Monthly SDK Downloads
Why it matters for your score
The largest MCP revision since launch removes session pinning at the transport layer and deprecates legacy HTTP+SSE transport with a 12-month offramp, meaning any tool or server built on session-based connections needs a migration plan now. TypeScript and Python SDKs each crossed 1 billion total downloads, and Tier 1 SDKs collectively receive close to half a billion downloads per month; the protocol is no longer experimental infrastructure. Businesses building or buying AI-connected tools should confirm their vendors are shipping against the 2026-07-28 spec, not the deprecated session model.
What to do this week
- 1
Audit your robots.txt against the full list of eight active AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended) and confirm you are not blocking retrieval-layer bots like Claude-SearchBot or OAI-SearchBot; use Faro's robots.txt tool at /tools/robots-txt to check your current configuration.
- 2
Run your site through Faro's AI readiness scan at /tools/ai-readiness-scan to establish a baseline citation rate you can classify against the IAB's new directional vs. decision-grade standard, the median blended rate across the industry benchmark was 4.1%, so compare your result against that benchmark before drawing conclusions.
- 3
If your business connects to any MCP-based tools or agent workflows, ask your vendor or developer to confirm their implementation targets the 2026-07-28 stateless spec and has a migration plan away from the deprecated HTTP+SSE transport before the 12-month offramp closes.