What Is This Note?
This is a dated note on how our own robots.txt treated AI crawlers between November 2025 and October 2026. It replaces a post that said we blocked 12 AI training crawlers in March 2026 and our citations went up. The git history shows the blocks were in place by November 2025, our configuration still allowed two training crawlers, and we have no citation data from before the change. So this note records the policy and withdraws the result.
How Did the Policy Change?
The dates and commits below come from the git history of public/robots.txt.
Our AI crawler policy over time
From the git history of robots.txt
AI rules added
first AI-specific rules
12 blocked
list complete
4 unblocked
selected training crawlers
TavilyBot
added to the allow list
| Date | Commit | Change |
|---|---|---|
| 31 Oct 2025 | 8ae6b225 | First AI crawler rules; CCBot, anthropic-ai, cohere-ai, Omgili and Omgilibot blocked |
| 5 Nov 2025 | 5c352eeb | Blocks extended to 12: Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Bytespider, Diffbot and img2dataset added |
| 16 May 2026 | c8723b5e | Allowed CCBot, Google-Extended, anthropic-ai and Applebot-Extended |
| 27 May 2026 | 437bcd90 | Allowed TavilyBot |
The 16 May commit explains the change in its own words: being known to future model generations mattered more to us than blanket protection, for a brand still earning ground in AI answers. That is a judgement, and it is ours. We have not measured its effect either.
What Do We Allow and Block Today?
As of 2 October 2026, we allow 13 AI crawlers and block 8:
- Allowed: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, GoogleOther, PerplexityBot, Perplexity-User, TavilyBot, CCBot, Google-Extended, Applebot-Extended.
- Blocked: cohere-ai, Meta-ExternalAgent, FacebookBot, Bytespider, Diffbot, Omgili, Omgilibot, img2dataset.
AI crawler user agents blocked in robots.txt
From the git history of public/robots.txt
Which Claims From the Original Post Are Withdrawn?
| Claim in the March 2026 version | What the record shows | Status |
|---|---|---|
| We blocked 12 AI training crawlers in March 2026 | All 12 were in place by 5 November 2025 | Corrected |
| The policy blocked training bots | GPTBot and ClaudeBot, both training crawlers, were allowed | Corrected |
| Citations went up after blocking | The citation monitor started 30 March 2026; no before data exists | Withdrawn |
What the original claimed, and what the record shows
March 2026 post versus the git history and monitor records
- We blocked 12 AI training crawlers in March 2026
- The policy blocked training bots
- Citations went up after blocking
- All 12 blocks were in place by 5 November 2025
- GPTBot and ClaudeBot, both training crawlers, were allowed
- No citation data exists from before 30 March 2026
For current advice on choosing crawlers by type, see Part 3, the GEO Playbook.
What This Series Covers
This is Part 7 of The AI Search Playbook, now a dated note. Part 3 covers crawler access as part of the full playbook.
Our AI Bot Policy: Questions Readers Ask
Common questions about this topic, answered.
Did blocking AI training bots raise Pixelmojo's citations?
We cannot show that it did. The full list of blocks was in place by 5 November 2025, and our citation monitor did not start recording until 30 March 2026, so there is no before-and-after data. The original version of this post said citations went up. We withdrew that claim on 2 October 2026.
Which bots did pixelmojo.io block in March 2026?
Twelve user agents: Google-Extended, CCBot, anthropic-ai, cohere-ai, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Bytespider, Diffbot, Omgili, Omgilibot and img2dataset. The full list had been in place since 5 November 2025 (five of them since 31 October 2025), not from March 2026 as the original post said.
Did the March 2026 policy block all training crawlers?
No. It allowed GPTBot and ClaudeBot, which are training crawlers, while blocking others. The original post described a policy of blocking training bots that its own configuration did not follow.
What changed in May 2026?
On 16 May 2026 we moved CCBot, Google-Extended, anthropic-ai and Applebot-Extended from blocked to allowed, and on 27 May 2026 we allowed TavilyBot. The commit message gives the reasoning: being known to future models mattered more to us than blanket protection, for a brand still earning ground in AI answers.
What does pixelmojo.io allow and block today?
As of 2 October 2026 we allow GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, GoogleOther, PerplexityBot, Perplexity-User, TavilyBot, CCBot, Google-Extended and Applebot-Extended, and block cohere-ai, Meta-ExternalAgent, FacebookBot, Bytespider, Diffbot, Omgili, Omgilibot and img2dataset.
Should I block training crawlers?
It is a content-use choice, separate from AI search. Training crawlers collect content that may be used to train models. Search crawlers are separate: in their crawler docs (checked 26 Sep 2026), OpenAI says sites that disallow OAI-SearchBot will not be shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce your visibility in user search results.
Where is the current advice on AI crawlers?
In Part 3 of The AI Search Playbook, the GEO Playbook, rewritten on 2 October 2026. It covers crawler types, our current policy and how to check that your server actually lets the bots through.
What changed in this post on 2 October 2026?
We cut it down to a dated note. The claim that citations went up is withdrawn, the March 2026 date for the blocks is corrected (all 12 were in place by 5 November 2025), and the post now records how the policy changed through May and October 2026.
