Only 133 of 10,099 Shopify Stores Block Any AI Crawler

Just 133 of 10,099 Shopify storefronts block any AI crawler in robots.txt — 1.32% — and almost none of them block the crawlers that build shopping answers.

The scan, run between 29 August and 2 September 2026, read /robots.txt on every store in a corpus of 10,099 known Shopify storefronts and asked twelve named crawlers one question each: may it fetch /products/? A store counts as blocking when the answer is no.

Twelve crawlers doing two different jobs

The study splits the twelve by when they read a page. Eight crawl ahead of time, to train or ground a model. Four fetch at the moment someone is asking a question.

Crawler	What it feeds	Fetches at answer time	Stores blocking
CCBot	Common Crawl	no	81
GPTBot	OpenAI training & retrieval	no	77
Bytespider	TikTok / Doubao	no	72
Amazonbot	Alexa+ / Rufus	no	60
Google-Extended	AI Overviews & AI Mode grounding	no	58
ClaudeBot	Claude retrieval & citations	no	53
Applebot-Extended	Apple Intelligence	no	48
meta-externalagent	Meta AI	no	47
ChatGPT-User	Live fetches during a ChatGPT chat	yes	11
PerplexityBot	Perplexity search & shopping	yes	10
OAI-SearchBot	ChatGPT search & shopping results	yes	6
Perplexity-User	Live fetches when a Perplexity user asks	yes	3

The study notes the twelve are the names merchants' block lists actually carry, not every answer-time agent that exists — Anthropic's and Google's live-fetch agents are not among them.

The lists are copied, and it shows

Of the 133 stores blocking anything, 130 do it with Disallow: / — the whole site, not just product pages. Three block /products/ alone. Fifty-four block exactly one crawler.

Then the tell: 28 stores block the same eight crawlers, verbatim — Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, meta-externalagent. Four more block those eight plus others. As the author puts it, identical lists do not arise independently; this one reads like a copied snippet, and it contains none of the four answer-time crawlers.

That eight-name list is exactly the training-and-grounding group. Whoever wrote it made a coherent decision — and it was a decision about training data.

OpenAI's pair is the clearest result in the dataset

OpenAI runs both kinds under different names. GPTBot trains. OAI-SearchBot is what ChatGPT's search and shopping results are built from.

Seventy-seven stores block GPTBot. Six block OAI-SearchBot, and every one of those six also blocks GPTBot. The reverse — keep the shopping crawler out, let the training crawler in — happens on zero stores.

The study's reading of that asymmetry is worth quoting in full, because it is the finding: no store in the corpus has decided to stay out of AI shopping answers. The stores that are out of them by robots.txt are there because a copied training opt-out happened to include the name.

Overall, 120 of the 133 block only training and grounding crawlers. Thirteen block at least one answer-time crawler; one of those blocks only answer-time crawlers, and twelve block both kinds.

The noai tag is a separate, smaller story

A second reading looked at product pages for a meta name="robots" tag carrying noai or noimageai. Eight stores carry one, and six of those eight block nothing at all in robots.txt.

By their names, six of the eight are musicians' merchandise stores — which, the author notes, again looks like one template rather than eight independent decisions. The study makes no claim about which crawlers honour the tag.

What the numbers cannot see

The method section is unusually candid about its own limits, and the caveat matters before anyone cites the 1.32%. A robots.txt is a request. A store can also block a crawler at the edge with a bot-management rule or a firewall, and that block is invisible to this scan because the scanner is not the crawler being blocked.

Every count in the study is therefore a floor, not a measurement of intent. The right reading is "at least this many", and the interesting structure is in the ratios rather than the absolute totals.

What to do with this if you run a site

The finding generalises past Shopify, because the copied list is not a Shopify artifact — it is a snippet that circulated when the only AI crawlers worth naming were training crawlers.

Three checks, in order. First, open your own robots.txt and sort every blocked name into the two columns above; if your list is the eight-name one, you have opted out of training and made no decision at all about retrieval. Second, decide the two questions separately — whether your content trains a model, and whether your pages can be fetched when a customer is actively asking — because one line of config currently answers both by accident. Third, check server logs rather than config alone, since robots.txt describes what should be allowed and logs show what actually arrives.

Note also that blocking retrieval crawlers is the one choice with an immediate, visible cost: ClaudeBot feeds Claude retrieval and citations, Google-Extended grounds AI Overviews and AI Mode, and OAI-SearchBot builds ChatGPT's shopping results. Those are the paths a customer question travels.

What to watch

Whether the circulated block list gets updated to name the answer-time crawlers — and if it does, whether merchants adopt the new version deliberately or copy that one too. The study is reproducible: it starts from curl -s https://EXAMPLE.com/robots.txt, which means the same corpus can be re-scanned later and the two readings compared directly.