Back to all posts

Is your website accidentally blocking AI?

By Wendy Harris·31 October 2026·5 minute read

An editorial illustration for the article Is your website accidentally blocking AI?.

Almost nobody decides to block AI. The setting arrived with a plugin, a security package, or a host's default, and nothing about the website looks any different afterwards. The site loads, the photographs are lovely, the booking button works. The only change is that a category of visitor can no longer read the page, and that visitor does not send a message to say so.

It is the quietest failure in this whole subject, because everything else keeps working.

Three kinds of AI visit, and only one of them is training

These are worth separating, because you might reasonably want a different answer for each.

The first is search-style crawling. OpenAI's OAI-SearchBot exists to surface websites in ChatGPT's search features, and Anthropic's Claude-SearchBot navigates the web to improve the quality of search results. These are the crawlers behind AI answers that cite their sources. If you want your business to come up when someone asks an assistant where to stay in your town, these are the ones that matter most.

The second is training. GPTBot collects content that may be used to train OpenAI's models, and ClaudeBot does the same for Anthropic. Blocking these is a perfectly reasonable position, and plenty of businesses hold it deliberately.

The third happens when a person asks about you directly. ChatGPT-User and Claude-User fetch a page because someone requested it in that moment. That is the closest thing to a customer typing your name into a search box.

The three usually get treated as one thing. A single line in a file, or one tick box in a security dashboard, can switch off all of them at once, including the ones you would have wanted.

The date worth knowing, and what it actually covers

On 1 July 2026, Cloudflare introduced three bot categories, Search, Agent, and Training. From 15 September 2026, for domains newly onboarding to Cloudflare, Training and Agent crawlers will be blocked by default on pages that display advertisements, while Search remains allowed by default.

Read the scope rather than the headline. It applies to new domains, not existing ones. It applies to pages carrying advertising, not to every page. Search is unaffected. Most visitor economy websites carry no advertising and are not newly onboarding, so most will not be touched by this at all.

It is still worth knowing, for two reasons. It is the clearest sign yet that access is becoming something decided by infrastructure defaults rather than by you. And it makes concrete a question that has been vague for two years: do you actually know what your site allows today?

Where accidental blocking comes from

Almost never from a decision. Usually from one of these four.

A robots.txt file written by a plugin or a theme. Plenty of SEO and security plugins now offer AI crawler blocking, some of them suggesting it during setup. It gets agreed to once and never revisited.

A bot protection setting at the host or the CDN. These are described in terms of scrapers and bad bots, and the category names rarely map onto the three kinds of visit above. Operators end up agreeing to something broader than they understood.

A holding page, staging password, or geographic restriction left in place after a rebuild. That blocks everyone equally, customers included, and it survives far longer than anybody expects.

Content that only appears after the page has loaded. Prices, opening hours, and menus often live inside a widget that fills in a moment later. It is not blocking, but the effect can be the same: whatever reads the page early finds the interesting parts empty.

The ten-minute check

Put your domain followed by /robots.txt into a browser, so yourbusiness.co.uk/robots.txt, and read what comes back. You are looking for two things: a blanket Disallow under User-agent: *, and any of the crawler names above listed individually. If GPTBot, ClaudeBot, OAI-SearchBot, or Claude-SearchBot appear in there, somebody made a decision on your behalf. Check that it is the decision you would have made.

Then ask whoever runs your hosting or your CDN a direct question: is bot protection switched on, and does it include AI crawlers? The phrasing matters, because "we block bad bots" is not an answer. What you want to know is which categories are blocked and who chose them.

Last, open one page, view the page source, and search it for a price or an opening time you know is on that page. If the text is there in the raw source, good. If it is not, that content is being added after the page arrives, and you should not assume every system waits for it.

That is the entire check. It costs nothing, it needs no developer, and it settles a question most businesses have never thought to ask.

A note on llms.txt

You may have been told to add an llms.txt file, a plain text summary of your site written for AI systems. Adoption has grown fast. Originality.ai tracked more than three million websites and found llms.txt instances rising 8.8 times in a year, reaching 36,120 by May 2026.

The usage evidence does not match the enthusiasm. Ahrefs looked at server logs across 137,000 domains and found that 97 percent of llms.txt files received no requests at all during May 2026. The largest single source of the requests that did arrive was SEO audit tools, at 21.7 percent.

Add one if you want to; it costs very little. Just do not file it under access solved. A summary nobody fetches does nothing for a site whose robots.txt is closed, and the file is no substitute for knowing what your own settings permit.

Access comes first

Every other kind of readiness work assumes something can reach the page. Clear pricing, structured information about your business, answerable opening hours, none of it counts for anything behind a closed door. That is why access is one of the categories in every glassmark scan, and why a Missing there matters more than a Missing anywhere else.

The good news is that this is the cheapest problem in the subject to fix. Unblocking a crawler is usually one line, one setting, and ten minutes of somebody's afternoon. It just has to be somebody who knows to look.

Read more

What does AI find when it reads your website? Free scan at glassmark.ai, no developer required.

Run your free scan