LLMs-txt and AI crawlers are already shaping how your business gets discovered, whether you’ve noticed the traffic or not. For most businesses whose growth depends on being found, blocking AI crawlers trades away future visibility for no measurable gain.

“Every client who asks us to block ChatGPT’s crawler is really asking a different question,” says Ciaran Connolly, founder of ProfileTree. “They want to know if they’ll lose control of their content. The honest answer is that you already have less control than you think, and blocking the bot doesn’t restore it. It just removes you from the conversation.”

This guide covers which AI crawlers actually visit business websites, what robots.txt does about them, what llms.txt adds on top, and the one case where blocking genuinely makes sense.

What Are AI Crawlers, Exactly?

AI crawlers are automated bots, similar in principle to Googlebot, that visit websites to collect content. Where they differ is in what happens to the content afterwards. A traditional search crawler indexes a page so it can appear in search results with a link back to the source. An AI crawler might do that too, but it might also feed the content into a training dataset, or pull it into a live answer generated for someone who never visits the original page at all.

That distinction matters because it changes the trade-off. When Google crawls a page for its index, the business gains access to organic traffic. When an AI system crawls a page for training, there’s no direct traffic exchange; the value shows up later, if at all, when that model recommends the business in a conversation months down the line. This is part of why the topic feels unsettled: the mechanism is new enough that the usual cost-benefit maths doesn’t map cleanly onto it yet.

It’s worth being specific about what “AI crawler” actually covers, because the term gets used loosely. There are three broad categories:

Training crawlers collect content to improve a future version of a model. GPTBot from OpenAI and CCBot (used by Common Crawl, which many AI labs draw from) fall here.

Search and retrieval crawlers fetch content in real time to answer a specific user query, similar to how a search engine fetches a page to build a snippet. PerplexityBot and OAI-SearchBot behave this way.

Indexing crawlers tied to a specific AI feature sit between the two. Google-Extended controls whether content can be used in Google’s AI-generated summaries and Gemini, separately from whether the page appears in normal Google search results.

Getting this distinction right matters when you’re deciding what to allow, because blocking one category doesn’t necessarily block the others, and a business’s actual goal (usually: “be found and cited accurately”) is served differently by each.

The Main Bots Worth Knowing

Most business owners have heard of ChatGPT and Claude but have never seen the crawler names attached to them. Here’s what’s actually showing up in server logs on a typical UK SME website:

User-agent	Company	What it feeds
GPTBot	OpenAI	Model training data
OAI-SearchBot	OpenAI	ChatGPT’s live search/browsing feature
ChatGPT-User	OpenAI	Real-time fetches when a user asks ChatGPT to browse a specific page
ClaudeBot	Anthropic	Model training data
Claude-User	Anthropic	Real-time fetches during a Claude conversation
PerplexityBot	Perplexity	Powers Perplexity’s answer engine and citations
Google-Extended	Google	Controls use in AI Overviews and Gemini training
Applebot-Extended	Apple	Controls use in Apple Intelligence features
CCBot	Common Crawl	A dataset many AI labs use for training
Bytespider	ByteDance	Training data, including for products used outside China

A business running a site audit for the first time is often surprised by the sheer number of these bots hitting the server every day, sometimes more frequently than Googlebot itself. That volume alone tells you something: these systems are actively indexing commercial websites right now, not in some hypothetical future.

Anyone running technical checks on a site as part of a wider SEO review should look at crawl logs alongside standard SEO services work, because bot traffic patterns are now a normal part of a healthy technical audit rather than a curiosity. A technical SEO guide covers how to read server logs for this kind of pattern if it’s not something the team has done before.

robots.txt and AI Crawlers

The mechanism for controlling any of this hasn’t changed. robots.txt, the same plain text file that’s controlled search engine access since the 1990s, is also how businesses tell AI crawlers whether they’re welcome. It sits at the root of a domain (profiletree.com/robots.txt, for example) and uses simple allow/disallow directives per user-agent.

To allow a specific AI crawler explicitly:

User-agent: GPTBot
Allow: /

To block a specific AI crawler from the entire site:

User-agent: GPTBot
Disallow: /

To block several crawlers in one file:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

To block AI crawlers from a specific section while leaving the rest open, useful for a private client portal or staging area:

User-agent: GPTBot
Disallow: /client-portal/

User-agent: ClaudeBot
Disallow: /client-portal/

A few practical points get missed here. First, robots.txt is a request, not a lock. Well-behaved crawlers from major AI companies respect it, but the file has no technical means of stopping a bot that ignores it; enforcement, where it exists, occurs through separate methods such as firewall rules or IP blocking at the server level. Second, each AI company documents its own user-agent strings and updates them periodically, so a robots.txt file written eighteen months ago may already be missing bots that didn’t exist then. A quick check against each company’s current documentation, done as part of routine website hosting and management, catches this before it becomes a gap.

Third, and this trips people up constantly: blocking a crawler in robots.txt is not the same as blocking that AI system’s users from finding you. If ChatGPT’s browsing feature is blocked but the underlying model was trained on data from Common Crawl that already includes the site, the content may still surface, just without a live check for accuracy. Businesses sometimes assume a robots.txt block gives them a clean opt-out. It doesn’t, not fully, and understanding that gap is part of making an informed decision rather than a reflexive one.

What llms.txt Actually Does

llms.txt is a proposed standard, not an official web protocol backed by a standards body, that suggests websites publish a plain markdown file at /llms.txt summarising the site’s most important content in a format built for AI systems to consume efficiently. Think of it as a curated table of contents aimed at language models rather than at search engine indexers, pointing to the pages a business most wants an AI system to understand and reference.

Where robots.txt says “you may or may not crawl this,” llms.txt says “if you do crawl this, here’s what matters and here’s how to understand it quickly.” The two files solve different problems, and most businesses will eventually want both.

A basic llms.txt file might look like this:

# ProfileTree

> Belfast-based digital marketing agency offering web design, SEO, 
content marketing, and AI training for SMEs.

## Services
- [SEO Services](https://profiletree.com/services/search-engine-optimisation/): 
Technical SEO, content strategy, and local search for UK and Ireland businesses.
- [Digital Marketing Strategy](https://profiletree.com/services/digital-marketing-strategy/): 
Audit, plan, deliver, and refine frameworks for SME marketing budgets.

## Guides
- [Technical SEO Guide](https://profiletree.com/technical-seo-guide/)
- [Schema Markup Guide](https://profiletree.com/schema-markup-guide/)

Here’s the caveat that gets left out of most articles on this topic: adoption of llms.txt by the major AI companies remains partial. As of early 2026, there is no confirmation that OpenAI, Anthropic, or Google actively parse and prioritise llms.txt files during crawling or generation. The proposal, first put forward by Answer.AI’s Jeremy Howard and documented at the llms.txt specification site, has been adopted voluntarily by a growing number of developer tools, documentation platforms, and SaaS companies, but there’s no requirement for any AI crawler to read it, and no penalty for ignoring it.

That doesn’t make the file worthless. It costs very little to produce, it forces a business to think clearly about which pages actually represent its expertise, and it positions the site to benefit if and when adoption becomes more standard. It’s a low-cost bet on a plausible future, not a proven lever that moves citations today. Framing it that way to a client avoids overpromising a result that the mechanism can’t currently guarantee.

Should You Block AI Crawlers?

For a typical SME service business, the case for allowing AI crawlers comes down to where buyers are actually looking for recommendations. When a business owner asks ChatGPT or Perplexity, “Who does WordPress web design in Belfast?” the businesses that appear in that answer are the ones whose content was crawlable. Blocking the bots that build those answers doesn’t protect anything for a service business; it just removes the business from a channel that’s growing every quarter.

There’s also a competitive dimension that rarely gets mentioned. If a business blocks GPTBot and its three direct competitors don’t, the AI system building an answer about local web design or SEO providers has three data sources and one gap. It doesn’t wait for the missing business to change its mind. It answers with what it has.

The content marketing team at ProfileTree tracks this directly through Bing’s AI citation reporting, and pages that receive meaningful AI citation volume tend to be those with clear, well-structured, genuinely useful content, not the ones that happen to be crawled by coincidence. Allowing the bots is necessary but not sufficient; the content still has to earn the citation on its own merits, which is where content marketing services and a properly built content marketing strategy do the actual work.

None of this means blocking is always the wrong call. There’s a genuine publisher exception, and it’s worth being honest about where the line sits.

When Blocking Actually Makes Sense

Blocking AI crawlers is a reasonable decision for a narrow set of businesses:

Paywalled or subscription content. A publisher whose entire revenue model depends on people paying to read articles has a direct commercial reason to keep that content out of free AI answers. If an AI system can summarise the paywalled article well enough that nobody needs to subscribe, the crawler is actively working against the business model.

Proprietary research or data products. A company selling access to a dataset, a benchmark report, or original research has a similar argument. If the value of the product is the information itself, letting a crawler absorb and redistribute it undermines the thing being sold.

Genuinely sensitive internal content. Client portals, staff intranets, or draft content that shouldn’t be public at all should be blocked from every crawler, AI or otherwise, though this is really a case for noindex and access controls rather than an AI-specific decision.

An SME service business, the kind that makes money by being hired for web design, SEO, digital marketing, or AI training, doesn’t fit any of these categories. The business isn’t selling the content itself; it’s using the content to demonstrate expertise so that people hire the business behind it. Blocking crawlers removes the demonstration without protecting any revenue. That’s the core distinction worth explaining clearly to a client who’s nervous about the idea: are you selling the words on the page, or the service they describe? Almost every SME is in the second category.

How to Check What’s Currently Crawling Your Site

Before making any change to robots.txt, it’s worth establishing what’s actually happening today. Server access logs show every crawler that’s hit the site, along with the user-agent and timestamp. Most hosting providers make these available through a control panel, and a competent website hosting and management provider should be able to pull a crawl report on request.

Google Search Console also shows crawl stats for Googlebot specifically, though it won’t show AI crawler activity from other companies. For that, log file analysis is the only reliable method; there isn’t yet a unified dashboard that shows activity from GPTBot, ClaudeBot, and PerplexityBot in one place, the way Google Search Console does for Google’s own crawlers.

Cross-referencing crawl activity against Google Analytics 4 referral data adds another layer of insight, since GA4 can show whether traffic is arriving from AI chat interfaces (ChatGPT, Perplexity, and Copilot all pass referral data in most configurations). A page with strong AI crawler visits but no corresponding referral traffic might be feeding training data without yet generating citations; a page with both is doing exactly what a business wants from its content.

Setting This Up Without Breaking Your SEO (shortened)

A few practical warnings before editing robots.txt on a live site. Test changes in a staging environment or use Google’s robots.txt tester first; a misplaced Disallow: / under User-agent: * blocks every crawler, including Googlebot, and can remove a site from search results within days.

Don’t confuse blocking AI crawlers with blocking AI-referred traffic. Bots like OAI-SearchBot and Claude-User fetch pages in real time because a user asked to see them; blocking those cuts off direct traffic rather than training data.

Treat this as ongoing maintenance, not a one-off fix. New crawlers appear as AI companies launch features, so review the file quarterly alongside broader SEO services work. Pair the technical setup with genuine structural work too: clear entity statements, FAQs, and schema markup, covered in a schema markup guide, matter more than the robots.txt setting on its own. Allowing crawlers gets a business into the room. Useful content is what gets it quoted.

What This Means for Your Business: LLMS-Txt and AI Crawlers

Allow the crawlers to publish an llms.txt file if the time cost is low, and keep building content worth citing. That’s the whole decision for most SMEs. Blocking protects nothing when the business sells a service rather than the words that describe it, and every quarter spent blocked is a quarter during which a competitor’s content fills the gap instead. Check crawl logs and referral data periodically, adjust as new bots emerge, and treat AI visibility as a standard line in the digital marketing strategy, not a speculative extra.

FAQs