AI Visibility ·
Should You Block AI Crawlers? A Straight Answer for Small Business
By Sam Shetty
CEO
Short answer
For most small businesses, blocking AI crawlers is a mistake. Blocking removes you from AI-generated answers without removing your competitors, so you lose visibility at the exact moment buyers are forming a shortlist. Publishers whose entire business is selling access to content have a genuine case for blocking. A business whose content exists to attract customers does not.
What blocking actually does
Adding `User-agent: GPTBot` and `Disallow: /` to robots.txt asks OpenAI's crawler not to access your site. Compliant crawlers respect it. What it does not do is remove you from answers built on other sources that mention you, or stop anyone quoting your business from a directory listing.
Who has a legitimate reason to block
News publishers, subscription research firms, stock libraries — businesses where the content is the product being sold. If someone reading your content instead of buying from you is a lost sale, blocking is a defensible commercial decision.
Why it's wrong for a service business
Your content exists to make people aware you can solve their problem. When an assistant answers "who should I hire for X in Sydney", you want to be one of the three names. Blocking guarantees you aren't, while your competitors are.
The crawlers worth naming explicitly
GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and Bingbot for Copilot. Being explicit in robots.txt is clearer than relying on a wildcard, and makes your intent auditable.
Training and retrieval are different things
Some crawlers gather training data; others fetch pages live to answer a question right now. Blocking the retrieval crawlers is what costs you visibility today. If you have a philosophical objection to training use specifically, `Google-Extended` and `GPTBot` are the levers, but understand you're trading present-day visibility for it.
What we do on client sites
Allow all of them, explicitly, and put the effort into being worth citing. That's the whole approach behind our AI visibility service, and it starts from the assumption that being findable beats being protected.
We fetch your page and check it live. Takes a few seconds, and no email is needed to see the result.
- Whether search engines and AI crawlers can index the page
- Title, meta, headings, canonical and mobile setup
- Core Web Vitals from real Chrome user data
- Whether contacting you takes one tap or three
- Every result shows the specific check behind it
Growth snapshot
your site
Measured live just now. Expand any row to see the checks behind it.
Want the parts we cannot automate?
AI Optimisation
Get cited by AI assistants, and get more out of AI in your campaigns.
Keep reading
Related articles
AI optimisation
How to Structure a Page So AI Assistants Will Quote It
Assistants don't quote your best paragraph. They quote your most liftable one. There's a difference, and it's structural.
Hasnain Hararwala ·
Read itAI optimisation
How to Build an AI Visibility Prompt Set for Your Business
You cannot improve AI visibility without measuring it, and measuring it means building your own benchmark. Here's how we construct one.
Hasnain Hararwala ·
Read itAI optimisation
Schema Markup That Actually Matters (and What to Skip)
There are hundreds of schema types. About six matter for a small business site, and one of the popular ones can get you a manual action.
Jisan Polara ·
Read itTalk to a strategist
Let's find out what's actually holding you back
Fifteen minutes on the phone, an honest read on your situation, and a number. If we're not the right fit we'll say so and point you somewhere better.
