Is Your Website Accidentally Blocking AI Crawlers? A 2026 Check for Singapore Businesses

Nobody sits down and decides to block AI from their website. It happens through accumulation.

Your robots.txt file was almost certainly last updated with Googlebot in mind. It does not account for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, or the dozen other AI crawlers that have appeared since 2023. AI crawlers use different user-agent strings from search engine crawlers, so a file that correctly allows Googlebot can still block every AI bot through a broad disallow rule.

Four common causes, all of them accidental:

A leftover staging rule. A Disallow: / written to keep a development site out of Google, which quietly went live with the production site.

Googlebot-only rules. A robots.txt that explicitly allows Googlebot and nothing else. Every AI crawler falls under the restrictive wildcard rule instead.

Security plugins and CDN configurations. This is the one most Singapore SMEs never think to check. Aggressive Cloudflare, Sucuri or WAF settings frequently treat OAI-SearchBot and PerplexityBot the same as malicious scrapers, challenging them before any content loads. Your robots.txt can be perfect and the CDN still blocks the bot.

CMS defaults. Some platforms and security plugins add blanket blocks for "unknown" crawlers automatically. Because AI bots are new, they get caught.

The result is asymmetric in a way worth stating plainly: if you accidentally block a retrieval bot, your brand disappears from the AI answers that crawler powers. There is no warning, no notification, and nothing in your analytics that tells you it happened.

Do this now, before reading further:

Step 1. Open yoursite.com/robots.txt in your browser.

Step 2. Look for these patterns:

User-agent: * / Disallow: /

This blocks everything, including every AI crawler.

User-agent: Googlebot / Allow: / / User-agent: * / Disallow: /

This allows only Google. Every AI crawler is blocked.

Step 3. Check whether any AI bots are named at all. If your file mentions only Googlebot and Bingbot, AI crawlers are governed by whatever your wildcard rule says.

Step 4. Check for path-level blocks. If AI crawlers can reach your homepage but /services/ or /blog/ is disallowed, they cannot cite the pages that actually matter commercially.

If you find nothing concerning, good — but robots.txt is only the first of several possible blockers, covered below.

AI crawlers are not interchangeable. Each major AI company runs separate bots for different purposes, and the distinction between training bots and retrieval bots is the one that determines your commercial exposure.

Retrieval and search bots — block these and you lose citations:

• GPTBot (OpenAI) — blocking it means your content is unlikely to appear in ChatGPT responses
• OAI-SearchBot (OpenAI) — powers ChatGPT's search function
• ClaudeBot and Claude-SearchBot (Anthropic)
• PerplexityBot — Perplexity is a major AI search engine; blocking this removes you from its answers
• Bingbot — worth flagging for Singapore businesses: ChatGPT's web search draws on Bing's index, so Bingbot matters for AI visibility beyond Bing's own modest search share
• Google-Extended — governs whether your content informs Google's AI features

Training bots — a legitimate business decision either way:

Some bots primarily gather data for model training rather than powering live citations. Training crawlers now account for roughly 67.5% of AI-driven crawl traffic by volume. Whether to allow them is a genuine choice: allowing them may help your brand appear in model knowledge over time, while blocking them protects your content from being used as training material. Reasonable businesses land on both sides.

The point is precision. A well-configured robots.txt in 2026 controls which bots access which content for which purposes — rather than applying blanket allow-all or block-all rules that either expose everything or remove you from AI search entirely.

If your check found a problem, the fix is straightforward. Add explicit Allow rules for the AI bots you want, placed above any wildcard rule — specific user-agent rules take precedence over the wildcard.

A minimal configuration allowing the major retrieval bots:

User-agent: GPTBot / Allow: / / User-agent: OAI-SearchBot / Allow: / / User-agent: ClaudeBot / Allow: / / User-agent: PerplexityBot / Allow: / / User-agent: Google-Extended / Allow: /

You do not have to allow everything. Common configurations include allowing search and retrieval bots while blocking pure training crawlers, or allowing AI crawlers site-wide but disallowing specific directories such as members-only areas. This is what robots.txt does well — per-bot, per-path control.

Then verify the fix actually took effect. Editing robots.txt does not help if a CDN or firewall is blocking the bot before it ever reaches the file.

If your robots.txt is clean and you still are not being cited, the blocker is usually further down the stack. AI crawlability is the technical foundation of GEO: if AI crawlers cannot access, render, parse and trust your pages, they cannot cite them. The problem is rarely one dramatic mistake — it is usually a stack of small technical decisions.

CDN and firewall rules. Cloudflare, Sucuri and similar services may challenge unfamiliar user agents before content loads. Check your WAF and bot-management settings specifically for AI user agents. This is the single most common blocker after robots.txt, and the one least likely to be on anyone's radar.

JavaScript rendering. If your key content — service descriptions, pricing, answers — only appears after client-side JavaScript executes, AI crawlers may never see it. A React or Vue site whose substance is absent from the raw HTML is functionally invisible to some crawlers. For Singapore businesses on modern frameworks, this deserves an explicit check.

Page structure and metadata. Weak heading structure, missing schema markup, and thin metadata reduce whether a crawler can parse and trust what it finds.

Internal linking. AI crawlers follow internal links to understand what your site is about and which pages matter. Orphaned or weakly linked pages get skipped.

A practical verification method: ask ChatGPT or Perplexity directly about your business and your service category in Singapore. If the answer is inaccurate, generic, or omits you entirely while naming competitors, something in this stack is not working. That test takes five minutes and is more informative than most dashboards.

Here is the argument for fixing this before writing another article.

Technical access is binary in a way content quality is not. Excellent content behind a blocked crawler produces zero citations. Adequate content behind an open crawler produces some. Every dollar spent on content while crawlers are blocked is spent on an asset that AI systems cannot reach.

In audit frameworks that score GEO readiness, robots.txt configuration is frequently the highest-weighted single check — the largest individual score gap available, and one of the few that can be closed in an afternoon.

For a Singapore SME, the sequencing is therefore: confirm access first, then structure, then content. Reversing that order is how businesses end up funding a content programme whose output is systematically excluded from the channel they were trying to enter.

Q1: How do I check whether my website is blocking AI crawlers?

Open yoursite.com/robots.txt in a browser. Look for User-agent: * followed by Disallow: /, which blocks everything including all AI crawlers, or a file that explicitly allows only Googlebot, which leaves every AI crawler governed by a restrictive wildcard rule. Also check for path-level blocks on directories like /services/ or /blog/ — crawlers that can reach your homepage but not your commercial pages cannot cite what matters. If robots.txt looks clean, check your CDN or firewall settings next, since services like Cloudflare and Sucuri often challenge AI user agents before content loads, regardless of what robots.txt says.

Q2: Which AI crawlers should a Singapore business allow?

At minimum, allow the retrieval and search bots that power live citations: GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's AI features), and Bingbot — the last matters more than its search share suggests, because ChatGPT's web search draws on Bing's index. Training-focused crawlers are a separate decision: they account for roughly 67.5% of AI crawl traffic, and blocking them protects content from training use while allowing them may help brand presence in model knowledge over time. Both positions are defensible.

Q3: Will allowing AI crawlers mean my content gets used for AI training?

Not necessarily, because you can separate the two. Retrieval bots (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) primarily power live citations in AI answers, while other crawlers focus on gathering training data. robots.txt supports per-bot rules, so a common configuration allows search and retrieval bots — preserving your ability to be cited — while blocking pure training crawlers. This is a legitimate business choice with reasonable arguments on both sides, and the important thing is making it deliberately rather than inheriting a blanket rule you never chose.

Q4: My robots.txt is fine, but I still don't appear in ChatGPT or Perplexity. Why?

robots.txt is only the first layer. The most common next blocker is CDN or firewall configuration, where services like Cloudflare or Sucuri challenge unfamiliar AI user agents before any content loads. After that: JavaScript rendering, where content that only appears after client-side JavaScript executes may never be seen by crawlers; weak page structure and missing schema, which reduce whether a crawler can parse and trust your content; and poor internal linking, since crawlers follow links to determine which pages matter. The blocker is usually a stack of small technical decisions rather than one dramatic error.

Q5: Should I fix crawler access before investing in content for AI search?

Yes, and the reason is that technical access is binary while content quality is not. Excellent content behind a blocked crawler generates zero citations, so any content investment made while crawlers are blocked funds an asset AI systems cannot reach. In GEO audit frameworks, robots.txt configuration is often the highest-weighted single check and one of the largest score gaps available — while also being one of the few fixable in an afternoon. The correct sequence for a Singapore SME is to confirm crawler access first, then page structure, then content production.

Mayson AI runs technical GEO audits for Singapore businesses — crawler access, CDN configuration, rendering, structure and schema — before recommending any content spend. If you want to know whether AI systems can actually reach your site, book a consultation.