Krolik Marketing

Is my website blocking ChatGPT? Check robots.txt and your firewall

Open yourdomain.com/robots.txt in your phone's browser. If you see `User-agent: OAI-SearchBot` followed by `Disallow: /`, you are blocking ChatGPT's search bot. If you don't see OAI-SearchBot at all, check for `User-agent: *` followed by `Disallow: /`. That blocks it too. A firewall or Cloudflare can block it even when this file is fine.

A robot stopped by a red barrier with a no-entry sign before it can reach a website.

Using Wix, Squarespace or Shopify? Your builder's steps are at the bottom.

robots.txt is a small text file that tells bots which parts of your site they may read. Anyone can open it.

If the bot names are new, see which AI bots visit my website.

Step 1: read the file

  1. Open yourdomain.com/robots.txt in your browser.
  2. Use your browser's Find option to look for OAI-SearchBot.
  3. Found it? Read the lines under it. `Disallow: /` (the whole site) means blocked. `Allow: /` or nothing means open.
  4. Not found? Look for `User-agent: *` (any bot) followed by `Disallow: /`. That means blocked.
  5. Nothing matches? This file is not blocking it.

A `User-agent:` line names a bot. The rules under it belong to that bot.

Found a block? Ask whoever built your site to remove those two lines.

www.yourdomain.com and yourdomain.com have separate files. Check both.

A bot with its own group ignores the `` group. So a `` block does not stop OAI-SearchBot if OAI-SearchBot has its own group.

A block on OAI-SearchBot alone looks like this:

1User-agent: OAI-SearchBot
2Disallow: /

Which blocks matter for ChatGPT

  • OAI-SearchBot decides whether your site is shown in ChatGPT's search answers. OpenAI says sites that block it "will not be shown".
  • GPTBot is the training bot. OpenAI says you can allow OAI-SearchBot and disallow GPTBot. That is a separate choice, yours to make.
  • ChatGPT-User visits when a person asks. OpenAI says robots.txt rules may not apply to it.

Step 2: Cloudflare or a firewall

A firewall (software that decides who may reach your site) can turn bots away even when robots.txt allows them.

So can a service in front of your site, such as Cloudflare. Cloudflare can add its own rules to what bots see.

To check it:

  1. In the Cloudflare dashboard, select Security.
  2. Select Settings.
  3. Filter by Bot traffic.
  4. Select AI Crawler and Agent access policies.
  5. Read the setting for Search, Agents and Training.

Cloudflare's docs say new domains start with all three on "Allow (do not block)".

If any of the three says Block, ChatGPT's search bot may be turned away. Ask whoever set it up before changing it.

Not sure? Ask your host or developer to look in your visit log for refused requests from OAI-SearchBot.

If you use a site builder

Wix

  1. Open SEO & GEO from your site's dashboard.
  2. Scroll down to Tools and settings.
  3. Click Robots.txt Editor.
  4. Click View File.

Wix says the default file lets search bots reach all your pages.

Squarespace

  1. Open the Settings panel.
  2. Click Crawlers.
  3. Look at the box Block known artificial intelligence crawlers.

It is unchecked by default. Ticking it blocks 26 AI bots, including GPTBot and ClaudeBot, but not OAI-SearchBot or ChatGPT-User.

Squarespace warns that excluding AI crawlers "might negatively impact your site traffic".

Shopify

Shopify calls its default file "optimal" for SEO and says bot management "is handled directly for stores on Shopify".

Its Support can't help with edits.

If you need to change it:

  1. In the Shopify admin, select Online Store.
  2. For your theme, click the menu icon.
  3. Select Edit code.
  4. Click Add a new template.
  5. Select robots.
  6. Click Create template.

What to leave alone

Don't delete lines you don't understand. And robots.txt is not a lock. Use a password for anything private.

Related

Last checked 2026-11-16.

Based on public documentation: RFC 9309; OpenAI; Google Search Central; Cloudflare; Wix Help Center; Squarespace Help.

All posts