
A bot is a program that reads web pages automatically. Each one gives its name when it visits.
That name shows up in your access log, the list your host keeps of every request to your site. I call it the visit log.
Not sure where yours is? See how to find your website's visit log.
If you only have 2 minutes
- OAI-SearchBot lets ChatGPT show your site. Good to see.
- ChatGPT-User means a person asked ChatGPT and it fetched your page. Good to see.
- GPTBot, ClaudeBot, meta-externalagent collect text to train AI models. Whether to allow them is your choice.
Some bots may ignore robots.txt (your site's do-and-don't list for bots), especially when a person asked for the page.
A name is only what the visitor claims. Google and Bing both say a bot's name can be faked.
In my log, 5 of the 929 requests labelled ChatGPT-User did not come from OpenAI's published addresses.
OpenAI and Perplexity publish lists of the addresses their bots use. Ask your developer to check a visit against the company's list.

OpenAI (ChatGPT)
Every description below is from the company's own page, 2026-11-16. It is what the company says, not something I tested.
- OAI-SearchBot is for search: it lets ChatGPT show websites in its search results.
- GPTBot collects text that may be used to train OpenAI's models.
- ChatGPT-User fetches a page when a person asks ChatGPT a question. OpenAI says it is not used to crawl the web automatically. Because a person starts the visit, OpenAI says robots.txt rules may not apply.
Perplexity
- PerplexityBot shows and links websites in Perplexity's search results. Perplexity says it is not used to collect content for AI models.
- Perplexity-User fetches a page when a person asks Perplexity a question. Because a person requested it, Perplexity says it generally ignores robots.txt.
Anthropic (Claude)
- ClaudeBot collects web content that could be used for training.
- Claude-User visits pages when a person asks Claude a question.
- Claude-SearchBot crawls the web to improve search result quality.
Meta
- meta-externalagent crawls to train models or to add content to Meta's catalog directly.
- meta-externalfetcher fetches a single link at a user's request. Meta says it may bypass robots.txt.
- meta-webindexer crawls to improve Meta AI search results.
Three more bots deal with ads and shared links, not AI search:
- OAI-AdsBot checks pages submitted as ads on ChatGPT.
- meta-externalads improves Meta's advertising products.
- facebookexternalhit reads a page someone shared on Meta's apps, to show its title and thumbnail.
Names you will also see
Googlebot and bingbot are Google's and Bing's normal search bots. You can ignore them for this.
What one log showed
On one site I built, OpenAI's bots made 93% of the AI-bot requests (7,080 of 7,630). Being read often is not the same as being recommended.
The full write-up is in what ChatGPT reads on a small business website.
No access to your log?
Cloudflare's AI Crawl Control shows which AI services request your content.
Related
- Is my website blocking ChatGPT? Check robots.txt and your firewall: how to let in the bots you want.
- How do I see ChatGPT traffic in Google Analytics (GA4)?: the people ChatGPT sends, as opposed to the bots.
Based on public documentation: OpenAI; Perplexity; Anthropic; Meta; Google; Bing.
