Can Google and ChatGPT read your site?
Your robots.txt decides which search engines and AI tools may visit your website. Check yours in seconds: who is allowed, who is blocked, the exact line behind each answer, and what to fix first.
We read your public robots.txt and sitemap only. Nothing on your site changes.
Reading your robots.txtโฆ
What to fix first
Worst first. Each one says what it means for your business and how to fix it.
Crawler by crawler
Whether each crawler may read your homepage, and the line in your file that decided it.
Your pages, as Googlebot sees them
Pages from your sitemap, tested against your Googlebot rules.
Your robots.txt
Highlighted lines are the ones that block a crawler above.
Get help with what we found
Want us to fix this for you?
Tell us what the tool found and we will tell you what it takes to fix it. The first call is free.
Get tips like these by email
Practical advice on getting found, getting faster and getting more enquiries, plus news and offers about WPfoss products. No more than once a fortnight, and one click to leave.
Optional. You do not need to sign up to see your results.
Want us to fix this for you? The first call is free.
Four kinds of crawler, four different decisions
Most robots.txt advice still assumes one visitor matters: Googlebot. Today your file is read by four kinds of bot, and blocking the wrong one has very different results.
- Search engines (Googlebot, Bingbot, Applebot) decide whether you rank. Google AI Overviews and Bing's AI answers draw on these same indexes.
- AI answer engines (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether ChatGPT search, Claude and Perplexity can find and cite your pages.
- AI assistants (ChatGPT-User, Claude-User, Perplexity-User) fetch a page live when someone asks an assistant to read it. OpenAI and Perplexity say these may fetch anyway, because a person asked.
- AI training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) decide whether your content may be used to train AI models.
Blocking a training bot is a licensing choice and does not remove you from AI answers. Blocking an answer engine removes you from the pages it can cite, and that is usually an accident: a "block all AI" snippet copied from a forum, which shuts out the search bots along with the training ones.
A robots.txt that keeps you in AI answers
If you want search engines and AI answers to find you, but not to have your content used for training, this is the pattern. Everything not named follows the first group; the training bots get a group of their own.
# Everyone, including search engines and AI answer engines User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Opt out of AI training only User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / Sitemap: https://yourbusiness.co.ke/sitemap.xml
Two details catch people out. A group that names a crawler replaces the User-agent: * group for that crawler; it does not add to it, so any rule you want every bot to follow has to be repeated in the named groups. And Google-Extended and Applebot-Extended never visit your site: they are switches that tell Google and Apple whether pages their normal crawlers already collected may train their models. Blocking them changes nothing in search.
Want this set up for you, or your whole site checked? Our SEO service covers it. Read the full guide: robots.txt and AI crawlers, explained.
Questions people ask
If I block GPTBot, will I disappear from ChatGPT?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot opts you out of training only. To stay in ChatGPT search answers, OAI-SearchBot must be allowed.
Does robots.txt keep a page out of Google?
Not reliably. robots.txt stops crawling, not indexing. A blocked page that other sites link to can still appear in results, without a description. To keep a page out of Google, allow crawling and add a noindex tag instead.
What happens if my robots.txt is missing or returns an error?
A 404 means no restrictions: every crawler may read everything. A server error (5xx) or a timeout is worse: Google stops crawling your site for the first 12 hours and keeps retrying the file.
How does a crawler decide which rule applies?
It uses only the group that names it, or the User-agent: * group if none does. The two are never combined. Inside that group the longest matching path wins, and on a tie Allow beats Disallow. Apple's crawler is the one exception we check for: with no Applebot group, it follows your Googlebot rules.
What is Content-Signal?
A line, introduced by Cloudflare, that states how your content may be used: search, ai-input (feeding AI answers) and ai-train (training models), each yes or no. It expresses a preference; not every crawler honours it.
How is the score worked out?
It measures how open your homepage is to the crawlers that bring visitors. Search engines count three times, AI answer engines twice and AI assistants once. Training bots are not counted, because blocking them is your choice. A robots.txt that cannot be read at all scores 0, because Google stops crawling.
Do you keep my data?
We keep the address you checked and a summary of the result, so we can improve the tool. You do not need to give us your name or email to see your results. See our privacy policy.