What Is robots.txt? How It Decides Whether Google and ChatGPT Can See Your Site

Someone in Mombasa asks ChatGPT for a good caterer for a wedding. It names three businesses. Yours is not one of them, and you will never know it was asked.

There are many reasons that can happen. One of them takes a minute to check and a minute to fix, and it is a small text file at the root of your website that most business owners have never opened: robots.txt.

This guide explains what the file does, who reads it today, the copied snippet that quietly removes Kenyan websites from AI answers, and how to check yours for free.

What robots.txt Is

robots.txt is a plain text file that sits at the top of your site, at an address like yourbusiness.co.ke/robots.txt. Before an automated visitor reads your pages, it reads this file to see where it is welcome.

The file is a list of groups. Each group names one or more crawlers on a User-agent line, then says what they may not read (Disallow) and, if needed, what they may (Allow). User-agent: * means "everyone I have not named". A Sitemap line points every crawler to your full list of pages.

User-agent: *
Disallow: /wp-admin/

Sitemap: https://yourbusiness.co.ke/sitemap.xml

That one says: everyone may read everything except the WordPress admin area, and here is the list of my pages. For years that was all most sites needed, because only one crawler really mattered.

Who Reads It Now: Four Kinds of Crawler

Today your file is read by four kinds of bot, run by different companies for different purposes. Each obeys its own User-agent name, so you can treat them differently.

Kind	Examples	What blocking it does
Search engines	Googlebot, Bingbot, Applebot	You drop out of search results. Google says Googlebot's rules cover all Google Search features.
AI answer engines	OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot	ChatGPT search, Claude and Perplexity cannot find and cite your pages.
AI assistants	ChatGPT-User, Claude-User, Perplexity-User	An assistant cannot open your page when a person asks it to.
AI training	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot	Your content is not used to train future AI models. You stay in search and AI answers.

Three facts from the companies' own documentation are worth knowing:

• OpenAI says that if you block OAI-SearchBot your site will not be shown in ChatGPT search answers, and that blocking GPTBot only opts you out of training. They are separate switches. (OpenAI crawlers)
• Google-Extended and Applebot-Extended never visit your site. They are switches Google and Apple check before using pages their normal crawlers already collected to train AI models. Google says Google-Extended does not affect your inclusion or ranking in Google Search. (Google crawlers, About Applebot)
• The assistants act for a person who asked, and two companies say they may not follow your rules. OpenAI says robots.txt "may not apply" to ChatGPT-User; Perplexity says Perplexity-User "generally ignores" it. Anthropic says its bots, including Claude-User, respect it. (Perplexity crawlers, Anthropic crawlers)

The Snippet That Hides You From AI Answers

When AI tools started reading the web, a lot of "block AI" advice went around, and many sites copied a snippet like this:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

The intention is "only Google may read my site". The effect is that every AI answer engine, Bing and Apple are shut out too, because they all fall into the * group. Your Google ranking is fine, so nothing looks wrong. Meanwhile ChatGPT search, Perplexity and Claude cannot cite you, and when someone asks them to recommend a business like yours, they recommend someone they can read.

If what you actually want is "do not train AI on my content", block the training bots by name and leave everyone else alone. That keeps you in search and in AI answers. The pattern is further down.

How a Crawler Reads the File

Two rules, from Google's robots.txt specification, explain most surprises. (Google's specification)

One group per crawler. A crawler uses only the group that names it. If nothing names it, it uses the User-agent: * group. The two are never combined. So if you write Disallow: /admin/ for everyone, then add a separate group for GPTBot with only Disallow: /private/, GPTBot ignores your /admin/ rule. Anything every bot must follow has to be repeated in every named group.

The longest matching rule wins. Inside a group, the rule with the longest matching path decides, and if an Allow and a Disallow are the same length, the Allow wins. That is why Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php works.

One exception we found while building our checker: Apple says Applebot follows your Googlebot rules when you have none for Applebot. If you block Googlebot from a folder, Siri and Spotlight lose it too.

What robots.txt Does Not Do

• It does not hide a page from Google. It stops crawling, not indexing. A blocked page that other sites link to can still appear in results, just without a description. To keep a page out of Google, let Google crawl it and add a noindex tag.
• It is not security. The file is public, so listing /secret-prices/ in it advertises the folder. Anything private needs a password, not a robots rule.
• It is a request, not a lock. Reputable crawlers follow it. Scrapers that do not care will not, and as above, some assistants acting for a person may not either.

When the File Itself Breaks Your Site

The worst robots.txt problems are not bad rules. They are a file that fails to load properly. Google's specification is specific about what happens:

• 404, file not found. Google assumes there are no restrictions. Nothing breaks, but you cannot point crawlers to your sitemap or opt out of training.
• A server error (5xx) or a timeout. Google stops crawling your site for the first 12 hours and keeps retrying the file. A flaky host can quietly slow your whole site's indexing this way.
• An HTML page instead of a text file. Common on website builders that send every unknown address to the homepage. Crawlers find no rules in it at all, so the rules you thought you had are not being read.
• A file over 500 KB. Google reads the first 500 KiB and ignores the rest.

One more that we see on Kenyan WordPress sites: a rule that blocks /wp-content/ or /wp-includes/. Google renders your pages the way a browser does. Block your theme's styles and scripts and it can misjudge your layout, including whether the page works on a phone.

A Sensible robots.txt for a Kenyan Business Site

For a typical WordPress business site that wants to be found on Google and in AI answers, but not used for AI training:

# Everyone, including search engines and AI answer engines
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Opt out of AI training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://yourbusiness.co.ke/sitemap.xml

Whether to block training at all is a business decision, not a technical one. Many businesses are happy for AI models to learn about them; a publisher whose articles are the product may not be. Either way, keep the answer engines and assistants open if you want to be recommended.

You may also see a Content-Signal line, introduced by Cloudflare. It states your preferences for three uses, search, ai-input (feeding AI answers) and ai-train, each yes or no. Cloudflare itself notes that not every crawler honours it, so treat it as a statement of intent alongside your rules, not instead of them. (Content Signals)

Check Yours in a Minute

Open yourbusiness.co.ke/robots.txt in a browser and you will see the file. Reading it correctly is harder, because of the group and longest-rule rules above.

We built a free tool for that. The WPfoss robots.txt checker reads your live file and tells you, for 14 crawlers across all four kinds, whether each can reach your homepage and which exact line decided it. It also tests real pages from your sitemap as Googlebot, catches a file that errors or comes back as a web page, flags blocked theme files, and lists what to fix first. No signup, and it changes nothing on your site.

If your file is fine, the next question is whether AI tools actually mention you, which is a different problem. Our guide on checking your AI search visibility covers that.

Frequently Asked Questions

If I block GPTBot, will I disappear from ChatGPT?

No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot opts you out of training only. To stay in ChatGPT search answers, OAI-SearchBot must be allowed.

Does robots.txt keep a page out of Google?

Not reliably. It stops crawling, not indexing, so a blocked page that other sites link to can still appear in results without a description. Use a noindex tag on a page Google is allowed to crawl.

Do I need a robots.txt at all?

No site is required to have one. Without it, every crawler may read everything. But a short file is worth having: it points crawlers to your sitemap and lets you decide about AI training.

Does blocking Google-Extended affect my Google ranking?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It only controls whether your content is used for Google's AI models.

Where do I edit robots.txt on WordPress?

WordPress serves a basic one automatically. SEO plugins such as Yoast and Rank Math include an editor for it, or a real robots.txt file uploaded to your site's root folder replaces the automatic one. After any change, run the checker again.