Why your website does not show up in ChatGPT answers

One bot decides

A single crawler is the gate for ChatGPT search, and you can check it in your robots.txt file.

Every website has a small text file called robots.txt. It tells visiting programs what they may and may not read. OpenAI says in its crawler docs that a site which blocks OAI-SearchBot in that file will not be shown in ChatGPT search answers, though it can still show up as a plain navigation link. OpenAI also says a change to that file takes about 24 hours to take effect for OAI-SearchBot, so if you fix a block today, check back tomorrow.

Search bots and training bots

OpenAI runs 2 different bots, and blocking one does not block the other.

OpenAI's crawler docs list 2 separate user agents, GPTBot for training its models and OAI-SearchBot for search. Blocking GPTBot stops your text from training future models. It does not, by itself, pull you out of ChatGPT search, and letting search in does not require letting GPTBot in.

There is a third one, called ChatGPT-User. OpenAI says it fetches a page only when a person asks ChatGPT to open that page, and because the person asked, robots.txt rules may not apply. It plays no part in search answers. So the decision about whether you appear rests on 1 bot, per OpenAI's docs, and that is OAI-SearchBot.

Who is knocking on your site

The other answer engines work the same way, with one bot for search and separate bots for training or fetching.

Perplexity's crawler docs say PerplexityBot decides who gets linked in Perplexity results, obeys robots.txt, and is not used for training, while Perplexity-User, which opens a page when a user asks, generally ignores robots.txt. Anthropic's support page lists 3 agents, ClaudeBot, Claude-SearchBot and Claude-User, and says all 3 honor a robots.txt disallow. The table sums it up.

Basis: every row comes from the OpenAI crawler docs, the Perplexity crawler docs, the Anthropic support page and Google Search Central. The 24 hours figure is OpenAI's own stated time for a robots.txt change to reach OAI-SearchBot.

Google is still the front door

Google says the only gate for its answer boxes is being indexed, which is plain old SEO.

Google Search Central states there are 0 extra requirements to appear in AI Overviews or AI Mode, 0 new machine readable files, 0 special AI text files and 0 special schema types. What you need is 2 things, per Google Search Central. Your page must be indexed, and it must be eligible to show in Search with a snippet. Google also says its answers run multiple related searches per question, which it calls query fan-out, so if you are not indexed, no fan-out finds you.

The traffic numbers back this up. Cloudflare measured crawler traffic across its whole network over 1 year, from one May to the next, and found Googlebot's share grew from 30% to 50% of all crawler requests, a 96% jump. In the same Cloudflare measurement, GPTBot grew from 2.2% to 7.7%, up 305% in request volume, while ClaudeBot fell from 11.7% to 5.4% and Bytespider fell from 22.8% to 2.9%. Cloudflare also found total AI and search crawler traffic rose 18% over that year, with a peak month 32% above the start, so Google is still the biggest visitor by far.

Blocking is rarely the problem

Most sites, even the biggest, have never written a rule about AI bots at all.

Cloudflare crawled the robots.txt files of the top 10,000 domains, could read 3,816 of them, and found only 546, about 14%, had any allow or disallow line aimed at an AI bot. In that same Cloudflare crawl, 312 domains disallowed GPTBot, 250 fully and 62 partially, while 61 explicitly allowed it, 18 fully and 43 partially. Those are big media and tech sites with legal teams. A plumbing company's site almost never has such a rule.

So if your site is missing from ChatGPT, the block is probably not the reason. Your page is not indexed by Google, or it does not say in plain words what you do, where you do it and what it costs, or the words only appear after a script runs. Vercel, a hosting company that sells server rendering, said in its own log study that 0 of the AI crawlers it watched ran JavaScript, while Googlebot did. That is 1 vendor's view of its own network, and the bots may have changed since, but plain text prices are the safe bet.

Being quoted is not a customer

Even a site that gets cited can be cited wrong, so the goal is words a machine cannot mangle.

The Tow Center at Columbia Journalism Review reports it tested 8 AI search tools on 1,600 news queries and found wrong sources, wrong links or made up citations in more than 60% of answers. I read the headline result, not the full report, so treat it as what the Tow Center claims. Either way, a machine can name you and still send the customer to the wrong page. The fix is short sentences about your service, your area and your price that any system has to quote word for word.

What we got wrong

I started this article thinking blocked crawlers were the story, and the sources say otherwise.

I expected to write that contractors were hiding from ChatGPT without knowing it. The Cloudflare crawl says only about 14% of the top 10,000 domains with a readable robots.txt mention AI bots at all, and small sites are far less likely to. Google says there is nothing extra to do for its answer boxes. The real gate is being indexed, same as always.

I also overstated invisible, since OpenAI says an opted out site can still appear as a navigation link, and the user triggered fetchers may ignore robots.txt. None of OpenAI, Anthropic or Perplexity publish how they rank pages, so anyone selling you a ranking secret is guessing. What stays human is the choice of whether your pages train models at all, and the honest sentence about what you charge.

Check your site yourself

You can run the whole check with a look at robots.txt and one pasted prompt.

Open your site's address with /robots.txt on the end. If a line names OAI-SearchBot with Disallow, that is the block, and OpenAI says a fix takes about 24 hours per its crawler docs. Then paste this into ChatGPT or Claude, with your own address in place of the example: "Read the website example.com. Tell me in plain words what this business does, where it works and what it charges. Then list anything a customer would need that you could not find." Whatever it could not find is what you write next, in plain text, on the page.

Do this yourself, it costs nothing but your time. If you would rather not, VuseDesk offers a website and SEO install, with the Contractor plan at $39.99 a month. Either way the rule is the same, say what you do in words a machine can quote.