How to own the internet: the five gates

We call the path from a crawler to a phone call the five gates. A page has to pass access, render, entity, substance and handoff, in that order. Miss one gate and the rest do not matter. Here is what each gate is and what the sources say.

Let the robots in

A crawler that cannot fetch your page cannot quote it.

The first gate is a plain text file called robots.txt. It tells each crawler whether it may read your site. Google documents 7 crawler names, such as Googlebot and Googlebot-Image, plus 1 control token called Google-Extended, on its common crawlers page. Google says on that same page that blocking Google-Extended only affects Gemini training and grounding and has 0 effect on whether you show in Search.

Anthropic says it runs 3 user agents, ClaudeBot for training, Claude-User for when a person asks Claude a question, and Claude-SearchBot for search quality, according to the Claude Help Center article on crawling. Anthropic also says all 3 follow robots.txt. OpenAI's crawler page would not load for us, so we can only report that OpenAI has described 3 agents of its own, OAI-SearchBot, ChatGPT-User and GPTBot. Check your robots.txt for each name first.

Show the page without scripts

If your words only appear after JavaScript runs, some bots see a blank page.

Google processes JavaScript pages in 3 phases, crawl, render and index, according to the Google Search Central page on JavaScript SEO basics. That page says a page can sit in the render queue for a few seconds but it can take longer. Google's own advice on that page is that server side rendering or pre-rendering is still a great idea because, in its words, not all bots can run JavaScript. Google also says Googlebot cannot reliably read pages that change content based on the hash part of the URL, which is 1 documented failure mode on that page.

Vercel, a company that sells hosting, claims its log study found that GPTBot, ClaudeBot and PerplexityBot fetched JavaScript files but did not run them, while Googlebot did. Treat that as a vendor claim, since the official OpenAI and Anthropic pages do not say either way. The test is simple. Turn off JavaScript in your browser and load your own home page, and if your phone number and your services vanish, you fail this gate.

Say who you are

Machines need your name, phone and address in a form they can read.

Google recommends 8 properties on Organization structured data, which are name, url, logo, telephone, address, sameAs, email and contactPoint, per the Google Search Central page on Organization markup. That page says the logo should be at least 112 by 112 pixels and the telephone should carry the country and area code. Google also says on that page to use the same name in the markup that you use as your site name. For address, Google says the more properties you give, the higher the quality of the result.

This is the entity gate. It is where most service sites fail. Your name, phone and address must match on the page, in the markup and on your Google Business Profile. One extra suite number in one place and the machine sees two businesses.

Put real numbers on the page

Pages with numbers, quotes and named sources get picked more.

A peer reviewed paper called GEO: Generative Engine Optimization, published at KDD by Aggarwal and co-authors, found that content edits can raise a page's visibility in generative engine answers by up to 40%. The paper's full tables, which we did not open for this article, are said to show that adding statistics, quotes and citations were the strongest edits and that keyword stuffing slightly hurt. We report that only as what the paper claims.

The long tail still matters. Wikipedia, YouTube and Reddit together made up only 15% of sources in AI summaries and 17% in standard results, in the Pew Research Center study of 68,879 searches. That leaves 85% of AI summary citations for ordinary sites. A plumbing site with real prices and named sources can be one of them.

Make the phone ring

The few people who arrive need one tap to call you.

Pew found that only 1% of visits with an AI summary included a click on a link inside that summary, in the same 900 adult panel. Pew also found people ended their browsing session 26% of the time after a page with an AI summary, versus 16% without one. So the handoff gate has to do more work than any other gate. Every visitor who does arrive should see a tappable phone number and a booking button before they scroll.

A tappable number uses the tel: link. The IETF standard for it is RFC 3966, which we did not open for this article, and it is said to require the global form to start with a plus sign and the country code. Put that link at the top of every page. Then answer the phone when it rings.

The five gates and what the primary sources say

Basis for the table. Google numbers come from Google Search Central developer pages. Anthropic numbers come from the Claude Help Center and are the vendor's own statement. The 40% figure is from the GEO paper abstract by Aggarwal and co-authors at KDD. Pew figures are from a Pew Research Center panel of 900 US adults and 68,879 Google searches.

What we got wrong

The five gates are the basics done right, not a new trade.

I used to pitch the five gates as a new system for AI search. Google's own page on AI features, which we did not open for this piece, is reported to say there are 0 special requirements or markup to show up in AI Overviews beyond normal Search eligibility. If that holds, the five gates are just the basics done properly.

I also leaned too hard on the render gate. Google itself renders JavaScript, per its JavaScript SEO basics page, so this gate only bites with other fetchers, and the only evidence they skip scripts is Vercel's own log data. The 40% GEO figure was measured on a test engine in a research setting, not on ChatGPT or Google as your customers use them. Pew's 8% and 1% click numbers tell me being quoted is not the same as getting a call.

Two things stay human. Someone has to decide which numbers are true enough to publish under a real name. And someone has to answer the phone when the tel link rings.

If you would rather not check five gates yourself, we will do it. The VuseDesk consult is $199 and it starts with an audit of your site against every gate. Book the audit and we send you the list of what fails and what to fix first.