# How to Make Your Website Searchable on Google and AI Search **URL:** https://mettevo.com/blog/article/how-to-make-your-website-searchable-on-google **Published:** 2024-05-07 **Updated:** 2026-11-11 **Author:** Oleg Silin **Category:** Blog | Mettevo, web development > Searchable on Google is half the job. Run 12 checkpoints for Google and AI engines such as ChatGPT and Perplexity, with pass and fail results from our own site. ![website searchable](https://stage.mettevo.com/wp-content/uploads/2024/05/1-How-to-make-your-website-searchable-on-Google.jpg) --- How to make your website searchable on Google: let Googlebot fetch every page, return a success status, keep noindex off, name one canonical address and publish a sitemap. That is half the job. The other half is access for artificial intelligence (AI) engines like ChatGPT, Perplexity and Gemini: clear bot rules, consistent entity data and quotable pages. A searchable website is a site whose pages a search system can fetch, read and keep in an index that it queries to answer people. Google says a page must be indexed to appear in its AI features, while ChatGPT, Perplexity and Claude send their own crawlers. mettevo.com is our testbed for both halves, and every number from it below names its date and source. ## **Key Takeaways** ·       Google lists three technical requirements for indexing: Googlebot is not blocked, the page returns Hypertext Transfer Protocol (HTTP) status 200 and the page has indexable content (Google Search Central). ·       ChatGPT search answers do not show sites that opt out of OAI-SearchBot, while disallowing GPTBot only signals that content should not train OpenAI's models (OpenAI documentation). ·       llms.txt is optional: Google Search ignores it, and a server-log study published June 15, 2026 found that 97% of the llms.txt files it examined received zero traffic in May 2026. ·       On mettevo.com, a September 28, 2026 crawl of 60 addresses found 58 pages answering 200 and none of the 12 AI bots blocked; index status and server answers to bots were not measured. ## **How do I make my website searchable on Google?** Google's technical requirements for Search are short: Googlebot is not blocked, the page works, meaning Google receives an HTTP 200 status code, and the page has indexable content. Searches for this exact phrase today return advice centred on Google: check indexing, allow indexing, submit to Search Console, build basic signals. None of it concerns AI crawlers, so I treat Google's answer as half the job. Robots.txt and noindex do different jobs. A Disallow rule stops crawling, but Google's robots.txt documentation says a disallowed page can still be indexed if other sites link to it. To keep a page out, I use noindex and leave the page crawlable, because Googlebot cannot read a tag on a page it may not fetch. I also list only canonical addresses that answer 200 in the sitemap, so sitemap, canonical tags and status codes agree. Eligibility is not indexing, so I verify. Google Search Console, Google's free tool for indexing and search reports, has the Uniform Resource Locator (URL) Inspection tool for one page and the Page indexing report for all pages. In my view, "Crawled - currently not indexed" on an important page is a content verdict, not a plumbing fault: the fetch worked, so I judge what the page adds before I touch robots.txt. On mettevo.com, robots.txt declares one sitemap and blocks general crawlers only from utility paths such as /wp-admin/, /search/ and tracking-parameter addresses. The sitemap lists about 449 addresses. In our September 28, 2026 crawl of 60 addresses, 58 pages answered 200, two answered 308, none lacked a canonical tag and the crawler marked all 58 indexable. ## **How do I make my website searchable in ChatGPT, Perplexity and Gemini?** AI engines are several doors, not one. AI Overviews and AI Mode need an indexed page, so the Google checkpoints still apply. Google describes Google-Extended as a robots.txt token for Gemini training and grounding that does not affect inclusion in Google Search. Earning citations from AI engines is generative engine optimization (GEO), and its first layer is plain access. ### **Which of the 12 AI bots should robots.txt allow?** My rule: search is distribution, training is licensing. The table lists the 12 agents we track and the job each operator documents. I would allow the five search and user agents (OAI-SearchBot, PerplexityBot, ChatGPT-User, Perplexity-User, Claude-User) on any commercial site; the other seven are a licensing call. **Agent** **Operator** **Documented job** **OAI-SearchBot** OpenAI Surfaces sites in ChatGPT search **ChatGPT-User** OpenAI Visits a page when a user asks **GPTBot** OpenAI Improves generative AI foundation models **PerplexityBot** Perplexity Surfaces and links sites in Perplexity search **Perplexity-User** Perplexity Visits a page when a user asks **ClaudeBot** Anthropic Collects content that may train models **Claude-User** Anthropic Retrieves a page for a user's query **Google-Extended** Google Opt-out token for Gemini training and grounding **Applebot-Extended** Apple Opt-out token for Apple model training **Bytespider** ByteDance No operator documentation found **CCBot** Common Crawl Builds an open archive of crawl data **meta-externalagent** Meta Crawls for AI training or product indexing OpenAI says robots.txt rules may not apply to ChatGPT-User because users trigger its visits, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. So a Disallow aimed at a user-triggered agent is a weak control. OpenAI also says its systems can take about 24 hours to adjust after a robots.txt update for OAI-SearchBot. Two search agents sit outside our 12: Anthropic's Claude-SearchBot and Meta's meta-webindexer. One group can serve every agent you allow on purpose: User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User User-agent: Perplexity-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: meta-webindexer Allow: / Disallow: /wp-admin/ Sitemap: https://example.com/sitemap.xml ### **Can those bots fetch your pages?** Robots.txt states a rule; the server's answer is a fact. A firewall can refuse an agent that robots.txt welcomes. Request your home page once per agent, using the agent's name as the user agent string, and compare each status with a browser-like request. Google says Google-Extended has no separate user agent string, and Apple says Applebot-Extended does not crawl webpages, so the loop skips both. A laptop tests user-agent rules only, not network-address checks: url=https://example.com/ printf 'browser '; curl -s -o /dev/null -w '%{http\_code}\\n' -A 'Mozilla/5.0' "$url" for ua in OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User Claude-User ClaudeBot GPTBot CCBot Bytespider meta-externalagent; do   printf '%s ' "$ua"; curl -s -o /dev/null -w '%{http\_code}\\n' -A "$ua" "$url" done Then run curl -s https://example.com/ | grep -c "Your main heading"; a result of 0 means the heading is not in the initial HTML as a literal substring - JavaScript may be adding it, a redirect may be intercepting the fetch, or the heading may be broken across tags. Inspect the raw body to confirm. ### **What should llms.txt contain?** llms.txt is a Markdown file served at /llms.txt. In the llmstxt.org proposal by Jeremy Howard, the only required element is a top-level heading with the site name; a blockquote summary and sections of links follow. In Common Crawl's July 2026 archive, only 49.90% of 584,107 llms.txt files had all three parts, and 22.56% had no links at all. The bar I hold ours to is a valid file with real links. Our llms.txt links a Markdown twin beside many pages; /seo/md, opened on November 10, 2026, is Markdown with a title, source address and update date. ### **Which structured data and entity facts do AI engines need?** AI engines need no special markup, and I would not sell it as a lever: Google's generative AI guidance says structured data is not required for generative AI search. We still publish Organization and Person markup. We add FAQPage markup for frequently asked questions (FAQ) only where the visible page holds those questions and answers, because markup states facts a crawler would otherwise guess. An entity, in search, is a company or person that machines identify by a consistent name, description and website. The posture I take is one description sentence, copied everywhere: home page title, About page, Organization markup, llms.txt and profiles such as LinkedIn. On mettevo.com, the home page title, About page and llms.txt use three labels: "Full-Service digital marketing agency", "your trusted digital agency partner" and "a digital agency delivering custom web development". Each is defensible; together they show drift without an error. ## **What are the 12 checkpoints for Google and AI engines?** The top half of the table affects whether Google can index you. The bottom half affects whether AI engines can reach and identify you. **#** **Checkpoint** **Pass when** **1** Googlebot not blocked No Disallow matches a page meant to rank **2** Status 200 Indexable pages answer 200; redirects end on a 200 page **3** No noindex No noindex in a robots meta tag or X-Robots-Tag header **4** Canonical Each page names one preferred address, normally its own **5** Sitemap Lists only canonical 200 addresses; robots.txt declares it **6** Indexed, not just eligible Search Console shows key pages as Indexed **7** Bot rules Search and user agents allowed on purpose; training agents decided **8** Bot responses Each agent gets the browser's status and the main text **9** llms.txt Valid Markdown at /llms.txt listing your best pages **10** Organization markup Organization, plus Person for authors, on every template **11** FAQPage markup Only where the visible page holds the questions and answers **12** Entity consistency Name, description and website agree across site and profiles ## **What does mettevo.com pass, and what does it still fail?** mettevo.com is the testbed, so it gets the same checklist. The September 28, 2026 crawl fetched 60 addresses with JavaScript rendering off, the home page twice, about 13% of the roughly 449 addresses the sitemap lists. I read robots.txt, llms.txt and sitemap.xml on November 10, 2026. We did not pull the Page indexing report or read server logs, so percentages describe the sample only. ·       Pass on our sample, 7 of 12: checkpoints 1 to 5, 7 and 9. Checkpoint 5 pass is scoped to the 60 URLs the crawler visited, not all ~449 in the sitemap. Checkpoint 2 counted two 308 responses as non-blocking; their destination status was not re-fetched. For checkpoint 7, robots.txt names 16 AI agents with Allow: /, including 11 of the 12, and none of the 12 is blocked (Claude-User is default-allowed, not explicitly named). For checkpoint 9, llms.txt is present. ·       Partial, 3 of 12: checkpoint 10 has Organization markup on 57 of 58 pages, and the blog index lacks it. Checkpoint 11 has FAQPage on 15 of 54 articles; we did not check whether the other 39 hold question blocks. Checkpoint 12 shows three company labels. ·       Not measured, 2 of 12: checkpoint 6, because we did not pull the Page indexing report, and checkpoint 8, because we probed rules, not server answers to bot user agents. The unmeasured rows are the ones a crawl cannot answer: indexing is Google's decision, and a bot's treatment is the server's. The Anthropic gap shows how bot lists age: our robots.txt names ClaudeBot, Claude-Web and anthropic-ai but not Claude-User or Claude-SearchBot, which Anthropic documents. Access stays open through the default group. Naming agents is cheap; keeping names current is the work. One page-level movement, with its limits. In Search Console (domain property mettevo.com, Pages tab, Web search type), our explainer on invalid web addresses went from 16 to 23 clicks, up 44%. The windows are September 15 to 21 and September 22 to 28, 2026. The page opens with a definition and carries Person and Organization markup. Seven extra clicks in one week is a small difference, and we did not isolate other changes, so we keep testing. ## **Why isn't llms.txt the shortcut vendors sell?** My position on llms.txt: publish it, but do not sell it. The llmstxt.org page dates the proposal September 3, 2024 and still calls it open for community input, and a proposal binds no engine to read the file. Two facts anchor that position. Google's generative AI guidance says Google Search itself does not use such files and that creating one will neither harm nor help visibility or rankings in Google Search. A server-log study published June 15, 2026 covered 137K domains using one search software vendor's analytics: 28% publish an llms.txt file, and 97% of those files received zero traffic in May 2026. The vendor notes its customers skew technical and that a fetch does not mean a read. A vendor selling llms.txt as a citation lever owes you two things: server-log lines showing an OpenAI, Perplexity or Anthropic agent fetching the file, and a citation that followed. You can check the first yourself. This command, for Nginx or Apache logs in the combined format, counts requests to /llms.txt by user agent: grep 'GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn I would publish the file anyway, because Google says it neither harms nor helps rankings and it is cheap to maintain. We cannot say which agents fetch ours, because we have not read our server logs. Build steps are in our llms.txt implementation guide. ## **Where should you start?** I would start with checkpoint 6, because Search Console shows which other checkpoints matter for your site. Checkpoints 7 and 8 come next, since they cost little and decide whether ChatGPT and Perplexity can reach you. To test whether engines then name you, run our AI visibility audit. To hand the work over, see our search engine optimization (SEO) services and what an AI SEO retainer includes. ## **FAQ** ### **How do I get my website to show up on Google?** Let Googlebot fetch your pages, make each return HTTP status 200, remove noindex, add a canonical tag and a sitemap, then confirm indexing in Google Search Console. Google says meeting its three technical requirements makes a page eligible, and that indexing is not guaranteed. ### **What is the Google 20% rule?** The Google 20% rule is not a search or ranking rule. It usually means Google's 20 percent time policy, which the founders' 2004 letter describes as employees spending effectively one day per week on projects they think will benefit Google. It does not affect indexing or ranking. ### **Can I take a picture of something and find it on Google?** Yes. On google.com, click the Search by image icon, then upload a file, drag an image into the search box or paste an image link. In Chrome, right-click an image on a page and choose Search with Google Lens. Google Search Help documents these options. ### **How to search within an entire website?** Use Google's site: operator. Type site: followed by the domain and your keywords, with no space after the colon, for example site:nytimes.com recipes. Google Search Help documents this syntax. The operator returns only pages Google has indexed from that domain, so it also works as a quick check of your own site. ### **How to use Google to search within a website?** Go to google.com and type site:yourdomain.com to list pages Google has indexed from that domain. Add a keyword after a space to find one topic, as in site:yourdomain.com pricing. If an expected page is missing, open the URL Inspection tool in Google Search Console to see whether the page is indexed and why not. ### **Can ChatGPT do web search?** Yes. OpenAI documents ChatGPT's search features and a dedicated crawler, OAI-SearchBot, that surfaces websites in them. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. A separate agent, ChatGPT-User, visits pages when a user asks, and GPTBot only signals whether content may train OpenAI's models.