AI Crawler Access: A Technical Checklist for ChatGPT, Perplexity, Google AI, and Copilot

A company can publish useful product pages, customer proof, and original data and still leave them behind a technical barrier. The page may look normal in a browser while a crawler receives a block, an error, a duplicate URL, or almost no usable text.

This guide covers public website access for ChatGPT Search, Perplexity, Google AI Overviews and AI Mode, and Microsoft Copilot and Bing. None of these checks guarantees indexing, citation, recommendation, or ranking.

Start with the pages that carry the answer#

Do not begin by grading every URL on the website. Select the pages that carry the information people or AI systems would need for the outcome you care about.

For a B2B SaaS or ecommerce company, that usually includes:

• the homepage and About page;
• core product, service, and category pages;
• pricing, packaging, availability, and policy pages;
• customer stories, reviews, and other proof;
• comparison, use-case, and objection pages;
• original data, expert explanations, or reference material.

The same principle applies to other goals. A hiring use case may depend on careers and culture pages. Customer support may depend on documentation and help content. Technical eligibility should be checked against the intended outcome, not a generic sitewide score.

Know which control applies to which platform#

The platforms do not publish one universal AI crawler or one special file that makes a site eligible everywhere.

Training controls are a separate decision. OpenAI documents GPTBot separately from OAI-SearchBot. Google says Google-Extended does not affect inclusion or ranking in Google Search. Do not open or close a training crawler and assume that the setting controls every search or answer surface from the same company.

Technical checklist#

Run the checks below on the selected pages. Save the URL, date, result, and evidence for each failure. A green homepage does not prove that a blocked pricing or product page is accessible.

1. Confirm that the public URL returns usable HTML#

The canonical page should return a successful response without requiring a login, CAPTCHA, cookie interaction, or geographic exception.

Check:

• the final URL returns HTTP 200;
• redirects terminate at the intended canonical URL;
• the response is the real page, not a soft error or challenge page;
• the title and main content are present;
• the same public URL works without an authenticated browser session.

A simple header check can reveal an obvious failure:

curl -I https://example.com/product/

Test the response with the relevant user-agent as an initial diagnostic:

curl -A "OAI-SearchBot" -L -I https://example.com/product/
curl -A "PerplexityBot" -L -I https://example.com/product/
curl -A "Googlebot" -L -I https://example.com/product/
curl -A "bingbot" -L -I https://example.com/product/

A user-agent string can be spoofed. These requests show how the public server responds to the label; they do not prove that a request came from the real platform. Use official IP-range or crawler-verification guidance and server logs when identity matters.

2. Check robots.txt and infrastructure together#

Robots.txt is only one access layer. A correct file can coexist with a CDN, WAF, rate limit, JavaScript challenge, CAPTCHA, login wall, or geographic rule that blocks the same request later.

An illustrative robots file may contain:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Sitemap: https://example.com/sitemap.xml

This is not a required template. A site with User-agent: * and Allow: / may already permit these crawlers. Path-specific rules, inherited groups, and security infrastructure must be checked in the site's actual configuration.

For each important page, verify:

• no robots rule blocks the relevant path;
• the CDN or WAF does not return 403, 429, or a challenge page;
• rate limits do not make repeated legitimate requests fail;
• authentication, age, location, or consent gates do not hide the main content;
• legitimate crawler traffic is not rejected solely because it lacks browser behavior.

3. Separate crawling from indexing controls#

Robots.txt controls crawling. It is not the correct tool for every indexing decision.

Inspect both HTML and response headers for:

<meta name="robots" content="noindex">

and:

X-Robots-Tag: noindex

Google notes that a crawler must be able to access a page to read its noindex directive. A URL blocked in robots.txt may still be known from other pages. OpenAI similarly says a disallowed URL may still surface as a title and link in ChatGPT Atlas when it is learned through another source. If exclusion is the goal, follow the platform's published indexing control rather than assuming robots blocking makes the URL unknown.

For pages intended to appear:

• remove accidental noindex;
• confirm the X-Robots-Tag header is not set by the CDN or framework;
• check that a staging or preview rule has not reached production;
• verify that the canonical page, not a duplicate, is indexable.

4. Make canonical signals agree#

A page can return 200 and still point search systems elsewhere.

Check that:

• the HTML canonical points to the preferred production URL;
• internal links use that same URL;
• the Sitemap includes the canonical URL;
• redirects, http/https, www/apex, trailing slashes, and URL parameters do not create conflicting versions;
• localized or mobile variants are connected correctly where relevant.

If the product page canonicals to the homepage, the page is not presenting itself as the preferred source for its own information.

5. Put the important answer in accessible text#

Google explicitly recommends making important content available in textual form for its AI features. Other platforms also need retrievable page content, even though their rendering systems and limits are not fully public.

Compare the initial HTML with the rendered page:

curl -L https://example.com/product/ > page.html

Then check whether page.html contains the core product name, description, features, pricing facts, and proof visible to the user.

Look for:

• essential facts loaded only after a user action;
• text rendered inside images without a textual equivalent;
• content available only after client-side API calls fail;
• tabs or accordions whose content never appears in the HTML or accessible tree;
• inconsistent facts between desktop, mobile, text, image, and video;
• buttons, forms, and menus without useful labels or states.

OpenAI's current publisher guidance also says ChatGPT Atlas uses ARIA labels and roles to interpret interactive elements. Accessibility work can therefore improve both human use and an agent's ability to understand what a control does. It is not a citation guarantee.

6. Keep discovery signals clean#

Sitemaps and internal links help search systems find preferred URLs. They do not force indexing.

For the XML Sitemap:

• include fully qualified canonical URLs;
• exclude redirects, errors, duplicates, and intentionally noindex pages;
• use an accurate <lastmod> value only for meaningful changes;
• place the Sitemap at a scope that covers the intended URLs;
• declare it in robots.txt and submit it in the relevant webmaster tool.

Google says it ignores Sitemap <priority> and <changefreq> values. It may use <lastmod> when the date is consistently accurate. Sitemap submission is a hint, not an indexing promise.

For Microsoft surfaces, IndexNow can notify participating search engines when a URL is added, changed, or deleted. A successful 200 response means the submission was received, not that the URL was indexed or cited. Automate IndexNow around real changes rather than submitting unchanged URLs repeatedly.

Also make sure important pages are linked from relevant navigation, category, documentation, or editorial pages. An orphan URL in a Sitemap is weaker discovery architecture than a page connected to the rest of the site.

7. Use structured data for a documented job#

Schema markup labels public facts such as organizations, products, offers, reviews, authors, and articles. Google can use supported markup to understand a page and make it eligible for particular rich results.

Check that:

• the Schema type is appropriate for the visible page;
• names, prices, availability, ratings, dates, and authors match visible text;
• the preferred canonical URL appears consistently;
• required and recommended properties for the targeted Google feature are present;
• the markup passes Google's Rich Results Test where applicable;
• the JSON-LD is valid and does not contain hidden claims.

Google explicitly says no special Schema.org markup or new AI text file is required for AI Overviews or AI Mode. OpenAI and Perplexity do not publish a special schema file that guarantees inclusion. Treat structured data as machine-readable accuracy and supported search implementation, not a shortcut to an AI recommendation.

For the investment decision behind Schema, see Does Schema Markup Improve AI Visibility?.

8. Verify in the tools each platform actually provides#

The output should be evidence, not a single technical score. Save the page, platform, date, status, observed content, and unresolved limitation.

What to do with the result#

Use the checklist to make one of three decisions.

A technical failure is confirmed#

Fix the smallest failure that blocks the intended page. Record the before and after response, then use the relevant platform tool or logs to confirm that the new version can be fetched. Allow time for recrawling. Do not promise a citation as the outcome of the fix.

Public access works, but the answer is missing#

Stop treating crawlability as the default explanation. Check whether the page answers the actual question, presents current and supportable facts, and carries enough customer, product, or third-party evidence to deserve use.

The platform provides too little evidence to decide#

Mark the result unknown. Preserve the page, date, query, platform, and response, then design a bounded check instead of converting uncertainty into a technical project.