A blanket block on AI crawlers can remove a B2B SaaS company from some AI search results at the moment a buyer compares vendors. OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers, although they may still appear as navigational links. A founder can protect model training preferences without making that search decision: OpenAI treats GPTBot and OAI-SearchBot as separate controls.
One bot policy can serve opposing goals
GPTBot is OpenAI's crawler for content that may contribute to foundation model training. OAI-SearchBot is its crawler for ChatGPT search results. OpenAI documents independent robots.txt settings for the two, so a training restriction does not require a search restriction.
ChatGPT-User serves certain requests initiated by ChatGPT users. OpenAI says it does not use that agent for automatic web crawling or to determine search inclusion. Marketing teams should therefore evaluate training access, search discovery, and user-requested retrieval as distinct decisions.
Perplexity documents separate crawlers and user agents for its services. CCBot belongs to Common Crawl, an independent web data repository; it isn't a Perplexity or OpenAI crawler. Blocking Common Crawl may limit access to its corpus, but it does not directly set a brand's inclusion in ChatGPT search or Perplexity results.
The commercial cost of missing source material
AI share of voice is the proportion of a defined set of AI answers that names or cites a brand. Define the denominator before reporting a percentage: a fixed panel of buyer prompts, repeated in specified products and markets, produces a measure a team can compare over time. Count brand mentions and source citations separately.
Consider a buyer asking ChatGPT to compare two customer support platforms for a regulated company. If one vendor's security documentation is accessible and the other's is blocked, the available source has a better chance of supporting the answer. The model can still mention the blocked vendor through other sources, so crawler access affects an opportunity rather than guaranteeing the outcome.
That distinction matters in a category with long sales cycles. A prospect may encounter a competitor in an AI answer, visit its comparison page, and place it on a shortlist before any visit reaches the absent brand's site. A drop in direct or organic sessions alone cannot establish that sequence; prompt observations, referral data, and CRM records can test it.
Set a public boundary the company can defend
A SaaS company should identify the material it wants buyers and agents to use: product capabilities, current pricing terms, integration limits, and security claims that have passed review. Keep customer records, private support tickets, unreleased features, and authenticated documents outside the public crawl surface. Publishing a concise account of approved facts creates a source the company can maintain as the product changes.
robots.txt is a crawler access signal, not an access control system. Public URLs may reach buyers or automated clients through other routes, while credentials and server permissions enforce private boundaries. Legal and security teams should review exposure at the document level before marketing changes bot rules.
The choice also has a timing cost. OpenAI says a change to its search crawler settings can take about 24 hours to affect its systems. A company that reverses a broad block should verify access and then watch results over repeated prompt runs rather than treat the first answer as a permanent verdict.
Give agents a maintained route to product facts
llms.txt is a proposed Markdown index that gives agents brief site context and links to selected documents. For a marketing team, its value lies in choosing which public pages explain the product, its limits, and its current position. The file can point an agent toward approved evidence when that agent elects to read it.
Its authority has a hard limit. The llms.txt proposal does not grant crawl permission or require any provider to fetch the file. Common Crawl's 2026 analysis notes that many published files receive no observed requests, so a brand should treat the index as an additional access path rather than a distribution channel with assured reach.
Google states that its Search systems, including generative AI features, do not use llms.txt as a special visibility signal. Keep the indexed HTML pages useful to buyers, maintain accurate source documents, and apply ordinary Search guidance. A Markdown index cannot repair weak evidence on the destination pages.
Assign ownership to product changes
Marketing can own the claims in the index, while product and security owners approve the linked facts. A pricing change should update the pricing source; a retired integration should leave the approved source list. Otherwise, an agent that follows the index can retrieve a cleanly formatted error.
GEOall's scanner checks whether the files exist and whether their structure is valid. That check establishes publication readiness for clients that use them. The business test starts after publication: inspect server requests to the index and linked pages, then compare the retrieved claims with the answers buyers receive.
Measure the competitor gap
Build a prompt panel from sales calls, search terms, and lost-deal notes. Include category selection prompts and direct brand comparisons; record the answer date, product, market, cited URLs, brand mentions, and competitors named. Preserve the prompt wording during each measurement period so a change in the panel doesn't masquerade as a change in visibility.
For example, if a brand appears in 18 of 100 observed category answers, its observed mention share is 18% for that panel. If 7 of those answers cite its own domain, its owned-source citation rate is 7% across the same 100 answers. Neither figure estimates every buyer query; both give the team a defined baseline for subsequent tests.
Segment losses by cause. A blocked page calls for an access decision; an outdated comparison page calls for a content correction. An answer that cites a third-party review instead of the vendor's current documentation gives the team a specific source gap to investigate.
Compare those observations with qualified AI referral sessions and opportunities in the CRM, using tagged landing pages where possible. Citation share can rise without producing pipeline, while a small set of high-intent prompts can produce valuable leads. Keep both measures visible when deciding whether to expand crawler access or revise the public product sources.
Discussion
No comments yet — start the discussion.