AI crawlers are programs AI companies use to fetch web pages, such as OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and Perplexity's PerplexityBot.
robots.txt can allow or block each one. Some fetch pages for model training and some fetch pages to answer a user in real time.
Most AI crawlers do not run JavaScript, so your text has to be in the raw HTML.
How Kaiwu measures it
Kaiwu checks robots.txt rules for the major AI crawlers and fetches pages without running scripts to confirm the text is readable. Together these make up the AI crawler access part of the GEO score, worth 15 points.
Start with an audit of your own site
Send us your domain and target markets. A person reviews the report before it reaches you, and every number in it states its source and method.