Guide
Let AI crawlers read your site
If an AI crawler cannot read your pages, engines have little to quote. Here is how to check, and what to fix.
Know who is visiting
Each AI company uses its own crawlers and rules. Examples include OpenAI's GPTBot, Perplexity's PerplexityBot and Anthropic's ClaudeBot. Google uses a robots.txt control called Google-Extended, and Microsoft Copilot draws on Bing's index. Check each company's documentation for the current names and what each controls.
Read your robots.txt
Open yourdomain.com/robots.txt. Look for rules that block these crawlers, either by name or with a broad rule that blocks everything. Decide on purpose which ones you want to allow.
Check your firewall and CDN
Some firewalls and CDNs block unknown bots by default. Check your bot-protection settings and server logs to see whether AI crawlers are being served or blocked.
Make pages readable
Some crawlers do not run scripts. If your key facts only appear after a script runs, put them in the page itself. Keep clean URLs and an up-to-date sitemap.
A note on llms.txt
Some sites publish an llms.txt file to describe their content to AI tools. It is optional, support varies and it does not guarantee anything. Do the basics first.
Where to go next
If you would rather have this done for you, our audit tests hundreds of your buyers' questions across six engines and ends with a ranked plan.