AI Crawler & llms.txt Checker

BLUF: Paste a domain below to check whether AI crawlers can access the homepage, robots.txt, llms.txt, optional llms-full.txt, and the important links listed inside the file.

AI crawler diagnostics

AI Crawler & llms.txt Checker

Paste a domain to check whether llms.txt, llms-full.txt, robots.txt, linked resources, and major AI crawler user agents are accessible.

What the Checker Reviews

Whether /llms.txt returns 200 and is served as plain text.
Whether /robots.txt includes the right Llms-txt: discovery line.
Whether tested AI user agents, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and PerplexityBot, can reach the homepage and /llms.txt.
Whether /llms-full.txt is healthy when it exists or is referenced.
Whether the public links inside llms.txt return successful responses.

If the checker finds a problem, the fix is usually in one of three places: the website files, the robots policy, or Cloudflare bot/security rules. I documented a real example here: how I fixed Cloudflare AI crawler blocking and llms.txt issues for a client.