WebDesy

Free LLMS.txt Checker for AI Crawlers

BLUF: Check whether ChatGPT, Claude, Perplexity, and other AI crawlers can access your homepage, robots.txt, llms.txt, optional llms-full.txt, and the important URLs listed inside the file.

Check if AI crawlers can access your site. Paste a domain below to test live crawler access and quickly spot whether the issue is in robots.txt, llms.txt, llms-full.txt, linked URLs, or edge rules such as Cloudflare.

AI crawler diagnostics

AI Crawler & llms.txt Checker

Paste a domain to check whether llms.txt, llms-full.txt, robots.txt, linked resources, and major AI crawler user agents are accessible.

Looking for more utilities like this? Visit the WebDesy SEO Tools page.

What the Checker Reviews

  • Whether /llms.txt returns 200 and is served as plain text.
  • Whether /robots.txt includes the right Llms-txt: discovery line.
  • Whether tested AI user agents, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, and PerplexityBot, can reach the homepage and /llms.txt.
  • Whether /llms-full.txt is healthy when it exists or is referenced.
  • Whether the public links inside llms.txt return successful responses.

Example Results

Healthy setup

  • /robots.txt returns 200 and points to /llms.txt.
  • /llms.txt returns 200 as plain text.
  • Major AI crawler user agents can reach the homepage and /llms.txt.
  • Important URLs listed inside llms.txt return successful responses.

Needs attention

  • A normal browser gets 200, but AI crawler user agents get 403.
  • Cloudflare or another edge layer blocks crawler requests.
  • /llms-full.txt is referenced but returns 404.
  • URLs listed in llms.txt redirect, 404, or time out.

What to Do If It Fails

  • If AI crawler user agents get 403, check Cloudflare AI Crawl Control, WAF rules, bot rules, and user-agent blocks.
  • If robots.txt is missing the discovery line, add the correct Llms-txt: URL and keep private paths blocked.
  • If /llms-full.txt returns 404, publish it as plain text or remove every reference to it.
  • If linked URLs fail, fix the listed URLs, replace them in llms.txt, or redirect them to relevant live pages.

Need help fixing a failed check? Contact WebDesy and send the copied report. I can review the website files, robots policy, Cloudflare AI Crawl Control, WAF rules, and llms.txt setup.

Sample Report

A healthy result should show PASS for homepage access, llms.txt access, plain-text content type, the Llms-txt: discovery line, AI crawler user-agent access, and linked URL checks.

AI crawler and llms.txt check
Result: PASS (100%)
Main checks:
- PASS: llms.txt access
- PASS: robots.txt Llms-txt line
- PASS: AI crawler user-agent access
- PASS: Linked URL checks

Related Guides

If the checker finds a problem, the fix is usually in one of three places: the website files, the robots policy, or Cloudflare bot/security rules. For the full walkthrough, see how I fixed Cloudflare AI crawler blocking and llms.txt issues for a client. If you are checking the robots layer first, start with how to use the robots.txt file with WordPress.

Alternative tool: You can also use HTTP Status to check HTTP status codes.

FAQ

What does the AI Crawler & llms.txt Checker test?

It checks whether the homepage, robots.txt, llms.txt, optional llms-full.txt, AI crawler user agents, and important links inside llms.txt return healthy public responses.

What is llms.txt?

llms.txt is a plain text file that helps AI systems discover important website content, preferred URLs, and crawler-facing context.

Does llms-full.txt have to exist?

No. If llms-full.txt is referenced from robots.txt or llms.txt, it should return 200 as text/plain. If the site does not publish a full file, references to it should be removed.

Why can llms.txt work in a browser but fail for AI crawlers?

A normal browser can receive a 200 response while Cloudflare, a WAF rule, or bot protection blocks AI crawler user agents such as GPTBot, ClaudeBot, OAI-SearchBot, or PerplexityBot.

Sitemap