AI SEO checker

Enter a page's URL. The checker fetches it the way a crawler does, without running JavaScript, and reports whether AI crawlers may read it, whether the site has an llms.txt and a sitemap, and how the page's titles, canonical, structured data and Open Graph tags hold up.

Checks one page plus its site's robots.txt, llms.txt and sitemap.

What it checks

AI crawlers. Your robots.txt is matched against each crawler for the exact path you entered, using the same rules crawlers follow: the most specific user-agent group, then the longest matching rule.

  • OAI-SearchBot: OpenAI, fetches pages for search and cited answers
  • GPTBot: OpenAI, collects content for model training
  • ClaudeBot: Anthropic, collects content for model training
  • PerplexityBot: Perplexity, fetches pages for search and cited answers
  • Google-Extended: Google, collects content for model training
  • CCBot: Common Crawl, collects content for model training

Site files. Whether /robots.txt, /llms.txt and a sitemap exist and look like what they claim to be, rather than an HTML page from a catch-all route.

The page. HTTPS, title length (about 15 to 60 characters), meta description length (about 50 to 160), exactly one <h1>, one absolute canonical URL, no noindex in the robots meta tag or X-Robots-Tag header, JSON-LD blocks that parse and declare a type, Open Graph title, description and image, how long the HTML took to arrive and how large it is.

Search crawlers and training crawlers

The crawlers are not all doing the same job. A search crawler fetches pages so an assistant can show and link to them in an answer. A training crawler collects text for future models. Blocking a search crawler takes you out of that product's answers; blocking a training crawler does not. So the checker fails a blocked search crawler and only warns about a blocked training crawler, because that can be a deliberate choice.

A robots.txt that allows AI search but opts out of training:

# Let AI search crawlers in, keep training crawlers out
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

After the check

Fix the failures first: a blocked search crawler, a noindex left over from staging, a missing title or H1, or JSON-LD that does not parse. Then work through the warnings. If the site has no llms.txt, the llms.txt generator will write a first draft from your sitemap.

Questions

What is AI SEO?
Making a page easy for AI search tools such as ChatGPT search, Perplexity, Claude and Google's AI features to fetch, understand and cite. Most of it is ordinary technical SEO: a crawlable page with a clear title, one topic per page, structured data and a canonical URL. The new part is checking that the AI crawlers themselves are not blocked.
Should I block GPTBot and the other AI crawlers?
Split them by job. Search crawlers such as OAI-SearchBot and PerplexityBot fetch pages so they can be shown and cited in answers; blocking them keeps you out of those answers. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content for model training; blocking them is a policy choice with no effect on classic search. That is why the checker fails a blocked search crawler and only warns about a blocked training crawler.
The checker says there is no H1, but I can see one. Why?
The checker reads the HTML your server sends and does not run JavaScript. If the heading only appears after scripts run, it is not in that HTML. Several AI crawlers fetch pages the same way, so content that needs JavaScript to appear may be invisible to them. Server rendering or static generation fixes it.
Does having an llms.txt affect AI visibility?
There is no public evidence that any major AI search product ranks or cites pages based on llms.txt. It costs little to publish and some tools read it, so the checker lists it as a warning, never a failure.
What does the checker not cover?
It checks one URL. It does not render JavaScript, measure Core Web Vitals, crawl the rest of the site or look at backlinks. Response time is measured from this server, so it includes the network distance between here and your host.
Do you store the URLs I check?
No. The server fetches the page, robots.txt, llms.txt and sitemap, runs the checks and returns the result. Nothing is saved. Requests identify themselves with a fabwaseem-tools user agent, and each visitor can run 10 checks in 10 minutes.