Skip to content
The Register

AI Crawler Audit — Are You Blocking ChatGPT?

Check whether your site is visible to AI search or accidentally blocking it. Audits robots.txt for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and 7 more, detects llms.txt, and returns an AI-readiness score with the exact fixes. No key, no login.

AI Crawler Audit — Are You Blocking ChatGPT? returns 14 documented output fields, including domain, aiReadinessScore, aiSearchReadiness, blocksAiSearch and blockedSearchBots. Every field carries a declared type, a written description and an example value. 45 runs are recorded on Apify.

Run it on ApifyPay per result · no subscription
14
Output fields
5
Inputs
45
Runs
2026-10-15
Last run

01 — output

What it returns

Every field the extractor writes to its dataset, with its type, a written description and an example value.

FieldTypeWhat it holdsExample
domainstringnullDomain name, without the scheme.example.com
aiReadinessScoreintegernullHow ready this site is for AI crawlers, from 0 to 100.0
aiSearchReadinessstringnullReadiness for AI search in words.<aiSearchReadiness>
blocksAiSearchbooleannullWhether the site blocks AI search crawlers.true
blockedSearchBotsstringnullSearch crawlers the site blocks.<blockedSearchBots>
blockedBotsstringnullUser agents the site blocks.<blockedBots>
hasLlmsTxtbooleannullWhether the site publishes an llms.txt file.true
llmsTitlestringnullTitle the site publishes in its llms.txt.<llmsTitle>
allowedBotCountintegernullHow many of the checked bots are allowed.0
totalBotsCheckedintegernullHow many bots were checked.0
sitemapCountintegernullHow many sitemaps were found.0
robotsStatusintegernullOutcome of reading robots.txt.0
recommendationsstringnullThe recommended actions for this row.["Declare a canonical URL","Publish a sitemap"]
statusstringnullOutcome recorded for this row.ok

02 — sample

One record

Assembled from the example values declared on the schema above, not invented at render time.

{
  "domain": "example.com",
  "aiReadinessScore": 0,
  "aiSearchReadiness": "<aiSearchReadiness>",
  "blocksAiSearch": true,
  "blockedSearchBots": "<blockedSearchBots>",
  "blockedBots": "<blockedBots>",
  "hasLlmsTxt": true,
  "llmsTitle": "<llmsTitle>"
}

03 — input

What you give it

InputTypeDescription
domainsarrayrequiredDomains or URLs to audit. Scheme, www and path are stripped automatically.
onlyBlockedSitesbooleanoptionalFilter the output down to the sites with a real problem — useful when auditing a long client list.
includeBotDetailbooleanoptionalAdds a botDetail array with the verdict and matched robots.txt rule for every AI crawler checked.
useProxybooleanoptionalRoute requests through Apify Proxy. Rarely needed — robots.txt is public.
maxItemsintegeroptionalHard limit, strictly enforced. You are never charged beyond this.

04 — run

Run AI Crawler Audit on your own input

It runs in the cloud on Apify. Nothing to install, and you are billed per result returned rather than per attempt.

Open on Apify ↗

05 — questions

Asked before running it

14 of 14 fields described

What does AI Crawler Audit — Are You Blocking ChatGPT? return?
It returns 14 documented output fields: domain, aiReadinessScore, aiSearchReadiness, blocksAiSearch, blockedSearchBots, blockedBots, hasLlmsTxt, llmsTitle, allowedBotCount, totalBotsChecked, sitemapCount and robotsStatus. Each field declares its type, a description of what it holds, and an example value. The remaining 2 are listed in the output schema below.
What input does AI Crawler Audit — Are You Blocking ChatGPT? require?
domains is required. A further 4 inputs are optional. The full input schema, with types and defaults, is below.
Does AI Crawler Audit — Are You Blocking ChatGPT? need an API key or login?
No. None of its inputs is a credential, and it reads only public data. Nothing to sign up for before the first run.
How do I run AI Crawler Audit — Are You Blocking ChatGPT??
It runs on Apify. Start it from its listing with the inputs above, or call the Apify API with the Actor's ID. Results land in a dataset you can export as JSON, CSV or Excel, or read through the Apify API into your own pipeline.