robots.txt · headers · meta · sitemaps
Instantly analyse robots.txt, crawl directives, AI bot permissions, meta robots, sitemaps and HTTP headers to understand how a website publishes its crawling preferences.
Why SiteAsker
Read what a site actually states about crawling, instead of guessing from its footer.
Every group, every rule, parsed the way a real crawler resolves them.
GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and Applebot, each read separately.
Listed sitemaps, plus a look at the usual address when none are declared.
X-Robots-Tag directives travel in headers and never appear in robots.txt.
index, noindex, follow, nofollow — taken straight from the homepage HTML.
One request, one structured report, and the raw exchange that produced it.
Sign in once and check as many sites as you need.
How it works
Paste any URL. A bare domain is enough — we work out the rest.
We request robots.txt, follow the redirects, read the response headers and the homepage meta tags.
A single verdict with a confidence rating, and every rule we found behind it.
FAQ