smart_toy
Robots.txt Checker
Fetch and parse a site's robots.txt, list its rules and sitemaps, and test whether a given URL is allowed for Googlebot, Bingbot or any crawler.
cloud
Runs on our edge servers
A browser cannot read another site's response, so a dotbunker edge server fetches it and reports back. Nothing is stored.
Data sources
HTTP · dotbunker edge