Development preview. dotbunker is still being built. Some of the data shown here, including registrar prices, is sample data and should not be relied on.

Robots.txt Checker

Fetch and parse a site's robots.txt, list its rules and sitemaps, and test whether a given URL is allowed for Googlebot, Bingbot or any crawler.

cloud

Runs on our edge servers

A browser cannot read another site's response, so a dotbunker edge server fetches it and reports back. Nothing is stored.

Data sources

HTTP · dotbunker edge