Ready for Agents

About our fetcher

When someone runs a scan, our fetcher visits that website a handful of times and then forgets what it saw.

User agentReadyForAgentsBot/1.0 (+https://ready-for-agents.pages.dev/bot/)
When it visitsOnly when a person submits that site's address on our scan page. It doesn't crawl on its own.
What it requestsThe homepage (first 64 KB) and /robots.txt (first 32 KB). Usually 2 requests per scan, plus DNS lookups.
LimitsUp to 64 KB and 8 seconds per request, at most 3 redirects, public HTTPS websites on port 443 only.
What it keepsNothing. Pages are read in memory to build the report, which goes back to the person who asked, and then they're discarded.
NetworkRequests come from Cloudflare's network.

Why it reads robots.txt but doesn't obey it

The scan is a one-time visit that a person asks for, like opening a page in a browser, so it fetches robots.txt in order to report on it rather than to follow it. If you'd like us to stop scanning your site, email support@liteflow.app with the domain.

Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.