About our fetcher
When someone runs a scan, our fetcher visits that website a handful of times and then forgets what it saw.
| User agent | ReadyForAgentsBot/1.0 (+https://ready-for-agents.pages.dev/bot/) |
|---|---|
| When it visits | Only when a person submits that site's address on our scan page. It doesn't crawl on its own. |
| What it requests | The homepage (first 64 KB) and /robots.txt (first 32 KB). Usually 2 requests per scan, plus DNS lookups. |
| Limits | Up to 64 KB and 8 seconds per request, at most 3 redirects, public HTTPS websites on port 443 only. |
| What it keeps | Nothing. Pages are read in memory to build the report, which goes back to the person who asked, and then they're discarded. |
| Network | Requests come from Cloudflare's network. |
Why it reads robots.txt but doesn't obey it
The scan is a one-time visit that a person asks for, like opening a page in a browser, so it fetches robots.txt in order to report on it rather than to follow it. If you'd like us to stop scanning your site, email support@liteflow.app with the domain.
Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.