Bot and crawler policy
How ScaleRankBot identifies itself, how often it asks for pages, what it keeps, and how to block it in one line.
01What the crawler is for
ScaleRankBot fetches pages so we can analyse a website: check whether pages can be indexed, read titles and headings, measure content depth and structure, and score how well the site answers the questions people ask about it.
It runs in two situations. Either the site owner asked for an analysis, or someone entered a URL into the free audit form on our home page. Nothing else triggers it. We do not crawl the open web speculatively and we do not build an index to sell.
02How to identify it
The crawler always sends this user agent string, unmodified:
Mozilla/5.0 (compatible; ScaleRankBot/1.0; +https://scalerank.io/bot-policy)
It never pretends to be a browser, never spoofs another crawler, and never rotates identities. If you see traffic claiming to be ScaleRankBot that does not match the string above, it is not us and we would like to know about it.
03Verifying that it is really us
A user agent string is trivial to forge, so verify by IP. Do a reverse DNS lookup on the requesting address, confirm it resolves to a hostname under our crawler domain, then do a forward lookup on that hostname and confirm it resolves back to the same address. That is the same two-step check the major search engines document, and it is the only one you should trust.
We publish the current address ranges on request. Email crawler@scalerank.io and we will send them.
04How it behaves
- One request at a time per host, with at least one second between requests. A typical audit is a few hundred pages and finishes in about a minute.
- Full audits are capped at 5,000 pages. Larger sites are sampled rather than crawled exhaustively.
- It requests HTML. It does not download images, video or font files, so it costs you very little bandwidth.
- It honours Retry-After and backs off on 429 and 5xx responses. Repeated errors stop the crawl entirely.
- It follows up to five redirects and then gives up.
- It never submits forms, never follows links that look like actions, and never attempts to reach anything behind a login.
- A given site is re-crawled at most once a day, and only while it is connected to an active account.
05robots.txt
We fetch and obey /robots.txt before anything else, and we cache it for no more than 24 hours. Directives aimed at ScaleRankBot take priority over wildcard rules. We respect Crawl-delay if you set one, up to a maximum of 30 seconds, and we honour noindex and nofollow in meta tags and headers.
There is one exception worth being explicit about. If you own a site and connect it to your own account, we will crawl pages you have disallowed in robots.txt, because you have directly asked us to audit them and a report that silently skips half the site is worse than useless. That only ever applies to a verified, connected property.
06Blocking it
To stop the crawler entirely, add this to your robots.txt:
User-agent: ScaleRankBot Disallow: /
To slow it down instead of blocking it:
User-agent: ScaleRankBot Crawl-delay: 10
Changes take effect within 24 hours. If you need it stopped sooner than that, email crawler@scalerank.io and we will block your domain at our end, usually the same day. We do not ask for a reason.
07What we keep from your pages
For a connected site, we store page HTML, response headers, extracted structure and derived scores for as long as the site is connected.
For a one-off audit run by someone typing a URL into our form, we keep the analysis for 90 days and then delete it. The report is addressed by an unguessable identifier and is excluded from search engines in our own robots.txt.
If you want a crawl of your site purged before that, write to crawler@scalerank.io from an address at the domain in question.
08What it does not do
- It does not collect training data. Nothing it fetches is used to train a model, ours or anyone else's.
- It does not republish your content, and it does not make your pages available to other customers.
- It does not attempt authentication, probe for vulnerabilities, or test inputs.
- It does not crawl a site on behalf of a competitor beyond what a public audit of a public page involves.
- It does not ignore a block once you have set one.
09Complaints and contact
Excess traffic, a crawl that should not have happened, a suspected impersonator, or a request for our IP ranges: all of it goes to crawler@scalerank.io. Include your domain and a few log lines with timestamps if you have them. We reply within one working day and act faster than that when a site is under load.