Bot information
The Site History crawler
We are a snapshot crawler that loads the pages a site owner has listed, once per schedule (usually once a day), like a desktop browser and like a phone.
We only visit domains whose owner has verified them with us, through Google Search Console, a confirmed company email address, or a DNS TXT record. See how verification works.
What we do
- Typical volume is your number of URLs x 2 page loads per day.
- We do not log in, submit forms or click, except a consent step the site owner configured.
- We abort analytics and ad measurement requests, so we do not show up in your numbers.
How to recognise us
| Signal | Value |
|---|---|
| User-agent token | SiteHistory/0.1 (+https://sitehistory.fyi/bot)A normal Chrome user agent ending in this token. |
| robots.txt token | SiteHistoryWe follow the rules in |
| Static IP address | 91.99.153.175 (IPv4, Hetzner, Falkenstein, Germany). Captures leave from this address only; we do not use IPv6 for captures. See the allowlisting page. |
| Optional secret header | X-Snapshot-Token: <your project token> |
How to allow us
The safest rule is always our IP addresses and the secret header together. Step by step instructions per protection vendor are on the allowlisting page.
How to stop us
- Remove the verification record or delete the project: captures stop at once.
- Or block our IP addresses.
Googlebot profiles: what they are not
If a site owner switches them on, they send Google's published Googlebot user-agent strings from our IP address. They show what the site's own code sends to a request with a Googlebot user agent (dynamic rendering, prerender services, user-agent switches, accidental cloaking). They cannot show what real Googlebot gets where the site or its CDN verifies Googlebot by IP or reverse DNS. Such requests may also trip "fake Googlebot" protections even on an allowlisted IP. Every such capture carries this note.
TODO The canonical wording is owned by legal (ticket E14-T07) and replaces this text.
Abuse contact
Report abuse or ask us to stop visiting your site: abuse@sitehistory.fyi (our legal details are in the imprint).