Example ticket
Quoted from the developer kit. Long fields are cut to their first sentence.
Developer kit v2.13.0
- Requirement ID
- DEV-SRV-01 (opens in a new tab)
- Level(01)
- MUST
- Area
- Server, status codes and crawling
- Requirement
- Let verified Google crawlers through firewalls, CDNs and bot protection
- Why(02)
- Google does not know that a firewall or bot-protection rule is blocking it; it only sees failing requests.
- How
- Allowlist Google's crawlers in WAF, CDN and bot-management rules by the IP ranges Google publishes for its common and special-case crawlers (common-crawlers.json, special-crawlers.json), or by reverse DNS to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com hosts; never by user-agent string alone (it is trivial to fake), and never by a whole googleusercontent.com host, where Google Cloud customers' machines resolve too.
- Test(03)
- Search Console Crawl stats report: host status shows no robots.txt fetch, DNS resolution or server connectivity failures, and no Search Console alert for DNS errors.
- Basis(04)
- DocumentedStated or supported by Google's documentation.
- Source
- Crawling December: CDNs and crawling (opens in a new tab)checked 4 October 2026
- Page tested
- your-site.com/…
- Status
- One of:
- Pass
- Finding
- n/a