Search crawlers are traffic you want. They index your storefront so customers can find you. The problem is that “Googlebot” in a user agent string is just text. Anything can claim to be Googlebot, and a lot of scraping and vulnerability-scanning traffic does exactly that to slip past simple bot rules.
Why blocking by user agent alone backfires
If you block anything that isn’t Googlebot by user agent, you’ll catch some scrapers. But if you trust anything that claims to be Googlebot, you’ve just handed impersonators a free pass, and that’s a common gap in basic bot-blocking setups.
What real verification looks like
Google publishes IP ranges for its crawlers, and legitimate Googlebot traffic will resolve back to Google-owned infrastructure on reverse DNS lookup. Real crawler behavior also has a shape: it requests robots.txt, it follows crawl-rate conventions, and it doesn’t execute JavaScript the way a browser session does.
Skyfense’s bot and crawler classification checks claimed crawler traffic against this fuller picture (network origin, request pattern, and behavior) rather than trusting the user agent string on its own. Legitimate search crawlers pass through untouched. Traffic pretending to be Googlebot to avoid scrutiny gets treated like any other unidentified bot, and scored accordingly.