Après l’activation d’un CDN, Googlebot peut accélérer son exploration alors que le cache est encore vide. Google explique comment limiter la charge sur le serveur d’origine sans bloquer ses robots.

Learn why CDN migrations trigger sudden Googlebot traffic spikes, how cold caches cause origin stampedes, WAF/CAPTCHA configuration traps, and 3 steps to pre-warm your cache safely.

Η Google εξηγεί, σε άρθρο και βίντεο της 11ης Νοεμβρίου 2026, γιατί μετά τη μετακίνηση ενός ιστότοπου σε CDN ο Googlebot μπορεί να ανεβάσει ξαφνικά τον ρυθμό…

Sites moving to a CDN can see Googlebot surge onto cold caches and origin servers. Google caps 429 or 503 brakes at 1-2 days. Why does a faster site get hit?

Curate Feeds | Make Collections | Customize Email Briefs

Sites moving to a CDN can see Googlebot surge onto cold caches and origin servers. Google caps 429 or 503 brakes at 1-2 days. Why does a faster site get hit?

Curate Feeds | Make Collections | Customize Email Briefs

Sites moving to a CDN can see Googlebot surge onto cold caches and origin servers. Google caps 429 or 503 brakes at 1-2 days. Why does a faster site get hit?

Which AI bots to allow on a WordPress or WooCommerce store and which to block: the bots that bring citations (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User, Googlebot, Bing...

Curate Feeds | Make Collections | Customize Email Briefs

Curate Feeds | Make Collections | Customize Email Briefs

Sites moving to a CDN can see Googlebot surge onto cold caches and origin servers. Google caps 429 or 503 brakes at 1-2 days. Why does a faster site get hit?

Gary Illyes and Chery Prommawin released the final video in the Non-Canonical Cabinet series, this one is on CDNs, content delivery networks. They discuss why migrating a website to a CDN can suddenly increase Googlebot crawling activity, why this may overload your origin server, and how to avoid accidentally blocking Google's crawlers.

Sites moving to a CDN can see Googlebot surge onto cold caches and origin servers. Google caps 429 or 503 brakes at 1-2 days. Why does a faster site get hit?

> Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com Two data sources most sites already have, and rarely put side by side: * **Your access logs** record every request Googlebot makes: which URLs, how often, what status code it got. * **Search Console's Performance report** shows which pages earn impressions and clicks. Each one alone is useful. Joined together, they answer a question neither can answer alone: **which URLs is Google spending crawl effort on that never show up in search?** That list is usually where the cheap technical SEO wins are: faceted URLs nobody meant to expose, endless calendar pages, old redirect chains, and thin templates that don't deserve to be crawled at all. This post walks through a short Python script that does the join, and how to interpret the result. ## What you need 1. **Access logs for the last 28 days or so** , in a common format (Nginx and Apache "combined" format work as-is). If you're behind a CDN, use the CDN's logs, since many Googlebot requests never reach your origin. 2. **A Search Console export of pages.** In Search Console, go to Performance, choose the same date range, switch to the "Pages" tab and export. You get a CSV with a "Top pages" column plus clicks and impressions. One caveat on the export: the web interface exports at most 1,000 rows. For larger sites, use the Search Analytics API or the bulk data export to BigQuery to get the full list. The script works the same way with any CSV that has a URL column and a clicks column. ## The script import csv, gzip, re, sys, glob from collections import Counter, defaultdict from urllib.parse import urlsplit LOG_GLOB, GSC_CSV, HOST = sys.argv[1], sys.argv[2], sys.argv[3] # e.g. "logs/access.log*" pages.csv example.com line_re = re.compile(r'"(?:GET|HEAD) (\S+) HTTP/[\d.]+" (\d{3}) .*"([^"]*)"$') def opener(p): return gzip.open(p, "rt", errors="replace") if p.endswith(".gz") else open(p, errors="replace") hits = Counter() statuses = defaultdict(Counter) for path in glob.glob(LOG_GLOB): for line in opener(path): m = line_re.search(line.rstrip()) if not m or "Googlebot" not in m.group(3): continue url, status = m.group(1), m.group(2) hits[url] += 1 statuses[url][status] += 1 clicks, impressions = {}, {} with open(GSC_CSV, encoding="utf-8-sig") as f: for row in csv.DictReader(f): u = urlsplit(row.get("Top pages") or row.get("Page") or row["page"]) key = u.path + (("?" + u.query) if u.query else "") clicks[key] = int(str(row.get("Clicks", 0)).replace(",", "")) impressions[key] = int(str(row.get("Impressions", 0)).replace(",", "")) def bucket(url): p = urlsplit(url) if p.query: return "params:" + ",".join(sorted(k.split("=")[0] for k in p.query.split("&"))) parts = [s for s in p.path.split("/") if s] return "/" + (parts[0] if parts else "") + "/*" rows = [] for url, n in hits.most_common(): rows.append((n, clicks.get(url, 0), impressions.get(url, 0), dict(statuses[url]), bucket(url), url)) total = sum(hits.values()) wasted = [r for r in rows if r[1] == 0] print(f"Googlebot requests: {total}, unique URLs: {len(hits)}") print(f"Requests to URLs with 0 clicks: {sum(r[0] for r in wasted)} " f"({100 * sum(r[0] for r in wasted) / max(total, 1):.0f}%)\n") by_bucket = Counter() for r in wasted: by_bucket[r[4]] += r[0] print("Top zero-click groups by crawl requests:") for b, n in by_bucket.most_common(15): print(f"{n:7d} {b}") print("\nTop 30 zero-click URLs:") for n, c, i, st, b, url in wasted[:30]: print(f"{n:6d} impr={i:<6d} {st} {url}") Run it: python3 crawl_vs_clicks.py "logs/access.log*" pages.csv example.com The `Googlebot` check matches on the user agent. Plenty of scrapers pretend to be Googlebot, so for anything you'll act on, verify the IPs with a reverse DNS lookup first. Our earlier post on telling real Googlebot from fakes covers that in detail. > **Want these crawl-vs-traffic gaps flagged automatically?** seotto.com runs technical SEO checks that catch indexing problems before they sit unnoticed in your logs. ## Reading the output The script prints three things: the share of Googlebot requests going to URLs with zero clicks, those requests grouped by pattern, and the top individual URLs. A zero-click URL isn't automatically a problem. Some pages are new, some are legitimately niche, and some exist for users rather than search (account pages, carts). What you're looking for is **patterns with a lot of crawling and no value** : ### Parameter groups If `params:color,size,sort` shows up near the top, Google is crawling combinations of your filters. Each combination is a separate URL. Fix by not linking crawlable URLs for filter combinations you don't want indexed, canonicalising to the unfiltered page, and keeping the parameter URLs out of your sitemap. ### Redirects and 404s URLs where the status counter is mostly `301` or `404` are being requested because something still points at them: internal links, your sitemap, or old external links. Update internal links and the sitemap. External links you can't fix are fine to leave, as long as the redirect is a single hop. ### Calendar, archive and pagination traps Groups like `/events/*` or `/archive/*` with thousands of requests and no clicks often mean an infinite space: "next month" links that go on forever, or tag archives multiplied by page numbers. Cap the range you link to, and stop linking to empty pages. ### Pages with impressions but no clicks These are different: Google thinks they're relevant enough to show, but nobody clicks. That's a title and snippet problem, not a crawling problem. Rewriting the title and meta description is the first thing to try. ## Running it regularly Run the script monthly with the same date range for both sources, and keep the outputs. The interesting number is the trend in "requests to URLs with 0 clicks". If a release suddenly adds a new parameter group to the top of the list, you'll see it the following month instead of a year later. Tools like seotto.com can track this kind of crawl-versus-value picture continuously, but the script is enough to start. ## Related reading * How to read your server logs like Googlebot's diary (with a short Python script) * Crawl budget probably isn't your problem: read your server logs before you blame it * How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python) * How to find orphan pages with Python: diff your sitemap against your internal links * Google renders your page with last week's JavaScript: a 10-point checklist for assets crawlers can actually fetch What did the biggest zero-click group turn out to be on your site? On e-commerce sites it's almost always sort and filter parameters.

Sites moving to a CDN can see Googlebot surge onto cold caches and origin servers. Google caps 429 or 503 brakes at 1-2 days. Why does a faster site get hit?

What 30 days of production server logs from GPTBot, ClaudeBot, and Meta-ExternalAgent taught us — and how to serve clean Markdown and Schema.org graphs on WordPress, Joomla, OpenCart, and Craft CMS.

Technical specifications and forensic logs analyzing search engine infrastructure, query routing, attribution mechanics, and structural subtraction models based on continuous SERP observations.