someone has discovered that if you use a fake googlebot useragent on tcrf, it will actually check that you're a real googlebot, and if not, bans you immediately, forever
August 1, 2026 at 4:24 PM
someone has discovered that if you use a fake googlebot useragent on tcrf, it will actually check that you're a real googlebot, and if not, bans you immediately, forever
Googlebot Slowdown Alert! 🚨 Many websites are experiencing massive server response times for Googlebot (up to 9000ms ☠️), causing crawl activity to drop. The likely reason? Google updated their IP addresses for crawling on Feb 4th 2025, but a few CDNs haven't caught up yet. 👇🏼
February 24, 2025 at 4:57 PM
Googlebot Slowdown Alert! 🚨 Many websites are experiencing massive server response times for Googlebot (up to 9000ms ☠️), causing crawl activity to drop. The likely reason? Google updated their IP addresses for crawling on Feb 4th 2025, but a few CDNs haven't caught up yet. 👇🏼
so like is it time for sites to start blocking googlebot? that's the obvious conclusion, right?
Google is transforming Search from a list of links into an AI-powered experience filled with conversational answers, autonomous agents, and interactive interfaces — a shift that could further reduce traffic to publishers across the web.
Google Search as you know it is over | TechCrunch
Google is transforming Search from a list of links into an AI-powered experience filled with conversational answers, autonomous agents, and interactive interfaces — a shift that could further reduce traffic to publishers across the web.
techcrunch.com
May 19, 2026 at 7:48 PM
so like is it time for sites to start blocking googlebot? that's the obvious conclusion, right?
PSA Googlebot tries to render pages to understand what they look like, to better understand the value & context of your site's content. This requires that your JS & CSS files be crawlable. Don't block these files for Googlebot in robots.txt, if you want Google to understand your site optimally.
April 9, 2026 at 4:55 PM
PSA Googlebot tries to render pages to understand what they look like, to better understand the value & context of your site's content. This requires that your JS & CSS files be crawlable. Don't block these files for Googlebot in robots.txt, if you want Google to understand your site optimally.
Do I use robots.txt or noindex to control GoogleBot?
February 6, 2025 at 7:38 PM
Do I use robots.txt or noindex to control GoogleBot?
So I'm thinking... how does google know it's supposed to be a PDF? how does it know the page count?
What happens if I spoof a googlebot UA?
What happens if I spoof a googlebot UA?
September 30, 2024 at 10:11 PM
So I'm thinking... how does google know it's supposed to be a PDF? how does it know the page count?
What happens if I spoof a googlebot UA?
What happens if I spoof a googlebot UA?
favicons don't show up in the URL Inspection Tool's "More Info > Page Resources" section because Googlebot hands them off to Googlebot-Image
you are now armed with forbidden knowledge
you are now armed with forbidden knowledge
March 17, 2026 at 11:25 AM
favicons don't show up in the URL Inspection Tool's "More Info > Page Resources" section because Googlebot hands them off to Googlebot-Image
you are now armed with forbidden knowledge
you are now armed with forbidden knowledge
Twitter: We will block access to users who aren't logged in
Actual malicious actors: We will pretend to be Googlebot and will still have access
Actual malicious actors: We will pretend to be Googlebot and will still have access
July 3, 2023 at 7:13 AM
Twitter: We will block access to users who aren't logged in
Actual malicious actors: We will pretend to be Googlebot and will still have access
Actual malicious actors: We will pretend to be Googlebot and will still have access
I read a study that said Googlebot is eating all the cookies on websites.
IS THIS TRUE, MARTIN?
IS THIS TRUE, MARTIN?
November 21, 2025 at 7:42 AM
I read a study that said Googlebot is eating all the cookies on websites.
IS THIS TRUE, MARTIN?
IS THIS TRUE, MARTIN?
N'empêche les IA vont poser un vrai défi à la cybersécurité informatique.
Je donne un exemple : Les captcha. Avant les captchas étaient faits pour distinguer humains et robots lorsqu'on entrait sur un site.
Par robot, on peut penser aux googlebot.
fr.wikipedia.org/wiki/Googlebot
Je donne un exemple : Les captcha. Avant les captchas étaient faits pour distinguer humains et robots lorsqu'on entrait sur un site.
Par robot, on peut penser aux googlebot.
fr.wikipedia.org/wiki/Googlebot
Googlebot — Wikipédia
fr.wikipedia.org
September 20, 2025 at 12:15 PM
N'empêche les IA vont poser un vrai défi à la cybersécurité informatique.
Je donne un exemple : Les captcha. Avant les captchas étaient faits pour distinguer humains et robots lorsqu'on entrait sur un site.
Par robot, on peut penser aux googlebot.
fr.wikipedia.org/wiki/Googlebot
Je donne un exemple : Les captcha. Avant les captchas étaient faits pour distinguer humains et robots lorsqu'on entrait sur un site.
Par robot, on peut penser aux googlebot.
fr.wikipedia.org/wiki/Googlebot
Gary and I wrote a bit about crawling - especially the resources embedded in a page, so here you go: developers.google.com/search/blog/...
Crawling December: The how and why of Googlebot crawling | Google Search Central Blog | Google for Developers
developers.google.com
December 3, 2024 at 3:42 PM
Gary and I wrote a bit about crawling - especially the resources embedded in a page, so here you go: developers.google.com/search/blog/...
well i guess this makes me a "web crawler" ha ha ha (catches GoogleBot in my web and sucks it dry) that one tasted like 41gb of jpegs
April 19, 2025 at 6:51 AM
well i guess this makes me a "web crawler" ha ha ha (catches GoogleBot in my web and sucks it dry) that one tasted like 41gb of jpegs
Wait… maybe the 403 in my CI/CD tests for my backend is not my fault after all?
I need to google more…
I need to google more…
Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites
Anecdotal report claims that blocking AI training bots with Cloudflare prevents Googlebot from indexing websites.
www.searchenginejournal.com
August 4, 2026 at 6:01 PM
Wait… maybe the 403 in my CI/CD tests for my backend is not my fault after all?
I need to google more…
I need to google more…
This is not a good thing.
Googlebot offers a value exchange. ChatGPT doesn’t.
Googlebot offers a value exchange. ChatGPT doesn’t.
Find out why Googlebot is no longer the only dominant crawler as OpenAI's ChatGPT-User takes the lead in web requests.
#Google #ChatGPT
#Google #ChatGPT
ChatGPT Now Crawls 3.6x More Than Googlebot: What 24M Requests Reveal
Find out why Googlebot is no longer the only dominant crawler as OpenAI's ChatGPT-User takes the lead in web requests.
#Google #ChatGPT
www.searchenginejournal.com
April 7, 2026 at 9:21 AM
This is not a good thing.
Googlebot offers a value exchange. ChatGPT doesn’t.
Googlebot offers a value exchange. ChatGPT doesn’t.
Trying to test a theory that Googlebot may be broken or is breaking. Failing exactly every other time is very not random.
If you have Google Search Console set up, can you do me a favor?
Scroll down to Settings, look for Crawling > robots.txt, click "Open Report" then click on [yoursite/robots.txt] (NOT the three dots) and click on "Versions >."
Was every fetch successful, or does it fail EXACTLY every other time?
Scroll down to Settings, look for Crawling > robots.txt, click "Open Report" then click on [yoursite/robots.txt] (NOT the three dots) and click on "Versions >."
Was every fetch successful, or does it fail EXACTLY every other time?
August 27, 2025 at 10:13 PM
Trying to test a theory that Googlebot may be broken or is breaking. Failing exactly every other time is very not random.
Cloudflare drops planned Googlebot block for sites refusing AI training #Cloudflare #Googlebot #AITechnology #SEO #WebDevelopment
Cloudflare drops planned Googlebot block for sites refusing AI training
17% of Cloudflare sites block AI training; those blocks now turn into a robots.txt rule that Bing plans to honour from early 2027. What does Block cost now?
ppc.land
September 27, 2026 at 10:26 AM
Cloudflare drops planned Googlebot block for sites refusing AI training #Cloudflare #Googlebot #AITechnology #SEO #WebDevelopment
ICYMI: Googlebot is not a program - Google engineers finally explain what it really is #Googlebot #SEO #Crawling #SaaS #DigitalMarketing
Googlebot is not a program - Google engineers finally explain what it really is
Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints.
ppc.land
March 13, 2026 at 9:35 PM
ICYMI: Googlebot is not a program - Google engineers finally explain what it really is #Googlebot #SEO #Crawling #SaaS #DigitalMarketing
as far as shitty networks i see frequently, "ace data centers" is definitely up there. it's one of the few fake googlebot providers that consistently returns its rdns (*•ips•acedatacenter•com). and it sure is frequent
August 20, 2026 at 6:47 PM
as far as shitty networks i see frequently, "ace data centers" is definitely up there. it's one of the few fake googlebot providers that consistently returns its rdns (*•ips•acedatacenter•com). and it sure is frequent
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. blog.cloudflare.com/from-googleb...
From Googlebot to GPTBot: Who’s crawling your site in 2025
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. This blog post explores crawling activity focused on AI and search web crawlers, and how 14% of top dom...
blog.cloudflare.com
July 1, 2025 at 12:15 PM
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. blog.cloudflare.com/from-googleb...
Oh that's just friggin' dirty, Google.
February 11, 2026 at 11:19 PM
Oh that's just friggin' dirty, Google.
i wonder how many different networks this operator is going to (ab)use; using multiple entire /24 subnets on this feels like such a waste
(fake crawler pretending to be googlebot)
(fake crawler pretending to be googlebot)
July 30, 2026 at 8:23 PM
i wonder how many different networks this operator is going to (ab)use; using multiple entire /24 subnets on this feels like such a waste
(fake crawler pretending to be googlebot)
(fake crawler pretending to be googlebot)
Cloudflare drops planned Googlebot block for sites refusing AI training #Cloudflare #Googlebot #AITechnology #SEO #WebDevelopment
Cloudflare drops planned Googlebot block for sites refusing AI training
17% of Cloudflare sites block AI training; those blocks now turn into a robots.txt rule that Bing plans to honour from early 2027. What does Block cost now?
ppc.land
September 27, 2026 at 10:26 AM
Cloudflare drops planned Googlebot block for sites refusing AI training #Cloudflare #Googlebot #AITechnology #SEO #WebDevelopment
this is quite a big deal and ramps up potential liability. it also is a move to put pressure on Google to split Googlebot into search and training (allowing people to block training but allow search).
August 7, 2026 at 9:36 AM
this is quite a big deal and ramps up potential liability. it also is a move to put pressure on Google to split Googlebot into search and training (allowing people to block training but allow search).
( the real answer is developers.google.com/crawling/doc... )
Verify Requests from Google Crawlers and Fetchers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers
You can check if a web crawler really is Googlebot (or another Google user agent). Follow these steps to verify that Googlebot is the crawler.
developers.google.com
August 1, 2026 at 7:02 PM
( the real answer is developers.google.com/crawling/doc... )
20年前は個人ホームページでもgooglebotはちゃんとクロールしてくれてたので、検索すれば一応ヒットした。今はページランクが低いドメインはまったくヒットしない。検索条件をめちゃくちゃピンポイントに絞るように指定しても、出ないものは出ない。インターネットが壊れた。
September 26, 2026 at 2:35 AM
20年前は個人ホームページでもgooglebotはちゃんとクロールしてくれてたので、検索すれば一応ヒットした。今はページランクが低いドメインはまったくヒットしない。検索条件をめちゃくちゃピンポイントに絞るように指定しても、出ないものは出ない。インターネットが壊れた。