⚠️Attention, Creatives:
GPTBot is the specific bot OpenAI uses to crawl for Generative AI content, it does crawl Youtube Studio!
Type this into all video description before you finish uploading a video and posting it would disallow it:
User-agent: GPTBot
Disallow: /
GPTBot is the specific bot OpenAI uses to crawl for Generative AI content, it does crawl Youtube Studio!
Type this into all video description before you finish uploading a video and posting it would disallow it:
User-agent: GPTBot
Disallow: /
February 19, 2025 at 4:55 PM
⚠️Attention, Creatives:
GPTBot is the specific bot OpenAI uses to crawl for Generative AI content, it does crawl Youtube Studio!
Type this into all video description before you finish uploading a video and posting it would disallow it:
User-agent: GPTBot
Disallow: /
GPTBot is the specific bot OpenAI uses to crawl for Generative AI content, it does crawl Youtube Studio!
Type this into all video description before you finish uploading a video and posting it would disallow it:
User-agent: GPTBot
Disallow: /
oh. that's GPTBot actively scraping all of my my forgejo repos. GPTBot scraping all my repos without my consent or knowledge
March 23, 2025 at 3:06 PM
oh. that's GPTBot actively scraping all of my my forgejo repos. GPTBot scraping all my repos without my consent or knowledge
Things we're all increasingly going to need to be thinking about:
"How to block OpenAI's new AI-training web crawler from ingesting your data"
https://www.zdnet.com/article/how-to-block-openais-new-ai-training-web-crawler-from-ingesting-your-data/
"How to block OpenAI's new AI-training web crawler from ingesting your data"
https://www.zdnet.com/article/how-to-block-openais-new-ai-training-web-crawler-from-ingesting-your-data/
![Image explaining how to disallow or customize GPTBot access to one's website:
Top box reads
Blocking GPTBot:
add GPTBot to your site's robot.txt
[begin code lines]
User - agent : GPTBot
Disallow : /
Bottom box reads
Blocking GPTBot from parts of your website:
add GPTBot to your site's robot.txt like this:
[begin code lines]
1 User - agent : GPTBot
2 Allow : / directory - 1 /
3 Disallow : /](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:ybkylffhwhn2an2ic2lxh76k/bafkreiamxfd4lmqs5be2my3x3jbvggmcplu4kmp75svrjuekrot2tscusm)
August 9, 2023 at 7:32 PM
Things we're all increasingly going to need to be thinking about:
"How to block OpenAI's new AI-training web crawler from ingesting your data"
https://www.zdnet.com/article/how-to-block-openais-new-ai-training-web-crawler-from-ingesting-your-data/
"How to block OpenAI's new AI-training web crawler from ingesting your data"
https://www.zdnet.com/article/how-to-block-openais-new-ai-training-web-crawler-from-ingesting-your-data/
Every second, OpenAI's gptbot is scanning textfiles.com, so don't worry, we're going to know how to make a red box forever.
textfiles.com
July 11, 2025 at 12:26 AM
Every second, OpenAI's gptbot is scanning textfiles.com, so don't worry, we're going to know how to make a red box forever.
Similar to approaches taken by search engines, you can now tell OpenAI not to use your site content in its AI models https://platform.openai.com/docs/gptbot
August 8, 2023 at 4:02 PM
Similar to approaches taken by search engines, you can now tell OpenAI not to use your site content in its AI models https://platform.openai.com/docs/gptbot
Welp, it turns out my site was getting slowed down massively because of Ai bots combing it for content. GPTbot had accessed it 1.34k times just this morning!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
October 30, 2025 at 1:21 PM
Welp, it turns out my site was getting slowed down massively because of Ai bots combing it for content. GPTbot had accessed it 1.34k times just this morning!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
# GPTBot : The bot made by OpenAI (ChatGPT) / le robot de OpenAI (ChatGPT)
User-agent: GPTBot
Disallow: /
# Bots used by Google for generative pictures (Bard, Vertex AI) / robots des moteurs d’IA générative de Google (Bard, Vertex AI)
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
# Bots used by Google for generative pictures (Bard, Vertex AI) / robots des moteurs d’IA générative de Google (Bard, Vertex AI)
User-agent: Google-Extended
Disallow: /
March 4, 2024 at 10:07 PM
# GPTBot : The bot made by OpenAI (ChatGPT) / le robot de OpenAI (ChatGPT)
User-agent: GPTBot
Disallow: /
# Bots used by Google for generative pictures (Bard, Vertex AI) / robots des moteurs d’IA générative de Google (Bard, Vertex AI)
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
# Bots used by Google for generative pictures (Bard, Vertex AI) / robots des moteurs d’IA générative de Google (Bard, Vertex AI)
User-agent: Google-Extended
Disallow: /
Who the hell is still making requests to that server that's not exposed anywh… oh. 🤔 Hey G. 🤖
February 19, 2025 at 1:54 AM
Who the hell is still making requests to that server that's not exposed anywh… oh. 🤔 Hey G. 🤖
the "ChatGPT"-only useragent is unique in this behavior compared to other UAs i've seen ("ChatGPT-User/1.0" / "OAI-SearchBot" / "GPTBot", etc..)
March 22, 2026 at 6:48 AM
the "ChatGPT"-only useragent is unique in this behavior compared to other UAs i've seen ("ChatGPT-User/1.0" / "OAI-SearchBot" / "GPTBot", etc..)
i am more than a little amused that in the software i am explicitly not using, openai is given a pass.
June 23, 2026 at 3:29 AM
i am more than a little amused that in the software i am explicitly not using, openai is given a pass.
OpenAI's GPTbot is hammering my old site in a way it causes a lot of heavy mysql query. Luckily the site is fronted by Fastly so one-click blocking of that IP, and the CPU usage is gone.
March 12, 2025 at 9:50 PM
OpenAI's GPTbot is hammering my old site in a way it causes a lot of heavy mysql query. Luckily the site is fronted by Fastly so one-click blocking of that IP, and the CPU usage is gone.
Now we can block one more of OpenAI's scraping bots:
https://searchengineland.com/gptbot-openais-new-web-crawler-430360
https://searchengineland.com/gptbot-openais-new-web-crawler-430360
GPTBot: OpenAI releases new web crawler
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
searchengineland.com
August 7, 2023 at 5:17 PM
Now we can block one more of OpenAI's scraping bots:
https://searchengineland.com/gptbot-openais-new-web-crawler-430360
https://searchengineland.com/gptbot-openais-new-web-crawler-430360
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. blog.cloudflare.com/from-googleb...
From Googlebot to GPTBot: Who’s crawling your site in 2025
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. This blog post explores crawling activity focused on AI and search web crawlers, and how 14% of top dom...
blog.cloudflare.com
July 1, 2025 at 12:15 PM
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. blog.cloudflare.com/from-googleb...
every now and then i forget i can just do this
February 25, 2025 at 3:13 AM
every now and then i forget i can just do this
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
October 28, 2025 at 1:41 PM
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
gptbot (and several other AI scraper bots) were spamming one of my servers so much i had to block it :/
February 19, 2025 at 1:58 AM
gptbot (and several other AI scraper bots) were spamming one of my servers so much i had to block it :/
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
September 28, 2023 at 4:34 AM
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
How about a robots.txt file disallowing the usual suspects (e.g. GPTBot, PerplexityBot)?
November 27, 2024 at 3:23 AM
How about a robots.txt file disallowing the usual suspects (e.g. GPTBot, PerplexityBot)?
every time your site gets crawled by the GPTBot useragent you take a drink
June 11, 2024 at 9:34 PM
every time your site gets crawled by the GPTBot useragent you take a drink
the different scraper bots all act differently. it's kind of neat
claude seems to wait until late nights (9:40 PM PT and later)
gptbot is all the time, whenever
facebook blarps out 100 requests in a single second and then vanishes for another hour
claude seems to wait until late nights (9:40 PM PT and later)
gptbot is all the time, whenever
facebook blarps out 100 requests in a single second and then vanishes for another hour
April 23, 2026 at 8:41 AM
the different scraper bots all act differently. it's kind of neat
claude seems to wait until late nights (9:40 PM PT and later)
gptbot is all the time, whenever
facebook blarps out 100 requests in a single second and then vanishes for another hour
claude seems to wait until late nights (9:40 PM PT and later)
gptbot is all the time, whenever
facebook blarps out 100 requests in a single second and then vanishes for another hour
The rise of the AI crawler 🤖 and their JS Rendering Capabilities - An insightful analysis from @vercel , showing:
* None of the major AI crawlers currently render JavaScript: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ...
* None of the major AI crawlers currently render JavaScript: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ...
December 18, 2024 at 3:38 PM
The rise of the AI crawler 🤖 and their JS Rendering Capabilities - An insightful analysis from @vercel , showing:
* None of the major AI crawlers currently render JavaScript: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ...
* None of the major AI crawlers currently render JavaScript: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ...
Just realized I haven't added a robots.txt to the Spectre.Console docs yet. Will have to do that tonight!
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
June 25, 2025 at 6:03 PM
Just realized I haven't added a robots.txt to the Spectre.Console docs yet. Will have to do that tonight!
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
Worth noting: I have some other instructions on how to add stuff to your robots.txt on other types of sites. The only one that doesn't give you real control is SquareSpace
www.foundationwebdev.com/2023/08/util...
www.foundationwebdev.com/2023/11/how-...
www.foundationwebdev.com/2023/08/util...
www.foundationwebdev.com/2023/11/how-...
Utilizing Robots.txt to Block AI Crawlers | Blocking ChatGPT
How can you block AI web crawlers like OpenAI's GPTBot? I go over robots.txt, what it does, and how to use it to block GPTBot.
www.foundationwebdev.com
November 16, 2024 at 8:53 PM
Worth noting: I have some other instructions on how to add stuff to your robots.txt on other types of sites. The only one that doesn't give you real control is SquareSpace
www.foundationwebdev.com/2023/08/util...
www.foundationwebdev.com/2023/11/how-...
www.foundationwebdev.com/2023/08/util...
www.foundationwebdev.com/2023/11/how-...
okay actually they do reject gptbot too. so probably ai stuff in general
May 10, 2026 at 10:12 AM
okay actually they do reject gptbot too. so probably ai stuff in general