This approach also works in scenarios where IP blocking won't. For example. Applebot is used for all Apple crawling, but Applebot-Extended in the robots.txt file opts out of AI (support.apple.com/en-us/119829). If you just block Applebot you're blocking search engine indexing and link previews too.
About Applebot - Apple Support
Learn about Applebot, the web crawler for Apple.
support.apple.com
January 21, 2025 at 5:41 PM
This approach also works in scenarios where IP blocking won't. For example. Applebot is used for all Apple crawling, but Applebot-Extended in the robots.txt file opts out of AI (support.apple.com/en-us/119829). If you just block Applebot you're blocking search engine indexing and link previews too.
4.
A salty playing some sonic on a applebot tv ( @20daysapple.bsky.social , @rexythetrexy.bsky.social )
A salty playing some sonic on a applebot tv ( @20daysapple.bsky.social , @rexythetrexy.bsky.social )
January 4, 2025 at 10:08 PM
4.
A salty playing some sonic on a applebot tv ( @20daysapple.bsky.social , @rexythetrexy.bsky.social )
A salty playing some sonic on a applebot tv ( @20daysapple.bsky.social , @rexythetrexy.bsky.social )
If you haven't updated your robots.txt to protect your work from scraping for "AI" training recently, there's probably a few bots you aren't blocking. Added Applebot-Extended this morning.
neil-clarke.com/block-the-bo...
neil-clarke.com/block-the-bo...
June 11, 2024 at 2:15 PM
If you haven't updated your robots.txt to protect your work from scraping for "AI" training recently, there's probably a few bots you aren't blocking. Added Applebot-Extended this morning.
neil-clarke.com/block-the-bo...
neil-clarke.com/block-the-bo...
"maybe i shouldn't block applebot, i should go look up and check what it does"
ah, of course. into the 403 list it goes
ah, of course. into the 403 list it goes
March 25, 2025 at 5:33 AM
"maybe i shouldn't block applebot, i should go look up and check what it does"
ah, of course. into the 403 list it goes
ah, of course. into the 403 list it goes
If you have an existing site and want to block AI, you can create your own robots.txt and add the blockers. We're going to be updating our list using this one for now (note the Applebot entry - will be sending a PR later today): github.com/ai-robots-tx...
github.com
January 21, 2025 at 5:41 PM
If you have an existing site and want to block AI, you can create your own robots.txt and add the blockers. We're going to be updating our list using this one for now (note the Applebot entry - will be sending a PR later today): github.com/ai-robots-tx...
Apple updates its Applebot documentation explaining what it means to block Applebot-Extended vs Applebot www.seroundtable.com/apple-update...
#apple #applebot #seo #ai
#apple #applebot #seo #ai
April 28, 2025 at 11:11 AM
Apple updates its Applebot documentation explaining what it means to block Applebot-Extended vs Applebot www.seroundtable.com/apple-update...
#apple #applebot #seo #ai
#apple #applebot #seo #ai
Stranger creatures still inhabit this world. Johnny Applebot...
May 13, 2025 at 5:54 AM
Stranger creatures still inhabit this world. Johnny Applebot...
【AI観測ログ】
iOS 27でSiriが対話型AIに変わる
昨日のAI実験室#10でApplebotだけが
CSS・JSまで取得することを確認しました
SiriがAIになるとき
回答を支えるのがApplebotのクロールです
知っておきたい3点
・Siri刷新の基盤はGemini（WWDC 2026・6月8日〜）
・Applebot-Extendedの拒否率はまだ6%
・robots.txtで学習利用だけ断てる
👇詳しくはブログで
blog.ai-kansoku.com/siri-ios27-a...
#Applebot #iOS27 #Siri #AIクローラー
iOS 27でSiriが対話型AIに変わる
昨日のAI実験室#10でApplebotだけが
CSS・JSまで取得することを確認しました
SiriがAIになるとき
回答を支えるのがApplebotのクロールです
知っておきたい3点
・Siri刷新の基盤はGemini（WWDC 2026・6月8日〜）
・Applebot-Extendedの拒否率はまだ6%
・robots.txtで学習利用だけ断てる
👇詳しくはブログで
blog.ai-kansoku.com/siri-ios27-a...
#Applebot #iOS27 #Siri #AIクローラー
iOS 27でSiriが変わる—Applebotの動き方と今後の展開
iOS 27でSiriが対話型AIに刷新されます。Applebotのクロールデータがどう使われるのか、Applebot-Extendedの拒否率がまだ6%にとどまる現状と、サイト運営者が今判断すべきrobots.txtの設定を整理しました。
blog.ai-kansoku.com
April 6, 2026 at 7:10 AM
【AI観測ログ】
iOS 27でSiriが対話型AIに変わる
昨日のAI実験室#10でApplebotだけが
CSS・JSまで取得することを確認しました
SiriがAIになるとき
回答を支えるのがApplebotのクロールです
知っておきたい3点
・Siri刷新の基盤はGemini（WWDC 2026・6月8日〜）
・Applebot-Extendedの拒否率はまだ6%
・robots.txtで学習利用だけ断てる
👇詳しくはブログで
blog.ai-kansoku.com/siri-ios27-a...
#Applebot #iOS27 #Siri #AIクローラー
iOS 27でSiriが対話型AIに変わる
昨日のAI実験室#10でApplebotだけが
CSS・JSまで取得することを確認しました
SiriがAIになるとき
回答を支えるのがApplebotのクロールです
知っておきたい3点
・Siri刷新の基盤はGemini（WWDC 2026・6月8日〜）
・Applebot-Extendedの拒否率はまだ6%
・robots.txtで学習利用だけ断てる
👇詳しくはブログで
blog.ai-kansoku.com/siri-ios27-a...
#Applebot #iOS27 #Siri #AIクローラー
Sadly, Apple is participating in the great looting of the internet, with its "AppleBot" tracker helping to train their models on "publicly available data." Apple isn't clear what that means, or whether they've already used your data. It's a disgrace.
wheresyoured.at/untitled/
wheresyoured.at/untitled/
June 17, 2024 at 5:11 PM
Sadly, Apple is participating in the great looting of the internet, with its "AppleBot" tracker helping to train their models on "publicly available data." Apple isn't clear what that means, or whether they've already used your data. It's a disgrace.
wheresyoured.at/untitled/
wheresyoured.at/untitled/
I'm a bit less sick right now so I've deployed a first attempt at rate-limiting bots (*especially* AppleBot) so they don't overwhelm our server.
While "a.i." pillagers are now blocked, automated search agents are becoming a problem in terms of load. They aren't scrappers per se, but load is load.
While "a.i." pillagers are now blocked, automated search agents are becoming a problem in terms of load. They aren't scrappers per se, but load is load.
September 22, 2026 at 8:19 AM
I'm a bit less sick right now so I've deployed a first attempt at rate-limiting bots (*especially* AppleBot) so they don't overwhelm our server.
While "a.i." pillagers are now blocked, automated search agents are becoming a problem in terms of load. They aren't scrappers per se, but load is load.
While "a.i." pillagers are now blocked, automated search agents are becoming a problem in terms of load. They aren't scrappers per se, but load is load.
Wake up babes, new version of Simple NoAI WordPress plugin just dropped. Adds more crawlers to deny including:
AppleBot-Extended, Bytespider, Cohere-ai, Diffbot, ImagesiftBot, PerplexityBot, FacebookBot, Omigili
Get 1.6.3 here: wordpress.org/plugins/simp...
AppleBot-Extended, Bytespider, Cohere-ai, Diffbot, ImagesiftBot, PerplexityBot, FacebookBot, Omigili
Get 1.6.3 here: wordpress.org/plugins/simp...
Simple NoAI and NoImageAI
This plugin very simply adds a line of code to your header that tells AIs not to use anything on your website for indexing.
wordpress.org
June 11, 2024 at 3:13 PM
Wake up babes, new version of Simple NoAI WordPress plugin just dropped. Adds more crawlers to deny including:
AppleBot-Extended, Bytespider, Cohere-ai, Diffbot, ImagesiftBot, PerplexityBot, FacebookBot, Omigili
Get 1.6.3 here: wordpress.org/plugins/simp...
AppleBot-Extended, Bytespider, Cohere-ai, Diffbot, ImagesiftBot, PerplexityBot, FacebookBot, Omigili
Get 1.6.3 here: wordpress.org/plugins/simp...
Ok actually, because robots.txt is optional and ignorable, here is our current, full AI bot blocking solution using the non-optional .htaccess mod_rewrite (see alt text)
![<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
# block AI bots
RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Ahrefsbot|AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|AspiegelBot|aspiegel|AwarioRssBot|AwarioSmartBot|Brightbot\s1.0|Bytespider|CCBot|chatgpt|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|DataForSeoBot|Datanyze|Diffbot|DotBot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo\sBot|Kinza|OPPO\sA33|magpie-crawler|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|Omgilibot|PanguBot|peer39_crawler|PerplexityBot|PetalBot|serpstatbot|spaziodati|Scrapy|SemrushBot|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade\sindexer\sbot|thesis-research-bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YandexBot|YouBot|zh-CN|zh_CN) [NC]
RewriteRule ^ – [F]
</IfModule>](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:eb2wskwyww57welmbn7c6mak/bafkreigdduympefft24ufhmoggotfu76lokjng5zt42khy4zvku75haj5i)
October 28, 2025 at 5:09 PM
Ok actually, because robots.txt is optional and ignorable, here is our current, full AI bot blocking solution using the non-optional .htaccess mod_rewrite (see alt text)
Results consistently show that none of the major AI crawlers currently render JavaScript.
However, Google's Gemini and AppleBot enable full JavaScript rendering, which may push others to follow.
However, Google's Gemini and AppleBot enable full JavaScript rendering, which may push others to follow.
December 17, 2024 at 6:16 PM
Results consistently show that none of the major AI crawlers currently render JavaScript.
However, Google's Gemini and AppleBot enable full JavaScript rendering, which may push others to follow.
However, Google's Gemini and AppleBot enable full JavaScript rendering, which may push others to follow.
Been getting a bunch of high load events from my little webservers the last couple weeks... traffic has increased by a factor of THIRTY.
3000% more traffic.
Checked the analytics and it's almost entirely ChatGPT, Claude and Applebot. So I guess I am setting up anubis on 73 domains this weekend.
3000% more traffic.
Checked the analytics and it's almost entirely ChatGPT, Claude and Applebot. So I guess I am setting up anubis on 73 domains this weekend.
March 18, 2026 at 4:26 PM
Been getting a bunch of high load events from my little webservers the last couple weeks... traffic has increased by a factor of THIRTY.
3000% more traffic.
Checked the analytics and it's almost entirely ChatGPT, Claude and Applebot. So I guess I am setting up anubis on 73 domains this weekend.
3000% more traffic.
Checked the analytics and it's almost entirely ChatGPT, Claude and Applebot. So I guess I am setting up anubis on 73 domains this weekend.
Search engine crawlers still outpace AI crawlers, but it's growing significantly.
Last month GPTBot, Claude, AppleBot, and PerplexityBot combined account for nearly 1.3 billion fetches—a little over 28% of Googlebot's volume.
Last month GPTBot, Claude, AppleBot, and PerplexityBot combined account for nearly 1.3 billion fetches—a little over 28% of Googlebot's volume.
December 17, 2024 at 6:16 PM
Search engine crawlers still outpace AI crawlers, but it's growing significantly.
Last month GPTBot, Claude, AppleBot, and PerplexityBot combined account for nearly 1.3 billion fetches—a little over 28% of Googlebot's volume.
Last month GPTBot, Claude, AppleBot, and PerplexityBot combined account for nearly 1.3 billion fetches—a little over 28% of Googlebot's volume.
私の消したお人形サイトにもApplebotはきてたな。
ただbot名も偽れるという話もちらっときいたので、そういうの今どうなってるのかしら。
ただbot名も偽れるという話もちらっときいたので、そういうの今どうなってるのかしら。
September 9, 2025 at 3:57 AM
私の消したお人形サイトにもApplebotはきてたな。
ただbot名も偽れるという話もちらっときいたので、そういうの今どうなってるのかしら。
ただbot名も偽れるという話もちらっときいたので、そういうの今どうなってるのかしら。
OK, bit more info about this. The problem is AppleBot, which is taking up 63gb of bandwidth and 100% of my website's resources, which was stopping it loading.
Apple's bots are becoming a serious problem – this from Reddit explains a bit more:
www.reddit.com/r/webdev/com...
Apple's bots are becoming a serious problem – this from Reddit explains a bit more:
www.reddit.com/r/webdev/com...
May 26, 2026 at 2:18 PM
OK, bit more info about this. The problem is AppleBot, which is taking up 63gb of bandwidth and 100% of my website's resources, which was stopping it loading.
Apple's bots are becoming a serious problem – this from Reddit explains a bit more:
www.reddit.com/r/webdev/com...
Apple's bots are becoming a serious problem – this from Reddit explains a bit more:
www.reddit.com/r/webdev/com...
Apple rewrites Applebot rules to feed Siri AI - what publishers must know #Apple #Siri #AI #Applebot #MachineLearning
Apple rewrites Applebot rules to feed Siri AI - what publishers must know
Apple updated its Applebot documentation on June 8, 2026, adding AI training controls, nosnippet rules, X-Robots-Tag support, and new crawl behaviour details.
ppc.land
June 10, 2026 at 7:01 PM
Apple rewrites Applebot rules to feed Siri AI - what publishers must know #Apple #Siri #AI #Applebot #MachineLearning
Apple rewrites Applebot rules to feed Siri AI - what publishers must know #Apple #Siri #AI #Applebot #MachineLearning
Apple rewrites Applebot rules to feed Siri AI - what publishers must know
Apple updated its Applebot documentation on June 8, 2026, adding AI training controls, nosnippet rules, X-Robots-Tag support, and new crawl behaviour details.
ppc.land
June 10, 2026 at 7:01 PM
Apple rewrites Applebot rules to feed Siri AI - what publishers must know #Apple #Siri #AI #Applebot #MachineLearning
Oh, the comics I could write with AppleBot
December 21, 2024 at 2:44 PM
Oh, the comics I could write with AppleBot
Add Applebot to the naughty list for git scraping.
will do a post in a couple mins with updated whack-a-mole blocker caddy config
will do a post in a couple mins with updated whack-a-mole blocker caddy config
September 24, 2026 at 1:10 PM
Add Applebot to the naughty list for git scraping.
will do a post in a couple mins with updated whack-a-mole blocker caddy config
will do a post in a couple mins with updated whack-a-mole blocker caddy config
AppleBot documentation updates to include AI, nosnippet controls, crawl delay, X-Robots-Tag, accessibility and more www.seroundtable.com/applebot-ai-...
#apple #siri #appleintelligence #ai #search #robots
#apple #siri #appleintelligence #ai #search #robots
June 10, 2026 at 11:11 AM
AppleBot documentation updates to include AI, nosnippet controls, crawl delay, X-Robots-Tag, accessibility and more www.seroundtable.com/applebot-ai-...
#apple #siri #appleintelligence #ai #search #robots
#apple #siri #appleintelligence #ai #search #robots
Noticed Applebot is scraping this website a lot. Including scraping JavaScript files. FML.
May 7, 2025 at 10:49 AM
Noticed Applebot is scraping this website a lot. Including scraping JavaScript files. FML.
applebot ane exasearch were both being extra naughty.
updated caddy config in second img alt txt
there are _alot_ of IPs with traditional browser IPs hitting this tho (so a cloaked set of scrapers).
A decent chunk of the lowered bars are also fediverse […]
[Original post on mastodon.social]
updated caddy config in second img alt txt
there are _alot_ of IPs with traditional browser IPs hitting this tho (so a cloaked set of scrapers).
A decent chunk of the lowered bars are also fediverse […]
[Original post on mastodon.social]
September 24, 2026 at 1:31 PM
applebot ane exasearch were both being extra naughty.
updated caddy config in second img alt txt
there are _alot_ of IPs with traditional browser IPs hitting this tho (so a cloaked set of scrapers).
A decent chunk of the lowered bars are also fediverse […]
[Original post on mastodon.social]
updated caddy config in second img alt txt
there are _alot_ of IPs with traditional browser IPs hitting this tho (so a cloaked set of scrapers).
A decent chunk of the lowered bars are also fediverse […]
[Original post on mastodon.social]
Za posledních 40 hodin, počet přístupů:
Bingbot 11.347 (srsly?!)
Googlebot 4.629
jiný bingbot 2.373
Bytespider 2.171
Applebot 1.789
GoogleOther (AI) 955
hosting monitoring 723
ChatGPT 664
Yandexbot 294
Meta External 268
Ahrefs 256
Go 233
BaiduSpider 191
Facebook External 131
MJ12bot 113
Bingbot 11.347 (srsly?!)
Googlebot 4.629
jiný bingbot 2.373
Bytespider 2.171
Applebot 1.789
GoogleOther (AI) 955
hosting monitoring 723
ChatGPT 664
Yandexbot 294
Meta External 268
Ahrefs 256
Go 233
BaiduSpider 191
Facebook External 131
MJ12bot 113
December 2, 2025 at 8:09 PM
Za posledních 40 hodin, počet přístupů:
Bingbot 11.347 (srsly?!)
Googlebot 4.629
jiný bingbot 2.373
Bytespider 2.171
Applebot 1.789
GoogleOther (AI) 955
hosting monitoring 723
ChatGPT 664
Yandexbot 294
Meta External 268
Ahrefs 256
Go 233
BaiduSpider 191
Facebook External 131
MJ12bot 113
Bingbot 11.347 (srsly?!)
Googlebot 4.629
jiný bingbot 2.373
Bytespider 2.171
Applebot 1.789
GoogleOther (AI) 955
hosting monitoring 723
ChatGPT 664
Yandexbot 294
Meta External 268
Ahrefs 256
Go 233
BaiduSpider 191
Facebook External 131
MJ12bot 113