/.codex/config.toml ... Slackbot-LinkExpanding/1.0
/actuator/mappings ... ChatGPT-User/1.0
/graphql/console ... GrokBot/1.0
/.hermes/.env ... facebookexternalhit/1.1
/.claude/settings.json ... TelegramBot/1.0
/login ... Amzn-SearchBot/1.0
Get. In. The. Sea.
/.codex/config.toml ... Slackbot-LinkExpanding/1.0
/actuator/mappings ... ChatGPT-User/1.0
/graphql/console ... GrokBot/1.0
/.hermes/.env ... facebookexternalhit/1.1
/.claude/settings.json ... TelegramBot/1.0
/login ... Amzn-SearchBot/1.0
Get. In. The. Sea.
facebookexternalhit/1.1
Facebookに投稿していないのになぜ?
調べると、Xに記事をシェアするたびに
MetaのクローラーがOGPを取得しに来ていました。
7日間92件の実測データで挙動を解説しています。
www.blog.ai-kansoku.com/facebookexte...
#AIクローラー #OGP #サーバーログ #facebookexternalhit
facebookexternalhit/1.1
Facebookに投稿していないのになぜ?
調べると、Xに記事をシェアするたびに
MetaのクローラーがOGPを取得しに来ていました。
7日間92件の実測データで挙動を解説しています。
www.blog.ai-kansoku.com/facebookexte...
#AIクローラー #OGP #サーバーログ #facebookexternalhit
🤓 me on reddit ↝
🤓 me on reddit ↝
https://x.com/robots.txt
```
User-agent: Googlebot
Allow: /*?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Allow: /i/api/
[…]
User-Agent: Google-Extended
Disallow: *
User-Agent: FacebookBot
Disallow: *
User-agent: facebookexternalhit
Disallow: * […]
https://x.com/robots.txt
```
User-agent: Googlebot
Allow: /*?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Allow: /i/api/
[…]
User-Agent: Google-Extended
Disallow: *
User-Agent: FacebookBot
Disallow: *
User-agent: facebookexternalhit
Disallow: * […]
https://www.byteseu.com/2147919/
Building the AI Economy: Texas at a Crossroads | Brownwood News facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) X-Middleton/1 9a1aa2aa5a1f9e07a77f9f49759a2a813d9c5e7d 1
https://www.byteseu.com/2147919/
Building the AI Economy: Texas at a Crossroads | Brownwood News facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) X-Middleton/1 9a1aa2aa5a1f9e07a77f9f49759a2a813d9c5e7d 1
Googlebot 11,448
AhrefsBot 9,128
Unknown robot identified by bot\* 7,723+
Applebot 2,719
SemrushBot 2,182
empty user agent string 2,408
Mediapartners-Google 2,324
bingbot 2,129
feed 2,016
Googlebot-Image 2,043
Go-http-client 1,670
facebookexternalhit 1,589
Googlebot 11,448
AhrefsBot 9,128
Unknown robot identified by bot\* 7,723+
Applebot 2,719
SemrushBot 2,182
empty user agent string 2,408
Mediapartners-Google 2,324
bingbot 2,129
feed 2,016
Googlebot-Image 2,043
Go-http-client 1,670
facebookexternalhit 1,589
🤓 me on reddit ↝
🤓 me on reddit ↝
Our data shows this Meta crawler affects 15% of sites and can spike to 30%+ of total traffic. Learn what it is and how to block it at the edge to save server resources 👉 redirection.io/blog/unders...
Our data shows this Meta crawler affects 15% of sites and can spike to 30%+ of total traffic. Learn what it is and how to block it at the edge to save server resources 👉 redirection.io/blog/unders...
Hopkinsville O'Charley's Closing Its Doors Wednesday | WKDZ
facebookexternalhit/1.1 (+ 9896b249217151b5c552c72e38f85376e3abbb54 1 Source link
Hopkinsville O'Charley's Closing Its Doors Wednesday | WKDZ
facebookexternalhit/1.1 (+ 9896b249217151b5c552c72e38f85376e3abbb54 1 Source link