this same song and dance was happening decades ago with other websites first. joystiq was unsearchable because, to quote their robotstxt, "Archive provides us no value."
March 17, 2026 at 10:49 PM
this same song and dance was happening decades ago with other websites first. joystiq was unsearchable because, to quote their robotstxt, "Archive provides us no value."
November 27, 2024 at 6:47 PM
Robots.txt Deep Dive: Advanced Configurations for Complex Websites www.boldoutlook.com/robotstxt-de...
#webdevelopment #digitalmarketing #webdesign #robotstxt
#webdevelopment #digitalmarketing #webdesign #robotstxt
Robots.txt Deep Dive: Advanced Configurations for Complex Websites
Gain full control of your site’s crawl behavior with this in-depth technical guide to advanced robots.txt configurations. Learn how to optimize crawl budgets, manage complex architectures, and orchest...
www.boldoutlook.com
November 12, 2025 at 4:08 PM
Robots.txt Deep Dive: Advanced Configurations for Complex Websites www.boldoutlook.com/robotstxt-de...
#webdevelopment #digitalmarketing #webdesign #robotstxt
#webdevelopment #digitalmarketing #webdesign #robotstxt
I've replaced the comments in my robots.txt with a detailed page on scrapers I block: seirdy.one/meta/scraper...
Includes info on the multiple blocking-approaches & criteria I use, commonly-blocked scrapers I allow, & more fact-checking than more comprehensive alternatives.
#scrapers #RobotsTxt
Includes info on the multiple blocking-approaches & criteria I use, commonly-blocked scrapers I allow, & more fact-checking than more comprehensive alternatives.
#scrapers #RobotsTxt
seirdy.one
December 10, 2024 at 5:44 AM
I've replaced the comments in my robots.txt with a detailed page on scrapers I block: seirdy.one/meta/scraper...
Includes info on the multiple blocking-approaches & criteria I use, commonly-blocked scrapers I allow, & more fact-checking than more comprehensive alternatives.
#scrapers #RobotsTxt
Includes info on the multiple blocking-approaches & criteria I use, commonly-blocked scrapers I allow, & more fact-checking than more comprehensive alternatives.
#scrapers #RobotsTxt
Hey @screamingfrog.co.uk Just a quick question, does the robots.txt testing tool in SF use the open source robots parser released by Google rather than the original specification so safe to assume that if a rule works in SF it is valid with Google? github.com/google/robot...
GitHub - google/robotstxt: The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11). - google/robotstxt
github.com
November 27, 2024 at 10:37 AM
Hey @screamingfrog.co.uk Just a quick question, does the robots.txt testing tool in SF use the open source robots parser released by Google rather than the original specification so safe to assume that if a rule works in SF it is valid with Google? github.com/google/robot...
State of my toolchain 2024
An overview of the #software, #hardware and other personal #productivity toolchains I’m using as of December 2024.
#Overleaf #LaTeX #academic #writing #planner #robotstxt
blog.kathyreid.id.au/2024/12/08/s...
An overview of the #software, #hardware and other personal #productivity toolchains I’m using as of December 2024.
#Overleaf #LaTeX #academic #writing #planner #robotstxt
blog.kathyreid.id.au/2024/12/08/s...
blog.kathyreid.id.au
December 8, 2024 at 6:08 AM
State of my toolchain 2024
An overview of the #software, #hardware and other personal #productivity toolchains I’m using as of December 2024.
#Overleaf #LaTeX #academic #writing #planner #robotstxt
blog.kathyreid.id.au/2024/12/08/s...
An overview of the #software, #hardware and other personal #productivity toolchains I’m using as of December 2024.
#Overleaf #LaTeX #academic #writing #planner #robotstxt
blog.kathyreid.id.au/2024/12/08/s...
Want a reliable way to validate your own robots.txt? This is excellent. I like how transparent it is built and the learnings from Google's own open-source robotstxt project.
There are other robots.txt checkers which don't do this/use this and are potentially inaccurate
There are other robots.txt checkers which don't do this/use this and are potentially inaccurate
Robots.txt parser based on Google's open source parser from Will Critchlow, CEO of SearchPilot
My interpretation of how Google parses robots.txt files using a fork of their robust open source parser
buff.ly
December 18, 2024 at 6:25 PM
Want a reliable way to validate your own robots.txt? This is excellent. I like how transparent it is built and the learnings from Google's own open-source robotstxt project.
There are other robots.txt checkers which don't do this/use this and are potentially inaccurate
There are other robots.txt checkers which don't do this/use this and are potentially inaccurate
robots.txt:t on jänniä, niitä voi ihan vapaasti lueskella ja niitä löytyy lähes jokaiselta sivustolta (ei omalta kuitenkaan 🫣)
#OSINT #robotstxt
#OSINT #robotstxt
How to use robots.txt for #OSINT
(article by twitter.com/Altio10076080)
- What is a `robots.txt` File
- Importance of `robots.txt` for OSINT & Cybersecurity Professionals
- How to find a Robot.txt file on a website
- How to Find Hidden Information in `robots.txt`
osintteam.blog/robots-txt-a...
(article by twitter.com/Altio10076080)
- What is a `robots.txt` File
- Importance of `robots.txt` for OSINT & Cybersecurity Professionals
- How to find a Robot.txt file on a website
- How to Find Hidden Information in `robots.txt`
osintteam.blog/robots-txt-a...
July 27, 2024 at 9:50 PM
robots.txt:t on jänniä, niitä voi ihan vapaasti lueskella ja niitä löytyy lähes jokaiselta sivustolta (ei omalta kuitenkaan 🫣)
#OSINT #robotstxt
#OSINT #robotstxt
#Development #Launches
Is Your Site Agent-Ready? · Scan your website for agent-friendly standards ilo.im/16c93a by Cloudflare
_____
#Website #AI #Agents #MCP #Commerce #Content #RobotsTxt #Sitemap #WebDev #Frontend
Is Your Site Agent-Ready? · Scan your website for agent-friendly standards ilo.im/16c93a by Cloudflare
_____
#Website #AI #Agents #MCP #Commerce #Content #RobotsTxt #Sitemap #WebDev #Frontend
Is Your Site Agent-Ready?
Scan your website to see if it's ready for AI agents. Check for llms.txt, MCP, agent skills, and other agent-friendly standards.
ilo.im
April 21, 2026 at 8:09 PM
#Development #Launches
Is Your Site Agent-Ready? · Scan your website for agent-friendly standards ilo.im/16c93a by Cloudflare
_____
#Website #AI #Agents #MCP #Commerce #Content #RobotsTxt #Sitemap #WebDev #Frontend
Is Your Site Agent-Ready? · Scan your website for agent-friendly standards ilo.im/16c93a by Cloudflare
_____
#Website #AI #Agents #MCP #Commerce #Content #RobotsTxt #Sitemap #WebDev #Frontend
Great piece from @ashleybelanger.bsky.social for @arstechnica.com that covers tools like #nepenthes which are providing a tarpit to trick #AI scrapers that ignore #robotstxt rules - very relevant to my presentation at @everythingopen.au on #TokenWars.
arstechnica.com/tech-policy/...
arstechnica.com/tech-policy/...
AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt
Attackers explain how an anti-spam defense became an AI weapon.
arstechnica.com
January 28, 2025 at 9:27 PM
Great piece from @ashleybelanger.bsky.social for @arstechnica.com that covers tools like #nepenthes which are providing a tarpit to trick #AI scrapers that ignore #robotstxt rules - very relevant to my presentation at @everythingopen.au on #TokenWars.
arstechnica.com/tech-policy/...
arstechnica.com/tech-policy/...
#Development #Explorations
Do URLs in prompts change LLM responses? · “It took me weeks of experimentation.” ilo.im/16e9uu by Paul Kinlan
_____
#Business #Prompting #URL #Content #Website #RobotsTxt #WebDev #Frontend #Backend
Do URLs in prompts change LLM responses? · “It took me weeks of experimentation.” ilo.im/16e9uu by Paul Kinlan
_____
#Business #Prompting #URL #Content #Website #RobotsTxt #WebDev #Frontend #Backend
does a url in a prompt steer an llm's output toward its content?
Testing whether opaque URLs act as pointers to memorized content, and finding that JavaScript-rendered sites may be missing from model training entirely
ilo.im
July 6, 2026 at 11:12 AM
#Development #Explorations
Do URLs in prompts change LLM responses? · “It took me weeks of experimentation.” ilo.im/16e9uu by Paul Kinlan
_____
#Business #Prompting #URL #Content #Website #RobotsTxt #WebDev #Frontend #Backend
Do URLs in prompts change LLM responses? · “It took me weeks of experimentation.” ilo.im/16e9uu by Paul Kinlan
_____
#Business #Prompting #URL #Content #Website #RobotsTxt #WebDev #Frontend #Backend
#Development #Analyses
Farewell to robots.txt (1994-2025) · “You were too good for this world.” ilo.im/167q2b by Henning Fries
_____
#SearchEngine #InternetArchive #Crawlers #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend
Farewell to robots.txt (1994-2025) · “You were too good for this world.” ilo.im/167q2b by Henning Fries
_____
#SearchEngine #InternetArchive #Crawlers #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend
Obituary: Farewell to robots.txt (1994-2025)
The voluntary compliance protocol that civilized the internet has departed, bids Henning Fries farewell.
ilo.im
October 21, 2025 at 8:20 AM
#Development #Analyses
Farewell to robots.txt (1994-2025) · “You were too good for this world.” ilo.im/167q2b by Henning Fries
_____
#SearchEngine #InternetArchive #Crawlers #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend
Farewell to robots.txt (1994-2025) · “You were too good for this world.” ilo.im/167q2b by Henning Fries
_____
#SearchEngine #InternetArchive #Crawlers #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend
What Is llms.txt, and Should You Care About It?, by @ahrefs@mastodon.social:
https://ahrefs.com/blog/what-is-llms-txt/
#ai #crawling #robotstxt
https://ahrefs.com/blog/what-is-llms-txt/
#ai #crawling #robotstxt
What Is llms.txt, and Should You Care About It?
ahrefs.com
April 22, 2025 at 7:30 PM
What Is llms.txt, and Should You Care About It?, by @ahrefs@mastodon.social:
https://ahrefs.com/blog/what-is-llms-txt/
#ai #crawling #robotstxt
https://ahrefs.com/blog/what-is-llms-txt/
#ai #crawling #robotstxt
https://netcontentseo.net/article/cloudflare-can-keep-robotstxt-aligned-with-its-firewall-but-it-may-also-replace-per-bot-control-with-three-broad-ai-categories-879 #Cloudflare #RobotsTxt #AICrawlers #TechnicalSEO (2/2)
September 19, 2026 at 11:27 PM
https://netcontentseo.net/article/cloudflare-can-keep-robotstxt-aligned-with-its-firewall-but-it-may-also-replace-per-bot-control-with-three-broad-ai-categories-879 #Cloudflare #RobotsTxt #AICrawlers #TechnicalSEO (2/2)
What fresh #AI hell is that?
grep claudebot *.log | grep -c robots.txt
3792
Why the heck is that crap requesting the robots.txt over and over again?
#Claude #ClaudeBot #ClaudeAI #robotstxt
grep claudebot *.log | grep -c robots.txt
3792
Why the heck is that crap requesting the robots.txt over and over again?
#Claude #ClaudeBot #ClaudeAI #robotstxt
February 15, 2024 at 9:48 PM
What fresh #AI hell is that?
grep claudebot *.log | grep -c robots.txt
3792
Why the heck is that crap requesting the robots.txt over and over again?
#Claude #ClaudeBot #ClaudeAI #robotstxt
grep claudebot *.log | grep -c robots.txt
3792
Why the heck is that crap requesting the robots.txt over and over again?
#Claude #ClaudeBot #ClaudeAI #robotstxt
OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly scraping its entire content, as the site's robots.txt file was misconfigured. A reminder of the importance of proper site configuration for web scraping. #OpenAI #WebScraping #Ecommerce #RobotsTxt #TechEthics #SiteManagement
January 11, 2025 at 7:22 AM
OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly scraping its entire content, as the site's robots.txt file was misconfigured. A reminder of the importance of proper site configuration for web scraping. #OpenAI #WebScraping #Ecommerce #RobotsTxt #TechEthics #SiteManagement
#Development #Findings
AI bots and robots.txt · How websites use robots.txt to set AI crawling rules ilo.im/166la4 by Paul Calvano
_____
#AI #Bots #Content #Website #UserAgents #RobotsTxt #Business #SEO #WebDev #Backend
AI bots and robots.txt · How websites use robots.txt to set AI crawling rules ilo.im/166la4 by Paul Calvano
_____
#AI #Bots #Content #Website #UserAgents #RobotsTxt #Business #SEO #WebDev #Backend
AI Bots and Robots.txt
There’s been a lot of discussion lately around AI crawlers and bots, which are used to train LLMs and/or fetch content on behalf of their users. In the past few weeks I’ve seen blog posts about the...
ilo.im
September 3, 2025 at 7:09 PM
#Development #Findings
AI bots and robots.txt · How websites use robots.txt to set AI crawling rules ilo.im/166la4 by Paul Calvano
_____
#AI #Bots #Content #Website #UserAgents #RobotsTxt #Business #SEO #WebDev #Backend
AI bots and robots.txt · How websites use robots.txt to set AI crawling rules ilo.im/166la4 by Paul Calvano
_____
#AI #Bots #Content #Website #UserAgents #RobotsTxt #Business #SEO #WebDev #Backend
useful article from @mallory.techpolicy.social.ap.brid.gy and @awdsome.bsky.social on the state of play of robots.txt, AI preference signals, and more: www.techpolicy.press/robotstxt-is... - also highlights that whatever happens with (c), there will be a tussle of technical measures & counters
Robots.txt Is Having a Moment: Here's Why We Should Care | TechPolicy.Press
Once a quiet piece of internet plumbing, robots.txt is now in the spotlight, write Audrey Hingle and Mallory Knodel.
www.techpolicy.press
April 4, 2025 at 4:55 PM
useful article from @mallory.techpolicy.social.ap.brid.gy and @awdsome.bsky.social on the state of play of robots.txt, AI preference signals, and more: www.techpolicy.press/robotstxt-is... - also highlights that whatever happens with (c), there will be a tussle of technical measures & counters
Tips to manage sitemap.xml and robots.txt on large sites for better crawling, indexing, and SEO.
#sitemap #RobotsTxt #seo #websitemanagement #crawloptimization
Read blog for more information- etasolution.in/how-to-handl...
#sitemap #RobotsTxt #seo #websitemanagement #crawloptimization
Read blog for more information- etasolution.in/how-to-handl...
How to Handle sitemap.xml and robots.txt for Large Websites
Learn how to optimize sitemap.xml and robots.txt for large websites to improve crawling, indexing, and SEO performance with best practices.
etasolution.in
August 26, 2025 at 10:47 AM
Tips to manage sitemap.xml and robots.txt on large sites for better crawling, indexing, and SEO.
#sitemap #RobotsTxt #seo #websitemanagement #crawloptimization
Read blog for more information- etasolution.in/how-to-handl...
#sitemap #RobotsTxt #seo #websitemanagement #crawloptimization
Read blog for more information- etasolution.in/how-to-handl...