Few, if any, web sites or web-based services have gone unscathed by the locust-like hordes of A [...]

Web sites are being increasingly beset by AI scraperbots — a problem that we have written abou [...]

Is your content being stolen? A guide to blocking scraper bots and protecting your articles using Cloudflare and simple firewall rules.

Few, if any, web sites or web-based services have gone unscathed by the locust-like hordes of AI crawlers looking to consume (and then re-consume) all of the world's content. The Anubis project is designed to provide a first line of defense that blocks mindless bots—while granting real users access to sites without too much hassle. Anubis is a young project, not even a year old. However, its development is moving quickly, and the project seems to be enjoying rapid adoption. The most recent release of Anubis, version 1.20.0, includes a feature that many users have been interested in since the project launched: support for challenging clients without requiring users to have JavaScript turned on.

Web sites are being increasingly beset by AI scraperbots — a problem that we have written about before, and which has slowly ramped up to an occasional…

Web sites are being increasingly beset by AI scraperbots — a problem that we have written about before, and which has slowly ramped up to an occasional de-facto DDoS attack. This has not gone uncontested, however: web site operators from around the world have been working on inventive countermeasures. These solutions target the problem posed by scraperbots in different ways; iocaine, a MIT-licensed nonsense generator, is designed to make scraped text less useful by poisoning it with fake data. The hope is to make running scraperbots not economically viable, and thereby address the problem at its root instead of playing an eternal game of Whac-A-Mole.

Few, if any, web sites or web-based services have gone unscathed by the locust-like hordes of AI crawlers looking to consume (and then _re-consume_) all of the world's content. The Anubis project is designed to provide a first line of defense that blocks mindless bots—while granting real users access to sites without too much hassle. Anubis is a young project, not even a year old. However, its development is moving quickly, and the project seems to be enjoying rapid adoption. The most recent release of Anubis, version 1.20.0, includes a feature that many users have been interested in since the project launched: support for challenging clients without requiring users to have JavaScript turned on.

Inside this week's LWN.net Weekly Edition: * Front: Systemd; AI scraperbots; Time-slice extension; FUSE regression; Multi-size THPs; Memcached; Meshtastic. * Briefs: Asahi leadership; Debian images; RISC-V Fedora; OpenSUSE; Mesa 25.0.0; Pi-hole v6; Quotes; ... * Announcements: Newsletters, conferences, security updates, patches, and more.

submitted by cm0002 to opensource 24 points | 4 comments https://lwn.net/Articles/1028558/