"Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."

A pseudonymous coder has created and released an open source “tar pit” to indefinitely trap AI training web crawlers in an infinitely, randomly-generating series of pages to waste their time and computing power. The program, called Nepenthes after the genus of carnivorous pitcher plants which trap and consume their prey, can be deployed by webpage owners to protect their own content from being scraped or can be deployed “offensively” as a honeypot trap to waste AI companies’ resources.

Registration bypass: https://archive.is/3tEl0

  • FaceDeer@fedia.io
    0·
    12 days ago

    No, a few million hits from bots is routine for anything that’s facing the public at all. Others have posted on this thread (or others like it, this article’s been making the rounds a lot in the past few days) that even the most basic of sites can get that sort of bot traffic, and that it’s just a simple recursion depth limit setting to avoid the “infinite maze” aspect.

    As for AI training, the access log says nothing about that. As I said, AI training sets are not made by just dumping giant piles of randomly scraped text on AIs any more. If a trainer scraped one of those “infinite maze” sites the quality of the resulting data would be checked, and if it was generated by anything remotely economical for the site to be running it’d almost certainly be discarded as junk.