Any protection against AI Crawlers and AI learning bots?
-
The forum has the robot.txt which is a basic protection against search bots, but AI bots are still a threat, so maybe someone can make a plugin or devs can integrated that into the next versions of the Forum? NodeBB joining Fediverse is excellent and I'm using NodeBB Forums for the exactly reason, I understand that project is like in pre-alpha but suggesting new things to be implemented is also a good thing.
-
The forum has the robot.txt which is a basic protection against search bots, but AI bots are still a threat, so maybe someone can make a plugin or devs can integrated that into the next versions of the Forum? NodeBB joining Fediverse is excellent and I'm using NodeBB Forums for the exactly reason, I understand that project is like in pre-alpha but suggesting new things to be implemented is also a good thing.
Shaknunic here's the catch-22... If you want to stop AI crawlers, you can stop the good ones using
robots.txtGitHub - ai-robots-txt/ai.robots.txt: A list of AI agents and robots to block.
A list of AI agents and robots to block. Contribute to ai-robots-txt/ai.robots.txt development by creating an account on GitHub.
GitHub (github.com)
If you want to stop the bad ones (which you probably do)... you can't
They masquerade as legitimate users, and they have more funding behind them than you have time. Every time you try to block them someone on the other end will adapt.
Unfortunately right now the only reliable solution is a hard challenge from CloudFlare for nearly all requests.
