CDNs like Cloudflare seem to mitigate things to some degree. I self host a little instance of Mediawiki. Before putting it behind Cloudflare I was getting a firehose of requests, now it's just Cloudflare's caching thingy doing it's thing.

I'm not sure it does anything about AI crawlers other than taking the load off the end server and on to the CDN, so it's probably not stopping the clankers from stealing your data.

Using Cloudflare has its own host of issues though, namely concentrating a bunch of stuff behind a single point of failure as was seen a few weeks ago.