Cloudflare Setup

[yasas@community.nodebb.org]

Hi all,

We've been using Cloudflare for a while now and atm we are in the free tier if that helps(and have turned off acceleration).

Since like last week we've seen a sudden hike in requests to the point where the server cannot handle it anymore.

These should be AI crawlers and we're trying to limit access to them but they seem to crawl into the forum somehow.

While we would want non-logged-in users/visitors to see the content, since no one can use the site anymore we have decided to only let in logged-in users without checks and for others, for a human check to go through.

The following is the rule we're using to identify logged in/actual users:

I was wondering if there is a better/more standardised/more accurate way to identify logged in users.

Thanks very much!

[d1re_w0lf@community.nodebb.org]

Since you are on Cloudflare, do you have their Block AI bots rules on?
They seem to help.

[julian@community.nodebb.org]

D1re_W0lf those rules do not seem to help, I have enabled them as well.

yasas we have been seeing the same behaviour starting perhaps a week ago. The bursts seem to happen for maybe half a day and then disappear.

Since we wanted the site to start working quickly, we opted for an allow-list approach. We turned on "I'm under attack" mode, which has some unfortunate side effects (namely, causing federation to stop).

These are our rules:

(any(http.request.headers["accept"][*] eq "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"")) or (any(http.request.headers["accept"][*] eq "application/activity+json")) or (http.request.method eq "POST" and starts_with(http.request.uri.path, "/inbox")) or (starts_with(http.request.uri.path, "/assets")) or (starts_with(http.request.uri.path, "/.well-known"))

This lets (respectively):

1. ActivityPub fetches
2. Same
3. ActivityPub publishes to the NodeBB inbox
4. Static assets
5. Certbot and webfinger (also for AP)

[yasas@community.nodebb.org]

D1re_W0lf Thanks for the response. Seem they do not work as Julian has confirmed.

Hi julian
Thanks a lot. Our current version is still NodeBB v3 so guess we may have to either make some tweaks or schedule a v4 upgrade soon. Thanks anyway Julian. Hope crawlers from search engines are still able to get through with the above setup yeah?

[omega@community.nodebb.org]

yasas Yep seen the same last week, not nodeBB install, but the server behind cloudflare showed a huge uptick, not served by cache either. SO I am guessing scanning the whole site or large parts.

IIRC the traffic was coming manly from swiss-german based servers but did not have the time to dig in. This is even with Block AI cralwers turned on.

In the recent past it's been Asia, like mostly Singapore.

The only way to handle this in Cloudflare is to watch and then put in place with geo-based challenge security rules in place afaict

[omega@community.nodebb.org]

You can also use this online tool to check servers when reviewing the logs in cloudflare, this is an example of one of those serves that showed up, the radar tracks it as 86.8% bot

Just a moment...

(radar.cloudflare.com)

Here can be seen the 6th of September surge on on the graph and it coincides with the server traffic

A lot of these bots are hammering wordpress exploits with targets such as /wp-content/index.php and so on, it might not be a bad idea to block by default if you have server load issues.

[julian@community.nodebb.org]

yasas turning on "I'm Under Attack" mode will likely block search engine crawlers as well.

It is meant to be used as a last resort.