Are you using the ActivityPub functionality? It might be related to that if only because that's what's new (among other smaller items) in v4.
If you toggle the global activitypub switch off, does that make memory usage stabilize?
Keep in mind that since you upgraded, ActivityPub is automatically disabled, so perhaps this is unrelated.