I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
-
@trwnh@mastodon.social it's not an exercise, not anymore, with the Fediverse Observatory!
@julian fedi observatory lists properties commonly used, right? that's a good start, at least.
-
generally agreed except
> you have to map the expanded representation to your internal structure which will likely be modeled after the compacted one
this is compaction but manual instead of using a jsonld processor to do it. maybe the more precise argument is "don't bother with auto/native compaction"?
with that said: you also lose out on flattening and framing, which are pretty cool features for transforming the serialization. if you don't care about those, ok fine
@trwnh @hongminhee i'm not entirely sure on what you mean (it's about 3am here) but compaction isnt that cheap.
flattening and especially framing are the most expensive, and expansion is the cheapest especially since all the other algorithms depend on it (though if you do expand manually before it'll take a fast path out)
my argument here is that, if you know the structure you're serializing to (i.e. if you're a contemporary AP implementation that isn't doing anything too fancy), you can directly serialize in compacted form and skip constructing a tree of JSON objects in your library and running the compaction algorithm over it. depending on how clever you(r libraries) get you may be able to directly write the JSON string directly, even.
from some brief profiling i've done this does show up as a hot code path in iceshrimp.net, one of my goals with Eventually replacing dotNetRdf with my own impl mentioned above is to, given i'm gonna have to mess with serialization anyhow, remove the JSON-LD bits there and serialize directly to compacted form which should help with large boosts and other bursts -
@trwnh @hongminhee i'm not entirely sure on what you mean (it's about 3am here) but compaction isnt that cheap.
flattening and especially framing are the most expensive, and expansion is the cheapest especially since all the other algorithms depend on it (though if you do expand manually before it'll take a fast path out)
my argument here is that, if you know the structure you're serializing to (i.e. if you're a contemporary AP implementation that isn't doing anything too fancy), you can directly serialize in compacted form and skip constructing a tree of JSON objects in your library and running the compaction algorithm over it. depending on how clever you(r libraries) get you may be able to directly write the JSON string directly, even.
from some brief profiling i've done this does show up as a hot code path in iceshrimp.net, one of my goals with Eventually replacing dotNetRdf with my own impl mentioned above is to, given i'm gonna have to mess with serialization anyhow, remove the JSON-LD bits there and serialize directly to compacted form which should help with large boosts and other bursts@kopper @hongminhee i mostly just mean that "directly serialize to compacted form" is basically just doing the compaction in your brain ahead-of-time then hardcoding it into your app. like it's still compaction just uh... once, using a wetware jsonld processor
-
@hongminhee from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
1. making fields which take multiple values explicit
2. always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow<str>, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts in parallel, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocol@kopper @hongminhee As the person probably most responsible for making sure json-ld stayed in the spec (two reasons: because it was the only extensibility answer we had, and because we were trying hard to retain interoperability with the linked data people, which ultimately did not matter), I agree with you. I do ultimately regret not having a simpler solution than json-ld, especially because it greatly hurt our ability to sign messages, which has considerable effect on the ecosystem.
Mea culpa
![😕](https://forum.wedistribute.org/assets/plugins/nodebb-plugin-emoji/emoji/android/1f615.png?v=8eaae6ed46b ":\")
I do think it's fixable. I'd be interested in joining a conversation about how to fix it.
-
@kopper @hongminhee As the person probably most responsible for making sure json-ld stayed in the spec (two reasons: because it was the only extensibility answer we had, and because we were trying hard to retain interoperability with the linked data people, which ultimately did not matter), I agree with you. I do ultimately regret not having a simpler solution than json-ld, especially because it greatly hurt our ability to sign messages, which has considerable effect on the ecosystem.
Mea culpa
I do think it's fixable. I'd be interested in joining a conversation about how to fix it.
I don't remember it that way.
We started the WG off with AS2 being based on JSON-LD, and I don't think we ever considered removing it.
I don't think it was a decision you made on your own. I'm not sure how you would, since you edited AP and not AS2 Core or Vocabulary.
-
I don't remember it that way.
We started the WG off with AS2 being based on JSON-LD, and I don't think we ever considered removing it.
I don't think it was a decision you made on your own. I'm not sure how you would, since you edited AP and not AS2 Core or Vocabulary.
I would be strongly opposed to any effort to remove JSON-LD from AS2. We use it for a lot of extensions. Every AP server uses the Security vocabulary for public keys.
-
I would be strongly opposed to any effort to remove JSON-LD from AS2. We use it for a lot of extensions. Every AP server uses the Security vocabulary for public keys.
@cwebber @kopper @hongminhee It would be a huge backwards-incompatible change for almost zero benefit. People would still make mistakes in their ActivityPub implementations (sorry, Minhee, but that's life on an open network). We'd need to adopt another mechanism for defining extensions, and guess what? People are going to make mistakes with that, too.
-
@cwebber @kopper @hongminhee It would be a huge backwards-incompatible change for almost zero benefit. People would still make mistakes in their ActivityPub implementations (sorry, Minhee, but that's life on an open network). We'd need to adopt another mechanism for defining extensions, and guess what? People are going to make mistakes with that, too.
@cwebber @kopper @hongminhee The biggest downside to JSON-LD, it seems, is that it lets most developers treat AS2 as if it's plain old JSON. That was by design. People sometimes mess it up, but most JSON-LD parsers are pretty tolerant.
-
@cwebber @kopper @hongminhee It would be a huge backwards-incompatible change for almost zero benefit. People would still make mistakes in their ActivityPub implementations (sorry, Minhee, but that's life on an open network). We'd need to adopt another mechanism for defining extensions, and guess what? People are going to make mistakes with that, too.
@evan @hongminhee @cwebber my argument is that json-ld is way more prone to mistakes. in iceshrimp.net, for example, we ship and preload several modified contexts in order to correct some mistakes on our end, and even then we encounter a lot of software that do not, for example, include the security context in their actors
if, as per my suggestion, property names were always written in expanded form, the only mistakes you could really do would be typos, and that would fail pretty loudly compared to the current status quo where most software accept it and some software silently fail. how are those developers meant to even be aware that this is a problem? -
@cwebber @kopper @hongminhee The biggest downside to JSON-LD, it seems, is that it lets most developers treat AS2 as if it's plain old JSON. That was by design. People sometimes mess it up, but most JSON-LD parsers are pretty tolerant.
@evan @cwebber @kopper @hongminhee Couldn’t we agree to standardize on expanded json-ld? We would not need any json-ld processor, we would not need to fetch or cache any context. There would be no way to shadow properties.
-
@evan @cwebber @kopper @hongminhee Couldn’t we agree to standardize on expanded json-ld? We would not need any json-ld processor, we would not need to fetch or cache any context. There would be no way to shadow properties.
@gugurumbe @hongminhee @evan @cwebber
from my brief tests, compacting with no context (which is basically expanded json-ld, with very minor differences) compresses better, but standardizing on expanded ld would still be better than the status quo. yes backwards compatibility would be broken, but pretty much any other solution to this problem beyond not solving it would end up breaking it anyway
i'm still unsure about certain aspects of json-ld such as everything having the capability for multiple values, but without any context defined it's at least explicit and implementations can take that into account where it's actually helpful (sec:publicKeycomes to mind) and ignore it where it isn't
(edit: ignore the last part, i just re-checked and compact-with-no-context collapses arrays with single values, expanded would be clearer here)
RE: not-brain.d.on-t.work/notes/aihftmbjpxdyb9k7 -
@evan @cwebber @kopper @hongminhee Couldn’t we agree to standardize on expanded json-ld? We would not need any json-ld processor, we would not need to fetch or cache any context. There would be no way to shadow properties.
@gugurumbe @cwebber @kopper @hongminhee AS2 requires compacted JSON-LD.
-
@gugurumbe @cwebber @kopper @hongminhee AS2 requires compacted JSON-LD.
There is no data format we can choose to eliminate programmer errors in online protocols. That's a quixotic aim.
-
There is no data format we can choose to eliminate programmer errors in online protocols. That's a quixotic aim.
-
@cwebber @kopper @hongminhee It would be a huge backwards-incompatible change for almost zero benefit. People would still make mistakes in their ActivityPub implementations (sorry, Minhee, but that's life on an open network). We'd need to adopt another mechanism for defining extensions, and guess what? People are going to make mistakes with that, too.
@evan @cwebber @kopper @hongminhee maybe a compromise approach could be to specify a simpler “json-ld as it is used in practice”, similar to what HTML5 was, that remains backward compatible while simplifying the spec to the point that it is actually feasible to implement
-
@gugurumbe @kopper I don't think that's the model of ActivityPub. It's made to allow reading remote objects.
Most implementations pre-load or compile in the external contexts. I agree, it's a big performance hit to load context URLs at runtime.
-
@gugurumbe @kopper I don't think that's the model of ActivityPub. It's made to allow reading remote objects.
Most implementations pre-load or compile in the external contexts. I agree, it's a big performance hit to load context URLs at runtime.
@evan @gugurumbe it's infeasible to preload all contexts, pretty much every pleroma instance hosts their own context on their own instance for example. then there is the obvious interop problems of how to handle contexts for new extensions your software is not aware of (though pretending like they're empty might work i guess?) -
I would be strongly opposed to any effort to remove JSON-LD from AS2. We use it for a lot of extensions. Every AP server uses the Security vocabulary for public keys.
@evan @kopper @hongminhee The problem is that signing json-ld is extremely hard, because effectively you have to turn to the RDF graph normalization algorithm, which has extremely expensive compute times. The lack of signatures means that when I boost peoples' posts, it takes down their instance, since effectively *every* distributed post on the network doesn't actually get accepted as-is, users dial-back to check its contents.
Which, at that point, we might as well not distribute the contents at all when we post to inboxes! We could just publish with the object of the activity being the object's id uri
-
@evan @kopper @hongminhee The problem is that signing json-ld is extremely hard, because effectively you have to turn to the RDF graph normalization algorithm, which has extremely expensive compute times. The lack of signatures means that when I boost peoples' posts, it takes down their instance, since effectively *every* distributed post on the network doesn't actually get accepted as-is, users dial-back to check its contents.
Which, at that point, we might as well not distribute the contents at all when we post to inboxes! We could just publish with the object of the activity being the object's id uri
@cwebber @hongminhee @evan admittedly, codeberg.org/fediverse/fep/src/branch/main/fep/8b32/fep-8b32.md does kind of solve this specific problem. the json canonicalization used there is much lighter than rdf canonicalization (which iceshrimp had to implement in dotNetRdf specifically for its ld signature support, so tooling availability is not really an excuse in favor of json-ld either!) -
@cwebber @hongminhee @evan admittedly, codeberg.org/fediverse/fep/src/branch/main/fep/8b32/fep-8b32.md does kind of solve this specific problem. the json canonicalization used there is much lighter than rdf canonicalization (which iceshrimp had to implement in dotNetRdf specifically for its ld signature support, so tooling availability is not really an excuse in favor of json-ld either!)
@kopper @hongminhee @evan Interesting... I guess it means you can't re-compact with a new outer context, but maybe that's fine
