I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
-
@mariusor I don't remember having such discussion. The SHOULD keyword is defined in RFC-2119:
This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
There are many valid reasons to not include
@context. We also have almost 10 years of implementation experience and by now full implications are very well understood: by ignoring this recommendation we make messages smaller and developer experience better. No downside at all.@silverpill regarding size, ActivityPub is such a verbose protocol that the hundred or so of raw bytes you save through omitting context, are most likely negligible through the prism of connection compression. So to me that's not entirely a "valid reason".
And as developer myself, I think that contexts, even in a non valid JSON-LD implementation, offer enough guidance for building a data vocabulary for them to have plenty of value.
Do you propose we replace contexts with Open API specifications, or how do we coordinate what's a valid vocabulary data object in a federated network? And how do you propose that others discover these specs?
-
@pintoch read this thread?
-
@silverpill regarding size, ActivityPub is such a verbose protocol that the hundred or so of raw bytes you save through omitting context, are most likely negligible through the prism of connection compression. So to me that's not entirely a "valid reason".
And as developer myself, I think that contexts, even in a non valid JSON-LD implementation, offer enough guidance for building a data vocabulary for them to have plenty of value.
Do you propose we replace contexts with Open API specifications, or how do we coordinate what's a valid vocabulary data object in a federated network? And how do you propose that others discover these specs?
@silverpill personally I feel like the various activity/object signing methods that get used in recent FEPs are more egregious from a size point of view, when the in spec behaviour for obtaining canonical versions of a resource is to fetch them from their server, instead of relying on random object signing that introduces so much more friction.
-
@silverpill personally I feel like the various activity/object signing methods that get used in recent FEPs are more egregious from a size point of view, when the in spec behaviour for obtaining canonical versions of a resource is to fetch them from their server, instead of relying on random object signing that introduces so much more friction.
@mariusor@metalhead.club I thought the whole point of signing objects or attaching proofs (none of which I do, mind you) are precisely to save the need to make a new request, which comes with its own overhead.
The good thing is fetching from canonical source will never go out of style.
Aside, it seems like I'm only getting Marius's posts, not silverpills. Makes for an interesting one-sided exchange
-
@mariusor@metalhead.club I thought the whole point of signing objects or attaching proofs (none of which I do, mind you) are precisely to save the need to make a new request, which comes with its own overhead.
The good thing is fetching from canonical source will never go out of style.
Aside, it seems like I'm only getting Marius's posts, not silverpills. Makes for an interesting one-sided exchange
@julian I noticed that your inbox endpoint returns 404s (my activities are delivered to personal inbox, not shared).
-
@hongminhee from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
1. making fields which take multiple values explicit
2. always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow<str>, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts in parallel, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocol -
@hongminhee from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
1. making fields which take multiple values explicit
2. always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow<str>, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts in parallel, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocol@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011 -
reposting so @julian sees this
"I noticed that your inbox endpoint returns 404s (my activities are delivered to personal inbox, not shared)." says @silverpill
-
@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011@kopper @hongminhee I'm glad I'm not the only one who noticed this.
-
@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011@hongminhee if i can give one piece of advice to devs who want to process JSON-LD: dont bother compacting. you already know the schema you output (or you're just passing through what the user gives and it doesn't matter to you), serialize directly to the compacted representation, and only run expansion on incoming data
expansion is the cheapest JSON-LD operation (since all other operations depend on it and run it internally anyhow), and this will get you all the compatibility benefits of JSON-LD with little downsides (beyond more annoying deserialization code, as you have to map the expanded representation to your internal structure which will likely be modeled after the compacted one) -
@julian I don't know as much as I'd like about AT Lexicons. That is, not so much how they work, but what the grand idea is? I don't even understand if Bluesky imagines them being mixed and matched JSON-LD style. I think not?
-
Manu, maker of JSON-LD who also helped with the AP Confs, made this nice video https://www.youtube.com/watch?v=vioCbTo3C-4
JSON-LD is a normative reference to ActivityPub. The context of AP is only 1 line, maybe 4 if you support the official extensions. It does not make anything much larger.
It is for example important if you want to consume the federated wikipedia, wikidata, European Broadcasting Union or these Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/ but also to know that e.g. mobilizon uses schema.org for addresses.
I give you an example, if you include
"mz": "https://joinmobilizon.org/ns#", "wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/"in your context, then you know about mobilizon extension but also the whole common knowledge of the world …
I like that, now you can support the whole vocabulary of wikipedia and wikidata which is just JSON-LD.
You get it in all the languages of the world including the properties name.
No problem, if others don't support it, but sad for users. -
Manu, maker of JSON-LD who also helped with the AP Confs, made this nice video https://www.youtube.com/watch?v=vioCbTo3C-4
JSON-LD is a normative reference to ActivityPub. The context of AP is only 1 line, maybe 4 if you support the official extensions. It does not make anything much larger.
It is for example important if you want to consume the federated wikipedia, wikidata, European Broadcasting Union or these Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/ but also to know that e.g. mobilizon uses schema.org for addresses.
I give you an example, if you include
"mz": "https://joinmobilizon.org/ns#", "wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/"in your context, then you know about mobilizon extension but also the whole common knowledge of the world …
I like that, now you can support the whole vocabulary of wikipedia and wikidata which is just JSON-LD.
You get it in all the languages of the world including the properties name.
No problem, if others don't support it, but sad for users.PS, I am using the official JSON-LD processor of Manu and contributors, if support in any language is lacking, we just speak to the JSON-LD Group (glad about the 2 webintents coming together now as well ) …
Cause we are social … -
@hongminhee if i can give one piece of advice to devs who want to process JSON-LD: dont bother compacting. you already know the schema you output (or you're just passing through what the user gives and it doesn't matter to you), serialize directly to the compacted representation, and only run expansion on incoming data
expansion is the cheapest JSON-LD operation (since all other operations depend on it and run it internally anyhow), and this will get you all the compatibility benefits of JSON-LD with little downsides (beyond more annoying deserialization code, as you have to map the expanded representation to your internal structure which will likely be modeled after the compacted one)@kopper@not-brain.d.on-t.work @hongminhee@hollo.social expansion is actually really annoying because the resulting JSON has annoyingly similar keys to lookup in a hashmap, wasting cache lines, and CPU time
-
reposting so @julian sees this
"I noticed that your inbox endpoint returns 404s (my activities are delivered to personal inbox, not shared)." says @silverpill
@liaizon@social.wake.st actually, oddly, I did receive @silverpill@mitra.social's response, so it seems to be I can access some replies from Mitra, but not all.
-
@kopper@not-brain.d.on-t.work @hongminhee@hollo.social expansion is actually really annoying because the resulting JSON has annoyingly similar keys to lookup in a hashmap, wasting cache lines, and CPU time
@natty @hongminhee i would imagine a Good hash algorithm wouldn't care about the similarity of the keys, no? -
@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011@hongminhee i put this in a quote but people reading the thread may also be interested: json-ld compaction does not really save that much bandwidth over having all the prefixes explicitly defined if you're gzipping (and you are gzipping, right? this is json. make sure your nginx gzip_types includes ld+json and activity+json)
RE: not-brain.d.on-t.work/notes/aihftmbjpxdyb9k7 -
@hongminhee that's great!
-
@hongminhee from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
1. making fields which take multiple values explicit
2. always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow<str>, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts in parallel, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocolhm, we really need to differentiate between users responsibility and dev responsibility.
Not sure if Hong saw the draft about the AP kv thing, it supports either JSON-LD fields _or_ as:attachment / as:context …
wtf do I want to say.user story:
We are working on 2 major and 3 projects fulltime which is
- federation of wikibase / wikidata
- federation of Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/
and these https://codeberg.org/Menschys/fedi-codebaseLet's say we want to federate a Country, then all the knowledge is sent in `attachment` with the fully qualified qikidata url in `context` [as:context - not @context ! - this is so confusing :)]
For example the according entries from the PressFreedomIndex `collection` (co-founder of freelens here
But anyway, the idea about having
"wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/" in the `@context` was that any user can consume and federate wikibase
incl.
🧵 1/2 -
hm, we really need to differentiate between users responsibility and dev responsibility.
Not sure if Hong saw the draft about the AP kv thing, it supports either JSON-LD fields _or_ as:attachment / as:context …
wtf do I want to say.user story:
We are working on 2 major and 3 projects fulltime which is
- federation of wikibase / wikidata
- federation of Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/
and these https://codeberg.org/Menschys/fedi-codebaseLet's say we want to federate a Country, then all the knowledge is sent in `attachment` with the fully qualified qikidata url in `context` [as:context - not @context ! - this is so confusing :)]
For example the according entries from the PressFreedomIndex `collection` (co-founder of freelens hereBut anyway, the idea about having
"wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/" in the `@context` was that any user can consume and federate wikibase
incl.
🧵 1/2incl.
- the properties in all the languages of the world
- the knowledge of the world in all the languages
- the wikidata relations and qualified statements including the nameMap etc. and all the urls to all wikiprojects incl. their languages and knowledgeHow else could I say to other softwares if they want all users qualified data, use wikidata vocabulary?
wikipedia, wikidata, EBU, Public Broadcasters, taxi data is _all_ JSON-LD …
