[Proposal] Protocol-agnostic PDS

[doriantaylor.com]

Hi folks, Dorian here.

Some of you may know me, and an even smaller subset may know that I’m working on an application server called Intertwingler, whose purpose is to create a durable substrate for making websites out of very large numbers of very small objects, with many many many (typed) links that do not break.

(See my Summer of Protocols paper, Retrofitting the Web for a background on Intertwingler, or a newsletter I just shipped that has a much more concise description.)

The short-term goal is the planning tool I’m calling Sense Atlas (that’ll be slow to load at first until sometime next week when I implement internal caching, lol), but in the longer term I am looking at Intertwingler (or likely a successor) to act as a general-purpose server for what I’ve termed “dense hypermedia” networks. I also have a plan to federate Intertwingler, though using little more than ordinary HTTP reverse proxying.

Intertwingler backs onto an RDF graph database for what I’m calling “transparent” resources, (ie structured data objects that do not correspond to any canonical string of bytes) and a content-addressable store for “opaque” (i.e., arbitrary byte segments the system can’t see into). It also functions roughly as a microservice bus, including an interface for transformation functions that can compose to operate over HTTP messages. I am calling this configuration “Semantic REST”.

Anyway, what I’m getting at is that Intertwingler has the basic ingredients for a PDS, albeit one that is potentially agnostic about what language it speaks. A use case for this would be a BlueSky PDS that was also a personal Mastodon instance, plus whatever else comes along in the future. The goal at this stage is mainly a flex just to say I can do it, but I anticipate it will yield valuable insights.

I see two immediate challenges:

• reconciling Lexicon with RDF, such that there can be a lossless pair of pure, deterministic functions that can map between them (or prove they can’t exist)
• reconciling HTTP/REST with WebSocket/XRPC (something I plan on exploring anyway—mapping message semantics to resources and back)

I’m not asking for anybody’s help at this time, just announcing that I’m considering doing this. Intertwingler is still pretty rough (I’m still a few months away from an installation candidate). But the goal is to make it indistinguishable from the point of view of the network from any other PDS.

PS: I submitted a talk to ATmosphereConf about this.

[erlend.sh]

I remember your work from the Summer of Protocols days; great to see you’re still at it!

While not the same thing, this seems related:

The Case For Universal Login and "Off-Protocol" Services

Noe this post is also available on Leaflet: For the last year, we have been working on Roomy with a clear integration to the ATProtocol. At the same time, we’ve also had a distinctly “off-protocol” strategy for data handling: we don’t store Roomy’s canonical data on the ATProto PDS. The reason for this is that the PDS because has shortcomings, such as a lack of private data, that make it unsuitable for Roomy’s needs. This is actually completely natural! Bluesky’s needs, such as distributin…

Speaking of RDF, we also had an insightful discussion about that not long ago:

Solid vs ATProto

Hey! I just recently read This is for Everyone by Tim Berners-Lee. It’s a phenomenal book and history of the web and I highly recommend it! He spends a good part of the book talking about SOLID: About Solid - Solid Project (not to be confused with the SOLID software design principles ) Looking into it it looks like it has super similar principles to ATProto, although it’s designed to be private by default. I’m still pretty new to ATProto, has there been any discussion on comparing/c…

[doriantaylor.com]

Definitely going to still be at it for a while. I intend for Intertwingler to be my hundred-year infrastructure.

But yes, I imagine the minimum footprint for a federated Intertwingler as a server that lives in my house (plus office if I had one), and another one in the cloud (for availability), and another eventually on my phone. And using VPNs (I rolled my own because I can do that but maybe eg Tailscale?) and ordinary boring HTTP caching/proxying, I can create a unified fabric of HTTP(s, obviously) address space that merges together information resources from the different concrete origins. The idea, at least operationally, is to take REST (qua Fielding’s own definition, not some third-hand blog post about it) to the extreme.

Like I just said on bsky, one goal of Intertwingler is to make it possible to physically possess, within your four walls, the authoritative (or only?) copy of a given piece of information; make it so somebody would have to physically break into your house to steal it. But also make it possible to share it out onto the network if you want. Sorta one-way mirror kinda thing.