Tag: federation

Posted on
by

Favorited Rocket.Chat Leverages The Matrix Protocol for Decentralized and Interoperable Communications (Rocket.chat press release 25-05-2022)

This press release by Rocket.chat is interesting to me for several reasons, despite it being written in marketing speak as press releases tend to do.

  • My company uses a self-hosted Rocket.chat instance for internal communication since 2019, for reasons of information hygiene.
  • It positions Matrix as the way of bringing federation and interoperability to Rocket.chat.
  • It leads with e-mail’s SMTP protocol as a useful example of federation.
  • It cites the interoperability requirements for chat that the EU Digital Markets Act requires as evidence that there’s a growing market need for both openness and data control in inter-company collaboration.
  • It states “Big Tech may say that this cross-communication isn’t technically feasible, but Matrix and others prove” otherwise.
  • With those statements it’s directly speaking against the gaslighting Meta did earlier about the EU Digital Markets Act

It seems to me even before its adoption the Digital Markets Act is showing signs of working as intended: breaking monopolistic behaviour by demanding a.o. interoperability.

HT Stephen Downes.

The Rocket.Chat adoption of Matrix makes it simple for organizations to easily connect with external parties, whether they’re using Rocket.Chat or any other Matrix compatible platform. This initiative is another step forward on Rocket.Chat’s journey to let every conversation flow without compromise and enable full interoperability with its ecosystem….The importance of openness and data control in inter-company collaboration is growing. The European Union’s recent Digital Markets Act to limit the market power of large online chat and messaging platforms is evidence of this need.

Rocket.chat press release

Posted on
by

Looking forward to reading Data Mesh by Zhamak Dehghani. She published this book last March with O’Reilly, but originated the concept in 2018. Received the book yesterday.
She suggests data mesh, as a decentralized sociotechnical paradigm drawn from modern distributed architecture that provides a new approach to sourcing, sharing, accessing, and managing analytical data at scale. In other words, moving beyond traditional data warehouses and lakes to a distributed set-up.

I think this is a key concept for the creation of the EU dataspace(s), where combining and working with data is envisioned as not requiring centralising that data, nor even having a copy of the data for your application. Only that way, through things like multi party computing and bringing your model to the data not the other way around, is it possible to bring data into play at European scale that is sensitive, personal or non-public. Only that way is it possible to work with data across domains and a diversity of data holding or using stakeholders. In my work on a national reference architecture for federated digital twins, the concept was applied to the data parts of the reference.

Posted on
by

Having created a working flow to generate OPML booklists directly from the individual book notes in my PKM system, I did the first actual run in production of those scripts today.

It took a few steps to get to using the scripts in production.

  • I have over 300 book note files in my Obsidian vault.
  • Of course most lacked the templated inline data fields that allow me to create lists. For the 67 fiction books I read in 2021 I already had a manual list with links to the individual files. Where needed I added the templated data fields.
  • Having added those inline fields where they were missing I can easily build lists in Obsidian with the Dataview plugin. Using this code


    results in

  • The same inline data fields are used by my scripts to read the individual files and build the same list in OPML
  • That gets automatically posted to my website where the file is both machine and human readable.

Doing this in production made me discover a small typo in the script that builds the OPML, now fixed (also in the GitHub repository). It also made me realise I want to add a way of ordering the OPML outline entries by month read.

Lists to take into production next are those for currently reading (done), non-fiction 2021, and the anti-library. That last one will be the most work, I have a very long list of books to potentially read. I will approach that not as a task of building the list, but as an ongoing effort of evaluating books I have and why they are potentially of interest to me. A way, in short, to extend my learning, with the list as a useful side effect. The one for currently reading is the least work, and from it the lists for fiction 2022 and non-fiction 2022 will automatically follow. The work is in the backlog, getting history to conform to the convention I came up with, not in moving forward from this point.

In parallel it is great to see that Tom Critchlow is also looking at creating such book lists, in JSON, and at digesting such lists from others. The latter would implement the ‘federated’ part of federated bookshelves. Right now I just point to other people’s list and rss feeds in my ‘list of lists‘. To me getting to federation doesn’t require a ‘standard’. Because JSON, OPML and e.g. schema.org have enough specificity and overlap between them to allow both publishers of lists and parsers or such lists enough freedom to use or discard data fields as they see fit. But there is definitely a discussion to be had on identifying that overlap and how to use it best. Chris Aldrich is planning an IndieWeb event on this and other personal libraries related topics next month. I look forward to participating in that, quite a number of interesting people have expressed interest, and I hope we’ll get to not just talk but also experiment with book lists.

TIL: In an xml style sheet, an xslt file, testing for the existence of an attribute is not the same as testing whether an attribute has content.

I have a machine readable file (OPML) with lines such as

<outline type="book" text="title by author" name="book title" author="author name" url="" comment="some remarks" authorurl="" inLanguage="en" />

I want to show that file in a browser for people to see, and I want to test for the attribute url in that line so that I can mark up the text with a web link in the human readable version.

What I first used to test for the presence of a link was:

<xsl:when test="@url">

I assumed this would test for the actual presence of a value for attribute ‘url’. But that is not correct. The statement actually tests whether the attribute ‘url’ is present in the line of machine readable code.

To test if the url attribute also has content in it, e.g. url=”https://weblink.tld/page” and not url=”” I need to check that the field is not empty:

<xsl:when test="@url!=''">

I experimentally create OPML lists of books I read (see why and how). For instance I have a ‘fiction 2021’ list. OPML files are meant to be read by machines, but I use the same file to make a human readable version. In the human readable version I want to link to a book and to an author if I have a direct link to a page about the book (by the publisher or author) and a link to the author’s website. I test for non-empty url attributes so I know when to construct a link in the human readable output. You can download the XSLT I use to do that and of course the OPML file that lists the fiction books for 2021 (also see right hand side column, in the feeds section).

Posted on
by

Writing it down may help in getting out of the loop…

I’m continuing my tinkering with federated bookshelves, for which I made an OPML based way of publishing both lists of books, as well as point to other people’s lists and to RSS feeds of content about books. I now changed my XSL style sheet to parse my OPML files to be able to also parse mentions of RSS feeds.

Meanwhile I read Matt Webb’s posting on using RSS (and OPML) a few more times, and I keep thinking, “yes, but where do you leave the actual data?”
Then I read Stephen Downes’ recent posting on distributing reading material and entire books for courses through RSS, and realised it gave me the same sense of not sounding quite right, like Matt’s posting. That feeling probably means I’m not fully understanding their argument.

RSS is a by design simple XML format as a way to syndicate web content, including videos and podcasts. Content is an important word here, as is syndication: if you have something where new material gets added regularly, an RSS feed is a good way to push it out to those interested.
OPML is another by design simple XML format as a way to share outlines. Outlines are content themselves, and outlines can contain links to other content (including further outlines). One of the common uses of OPML is to share a list of RSS feeds through it, ‘these are the blogs I follow’.

In Matt’s and Stephen’s posts I think there are examples that fail to satisfy either the content part of RSS, or the syndication of new content part. In Matt’s case he talks about feeds of postings about books, like my book category in this site, which is fine, but also in terms of lists of books, which is where I struggle: a list doesn’t necessarily list pieces of content, let alone pieces of web content which RSS seems to require. It more likely is just a list. At the same time he mentions OPML as ‘library’, to use to point to such lists of books. Why would you use OPML for the list of lists, but not for the lists themselves, when those book lists themselves have no content per book, only a number of data attributes which aren’t the content items but only descriptions of items? And when the whole point of OPML outlines is branching lists? When a library isn’t any different from a list, other than maybe in size? Again it is different for actual postings about books, but you can already subscribe to those feeds as existing rivers of content, and point to those feeds (in the same OPML, as I do in my experimental set-up now as well).
In Stephen’s posting he talks about providing the content of educational resources through RSS. He suggests it for the distribution of complete books, and for course material. I do like the idea of providing the material for a course as a ‘blob’. We’re talking about static material here, a book is a finished artefact. Where then is the point in syndication through RSS (other than maybe if the book is a PDF or EPUB or something that might be an enclosure in a RSS feed)? Why not provide the material from its original web source, with its original (semantic) mark-up? Is it in any way likely that such content is going to be read in the same tool the RSS feed is loaded into? And what is the ‘change’ the RSS feed is supposed to convey here, when it’s a one-off distribution and no further change beyond that moment of distribution is expected?

OPML outlines can have additions and deletions, though at a slower pace than e.g. blogs. You could have an RSS feed for additions to an OPML outline (although OPML isn’t web content). But you could also monitor OPML outlines themselves for changes (both additions and deletions) over time. Or reload and use the current version as is, without caring about the specific changes in them.

The plus side of OPML and RSS is that there are many different pieces of code around that can deal with these formats. But most won’t be able to deal as-is with adding data attributes that we need to describe books as data, but aren’t part of the few basic mandatory attributes RSS and OPML are expected to contain. Both RSS and OPML do allow for the extension of attributes, if you follow existing name spaces, such as e.g. schema.org’s for creative works, which seems applicable here (both for collections of books, i.e. a shelf or a library, as well as books themselves). If the use of RSS (and OPML for lists of RSS files) is suggested because there’s an existing eco-system, but we need to change it in a way that ensures the existing ecosystem won’t be able to use it, then where’s the benefit of doing so? To be able to build readers and to build OPML/RSS creators, it is useful to be able to re-use existing bits and pieces of code. But is that really different from creating ones own XML spec? At what point are our adaptations to overcome the purposeful simplicity of OPML and RSS destroying the ease of use we hope to gain from using that simplicity?

Another thing that I keep thinking about is that book lists (shelves, libraries) and book data, basically anything other than web published reviews of books, don’t necessarily get created or live on the web. I can see how I could easily use my website to create OPML and RSS feeds for a variety book lists. But it would require me to have those books and lists as content in my website first, which isn’t a given. Keeping reading lists, and writing reading notes, are part of my personal knowledge management workflow, and it all lives in markdown textfiles on my local harddrive. I have a database of e-books I own, which is in Calibre. I have an old database of book descriptions/data of physical books I owned and did away with in 2012, which is in Delicious Library. None of that lives on the web, or online in any form. If I am going to consistently share bookshelves/lists, then I probably need to create them from where I use that information already. I think Calibre has the ability to work with OPML, and has an API I could use to create lists.
Putting that stuff first into my website in order to generate one some or all of XML/OPML/RSS/JSON from it there, is work and friction I don’t want. If it is possible to automatically put it in my website from my own local notes and databases, that is fine, but then it is just as possible to automatically create all the XML/OPML/RSS/JSON stuff directly from those local notes and databases as well. Even if I would use my website to generate sharable bookshelves, I wouldn’t work with other people’s lists there.

I also think that it is very unlikely that a ‘standard’ emerges. There will always be differences in how people share data about books, because of the different things they care about when it comes to reading and books. Having namespaces like schema.org is useful of course, but I don’t expect everyone will use them. And even if a standard emerges, I am certain there will be many different interpretations thereof in practice. It is key to me that discoverability, of both people sharing book lists and of new to me books, exists regardless. That is why I think, in order to read/consume other people’s lists, other than through the human readable versions in a browser/reader, and to tie them into my information filtering and internal tools/processes, I likely need to have a way to flexibly map someone else’s shared list to what I internally use.

I’m not sure where that leaves me. I think somewhere along these lines:

  • Discovery, of books and people reading them, is my core aim for federation
  • OPML seems useful for lists (of lists)
  • RSS seems useful for content about books
  • Both depend on using specific book related data attributes which will have limited standardisation, even if they follow existing namespaces. It is impossible to depend on or assume standardisation, something more flexible is needed
  • My current OPML lists points to other lists by me and others, and to RSS feeds by me and others
  • I’m willing to generate OPML, RSS and JSON versions of the same lists and content if useful for others, other than templating there’s no key difference after all
  • Probably my website is not the core element in creating or maintaining lists. It is for publishing things about books.
  • I’m interested in other people’s RSS feeds about books, and will share my list of feeds I follow as OPML
  • I need to figure out ways to create OPM/RSS/JSON etc directly from where that information now lives in my workflow and toolset
  • I need to figure out ways to incorporate what others share with me into my workflow and toolset. Whatever is shared through RSS already fits existing information strategies.
  • For a limited number of sources shared with me by others, it might make sense to create mappings of their content to my own content structures, so I can import/integrate them more fully.

Related postings:
Federated Bookshelves (April 2020)
Federated Bookshelves Revisited (April 2021)
Federated Bookshelves Proof of Concept (May 2021)
Booklist OPML Data Structure (May 2021)

Bookmarked This is Fine: Optimism & Emergency in the P2P Network by Cade Diehm

Another good find by Neil Mather for me to read a few times more. A first reaction I have is that in my mind p2p networks weren’t primarily about evading surveillance, evading copyright, or maintaining anonymity, but one of netwerk-resilience and not having someone with power over the ‘off-switch’ for the entire network. These days surveillance and anonymity are more important, and should gain more attention in the design stage.

I find it slightly odd that the dark web and e.g. TOR aren’t mentioned in any meaningful way in the article.

Another element I find odd is how the author talks about extremists using federated tools “Can or should a federated network accept ideologies that are antithetical to its organic politics? Regardless of the answer, it is alarming that the community and its protocol leadership could both be motivated by a distrust of centralised social media, and be blindsided by a situation that was inevitable given the common ground found between ideologies that had been forced from popular platforms one way or another.”
It ignores that with going the federated route extremists loose two things they enjoyed on centralised platforms: amplification and being linked to the mainstream. In a federated setting I with my personal instance, and any other instance decides themselves whom to federate with or not. There’s nothing for ‘a federated network to accept’, each instance does their own acceptance. There’s no algorithmic rage-engine to amplify the extreme. There’s no standpoint for ‘the federated network’ to take, just nodes doing their own thing. Power at the edges.

Also I think that some of the vulnerabilities and attack surfaces listed (Napster, Pirate Bay) build on the single aspect in that context that still had a centralised nature. That still held some power in a center.

Otherwise good read, with good points made that I want to revisit and think through more.

…driven by the desire for platform commons and community self-determination. These are goals that are fundamentally at odds with – and a response to – the incumbent platforms of social media, music and movie distribution and data storage. As we enter the 2020s, centralised power and decentralised communities are on the verge of outright conflict for the control of the digital public space. The resilience of centralised networks and the political organisation of their owners remains significantly underestimated by protocol activists. At the same time, the decentralised networks and the communities they serve have never been more vulnerable. The peer-to-peer community is dangerously unprepared for a crisis-fuelled future that has very suddenly arrived at their door.

Cade Diehm