17 reactions on “Booklist OPML Data Structure

Wiki

opml

”

1. After I wrote about federated bookshelves again two weeks ago, I decided to build a proof of concept. A proof of concept for providing an OPML file that contains a list of books, in a way that can be parsed by others. I roughly follow Tom Critchlow’s “spec”.
   Because I am making up my own data attributes (although I follow schema.org where possible), I decided to not just create an OPML list, but also to add an XSL template so that OPML is not just machine readable but also human readable in a browser.
   The general idea is I have a list that contains lists of books. A list of books can contain books directly, or only be a link to that list of books. A list of books can be one of my own lists on my own domain, or it can be a list published by someone else on a different web address. This allows me to point to other people’s lists when it is somehow relevant to me.
   A book in a list I provided with data attributes like title, author and urls for the book and author, and again fields pointing to other people, like the url for the list I may have found the book, or the url for the person / review which was my recommendation.
   This allows discovery for both you and me. It makes it work like social software: in triangles, where you can navigate from a person, to a piece of content (a book or list), and to a piece of metadata which is itself the url of another list, or the url of another person, that then have their own metadata pointing to others etc.
   Because my lists are structured opml, I should be able to automatically create list files from my own book notes.
   Let’s have a look at the proof of concept:
   I have an OPML file, called ‘books.opml‘

   At the top of that screenshot you can see the opml file calls a XSL stylesheet, named test. I created it by adapting the similar set-up I have for my OPML blogroll. Because of that stylesheet the book list is human readable in a browser and looks like this:

   What you see is first some info about me as the creator of the list. It has a link to this list, which is my main list, and a link to my site.
   Below it is a list of book lists.

   The first of those book lists, called ‘Fiction I read in 2021’ doesn’t have its own url, and the books are shown directly. Those books may have a link to the book, to the author, some notes, or a link to who recommended it to me, or in which list I found it. It also has a short list description at the top.
   Underneath it is another list by me, called ‘Current non-fiction anti-library’, that is just linked.
   Underneath that are two lists, created by Tom Critchlow and Tom Macwright, both of which are just links. The list item in the outline has an author attribute, and if it’s not my name it gets shown as a ‘followed’ book list. Theoretically if an external link is an OPML file itself, I could include it and show it right here.
   Now if you click my other book list, the anti-library list (read here what an anti-library is), that is another opml file.

   It calls the same XSL stylesheet as the other list, and renders in the browser as

   What you can see in this second list is that it starts with the same link to the main list and info about me, and then shows how the list itself has both a URL and a description. It contains books, and see how one of those books has a link to the book itself. (I don’t link to Amazon or Goodreads, so won’t have links for most books, only if there’s a link to the publisher’s or author’s page on that book)
   In the description of the last book you also see how it has a link to the list (Tom Macwright’s) in which I found it, as well as a link to a posting that served as recommendation for me.
   For now what I like is that these lists bring their own viewer with it (your browser using my stylesheet), and can point to other people’s lists both directly as a list in my own collection, or as a reference for one of the books. Without having to make assumptions about other people’s lists or parse them somehow, it still allows connecting to them (federating), and discovery. My own lists use schema.org terms for book lists (collection) and books (book). Three attributes I cannot place within schema.org terms, at least not without adding additional subnodes in the outline: author url, reference list (url of the list I found a book in), and reference url (the person or posting that recommended the book to me).
   UPDATE: I described the data structure for booklists I used.

2. After I built a proof of concept of using OPML to share and federate book lists yesterday (UPDATE: description of the data structure for booklists), Tom Chritchlow asked me about subscribing to OPML lists in the comments. I also reread Matt Webb’s earlier posting about using OPML and RSS for book lists.
   That results in a few remarks and questions I’d like to make and ask:

   OPML serves 2 purposes

   In the words of Dave Winer, opml’s creator, OPML is meant as a “transparently simple, self-documenting, extensible and human readable format that’s capable of representing a wide variety of data that’s easily browsed and edited” to create and manipulate outlines, i.e. content structured hiearchically / tree-like.
   the format is a way to exchange such outlines between outliner tools.

   In other words OPML is great for making (nested) lists, and for exchanging them. I use outlines to build my talks and presentations. It could be shopping lists like in Doug Engelbart’s 1968 ‘mother of all demos’. And indeed it can be lists of books.
   A list I regard as an artefact in itself. A list of something is not just iterating the somethings mentioned, the list itself has a purpose and meaning for its creator. It’s a result of some creative act, e.g. curation, planning, writing, or desk research.
   A book list I regard as a library, of any size. The list can be as short as the stack on my night reading table is high, as long as a book shelf in my home is wide, or as enormous as the full catalogue of the Royal Library. Judging by Tom Critchlow’s name for his booklist data ‘library.json‘ he sees that similarly.
   A book list, as I wrote in my posting about the proof of concept, can have books in them, and other book lists by myself or others. That is where the potential for federation lies. I can from a book point to Tom’s list as the source of inspiration. I could include one of Tom’s booklists into my own booklists.
   A list of books is different from a group of individual postings about books as also e.g. presented on my blog’s reading category page. I blog about books I read, but not always. In fact I haven’t written any postings at all this year, but have read 25 books or so since January 1st. It is easier to keep a list of books, than to write postings about each of the books listed. This distinction is expressed too in Tom Macwright’s set-up. There’s a list of books he’s read, which points to pages with a posting about an entry in that list, but the list is useful without those postings.
   The difference between booklists as artefacts and groups of postings about books that may also be listed has impact on what it means to ‘subscribe’ to them.

   A book list, though it can change over time, is a steady artefact. Books may get added or removed just like in a library, but those changes are an expression of the will of its maker, not a direct function of time.
   My list of blogsposts about books, in contrast is fully determined by time: new entries get added on top, older ones drop off the list because the list has a fixed length.
   OPML is very suited for my lists as artefacts
   RSS is very suited for lists as expression of time, providing the x most recent posts
   Subscribing to RSS feeds is widely available
   Subscription is not something that has a definition for OPML (that you can use OPML to list RSS subscriptions may be confusing though)

   Inclusion however is a concept in OPML: I can add a list as a new branch in another list. If you do that once you only clone a list, and go your own seperate way again. You could also do it dynamically, where you always re-import the other list into your own. Doing it dynamically is a de-facto subscription. For both however, changes in the imported list are non-obvious.
   If you keep a previously seen copy and compare it to the current one, you could monitor for changes over time in an OPML list (Inoreader did that in 2014 so you could see and subscribe to new RSS feeds in other people’s OPML feed lists, also see Marjolein Hoekstra’s posting on the functionality she created.).

   I am interested in both book lists, i.e. libraries / bookshelves, the way I am interested in browsing a book case when I visit somebody’s home, and in reading people’s reviews of books in the form of postings. With OPML there is also a middle ground: a book list can for each book include a brief comment, without being a full review or opinion. In the shape of ‘I bought this because….’ this is useful input for social filtering for me.
   While interested in both those types, libraries, and reviews, I think we need to treat them as completely different things, and separate them out. It is fine to have an OPML list of RSS feeds of reviews, but it’s not the same as having an OPML book list, I think.
   I started at the top with quoting Dave Winer about OPML being a “simple, self-documenting, extensible and human readable format that’s capable of representing a wide variety of data that’s easily browsed and edited“. That is true, but needs some qualification:

   While I can indeed add all kinds of data attributes, e.g. using namespaces and standardised vocabularies like schema.org, there’s no guarantee nor expectation that any OPML parser/reader/viewer would do anything with them.
   This is the primary reason I used an XSL template for my OPML book lists, as it allows me to provide a working parser right along with the data itself. Next to looking at the raw file content itself, you can easily view in a browser what data is contained in it.
   In fact I haven’t seen any regular outliner tool that does anything with imported OPML files beyond looking at the must have ‘text’ attribute for any outline node. Tinderbox, when importing OPML, does look also at URL attributes and a few specific others.
   I know of no opml viewer that shows you which attributes are available in an OPML list, let alone one that asks you whether to do something with them or not. Yet exploring the data in an OPML file is a key part of discovery of other people’s lists, of the aim to federate booklists, and for adopting better or more widely shared conventions over time.
   Are there generic OPML attribute explorers, which let you then configure what to pay attention to? Could you create something like an airtable on the fly from an OPML list?
   Monitoring changes in OPML list you’re interested in is possible as such, but if OPML book lists you follow have different structures it quickly becomes a lot of work. That’s different from the mentioned Inoreader example because OPML lists of RSS feeds have a predefined expected structure and set of attributes right in the OPML specification.
   Should it be the default to provide XSL templates with OPML files, so that parsing a list as intended by the creator of the list is built right into the OPML list itself?
   Should we ‘dumb down’ lists by moving data attributes of an outline node to a sub-node each? You will reduce machine readability in favor of having basic OPML outliners show all information, because there are no machines reading everything yet anayway.

   I think for the coming weeks I’ll be on the lookout for sites that have book lists and book posting feeds, to see what commonalities and differences I find.

3. Saw it, will use it, thank you—and know OPML from waaaaay back in the RSS wars… I mean days… / I saw that Dave W. already jumped into the mentions too. 🙂
   
   
   

4. Writing it down may help in getting out of the loop…
   I’m continuing my tinkering with federated bookshelves, for which I made an OPML based way of publishing both lists of books, as well as point to other people’s lists and to RSS feeds of content about books. I now changed my XSL style sheet to parse my OPML files to be able to also parse mentions of RSS feeds.
   Meanwhile I read Matt Webb’s posting on using RSS (and OPML) a few more times, and I keep thinking, “yes, but where do you leave the actual data?”
   Then I read Stephen Downes’ recent posting on distributing reading material and entire books for courses through RSS, and realised it gave me the same sense of not sounding quite right, like Matt’s posting. That feeling probably means I’m not fully understanding their argument.
   RSS is a by design simple XML format as a way to syndicate web content, including videos and podcasts. Content is an important word here, as is syndication: if you have something where new material gets added regularly, an RSS feed is a good way to push it out to those interested.
   OPML is another by design simple XML format as a way to share outlines. Outlines are content themselves, and outlines can contain links to other content (including further outlines). One of the common uses of OPML is to share a list of RSS feeds through it, ‘these are the blogs I follow’.
   In Matt’s and Stephen’s posts I think there are examples that fail to satisfy either the content part of RSS, or the syndication of new content part. In Matt’s case he talks about feeds of postings about books, like my book category in this site, which is fine, but also in terms of lists of books, which is where I struggle: a list doesn’t necessarily list pieces of content, let alone pieces of web content which RSS seems to require. It more likely is just a list. At the same time he mentions OPML as ‘library’, to use to point to such lists of books. Why would you use OPML for the list of lists, but not for the lists themselves, when those book lists themselves have no content per book, only a number of data attributes which aren’t the content items but only descriptions of items? And when the whole point of OPML outlines is branching lists? When a library isn’t any different from a list, other than maybe in size? Again it is different for actual postings about books, but you can already subscribe to those feeds as existing rivers of content, and point to those feeds (in the same OPML, as I do in my experimental set-up now as well).
   In Stephen’s posting he talks about providing the content of educational resources through RSS. He suggests it for the distribution of complete books, and for course material. I do like the idea of providing the material for a course as a ‘blob’. We’re talking about static material here, a book is a finished artefact. Where then is the point in syndication through RSS (other than maybe if the book is a PDF or EPUB or something that might be an enclosure in a RSS feed)? Why not provide the material from its original web source, with its original (semantic) mark-up? Is it in any way likely that such content is going to be read in the same tool the RSS feed is loaded into? And what is the ‘change’ the RSS feed is supposed to convey here, when it’s a one-off distribution and no further change beyond that moment of distribution is expected?
   OPML outlines can have additions and deletions, though at a slower pace than e.g. blogs. You could have an RSS feed for additions to an OPML outline (although OPML isn’t web content). But you could also monitor OPML outlines themselves for changes (both additions and deletions) over time. Or reload and use the current version as is, without caring about the specific changes in them.
   The plus side of OPML and RSS is that there are many different pieces of code around that can deal with these formats. But most won’t be able to deal as-is with adding data attributes that we need to describe books as data, but aren’t part of the few basic mandatory attributes RSS and OPML are expected to contain. Both RSS and OPML do allow for the extension of attributes, if you follow existing name spaces, such as e.g. schema.org’s for creative works, which seems applicable here (both for collections of books, i.e. a shelf or a library, as well as books themselves). If the use of RSS (and OPML for lists of RSS files) is suggested because there’s an existing eco-system, but we need to change it in a way that ensures the existing ecosystem won’t be able to use it, then where’s the benefit of doing so? To be able to build readers and to build OPML/RSS creators, it is useful to be able to re-use existing bits and pieces of code. But is that really different from creating ones own XML spec? At what point are our adaptations to overcome the purposeful simplicity of OPML and RSS destroying the ease of use we hope to gain from using that simplicity?
   Another thing that I keep thinking about is that book lists (shelves, libraries) and book data, basically anything other than web published reviews of books, don’t necessarily get created or live on the web. I can see how I could easily use my website to create OPML and RSS feeds for a variety book lists. But it would require me to have those books and lists as content in my website first, which isn’t a given. Keeping reading lists, and writing reading notes, are part of my personal knowledge management workflow, and it all lives in markdown textfiles on my local harddrive. I have a database of e-books I own, which is in Calibre. I have an old database of book descriptions/data of physical books I owned and did away with in 2012, which is in Delicious Library. None of that lives on the web, or online in any form. If I am going to consistently share bookshelves/lists, then I probably need to create them from where I use that information already. I think Calibre has the ability to work with OPML, and has an API I could use to create lists.
   Putting that stuff first into my website in order to generate one some or all of XML/OPML/RSS/JSON from it there, is work and friction I don’t want. If it is possible to automatically put it in my website from my own local notes and databases, that is fine, but then it is just as possible to automatically create all the XML/OPML/RSS/JSON stuff directly from those local notes and databases as well. Even if I would use my website to generate sharable bookshelves, I wouldn’t work with other people’s lists there.
   I also think that it is very unlikely that a ‘standard’ emerges. There will always be differences in how people share data about books, because of the different things they care about when it comes to reading and books. Having namespaces like schema.org is useful of course, but I don’t expect everyone will use them. And even if a standard emerges, I am certain there will be many different interpretations thereof in practice. It is key to me that discoverability, of both people sharing book lists and of new to me books, exists regardless. That is why I think, in order to read/consume other people’s lists, other than through the human readable versions in a browser/reader, and to tie them into my information filtering and internal tools/processes, I likely need to have a way to flexibly map someone else’s shared list to what I internally use.
   I’m not sure where that leaves me. I think somewhere along these lines:
   Discovery, of books and people reading them, is my core aim for federation
   OPML seems useful for lists (of lists)
   RSS seems useful for content about books
   Both depend on using specific book related data attributes which will have limited standardisation, even if they follow existing namespaces. It is impossible to depend on or assume standardisation, something more flexible is needed
   My current OPML lists points to other lists by me and others, and to RSS feeds by me and others
   I’m willing to generate OPML, RSS and JSON versions of the same lists and content if useful for others, other than templating there’s no key difference after all
   Probably my website is not the core element in creating or maintaining lists. It is for publishing things about books.
   I’m interested in other people’s RSS feeds about books, and will share my list of feeds I follow as OPML
   I need to figure out ways to create OPM/RSS/JSON etc directly from where that information now lives in my workflow and toolset
   I need to figure out ways to incorporate what others share with me into my workflow and toolset. Whatever is shared through RSS already fits existing information strategies.
   For a limited number of sources shared with me by others, it might make sense to create mappings of their content to my own content structures, so I can import/integrate them more fully.
   Related postings:Federated Bookshelves (April 2020)Federated Bookshelves Revisited (April 2021)Federated Bookshelves Proof of Concept (May 2021)Booklist OPML Data Structure (May 2021)

5. @ton @dredmorbius I see now, thanks!

6. @ton I’m a bit confused / unclear on what your workflow / sources / tools / process / product(s) / goal(s) are here.Yes, Pandoc is a conversion tool, which converts to and from numerous document-ish formats. It is often used with Markdown as an input format, but by no means exclusively, and supports numerous other light-weight and heavy-weight markup languages, as well as numerous application-specific formats, for both input and output.(Input and output formats vary somewhat, be aware.)Pandoc is immensely flexible and powerful, and I strongly recommend reading at least an overview. The project documentation is excellent.Pandoc is not what you use to author or create a document, but once you’ve created one through the editor (or process) of your choice, you can then flexibly convert between formats with Pandoc.If you’re looking to outline directly from an existing source (e.g., notetaking from a book or article), could you specify that a bit more clearly?Otherwise, I’m assuming you’re using an editor to create an outline and can annotate that with the specific attributes you’d like to output. Again, depending on the input markup used, you should have quite extensive flexibility, and Pandoc itself can be further extended through templates, preprocessors, and postprocessors.I’m familiar with Linux / Unix and its toolchains, mostly command-line / terminal-based tools, and prefer those. If you prefer other tools, please specify. If you want specific suggestions on the Linuxy / Unixy side, I can make suggestions or discuss options.@edsu

7. Today at 14:07 it is exactly 19 years ago I published the first post on this blog. Back then I already mention how connecting to others, conversation, is the key thing I’m aiming for. I’ve always been a prolific note maker (going back to primary school even, buying my own notepads). With the launch of my weblog it became a more public thing as well as a means to engage with others.
   In recent years I’ve marked the occasion by reflecting on my blogging and practices (see the 18, 17, 16 years edition), and long ago I marked the 3rd and 5th anniversary both extolling the value of the conversations and connections this blog helped create.
   This year, as most of last year was spent working from home. It meant a similar internal oriented focus when it comes to my note making and blogging.
   I haven’t spend time on IndieWeb community organising for instance, didn’t feel the energy for it either. I did make steps towards making this blog much less dependent on third parties:

   I stopped embedding Flickr images in my blog, replacing them with locally hosted copies while linking to the original. Most postings now no longer have Flickr embeds, some 150 still do, which I am slowly bringing down to 0.
   I removed all video embeds, replacing them with stills and links
   I slowly replaced a number of Slideshare decks, but not all yet. There are no actual slideshare embeds active anymore on my blog, as I deleted my account, but the now non-functional embeds still ‘call’ those web adresses. I’m self-hosting my slides on tonz.nl (Dutch), and tonz.eu (English)

   I experimented with sharable bookshelves for my blog, but there’s a connection missing with my internal note taking. I’d very much like to directly generate my book lists and book posts directly from my own notes. I haven’t actually posted about books here since January, a fact I dislike.
   That brings me to the note making part. I have completely removed myself from Evernote, replacing it with a local collection of notes in markdown. I’ve kept them separate of the notes collection I actually work with, but import specific notes when I need them. I also, based on an example from fellow Obsidian user Wouter Groeneveld, started scanning my paper notebooks from over the years, creating indexes for them, and thus making them connect to my ongoing work and notes. My use of Obsidian to maintain those markdown notes continues undiminished. The speed of creating new conceptual nodes has slowed a lot, having mined most of my old blogposts for their content. I am now slowly evolving my ways of digesting and adding new knowledge and thoughts. In terms of volume, there are now some 5k notes, of which 1k6 are conceptual, 1k are ‘collected stuff’ with just a few added remarks of why I find them interesting, and some 2k5 work related notes.
   In general I would like to see a more direct connection between my notes and my blogging, and ‘wiki’ pages on this site. I’m not sure yet what I’d like so I need to experiment. In the past months I have been contributing to two GitHub hosted sites using Respec, where the site is directly created from my notes. This works really well, but as those are public pages I do keep the corresponding notes in a different place than my ‘real’ notes. I do want to maintain the difference between public and private, as it influences my writing, but I do not necessarily want to keep the public notes in a separate location from the others.
   Coincidentally, around note making, I did do some outreach and hosted two ‘Dutch language Obsidian user meet-ups‘. The third is due to take place in two weeks.
   For the coming time this note-to-blog pipeline, and making it easier for myself to post, will be my area of attention I think. Let’s see next year around this time, when I hit the two decade mark with this blog, how that went.
   How I took notes in 2006, on a locally hosted wiki

8. Drummers by Alper Çuğun, license CC BY
   Drummer
   Drummer is a new outliner tool for blogging launched last month, created by Dave Winer. As it is popping up in various places, connections are built to other tools (like Microblog), I found myself rolling the topic of outlines around in my head.
   I have an somewhat ambivalent attitude towards outlining. Actually I need to split that up in being ambivalent on outlining, on outliners, and on OPML, the standard for exchanging outline files. Some remarks on all three things.
   Outlines
   Outlines are very useful. I use them for braindumps, idea generation, project plannning and design, and when making the storyline for my presentations. They’re great because you can quickly write out many points and then start shifting things around, changing the order, placing items in branches (or chapters) etc.
   Outlines are limited as well, because they are hierarchical and linear in nature. This is similar for me with mind maps. They require a beginning and an end, main or central points with sub points etc. Even though you could link or include other outlines on a branch it still is just another part of a tree structure. A large amount of the information I work with is not like that, and a lot of my work processes (especially those where structure needs to emerge from the information, not to be applied to information) are not like that. Stuff is always linked, but those links can loop back, unlike in outlines. In my description of my personal knowledge management system last year I wrote about the distinction between the parts of it that are organised as networks, and the parts that are hierarchical. This is the same thing, at a lower level of aggregation.
   Fast Drummer by Hsing Wei, license CC BY
   Outliners
   Outliners are great tools, and I use them regularly. When done well you can seamlessly move items around, changing the order, nesting them, or moving them several levels up. Dave Winer talks about moving things around ‘on rails’, and indeed that is what it feels like in a well working outliner, an almost frictionless rearranging of things.
   I have used a variety of outliner tools over the years. E.g. I used to make my presentation outlines in Cloudliner, which could sync with Evernote. Once the outline was there, I’d move to Evernote to flesh out the story in more detail.
   Currently I mostly work in Obsidian, which isn’t a full outliner in the sense that it lacks the ‘on rails’ features, although by now it is way better at outlining than earlier through the use of hotkeys. (See this video by Nick Milo on outlining with Obsidian)
   Tinderbox is also an outliner I use, which is also able to work non-hierarchically: I can start there with adding notes visually, and then switching to an outline view of the same information to do the ordering and branching I want.
   Where outliners are less great in my eyes is how they generally imply that all outlines are glorified shopping lists. When writing anything it is about creating prose. Sentences need to flow into each other. Outliners in their interface however suggest that every item in an outline is short. A brief statement at most, definitely not something like a full sentence or even a paragraph. This is where for me friction originates, outliner UIs imply bullet lists. The mental model of a bullet list clashes with that of a text, even if well structured. I don’t think outlines need to be bullet lists, but outliner tools apparently do. Obsidian is the only exception, as it works fully in notes, and you can mix up longer pieces of prose with lists of short items, and have bullets as long as a novel if you want. (This is what I gain for Obsidian not having the ‘on rails’ experience)
   Even the original demonstration of an outliner, in Doug Engelbart’s famous 1968 demo, shows that clash to me. It demonstrates the power of outliners: rearranging items at will, moving an item to become a sub-item or a sub-item to become a main item, stacking lists multiple levels deep, as well as adding the link to another outline as item in an outline and it being navigable. But the content of the outline demonstrated is short, a shopping list and a task list.
   Drummers by George N, license CC BY
   OPML
   Dave Winer has been developing outliners for a long time, and we also have to thank him for the OPML standard, meant to exchange outlines in XML. Most people that use RSS readers are familiar with OPML because it is the common format in which you can export and import lists of RSS subscriptions.
   Outliners are usually capable of exporting OPML. Most outliner tools however only export to OPML, and don’t store them in files that way or in some other text based format (for the ‘on rails’ effect they regularly keep outlines not as files but in an internal database. Obsidian works on files, and thus doesn’t have the ‘on rails’ style). This means there’s no seamless transition from an outliner tool to another tool, or vice versa, nor a good way to switch between different text based formats such as markdown or OPML. Drummer is an exception, it natively stores everything in OPML files.
   Another issue I have with outliners is in how they deal with importing OPML. OPML is an extensible format, which allows adding data attributes to the text information in the outline. This allows me to attach meaning to content that is machine readable. It’s what I did for my book lists for instance. Outliners however in general never attempt to check if such data attributes are present. OPML is short for Outline Processor Markup Language, yet outliners never do some actual processing upon import.
   I don’t expect general outliners to be able to do something with such attributes, but I would expect general outliners to at least alert me to their presence in an import, and if possible ask me if I’d like to explore them or do something with them. That general outliners only look at the mandatory parts of the OPML file means they never even look if there’s other semantic information present in the OPML file, though the standard supports it.
   Yes, you could create your own outliner tool that reads specific additional data attributes but no regular user would be able to. Tinderbox that I mentioned, allows me to set a wide variety of attributes to notes, and I can create an OPML template in Tinderbox that includes (parts of) them in OPML exports. As far as I can tell though it doesn’t support templating OPML imports. Without this there’s no chance of an OPML using ecosystem evolving, and there hasn’t. The lack of interoperability means novel use cases for OPML always need to come with their own bespoke outliner. This is why I add XSLT to my OPML files for RSS subscriptions and for books, basically packaging a reader right inside the file: it makes them human readable in their entirety and independent from outliners.
   Hear The Drummer Get Wicked
   I think this is how Dave blogs: He opens up an outline at the start of the day and adds items (thoughts, annotated links, bookmarks, comments) to it during the day which get pushed to his blog. Every line has its own permalink, but it’s a single post for the day, evolving during the day and fixed thereafter. I like the ease of use I can imagine that brings to him.
   This is much like how I create my day logs in my personal notes. I open it up first thing behind my laptop in the morning, and during the day it’s my jumping off point as well as where I write things down first. What gets bigger or has more permanent value is then split off in its own note, with the note linked in the day log it sprouted from. (I create my weekly notes on this blog from collating the day logs and then picking the things I want to mention from). I wouldn’t post my day logs though, they’re not fit for publication.
   Does Dave also have a personal day log, or is that another branch on his daily outline that doesn’t get published?
   The reduction of friction between note making and blogging Dave Winer’s workflow suggests sounds valuable though, and Drummer therefore a tool to watch.
   Drummer by Jonas Bengtsson, license CC BY

9. TIL: In an xml style sheet, an xslt file, testing for the existence of an attribute is not the same as testing whether an attribute has content.
   I have a machine readable file (OPML) with lines such as
   <outline type="book" text="title by author" name="book title" author="author name" url="" comment="some remarks" authorurl="" inLanguage="en" />
   I want to show that file in a browser for people to see, and I want to test for the attribute url in that line so that I can mark up the text with a web link in the human readable version.
   What I first used to test for the presence of a link was:
   <xsl:when test="@url">
   I assumed this would test for the actual presence of a value for attribute ‘url’. But that is not correct. The statement actually tests whether the attribute ‘url’ is present in the line of machine readable code.
   To test if the url attribute also has content in it, e.g. url=”https://weblink.tld/page” and not url=”” I need to check that the field is not empty:
   <xsl:when test="@url!=''">
   I experimentally create OPML lists of books I read (see why and how). For instance I have a ‘fiction 2021’ list. OPML files are meant to be read by machines, but I use the same file to make a human readable version. In the human readable version I want to link to a book and to an author if I have a direct link to a page about the book (by the publisher or author) and a link to the author’s website. I test for non-empty url attributes so I know when to construct a link in the human readable output. You can download the XSLT I use to do that and of course the OPML file that lists the fiction books for 2021 (also see right hand side column, in the feeds section).

10. As a form of WAB* I’ve made it easier for myself to update my OPML book lists. I created those lists earlier this year as a proof of concept of publishing federated bookshelves. Updating OPML files residing on my hosted webserver is not a fun manual task. Ultimately I want to automate pushing lists from my personal working environment (notes in Obsidian) to my site. Given my limited coding skills I made a first easier step and created a webform in a php script that allows me to add a book to an opml list. It has a drop-down menu for the various OPML lists I keep (e.g. fiction2021, non-fiction2021, currently reading, anti-library), provides the right fields to add the right OPML data attributes, and then writes them to the correct list (each list is a separate file).
    That now works well. Having a way to post to my book lists by submitting a form now, I can take the next step of generating such form submissions to replace manually filling out the form.
    * Work Avoiding Behaviour, a continuation of the SAB, Study Avoiding Behaviour that I excelled in at university. WAB seems to fit very well with the current locked down last days until the end of year. The Dutch terms ‘studie/werk ontwijkend gedrag’ SOG/WOG lend themselves to the verb to ‘sog’ and to ‘wog’. Yesterday when Y asked E what she had been doing today, E said ‘I’ve been wogging’, and I realised I had been too.

11. I have a working proof of concept to take individual book notes from Obsidian, turn them into an OPML list of books, and publish them on this webserver. As I had time off these past days I’ve allowed myself to do some code tinkering, resulting in the set-up shown in the image below.
    A sketch of my set-up, made in Excalidraw within Obsidian. The blue items now exist, the grey items are still to be done.
    The workflow is now as follows:

    Within Obsidian I have made a template for book notes, which has a number of inline data fields (shaped ‘field:: value’). These fields contain the same attributes that I use in my OPML files, using the data structure I made earlier. It also contains one additional field, the booklist it is part of.
    When I first create a new book note I use the template and fill out the inline data fields. If it changes status (to read, reading, read) I update the attributes if needed. Next to those data fields it can contain anything else (e.g. my Kindle hihglights and remarks end up in those book notes too.)
    I can create lists of books in Obsidian using the Dataview plugin, which can find and interpret the inline data fields.
    I can run a PHP script, on my laptop, that iterates through all the files in the folder that contains my book notes. It reads the inline data fields and turns them into OPML lines with the same attributes. It saves it in the correct OPML file using the booklist field. This means that when I move a book in Obsidian from my “anti-library” to the “currently reading” list and then to the “non-fiction 2021” or “fiction 2021” list by changing that single booklist data field, that will get reflected in the OPML as well. The OPML files are saved in a folder, and both human and machine readable.
    I have a second PHP script that also runs locally on my laptop, that iterates through the files in the folder. For each of the .opml files it finds that have changed in the past week, it will get the filename and the file content. It then sends those two data fields (and an access code) to a script on my web server as POST form data.
    The script on my web server accepts POST form data and if the access code is ok, will save the submitted file content using the submitted file name. After that the OPML files on my webserver have the same content as my Dataview overviews within Obsidian, and are fully based on the inline data fields in my individual book notes.

    I’ve tested this flow and it works correctly. There’s one important improvement to still make. It currently goes through all my book notes and creates all opml files anew. I want to change that to start from the recently changed book notes and then generate the corresponding opml files. For now it is fast enough locally to not be an issue though that it iterates through the entire folder of book notes.
    A second step to take is an addition: to render the same information as JSON files. Dave Winer’s OPMLpackage is likely useful here. Early on there was some discussion on which format to use, and I don’t see a need to choose. I’ve created it using my preference, but the same information can be formatted differently in parallel if it aids usage and federation.
    To fully automate this, I still need to set a cron job that calls the first and second local script in turn, every other week or so.
    Now that it all works, I will need to see how it goes in practice when I pick a new book, or finish one.
    I also need to clean up the code (removing the tests I added in various steps) and translate some of the comments in English. Then I can share I’ve shared the scripts on GitHub, so others can use it for inspiration.
    Future steps may include generating book postings in my blog here, directly based on Obsidian notes as well.

12. This Article was mentioned on adhoc.systems

13. This Article was mentioned on maggieappleton.com

14. Bookmarked Interoperable Personal Libraries and Ad Hoc Reading Groups by Maggie Appleton
    I somehow missed Maggie Appleton’s blogpost (bookmarked above) about the IndieWeb pop-up session on personal libraries of a few weeks ago. During the session I found her suggestion for ad-hoc reading clubs very interesting, as an application of having book lists on your site. I first and foremost think about discovery in the context of publishing book lists: if I enjoy your blog, or know you and you share book lists those may contain good suggestions to read. Discovery is also why in my ‘data format‘ for such lists I allow for sharing the URLs of lists of others, as well as share the URL of where I found the recommendation for a specific book. What Maggie Appleton suggests is something else and interesting: what if you could see when several others in your network are also currently reading the same book you are reading, allowing an ad-hoc book reading club for that book. It would require a way to compare lists you follow. My sharing of lists I follow is a useful start for it I think, but you’d add a match detection layer on top of it. Whether that matching needs to take place in my site, I don’t know. To me it feels like a personal tool perhaps, alerting me to other readers, and allowing me to privately think about whether I’d want to form an ad-hoc book club with them at that time.

    An idea I brought to the event and ended up hosting a session on was ad hoc reading groups – discussions and meetups facilitated by our public bookshelves.
    Maggie Appleton

15. I want to make it easy to publish lists of books I am reading and have read, or any other list. And do so without using centralised platforms like e.g. Goodreads (Amazon). A book list is a small library.
    The route I am currently on, is publishing a machine readable list others can easily incorporate. These lists are in OPML, an exchange format for outlines. It’s the same format generally used to share lists of RSS feed subscriptions.
    Current situation and usage: automated lists
    Currently I am able to directly automatically create the lists in OPML from my individual book notes in Obsidian.md (which I use for PKM).
    In Q1 2022 I experienced that creating lists and posting them works nicely and smoothly, with no friction. I do currently only create a few lists (fiction and non-fiction in the running year, antilibrary). I’m also working through the books I’ve read in the last decade or so, and gradually creating those lists. I’m not generating those as OPML however, they currently are just a list in my own notes.
    Next steps: consuming other lists
    Next steps will look at how to do the federating itself: how can I ‘consume’, or even include in my own lists, the OPML, ActivityPub or JSON lists of others in a meaningful way? I think a first step is consuming one list published by someone else, treating it as a recommendation list perhaps or some other form of input, much like I’m reading feeds. It might be useful to be able to pick out mentions about books I’ve already read, are in my anti-library, match an author I like, or match my interests while being unknown to me. I suspect a slightly tweaked parser for every new list might be needed, as using a list depends both on format and on content fields.
    Ealier steps: proof of concept and data model
    In 2020 I came across a posting by Tom Critchlow on this topic, and a year later I started looking into using OPML to create the lists.
    I created a proof of concept, with a data format.
    Using that I created a webform to update a book list by hand with a new entry.
    Then I automated generating the lists (code on GitHub).
    All as proofs of concept.

16. A summary overview of changes I made to this site, to make it more fully a indieweb hub / my core online presence. The set-up of my WordPress installation also has been described.
    Theme related tweaks

    Created child theme of Sempress, to be able to change appearance and functions
    Renamed comments to reactions (as they contain likes, reposts, mentions etc.)
    in the entry-footer template and the comments template
    Removed h-card microformats, and put in a generic link to my about page for the author in the Sempress function sempress_posted_on. Without a link to the author mentions show up as anonymous elsewhere.
    Removed the sharing buttons I used (although they were GDPR compliant using the Sharriff plugin, but they got in the way a lot I felt.
    Added a few menu options for various aspects of my postings (books, check-ins, languages)
    Introduced several categories to deal with different content streams: Dutch, German for non-English postings, Day to Day for things not on the home page, Plazes for check-ins, Books for ehh books, RSS-Only for unlisted postings, and Micromessage for tweets I send from the blog. This allows me to vary how I display these different types of things (or not)
    Displaying last edited and created dates to (wiki)pages
    Added a widget with projects I support
    Added to the single post template a section that mentions and links the number of Hypothes.is annotations for that post, where they exist.

    Functionality related tweaks

    Started creating pages as a wiki-like knowledgebase, using page categories to create the wiki structure
    To show excerpts from webmentions I changed the template for a webmention in the Semantinc backlinks plugin, class-linkbacks-handler.php
    Added a plugin to display blogposts on the same date in previous years.
    Added plugin Widget Context to remove recent posts and comments from individual posting’s pages, as they cause trouble with parsing them for webmentions.
    Using categories as differentiator I added language mark-up to individual postings, category archives. Also added automatic translation links to non-English postings in the RSS feed (not on the site). On the front page non-English postings have language mark-up around the posting.
    Added a blogroll that is an OPML file with a stylesheet, so it can be equally read by humans and machines.
    Added an extra RSS feed for comments that excludes webmentions and ping/trackbacks
    Added a /feeds page
    Added a Now page
    Added a Hello page
    Added a way to share book lists / feeds.
    Stopped embedding slide decks, and stopped embedding new Flickr photos (as well as removed older embeds, currently 23 postings between January 2013 and July 2018 still have them, and 22 postings from June 2006 to July 2009)
    Removed all affilliate links to Amazon books as it entails tracking
    Added an Index (using a plugin)
    Added my own basic check-in and Dopplr style posts

    Other tweaks

    Set up 2 additional WordPress instances for testing purposes (Proto and Meso)

17. I’ve reconfigured my book list automation so it also publishes them to the book section on my blog.
    About a year ago, late 2021, I made it easy for myself to publish lists of books I’ve read in OPML. Before I first made those lists in early 2021 I would publish postings about some of my reading in the book section of this site. Switching to making OPML lists based on my internal notes meant I had stopped posting to the books section. There’s however no reason why I wouldn’t do both. The OPML lists are hardly discoverable, and posting them here as well means they are being shared in RSS too. Yet, I don’t want to do things twice, so tonight I automated it.
    The script that goes through my internal book notes on my laptop to create the OPML lists, now also creates posts for each book in my site. Not for all books, just for book notes that have changed in the last week and are part of the current year’s reading. It builds on my existing personal Micropub client I use to post other things to this site too. Meaning that if I run the script weekly it will automatically post any books I’ve finished that week. Today at the start I set the script to any book read this year, so this year’s reading list is up to date.

Comments are closed.

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Learn More)

Mentions

• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra
• 💬 Ton Zijlstra

Bookmarks

• 🔖 Zachary Dunn

Likes

• 👍 Doc Edward Morbius ⭕​
• 👍 Ed Summers
• 👍 Bopuc