I have a working proof of concept to take individual book notes from Obsidian, turn them into an OPML list of books, and publish them on this webserver. As I had time off these past days I’ve allowed myself to do some code tinkering, resulting in the set-up shown in the image below.
A sketch of my set-up, made in Excalidraw within Obsidian. The blue items now exist, the grey items are still to be done.
The workflow is now as follows:
- Within Obsidian I have made a template for book notes, which has a number of inline data fields (shaped ‘field:: value’). These fields contain the same attributes that I use in my OPML files, using the data structure I made earlier. It also contains one additional field, the booklist it is part of.
- When I first create a new book note I use the template and fill out the inline data fields. If it changes status (to read, reading, read) I update the attributes if needed. Next to those data fields it can contain anything else (e.g. my Kindle hihglights and remarks end up in those book notes too.)
- I can create lists of books in Obsidian using the Dataview plugin, which can find and interpret the inline data fields.
- I can run a PHP script, on my laptop, that iterates through all the files in the folder that contains my book notes. It reads the inline data fields and turns them into OPML lines with the same attributes. It saves it in the correct OPML file using the booklist field. This means that when I move a book in Obsidian from my “anti-library” to the “currently reading” list and then to the “non-fiction 2021” or “fiction 2021” list by changing that single booklist data field, that will get reflected in the OPML as well. The OPML files are saved in a folder, and both human and machine readable.
- I have a second PHP script that also runs locally on my laptop, that iterates through the files in the folder. For each of the .opml files it finds that have changed in the past week, it will get the filename and the file content. It then sends those two data fields (and an access code) to a script on my web server as POST form data.
- The script on my web server accepts POST form data and if the access code is ok, will save the submitted file content using the submitted file name. After that the OPML files on my webserver have the same content as my Dataview overviews within Obsidian, and are fully based on the inline data fields in my individual book notes.
I’ve tested this flow and it works correctly. There’s one important improvement to still make. It currently goes through all my book notes and creates all opml files anew. I want to change that to start from the recently changed book notes and then generate the corresponding opml files. For now it is fast enough locally to not be an issue though that it iterates through the entire folder of book notes.
A second step to take is an addition: to render the same information as JSON files. Dave Winer’s OPMLpackage is likely useful here. Early on there was some discussion on which format to use, and I don’t see a need to choose. I’ve created it using my preference, but the same information can be formatted differently in parallel if it aids usage and federation.
To fully automate this, I still need to set a cron job that calls the first and second local script in turn, every other week or so.
Now that it all works, I will need to see how it goes in practice when I pick a new book, or finish one.
I also need to clean up the code (removing the tests I added in various steps) and translate some of the comments in English. Then I can share I’ve shared the scripts on GitHub, so others can use it for inspiration.
Future steps may include generating book postings in my blog here, directly based on Obsidian notes as well.
Wat een mooie stap Ton! Tof dat je een koppeling tussen Obsidian en OPML weet te maken. Ik denk dat je hiermee zeker in de Drummer community ook waardevol kunt zijn. Er zijn steeds meer bewegingen om lokale PKM systemen te koppelen aan outliners en blogsystemen. Een mooie stap voorwaarts! Ik ben benieuwd naar je scripts.
Hey Ton great to see you pushing this forward. I’m working on a json/opml parser right now actually too!
Making OPML bookshelves directly from my individual #obsidianmd book notes, proof of concept.
Having created a working flow to generate OPML booklists directly from the individual book notes in my PKM system, I did the first actual run in production of those scripts today.
It took a few steps to get to using the scripts in production.
I have over 300 book note files in my Obsidian vault.
Of course most lacked the templated inline data fields that allow me to create lists. For the 67 fiction books I read in 2021 I already had a manual list with links to the individual files. Where needed I added the templated data fields.
Having added those inline fields where they were missing I can easily build lists in Obsidian with the Dataview plugin. Using this code
results in
The same inline data fields are used by my scripts to read the individual files and build the same list in OPML
That gets automatically posted to my website where the file is both machine and human readable.
Doing this in production made me discover a small typo in the script that builds the OPML, now fixed (also in the GitHub repository). It also made me realise I want to add a way of ordering the OPML outline entries by month read.
Lists to take into production next are those for currently reading (done), non-fiction 2021, and the anti-library. That last one will be the most work, I have a very long list of books to potentially read. I will approach that not as a task of building the list, but as an ongoing effort of evaluating books I have and why they are potentially of interest to me. A way, in short, to extend my learning, with the list as a useful side effect. The one for currently reading is the least work, and from it the lists for fiction 2022 and non-fiction 2022 will automatically follow. The work is in the backlog, getting history to conform to the convention I came up with, not in moving forward from this point.
In parallel it is great to see that Tom Critchlow is also looking at creating such book lists, in JSON, and at digesting such lists from others. The latter would implement the ‘federated’ part of federated bookshelves. Right now I just point to other people’s list and rss feeds in my ‘list of lists‘. To me getting to federation doesn’t require a ‘standard’. Because JSON, OPML and e.g. schema.org have enough specificity and overlap between them to allow both publishers of lists and parsers or such lists enough freedom to use or discard data fields as they see fit. But there is definitely a discussion to be had on identifying that overlap and how to use it best. Chris Aldrich is planning an IndieWeb event on this and other personal libraries related topics next month. I look forward to participating in that, quite a number of interesting people have expressed interest, and I hope we’ll get to not just talk but also experiment with book lists.
Ton Ziljstra is doing some more work on federated bookshelves (additional post here and Github stuff here). Chris Aldrich is also organizing a IndieWeb pop-up session on this topic. I reviewed the my federated bookshelves Github repo today, I came up with requirements and use cases for an app called Booklist Browser. I think my next step is to work on class diagrams…
Ton Ziljstra is doing some more work on federated bookshelves (additional post here and Github stuff here). Chris Aldrich is also organizing a IndieWeb pop-up session on this topic. I reviewed my federated bookshelves Github repo today, I came up with requirements and use cases for an app called Booklist Browser. I think my next step is to work on class diagrams…
This Article was mentioned on zblesk.net
I want to make it easy to publish lists of books I am reading and have read, or any other list. And do so without using centralised platforms like e.g. Goodreads (Amazon). A book list is a small library.
The route I am currently on, is publishing a machine readable list others can easily incorporate. These lists are in OPML, an exchange format for outlines. It’s the same format generally used to share lists of RSS feed subscriptions.
Current situation and usage: automated lists
Currently I am able to directly automatically create the lists in OPML from my individual book notes in Obsidian.md (which I use for PKM).
In Q1 2022 I experienced that creating lists and posting them works nicely and smoothly, with no friction. I do currently only create a few lists (fiction and non-fiction in the running year, antilibrary). I’m also working through the books I’ve read in the last decade or so, and gradually creating those lists. I’m not generating those as OPML however, they currently are just a list in my own notes.
Next steps: consuming other lists
Next steps will look at how to do the federating itself: how can I ‘consume’, or even include in my own lists, the OPML, ActivityPub or JSON lists of others in a meaningful way? I think a first step is consuming one list published by someone else, treating it as a recommendation list perhaps or some other form of input, much like I’m reading feeds. It might be useful to be able to pick out mentions about books I’ve already read, are in my anti-library, match an author I like, or match my interests while being unknown to me. I suspect a slightly tweaked parser for every new list might be needed, as using a list depends both on format and on content fields.
Ealier steps: proof of concept and data model
In 2020 I came across a posting by Tom Critchlow on this topic, and a year later I started looking into using OPML to create the lists.
I created a proof of concept, with a data format.
Using that I created a webform to update a book list by hand with a new entry.
Then I automated generating the lists (code on GitHub).
All as proofs of concept.
Starting in 2010 I have posted an annual ‘Tadaa’ list, a list of things that made me feel I had accomplished something that year. I started doing it in 2010 because I tend to forget things I did after completion. Like last year I didn’t feel much like writing this. It seemed a greyish year, passing in the shadow of the war that Russia wages on Ukraine. A year where Covid is still very much around us, yet things sort-of returned to normal. But for a different value of normal, a somewhat twisted normal, a parallel one. An appearance and pretense of normal perhaps more than an actual normal. An intransitive year almost, taking me from 2021 to 2023, but without object. Or maybe it’s because the last few months were extremely busy, pushing through more than being in the here and now, which sapped the colour from the months preceding it. Which is as good a reason as any to try and list the things that did bring a sense of accomplishment. I do have my day logs from the entire year, as well as kept up posting week notes here, so I can look back at what went on these past 12 months.
So here goes, in no particular order:
The European High Value Data list has become law in December. Two years ago I had a defining influence on the data it lists for earth observation, environment and meteorology. Even if the implementation period is 16 months and some datasets may get a temporary exemption for another two years, and even if it doesn’t go far enough (mostly on company information) to the taste of many, it is an important milestone. It draws the line under discussions about paywalls and exclusive access rights that were already old when I got involved in open data in 2009, in favor of mandatory pro-active publication for all to use freely. I’m glad I could translate my experience in this field into something now enshrined much more solidly in EU law.
We took regular breaks as a family. We started the year in Luzern, spent a week in Limburg in April, spent three weeks in Bourgogne doing most of nothing. Had weekend trips, to various musea for instance. One of the things E and I decided, while hanging out in front of our tent in the Bourgogne last summer, was to mark all school holidays in our own calendar in the coming year, to either take them off ourselves, or to keep them free of work appointments. I think it should be possible without impacting my output, but it will require careful planning.
I’ve kept an actualised guide about the incoming EU data legislation in Dutch for a client. It gets automatically generated directly from my own working notes in Obsidian which appeals to me in terms of nerdy workflow, and it is highly used by Dutch government data holders and regularly mentioned as a very useful resource which speaks to its utility.
I enjoyed homecooking a few software tools. Early in the year I adapted my OPML booklists so they are generated directly from my own book notes. (Although the negative side effect has been I did not blog about my reading at all, which I intend to change soon) I particularly enjoyed enabling myself to post through Micropub to my various websites. Through it I can post from various sources bypassing the WordPress back-end, inluding directly from my local notes in Obsidian, and from my feedreader. Every time feels like magic despite the fact I wrote the scripts myself. I think that sense of magic stems from the reduction of friction it affords.
I helped the foundation I chair through a inconvenient period of administrative issues. Nothing serious in itself, but right at a moment where it did have consequences for the team, which I was able to cushion. We also extended the number of board members, laying a better fundament for the coming years.
The influx of many new users into the Fediverse spurred my involvement in the use and governance of Mastodon. I helped plan a governance structure for the largest Dutch instance, and intend to help out in the coming year as well. We’re building a non-profit legal entity around it, and secured initial funding for that from a source in line with that non-profit status. I enjoyed also kicking off some discussion within the Dutch forum for standards that prescribes the mandatory standards for the Dutch public sector.
I keynoted at BeGeo, the Belgian annual conference of the geo-information sector, at the invitation of the Belgian national geographics institute. It was fun to create the story line for it, as well as enjoyed the sense of traveling and meeting with a professional community I’m normally not part of. It’s the type of thing I often did for years, and I miss it I noticed. Something to look out for in the coming months.
My company had a great year, apart from a hick-up after the summer, to the occasion of which the team rose fantastically. We grew despite that hick-up, adding two new team members in May and September, and signed an additional new hire in December. As of February we will be ten people. The work we’re doing is highly interesting, around digital ethics, data governance mostly, engaging new clients frequently. Our team is a great group of people, and I think we all take good care of eachother. We completed the 11th year of my company which I think is already an amazing run. For next year our portfolio is already mostly filled.
During the pandemic lock-downs in 2021 we hired cabins for all team members at a holiday park to work and hang out together for a week while maintaining social distancing advice. We realised we wanted to do that yearly regardless of pandemics, and did so in 2021 again. It’s an important thing for both the social and professional dimensions of our company.
I took my homecooked projects as the starting point for a presentation at WordCamp Netherlands to plead for more general adoption of IndieWeb principles, specifically webmention and microformats in WordPress which met with good responses and helped spur on at least one coder to finish and publish a plugin. I’m mostly a boundary spanner in these settings, at the edge of communities, in this case the WordPress community, and being able to bring a story and suggestions for change into a commmunity from another context and see it getting a response is something I enjoy.
Seeing Y grow and thrive, in school, socially, reading, swimming, skating.
Decided to join my old fraternity on their 30th anniversary trip to Montenegro, and am glad I did. Montenegro is a beautiful and rugged country.
I’ve been writing in this space continuously for twenty years now. Even if my writing here in the past few months has been less frequent, an expression of how busy it was in other aspects of my life, blogging has been a constant and a key to creating new conversations, connections, ideas and experiments.
I explored new tools to integrate in my personal workflow, like annotating with Hypothes.is, using machine translation (DeepL) and AI text and image generators. This as starting point for turning them into personal software tools in future months.
We spent some days around New Year in Switzerland, visiting dear friends. As years go by, such things become more important, never less. The simple fact of time passing means old friendships carry ever more context and meaning.
Ever onwards! (After having the first week of January off and spending it with the three of us that is.)
E and Y discussing artworks in the Rijksmuseum Twenthe. A great way to spend time together.