Tag: microsub

Posted on
by

I use a personalised feedreader (running on top of a self-hosted instance of FreshRSS‘s API that handles the RSS subscriptions) since about 4 years.
My feedreader allows me to interact with the Web, not just read it. I can post to this blog (and a few other websites) directly from it and keep reading my feeds. Same for adding an annotation to Hypothes.is, and for adding a note in markdown to my filesystem in the folder where Obsidian lives.

Recently I mentioned I want to make my habit of annotating web postings in my Hypothes.is easier to keep up.
As I wrote then:

… currently from within my feedreader I can post to either my blog or to Hypothes.is, but not both. I want to change that, so that the same thing can serve two purposes simultaneously.

I now have adapted my feedreader interface and related scripts to do just that.
It can post to a few websites AND to hypothes.is AND to Obsidian all at the same time now. It used to be either just one of the sites, hypothes.is or Obsidian. Posting to both hypothes.is and Obsidian simultaneously won’t happen a lot in practice as my hypothes.is annotations already end up in Obsidian anyway. I use the saving to Obsidian mostly to capture an entire posting, where I use hypothes.is in my feedreader to just initially bookmark a page so I might return later to annotate more. The current version of the response form in my feedreader is shown below.

One element I added to the interface that I haven’t coded yet in the back-end: posting to my personal and/or my business Mastodon accounts. [UPDATE I did that now too]. When Now that is done, I can write to all the places I write the web, right from where I read it, as in Tim Berners Lee’s original vision:

The idea was that anybody who used the web would have a space where they could write and so the first browser was an editor, it was a writer as well as a reader. Every person who used the web had the ability to write something. It was very easy to make a new web page and comment on what somebody else had written, which is very much what blogging is about.

Tim Berners Lee in a BBC interview in 2005

Posted on
by

Web pages are predictably untrustworthy to remain online as they were when you dropped by to see them. They can change while their address remains the same, can redirect an address to someplace else entirely, and entire websites can disappear meaning there’s no response when attempting to visit an address.

This means that when I link to something here, there’s no guarantee at all that when you click such a link that you actually get to see what I thought I’ve linked to. And no, screenshots don’t help: fake is easy and we need to back up our words with hyperlinks.

The same is true for stuff I don’t link to here, but save to my archive. That’s why I don’t just save URLs but the entire article for future reference to my markdown notes. That still means the actual source might disappear, without me having a way of proving what I saved is what I saw. This not only is relevant to the content itself, but also for instance for licensing information. There are photos in this blog that were openly licensed when I used them, but no longer, leaving it impossible for me to prove I still can use the image because of the license at the time.

This makes an archiving service useful, like Archive.org. I can use that to store URLs I find interesting and I do that with some regularity. It is why I am a monthly donor to the Web Archive, I’d like it to remain a more robust reference point on the web.

Currently I have one way of adding web pages to an archive, using the Wayback Machine add-on in my browser. The same add-on helps me find previous versions of a page already archived, tweets about that page, and annotations made by others. Very useful, during browsing.


The Web Archive browser add-on bookmarklet

Writing blogposts and saving webpages as markdown in my local notes, or starting to annotate a page in Hypothes.is are another matter however. There I’d like to automate getting or creating an archive link.

In all cases it would need to be an archive link next to the original. If I link to something in a blogpost here I want to still send WebMentions to the linked site, and that requires the link to the original to be in my posting. Similarly for my notes, I want to have the original url as well, although it would be reconstructable from the archive link. For online social annotations in Hypothes.is, the original link is needed because that is how you find other people’s annotations alongside your own. The last one is probably easiest, by using the browser add-on manually and adding the result as a first annotation for instance.


An archive link as first Hypothes.is annotation

On the page in the IndieWeb wiki about using the Internet Archive there are some code snippets to be found to use with the Archive’s API, or using the basic string to save something https://web.archive.org/save/urlhere. It also mentions bloggers who either send the URLs they mention, or their own postings, or both (e.g. when sending a WebMention) to the Internet Archive.
When posting to my blog from my local markdown notes I could potentially add a function to the markdown-to-html parser I use where it detects external links, runs them through the Archive and writes the html for both the direct and the archived link.

For saving web articles as local notes in markdown there are several options to explore:

  • When saving from the browser using the Markdownload add-on, first saving and copying the saved url using the Achive add-on, then pasting that archive link in the dialog box.
  • Adding [Opslaan in Internet Archive](https://web.archive.org/save/{baseURI}) to the Markdownload template so I can directly save a URL from within the local note later, if wanted. I added this experimentally, to see if I would actually use it like this.
  • When saving to notes from my microsub feedreader I could add a function to the html-to-markdown parser I use there to run external links through the Archive and write the Archive link in markdown after the original link.

Posted on
by

I extended the capabilities of my microsub feed reader with the option to save web articles directly from the reader to my Obsidian notes in markdown format.

Until now if I wanted to save an entire article I found in my feed reader, I would open it in the browser and then use the markdownclipper browser add-on to add some context and then save the article in markdown in my notes. I wanted to cut out that step of opening it in the feed reader, by saving it directly to my markdown notes. In my feedreader I already have a response form to e.g. post a reply to a posting on my own site. Posting it to my notes means adding a path to how I process that form.

I had to find a suitable script for converting HTML to MarkDown first. Which I found in PHP League’s HTML-to-Markdown, as suggested by Jan Boddez. It requires Composer which I already had installed on my laptop.

I tweaked my feed reader’s response form to also (as a hidden field) include the original HTML of a posting (using htmlentities to stuff it into a form field value). The script that processes the form I altered to both have a path for posting to websites (using micropub) and a new path to make a note in Obsidian, which is then saved as a .md file to the folder I store all clipped articles in.
To make a note I shape the available input the same way I template clipping things from the browser. At the top is my rationale for clipping something and reference to the source, followed by the original posting after which I add some keywords as tags and again the reference to the source.

In the images below you see the corresponding elements marked both as they appear in the reader as well as the resulting note.

The article as shown in my feed reader:

1: the original HTML content from a feed
2: title of the article (prefilled by my feed reader)
3: name of the author (prefilled by my feed reader)
4: original article’s URL (prefilled by my feed reader)
5: the reason and context why I am saving this to notes (also used to write a reply to a post, or the reason for bookmarking something if it will be posted on my site)
6: a quote I want to highlight
7: keywords that will become tags or categories on my site, and tags in my notes
8: selector for which site to post to (zyl is my blog), or ‘obs’ for making a note in Obsidian

Except for that last one those numbers are marked on the image of the resulting markdown note.

The resulting note in Obsidian:

1: the original HTML content from a feed shown in Markdown as the main body of the note
2: title of the article, both shown as part of the content of the note, as well as the title of the note (where a timestamp is added)
3: name of the author (mentioned with the source both at the top and bottom)
4: original article’s URL (mentioned with the author both at the top and bottom)
5: the reason and context why I am saving this, always at the top as it helps me process the content better
6: a quote I wanted to highlight
7: keywords that have become hashtags

(This posting was also written in my notes and, except for the images, posted directly from Obsidian to my site. Meaning I can both automatically move material into Obsidian, as well as automatically move material out of Obsidian. I quite enjoy the feeling of using that ‘magic’.)

Posted on
by

Today I posted a bookmark directly to this blog, straight from a feed reader I created myself. A few days ago I already posted about how I am building a Microsub client, which basically is the front-end of a feed reader: it’s the part where you actually read the content from the RSS feeds you subscribe to. It does not in itself subscribe to any feeds, or store what it gets from those feeds. That’s the Microsub server, and I use an existing WordPress plugin called Yarns for it. By splitting the server (subscribing, fetching and storing) from the client (presenting), making the feed reader becomes building a website that can show me whatever is stored in the server. It makes it possible to shape my feed reader the way I want it. I had the basic fetching and presenting part done earlier this week. The next step was interacting with the things I read from within the reader, having a reply button underneath each posting, with which I can post to this blog. I’ve got that part working now too. To build it I re-used things I previously already created. For instance I already had a form, to formulate a response (originally stuck on the back-end of another feedreader), and a way of posting directly to this site. So adding this was more a remix than coding something new. The only new additions were the ‘glue’ needed to join the existing parts together, and a bit of Javascript to be able to process the form in the browser, while staying on the same page as where I’m reading feeds. And it works!

Here’s a posting by Peter, as presented in my feed reader:

Note the button underneath which says in Dutch ‘respond to the post above’. Each posting has a button like that.
Clicking such a button reveals a form (which is at first hidden, and revealed by a line of Javascript).

In the form I specify what type of response it is (reply, bookmark, or favourite), and I write my reaction, and can add a quote from the article above which is useful if I’m responding to something specific, and provides context for readers on my own site who aren’t aware of the posting I’m responding to. It also allows me to change the title, author name and URL of the original post. This sometimes is useful when the title is very long, the name in the feed is a user name (e.g. ‘siteadmin’), or when the URL contains appended tracking elements that I want to remove.
Underneath is another button saying ‘Process’ in Dutch.

Hitting that button activates a little bit of Javascript to call the PHP script that processes the form, without leaving the page. It means that once I’ve clicked the button I can go on reading in the same page. The script that silently processes the form, turns what I’ve submitted into proper HTML for a posting, and then sends it on to my blog. Where it ends up as:

On my blog you get to see what the processing script has created, including e.g. the icon for a bookmark (or reply or favourite).

I’m pleased with how this works now, and I can start adding a few things. Like adding optional titles to a reaction (as I’ve done in this posting), or to add multiple tags or categories. For these too I have existing code sections I can re-use. Another element is being able to subcribe, unsubscribe and mark things as read from within the Microsub client.
After that it will become more difficult, if I want to work down my list of things an ideal feed reader should be capable of.
Part of that may require storing feed items client side, or maybe even also building my own Microsub server. So it’s a potential rabbithole I want to approach with caution. For now I will try this as my new workflow for a week or two.

(also posted to Indienews)

Posted on
by

I’ve started creating my own feed reader. Which I find is a pretty wild thing to say for me, given my limited coding skills.

Last month I created my own Micropub client. Micropub allows me to post directly to this site, without using the WordPress admin back-end for it. In fact I’m writing this as plain text, and at the end will hit a keyboard short cut for it to appear on my site. Part of the things I post to this site however are responses to posts by other people. I follow those people through their RSS feeds. Using Micropub I could post my responses directly while I’m reading the post I’m responding to, provided there’s a ‘reply’ button in my feed reader.

Feed readers usually don’t have such reply options. In fact, feed readers don’t have a whole range of functionality I’d like them to have. Building my own feed reader as a generic application probably would be hard. But as personal software tool, a ‘narrow band‘ tool that caters just to me it becomes easier to do. I’m pretty predictable to myself, and my workflows are known to me, so there are not many ‘what-ifs’ to cater for. My preferences are the default. As a result such a local tool could be more versatile, and much smarter in responding to my wishes than any fancy application, because I know what I want.

I said at the start I don’t know much about coding, but with some effort I can find my way in PHP well enough, and have been hand coding web pages ever since 1993.
Microsub is a web standard that makes a key difference: it splits the part where feeds are retrieved from other websites and stored in a database, from the part that presents the contents of the database. The retrieving and storing part is the Microsub server, for which I use a WordPress plugin called Yarns, but there are various others (and you could also build your own). The server has an API that allows you to query the content it stores. The presentation and reading part is the Microsub client. The client sends queries to the server and shows the results on screen. And that’s the part I’m building. Because building that part is basically like building a website, that every now and then requests something from a database.

The first step has been taken. I run a php script locally that shows me a webpage like in the image.


Screenshot of my feed reader to be, channels on the left, fetched feed items on the right.

It correctly grabs the groups of feeds and feed content from the Microsub server, and shows them on screen. It can of course be made to look nicer, but the basic concept works.
To this, unlike other feed reading apps, I can add my own response buttons. As links, buttons, or a form below each post. For that I already have scripts I can reuse, from when I was making my Micropub client, the bit that does the posting to my site.

Then I will be able to write my reactions directly underneath the bit I’m reading. Strengthening how my blog allows me to have distributed conversations. Getting that bit closer to the read/write web as it was envisioned.

For many of the other elements on my list of ideal feed reader features, I probably have to do some local storage. Specifically for things like visualising the activity of a feed, or doing things like showing me the topics people in a specific community are talking about this week, local storage might be needed.
I have a basic roadmap of steps to take, more or less following my post about this from last year around this time. For now this first step, a proof of concept that allows me to read feeds, is done.

In reply to a posting by Jan Boddez

These plugins, you create them in WordPress? I’d be interested to see an overview posting of what you’ve created thusfar. To see if it gives me any ideas I can act on with my low coding skills. (Although on the connective tissue of micropub and microsub, I’m likely more interested in the actual code and its workings itself)

Moving more and more of my feed reader’s functionality into separate plugins, which can be (de)activated on the fly. So far, I’ve got…

Jan Boddez