Tag: microsub

Posted on
by

Web pages are predictably untrustworthy to remain online as they were when you dropped by to see them. They can change while their address remains the same, can redirect an address to someplace else entirely, and entire websites can disappear meaning there’s no response when attempting to visit an address.

This means that when I link to something here, there’s no guarantee at all that when you click such a link that you actually get to see what I thought I’ve linked to. And no, screenshots don’t help: fake is easy and we need to back up our words with hyperlinks.

The same is true for stuff I don’t link to here, but save to my archive. That’s why I don’t just save URLs but the entire article for future reference to my markdown notes. That still means the actual source might disappear, without me having a way of proving what I saved is what I saw. This not only is relevant to the content itself, but also for instance for licensing information. There are photos in this blog that were openly licensed when I used them, but no longer, leaving it impossible for me to prove I still can use the image because of the license at the time.

This makes an archiving service useful, like Archive.org. I can use that to store URLs I find interesting and I do that with some regularity. It is why I am a monthly donor to the Web Archive, I’d like it to remain a more robust reference point on the web.

Currently I have one way of adding web pages to an archive, using the Wayback Machine add-on in my browser. The same add-on helps me find previous versions of a page already archived, tweets about that page, and annotations made by others. Very useful, during browsing.


The Web Archive browser add-on bookmarklet

Writing blogposts and saving webpages as markdown in my local notes, or starting to annotate a page in Hypothes.is are another matter however. There I’d like to automate getting or creating an archive link.

In all cases it would need to be an archive link next to the original. If I link to something in a blogpost here I want to still send WebMentions to the linked site, and that requires the link to the original to be in my posting. Similarly for my notes, I want to have the original url as well, although it would be reconstructable from the archive link. For online social annotations in Hypothes.is, the original link is needed because that is how you find other people’s annotations alongside your own. The last one is probably easiest, by using the browser add-on manually and adding the result as a first annotation for instance.


An archive link as first Hypothes.is annotation

On the page in the IndieWeb wiki about using the Internet Archive there are some code snippets to be found to use with the Archive’s API, or using the basic string to save something https://web.archive.org/save/urlhere. It also mentions bloggers who either send the URLs they mention, or their own postings, or both (e.g. when sending a WebMention) to the Internet Archive.
When posting to my blog from my local markdown notes I could potentially add a function to the markdown-to-html parser I use where it detects external links, runs them through the Archive and writes the html for both the direct and the archived link.

For saving web articles as local notes in markdown there are several options to explore:

  • When saving from the browser using the Markdownload add-on, first saving and copying the saved url using the Achive add-on, then pasting that archive link in the dialog box.
  • Adding [Opslaan in Internet Archive](https://web.archive.org/save/{baseURI}) to the Markdownload template so I can directly save a URL from within the local note later, if wanted. I added this experimentally, to see if I would actually use it like this.
  • When saving to notes from my microsub feedreader I could add a function to the html-to-markdown parser I use there to run external links through the Archive and write the Archive link in markdown after the original link.

Posted on
by

I extended the capabilities of my microsub feed reader with the option to save web articles directly from the reader to my Obsidian notes in markdown format.

Until now if I wanted to save an entire article I found in my feed reader, I would open it in the browser and then use the markdownclipper browser add-on to add some context and then save the article in markdown in my notes. I wanted to cut out that step of opening it in the feed reader, by saving it directly to my markdown notes. In my feedreader I already have a response form to e.g. post a reply to a posting on my own site. Posting it to my notes means adding a path to how I process that form.

I had to find a suitable script for converting HTML to MarkDown first. Which I found in PHP League’s HTML-to-Markdown, as suggested by Jan Boddez. It requires Composer which I already had installed on my laptop.

I tweaked my feed reader’s response form to also (as a hidden field) include the original HTML of a posting (using htmlentities to stuff it into a form field value). The script that processes the form I altered to both have a path for posting to websites (using micropub) and a new path to make a note in Obsidian, which is then saved as a .md file to the folder I store all clipped articles in.
To make a note I shape the available input the same way I template clipping things from the browser. At the top is my rationale for clipping something and reference to the source, followed by the original posting after which I add some keywords as tags and again the reference to the source.

In the images below you see the corresponding elements marked both as they appear in the reader as well as the resulting note.

The article as shown in my feed reader:

1: the original HTML content from a feed
2: title of the article (prefilled by my feed reader)
3: name of the author (prefilled by my feed reader)
4: original article’s URL (prefilled by my feed reader)
5: the reason and context why I am saving this to notes (also used to write a reply to a post, or the reason for bookmarking something if it will be posted on my site)
6: a quote I want to highlight
7: keywords that will become tags or categories on my site, and tags in my notes
8: selector for which site to post to (zyl is my blog), or ‘obs’ for making a note in Obsidian

Except for that last one those numbers are marked on the image of the resulting markdown note.

The resulting note in Obsidian:

1: the original HTML content from a feed shown in Markdown as the main body of the note
2: title of the article, both shown as part of the content of the note, as well as the title of the note (where a timestamp is added)
3: name of the author (mentioned with the source both at the top and bottom)
4: original article’s URL (mentioned with the author both at the top and bottom)
5: the reason and context why I am saving this, always at the top as it helps me process the content better
6: a quote I wanted to highlight
7: keywords that have become hashtags

(This posting was also written in my notes and, except for the images, posted directly from Obsidian to my site. Meaning I can both automatically move material into Obsidian, as well as automatically move material out of Obsidian. I quite enjoy the feeling of using that ‘magic’.)

Posted on
by

Today I posted a bookmark directly to this blog, straight from a feed reader I created myself. A few days ago I already posted about how I am building a Microsub client, which basically is the front-end of a feed reader: it’s the part where you actually read the content from the RSS feeds you subscribe to. It does not in itself subscribe to any feeds, or store what it gets from those feeds. That’s the Microsub server, and I use an existing WordPress plugin called Yarns for it. By splitting the server (subscribing, fetching and storing) from the client (presenting), making the feed reader becomes building a website that can show me whatever is stored in the server. It makes it possible to shape my feed reader the way I want it. I had the basic fetching and presenting part done earlier this week. The next step was interacting with the things I read from within the reader, having a reply button underneath each posting, with which I can post to this blog. I’ve got that part working now too. To build it I re-used things I previously already created. For instance I already had a form, to formulate a response (originally stuck on the back-end of another feedreader), and a way of posting directly to this site. So adding this was more a remix than coding something new. The only new additions were the ‘glue’ needed to join the existing parts together, and a bit of Javascript to be able to process the form in the browser, while staying on the same page as where I’m reading feeds. And it works!

Here’s a posting by Peter, as presented in my feed reader:

Note the button underneath which says in Dutch ‘respond to the post above’. Each posting has a button like that.
Clicking such a button reveals a form (which is at first hidden, and revealed by a line of Javascript).

In the form I specify what type of response it is (reply, bookmark, or favourite), and I write my reaction, and can add a quote from the article above which is useful if I’m responding to something specific, and provides context for readers on my own site who aren’t aware of the posting I’m responding to. It also allows me to change the title, author name and URL of the original post. This sometimes is useful when the title is very long, the name in the feed is a user name (e.g. ‘siteadmin’), or when the URL contains appended tracking elements that I want to remove.
Underneath is another button saying ‘Process’ in Dutch.

Hitting that button activates a little bit of Javascript to call the PHP script that processes the form, without leaving the page. It means that once I’ve clicked the button I can go on reading in the same page. The script that silently processes the form, turns what I’ve submitted into proper HTML for a posting, and then sends it on to my blog. Where it ends up as:

On my blog you get to see what the processing script has created, including e.g. the icon for a bookmark (or reply or favourite).

I’m pleased with how this works now, and I can start adding a few things. Like adding optional titles to a reaction (as I’ve done in this posting), or to add multiple tags or categories. For these too I have existing code sections I can re-use. Another element is being able to subcribe, unsubscribe and mark things as read from within the Microsub client.
After that it will become more difficult, if I want to work down my list of things an ideal feed reader should be capable of.
Part of that may require storing feed items client side, or maybe even also building my own Microsub server. So it’s a potential rabbithole I want to approach with caution. For now I will try this as my new workflow for a week or two.

(also posted to Indienews)

Posted on
by

I’ve started creating my own feed reader. Which I find is a pretty wild thing to say for me, given my limited coding skills.

Last month I created my own Micropub client. Micropub allows me to post directly to this site, without using the WordPress admin back-end for it. In fact I’m writing this as plain text, and at the end will hit a keyboard short cut for it to appear on my site. Part of the things I post to this site however are responses to posts by other people. I follow those people through their RSS feeds. Using Micropub I could post my responses directly while I’m reading the post I’m responding to, provided there’s a ‘reply’ button in my feed reader.

Feed readers usually don’t have such reply options. In fact, feed readers don’t have a whole range of functionality I’d like them to have. Building my own feed reader as a generic application probably would be hard. But as personal software tool, a ‘narrow band‘ tool that caters just to me it becomes easier to do. I’m pretty predictable to myself, and my workflows are known to me, so there are not many ‘what-ifs’ to cater for. My preferences are the default. As a result such a local tool could be more versatile, and much smarter in responding to my wishes than any fancy application, because I know what I want.

I said at the start I don’t know much about coding, but with some effort I can find my way in PHP well enough, and have been hand coding web pages ever since 1993.
Microsub is a web standard that makes a key difference: it splits the part where feeds are retrieved from other websites and stored in a database, from the part that presents the contents of the database. The retrieving and storing part is the Microsub server, for which I use a WordPress plugin called Yarns, but there are various others (and you could also build your own). The server has an API that allows you to query the content it stores. The presentation and reading part is the Microsub client. The client sends queries to the server and shows the results on screen. And that’s the part I’m building. Because building that part is basically like building a website, that every now and then requests something from a database.

The first step has been taken. I run a php script locally that shows me a webpage like in the image.


Screenshot of my feed reader to be, channels on the left, fetched feed items on the right.

It correctly grabs the groups of feeds and feed content from the Microsub server, and shows them on screen. It can of course be made to look nicer, but the basic concept works.
To this, unlike other feed reading apps, I can add my own response buttons. As links, buttons, or a form below each post. For that I already have scripts I can reuse, from when I was making my Micropub client, the bit that does the posting to my site.

Then I will be able to write my reactions directly underneath the bit I’m reading. Strengthening how my blog allows me to have distributed conversations. Getting that bit closer to the read/write web as it was envisioned.

For many of the other elements on my list of ideal feed reader features, I probably have to do some local storage. Specifically for things like visualising the activity of a feed, or doing things like showing me the topics people in a specific community are talking about this week, local storage might be needed.
I have a basic roadmap of steps to take, more or less following my post about this from last year around this time. For now this first step, a proof of concept that allows me to read feeds, is done.

In reply to a posting by Jan Boddez

These plugins, you create them in WordPress? I’d be interested to see an overview posting of what you’ve created thusfar. To see if it gives me any ideas I can act on with my low coding skills. (Although on the connective tissue of micropub and microsub, I’m likely more interested in the actual code and its workings itself)

Moving more and more of my feed reader’s functionality into separate plugins, which can be (de)activated on the fly. So far, I’ve got…

Jan Boddez

Posted on
by

Two years ago I wrote about the features I’d like to see in an ideal feed reader. Today I’ve tried to sketch out the various components, based on the various IndieWeb protocols, and then match the features I listed previously to the component that I think should provide it.

Separating feed reading into IndieWeb components

Feeds from blogs, to the left in the sketch below, end up in a microsub server, whose function is to fetch and store feeds. A microsub client is what presents those fetched feeds to me. A regular feed reader usually does both these things, but splitting them like the IndieWeb does allows multiple clients to use the same collection of feeds and feed items. It allows a wider diversity of readers, if the fetching is dealt with separately.
The microsub client also contains a micropub client. This allows having action buttons underneath each item presented in your microsub client. Hitting a button posts your reply or remarks, or shares something to your own website, and through your site to e.g. Twitter or Mastodon. Ideally it would be possible to have different websites to share to, next to storing locally in a specified format. The website that receives such an action has a micropub endpoint, and may need authentication through IndieAuth.

Such a reader set-up should be able to run locally on my laptop, as well as on a basic hosting package, so likely php / mysql based. Locally because I want to run my stuff local first, but the same thing online, even if a home server, because I’m not always working on my laptop and would like access from my tablet too, and point my Android Indigenous app to my subscriptions. Locally I would not need authentication from the microsub client to the microsub server, but in all other situations I would.

The sketch above is completely congruent with how I sketched my information gathering and filtering process in 2005:

filter1.jpg

Matching the ideal functionality to IndieWeb components

When I match the features of my ideal feed reader to those various IndieWeb components I think this is what results:

Microsub server needs to be able to :
* use various kinds of feeds (rss, atom, json, h-feed)
* allow folders (so I can arrange feeds on social distance)
* recognize and store tags if feed items have them
* allow me to tag _feeds_, really meaning tagging authors
* keep track of posting frequency, last post seen of feeds
* keep track of tags or predefined topics mentions/frequency
* pull in machine translations by default for certain feeds and store them with the orginal item

Microsub client needs to be able to:
* present the feed items in the server’s folder structure as a long list (the classic feed reader view)
* present views based on patterns in current feed items: what’s hot, what’s unique? Also set against social distance. (Topics discussed in my communities today)
* present views based on feed tags (show me all Germany based blog items of this morning, show me every feed of from EU based coders)
* present views based on feed tags and item tags: show me Germany based blog items talking about topic X.
* show full text search results of all items mentioning a certain topic.
* store full text search queries
* present visually which topics seem to be hot in which community, or where the frequency of mentioning a topic has changed
* provide a search of feeds (not feeds content): do I already have this feed in my list, where’s the feed of author Y?
* pull in a machine translation on request

Micropub client needs to be embedded in the microsub client and should support:
* saving an article as markdown or as html, to disk, to Zotero
* creating a todo from it by amending a textfile,
* bookmarking it, either to my blog or some other target
* sharing something about it to my blog, to Mastodon through my blog
* replying to it, through my blog to Mastodon, Twitter, other blogs
* allow configuring new actions.
* choosing from multiple micropub endpoints

What do you think, should some of these features be provided by other components than they’re currently listed? Are features missing that you’d like to see in your ideal feed reader?