Category Archives: opendata

Netherlands National Disclosure Day

Every 1st Tuesday of the new year is Disclosure Day (Openbaarheidsdag) in the Netherlands, the day when the National Archive opens up material and collections whose access restrictions have expired. It is generally a day approached with a sense of festivity by archivists (openness is their core public service and they’re also eager to show what cool stuff they have on record), and the hashtag #openbaarheidsdag shows a variety of tweets. Next to the National Archive, a range of local and regional archives participate, amongst which is the Frisian regional archive Tresoar. With both institutions I had the pleasure of working together last year (related to openness but unrelated to disclosure day). There is some momentum building to turn it into a nationally observed day by all archives, to celebrate the work and value of archives.

HET NATIONAAL ARCHIEF OP EEN ZONNIGE DAG
Photo of the National Archive building next to The Hague Central Station, by 23Archiefdingen CC BY-SA

In general the material opened up for the general public on a Disclosure Day wasn’t fully secret, but access was restricted to e.g. researchers. Some of it, such as personal correspondence and material from private archives however hasn’t been seen by anyone for multiple decades. Various time limits apply. Government documents are transferred to the National Archive after 20 years and most of it will be publicly accessible then right away (before that transfer the relevant Ministry itself determines openness, and freedom of information regulations apply). Where things like the privacy of living persons or national security are involved access can be restricted for a maximum of 75 years. That limit is set at the start, and per the first January after that limit expires, it becomes public. So for this year, it means that 1997 (20 yrs), 1967 (50 yrs) and 1942 (75 yrs) are the years for which new material is being published (as 50 and 75 years are commonly used time limits on access, and 20 yrs the legal default).

The amount of material published each year is substantial. Just the index of material published this year by the National Archives is over 1200 pages, and only the table of contents of that index of material is already 50 pages.


Lots of manual work involved, such as this tweet by National Archive employee Maartje v.d. Kamp illustrates. She says “green dots mark restricted access, disclosure day means scraping lots of dots from archive boxes”

For this year the collection of newly accessible material contains the minutes of Cabinet meetings of the mid 1990’s (in which the Prime Minister berates his Ministers for leaking and carelessly letting opinions be publicly known), the personal notes of the researcher who on behalf of the government looked into the riots during the 1966 wedding of future Queen Beatrix, as well as personal notes from the same researcher when looking into the war history of a government minister alleged to have served in the SS. These haven’t been opened at all, since then, and the National Archive itself wondered what is in it that made the researcher claim years of secrecy for it. Also the diaries of the first woman Minister in the Netherlands and a prime minister in the late 1940’s became public.

Of special interest this year is the disclosure of the Dutch Trustfund (Nationaal Beheersinstituut) that between 1945 and 1967 managed all the Dutch funds and assets seized from German nationals after the war and from Dutch (suspected) collaborators and profiteers until after their court verdicts, as well as funds and assets of missing persons (such as jewish people deported by the nazis during the war, or in hiding and not yet returned) until their fate was known. The index of all persons for whom assets were managed, or from whom assets were seized, contains some 180.000 names, and is available online. The Dutch Trustfund archive itself hasn’t been digitised yet (as it wasn’t public yet), and stretches 2.5km of shelves.
Out of curiosity I did a quick search to look for my grandparents but none of them show up in the index. So there wasn’t some never shared family secret. 😉

Collaborating E.J. Voûte, mayor of Amsterdam during the years of occupation

E.J. Voute, collaborating mayor of Amsterdam during the nazi occupation on the photo above (source Spaarnestad archive, within the National Archive. Photo taken during his trial, 25 April 1947), and the location in the list of the Dutch Trustfund archives where he’s mentioned.

Partnering with Open Belgium Conference

Part of my 2018 plans is to do a bit more in Belgium with my company The Green Land. We’re looking for partners to work with locally. To gain a bit more visibility we are a sponsoring partner for the upcoming Open Belgium Conference, that takes place on March 12th in Louvain-la-Neuve. Get your tickets now and see you in Belgium this spring!

Playing With Q-GIS

Most of my open data work is with government entities to help change their processes, routines and perceptions to ensure steps towards open by design. I almost never really work with open data itself during those activities. So I decided to accept the challenge we ourselves issued with the launch of the Frisian Open Data Platform.

The challenge was to “find out what the planting year was of the monumental tree that is nearest the street light with the provincial ID number 696502”. Finding that out needed to be done by using data from the Frisian Open Data Platform.

Figuring out which data to use was easy. There is a provincial data set that contains the position and ID’s of all street lights for roads where the province is responsible (other roads can be the responsibility of a municipality, or the national government). There is another data set of the city of Leeuwarden that contains all trees of interest within the city limits. If the street light with the right ID is within city limits, it should be possible to answer the question with the tree data set of Leeuwarden.

So what I did was first look in the provincial data set for the right ID. I copied the coordinates that data set gives for that ID into Google Maps, to see where it is on the map, and it turned out to indeed be within Leeuwarden city limits. So the Leeuwarden tree list contains the answer I’m looking for.

Then I started up Q-GIS, which is an open source geo-data viewer (and in fact, a very capable open source GIS *editor* too, as Peter says in the comments). It is possible to connect a CKAN data portal, such as the Frisian platform is, to Q-GIS. Under the menu-option Plugins in Q-GIS one can install a CKAN plugin, which gives you a CKAN logo button in Q-GIS. Pressing that prompts a dialog in which you can specify the right address for the CKAN server you want to use. This was specified on the Frisian platform as https://ckan.dataplatform.nl/api/3/. I also needed to add a default folder that can be used to keep necessary files.

Now I could search within all the Frisian open data platform data sets right within Q-GIS, using that plugin. I first loaded a map of the Netherlands (the TOP10NL map, which is the most detailed map the Dutch Cadastre provides, as a zipfile of 2GB). I used the PDOK Dutch open geoportal for this, for which I had already previously installed the PDOK plugin, in similar ways as the CKAN plugin). Then I added the Provincial street light list, and the Leeuwarden tree list as layers on the map. I then scrolled the map to the location I had previously checked out in Google Maps.

In the screenshot below you see green dots on the red road. Those are provincial street lights. The rightmost green dot is the one we’re looking for. A bit further to the right you see a row of purple dots. Those are the trees, and one of these is nearest our green dot. Now, I visually judged which purple dot is the nearest, although you could calculate it from the coordinates in the data. Also there is some room for error, as most of the trees in that row were planted at the same time as it turns out. By clicking on one of the dots in Q-GIS you can see the data fields and labels attached to it, and that gave me the year of planting.


The map of Leeuwarden, with the street lights as green dots on the red road in the middle, and the monumental trees as purple dots.


At the top you see the depicted map layers (Dutch map top10nl, trees in ‘bomen’, street lights in ‘provfriesland’), below that, when you highlight a specific purple dot, under identification results (‘identificatieresultaten’), PLANTJAAR is the field with the year of planting.

End of an Era in Flanders Open Data

Last week saw an end of an era. The program manager for open data of the Flemish government retired. While parts of the work will go on, no direct successor will be named to the role. At the annual conference of Information Flanders (#tiv2017), Noël van Herreweghe after 6 years of being the driving force behind Flanders’ open data team, said his goodbye during the opening plenary. His main and clearly heard message was that much is still to be done, and we’ve barely started on the path towards open by design. I hope the Flemish government and civil service will take this to heart. Now is not the time to reduce efforts, as the transition is only just in motion.


Noël telling us and the Flemish government to stay the course (Tweet and photo by @toon, Toon Vanagt)

In the past 6 years Flanders has taken several steps that I think the Netherlands should follow. Based on the underlying legal framework, the Flemish government has taken pre-emptive decisions for all government entities within their scope about in what ways data can and should be published. It is no longer up to the individual agencies, if you decide to publish you must follow the established principles. In the Netherlands that is all still voluntary, and the principles are put forward as guidelines, not as must-follow rules. Similarly the Flemish government has adopted a URI strategy, using both machine and human readable URI conventions, which in the Netherlands is lacking.

It’s been a pleasure to work with Noël and his team in these past 6 years. Whether it was in helping decide on which local and regional open data projects to fund from the Flemish government, translating research on the economic impact of open data to the Flemish and Belgian context, providing scenario’s to the Flemish Chancellary for opening up Flemish consolidated laws and regulations as open data, or providing open data training together with Noel to a joint session of the Dutch and Belgian/Flemish supreme audit authorities.

For each of those 6 years my colleague Paul, representing the Dutch government open data team, and I participated in the Flemish open government days, and its successor the annual Information Flanders Meet-up. It gave us the opportunity to keep comparing Dutch and Flemish open data efforts, to learn from each other as well as laugh about the differences. A fixed feature on the agenda was eating a Portuguese fish soup the evening before the event in Brussels with Noël and his colleagues.

Portuguese Fish Soup Open data dag Vlaanderen
A ‘small bowl’ of fish soup, 2012 and 2015 editions

As Noël said, the work isn’t remotely done, and judging from the conversations we had with Noël last week, he isn’t likely to stop being active either. So I trust we will find ways of working together again in a different setting in the near future.

Frisian Open Data Platform a Unique Collaboration

Yesterday saw the Frisian Open Data Platform go live. Initiated by the Province Fryslân, the City of Leeuwarden (the province’s capital), and the regional historic center and archive Tresoar it is a unique collaborative effort in the Netherlands. The Province initially proposed to create the platform, based on the notion that open data is more useful if it is available from local governments across the region, and seeking to avoid every local government needing to create their own infrastructure for publishing (especially an issue with smaller municipalities), and that creating a single point to search for regional and local data makes it more likely that data will be used. The Province invites all other Frisian government entities to participate in the platform.


Screenshot of the portal

Data can be hosted in the platform (the Province does this, but also very useful for smaller entities), but those that want to maintain their own infrastructure or already do can also use it as a register and increase findability that way (e.g. Tresoar has been publishing a lot of material, also in the form of linked data for a long time already).

The platform was launched during Connect.FRL, a conference bringing IT employers and students together to try and keep more talent in the region. There all three initiators presented themselves together and also provided insight how they currently use data internally.

Fries Open Data Platform / Connect.frl
The stand of the platform initiators at the Connect.FRL conference

The launch coincides with a challenge for students and others to solve a specific riddle with the now published data, to suggest a concept of how the data can be used, and to create a prototype.

I’ve been working with the City of Leeuwarden in 2012, as well as with the Province in the past 2 years. Employees of Tresoar attended our Mastercourse open data last year, initiated by the National Archives. All that combines now in together delivering this platform. It is especially great to note how currently there is palpable energy within all three participants to move this forward. In the coming time we will work to bring more participants to the table, expand the data on offer, and further align open data efforts in the region.

Fries Open Data Platform / Connect.frl
The banner for the open data challenge

“Privacy is Cultural”

Yesterday my colleague Paul and I visited the annual conference organized by the Flemish government’s information management / IT office. We were there to speak about the open data experiences of the Netherlands.

The upcoming GDPR, Europe’s new privacy regulations, was mentioned and discussed a lot. Such pan-European laws suggest that there is a generic way to approach a topic like privacy, or even an objective one. Nonetheless the actual perception of privacy is strongly culturally determined as well, Toon van Agt remarked during his presentation, and pointing to us Dutchies sitting on the front row. He gave the example of how in the Netherlands real estate transaction prices and mortgages on a house are publicly available (if not yet as open data I must add. Transaction prices are available as open data in the UK, afaik). Where in the Netherlands this is regarded as necessary to be able to determine who you’re dealing with if you buy or sell a house, in Belgium it would be unthinkable. In my own presentation I showed how open data from the license plate register is used in the Netherlands to prevent theft of petrol at gas stations. Again unthinkable in Belgium, mostly because of the fundamental difference that license plates in the Netherlands are connected to a car (and the car to an owner), and in Belgium to the car owner (and the owner to a car). Calvinism was put forward as a determining difference, resulting in Dutch window curtains being open, so everyone can see a) we have nothing to hide and/or b) we have the coolest stuff in the street :). Similarly the tax amounts and incomes of Norwegians are famously public, whereas in the Netherlands asking how much someone earns or even worse touting how much you earn yourself, is frowned upon and not suitable for polite conversation.

It would be interesting to create an overview of socially acceptable and unacceptable forms of transparency across Europe. To learn where further opportunities for open data are to be found, as well as to see where social barriers can be expected.

The wonderful windows open houses on the Dutch( Volendam) 4 2017-09-23_15-15-25_ILCE-6500_DSC03304
The quintessential difference between Belgium (r) and the Netherlands (l): curtains open or closed. Photos by Miguel Discart and magalibobois

Dutch Provinces Looking for Open Data Inspiration

Last week ten of the twelve Dutch Provinces met at the South-Holland Provincial government to discuss open data, and exchange experiences, seeking to inspire each other to do more on open government data. I participated as part of my roles as open data project lead for both the Province of Overijssel, and the Province Fryslân.

There were several topics of discussion.

  • The National Open Government Action Plan (part of the OGP effort), a new version of which is due next spring, and for which input is currently sought by the Dutch government.
  • A proposal by the team behind the national open data platform to form a ‘high value data list’ for provincial data sets.
  • Several examples were discussed of (open) data being used to enhance public interaction.

I want to briefly show those examples (and might blog about the other two later).

Make it usable, connect to what is really of significance to people
Basically the three examples that were presented during the session present two lessons:

1) Make data usable, by presenting them better and allow for more interaction. That way you more or less take up position half-way between what is/was common (presenting only abstracted information), and open data (the raw detailed data): presenting data in a much more detailed way, and making it possible for others to interact with the data and explore.

2) Connect to what people really care about. It is easy to assume what others would want to know or would need in terms of data, it is less easy to actually go outside and listen to people and entrepreneurs first what type of data they need around specific topics. However, it does provide lots of vital clues as to what data will actually find usage, and what type of questions people want to be able to solve for themselves.

That second point is something we always stress in our work with governments, so I was glad to hear it presented at the session.

There were three examples presented.

South-Holland put subsidies on a map
The Province of South-Holland made a map that shows where subsidies are provided and for what. It was made to better present to the public the data that exists about subsidies, als in order to stimulate people to dive deeper into the data. The map links to where the actual underlying data should be found (but as far as I can tell, the data isn’t actually provided there). A key part of the presentation was about the steps they took to make the data presentable in the first place, and how they created a path for doing that which can be re-used for other types of data they are seeking to house in their newly created data warehouse. This way presenting other data sources in similar ways will be less work.


The subsidy map

Gelderland provides insight into their audit-work
Provinces have a task in auditing municipal finances. The Province of Gelderland has used an existing tool (normally used for presenting statistical data) to provide more detail about the municipal finances they audited. Key point here again was to show how to present data better to the public, how that plays a role in communicating with municipalities as well, and how it provides stepping stones to entice people to dive deeper. The tool they use provides download links for the underlying data (although the way that is done can still be significantly improved, as it currently only allows downloads of selections you made, so you’d have to sticht them back together to reconstruct the full data set)



Screenshot of the Gelderland audit data tool

Flevoland listens first, then publishes data
The last example presented was much less about the data, and much more about the ability to really engage with citizens, civil society and businesses and to stimulate the usage of open data that way. The Province Flevoland is planning major renovation work on bridges and water locks in the coming years, and their aim is to reduce hindrance. Therefore they already now, before work is starting, are having conversations with various people that live near or regularly pass by the objects that will be renovated. To hear what type of data might help them to less disrupt their normal routines. Resulting insights are that where currently plans are published in a generic way, much more specific localized data is needed, as well as much more detailed data about what is going to happen in a few days time. This allows people to be flexible, such as a farmer deciding to harvest a day later, or to move the harvest aways over water and not the road. Detailed data also means communicating small changes and delays in the plans. Choosing the right channels is important too. Currently e.g. the Province announces construction works on Twitter, but no local farmer goes there for information. They do use a specific platform for farmers where they also get detailed data about weather, water etc, and distributing localized data on construction works there would be much more useful. So now they will collaborate with that platform to reach farmers better. (My company The Green Land is supporting the Province, 2 municipalities and the water board in the province, in this project)


Overview of the 16 bridges and waterlocks that will be renovated in the coming years


Various stakeholders around each bridge or waterlock are being approached

Launching the Malaysia Open Data User Group

I spent the last week in Kuala Lumpur to support the Malaysian Administrative Modernisation and Management Planning Unit (MAMPU) with their open data implementation efforts (such as the Malaysian open data portal). Specifically this trip was about the launch of the Malaysia Open Data User Group (MODUG), as well as discussions with MAMPU on how we can help support their 2018 and 2019 open data plans. I was there together with my World Bank colleague Carolina Vaira, and with Baden Appleyard, a long time long distance friend of my company The Green Land. As he is from Australia, working together in Malaysia means meeting sort-of half way.

The MODUG comes from the action plan presented last May, after our Open Data Readiness Assessment last year, which I helped bring about when I first visited in spring 2015 as part of the Malaysian big data advisory board. In the action plan we suggested creating an informal and trusted place for government organisations to discuss their practical issues and concerns in creating more open data, learn from each other, and collaborate on specific actions as well as formulating government good practice. Similarly it called for creating a similar space for potential users of government open data, for individuals, coding community, NGO’s and civil society, academia and the business community. Next to having these two places where both government and non-government can discuss their questions and issues amongst themselves, regular interaction was proposed between the two, so that data custodians and users can collaborate on creating social and economic value with open data in Malaysia. The MODUG brings these three elements under one umbrella.

Last Tuesday MAMPU held an event to launch the MODUG, largely moderated by Carolina and me. MAMPU is within the remit of General Affairs Minister within the Prime Minister’s office, Joseph Entulu Belaun. The Minister officially opened the event and inaugurated the MODUG (by cutting a ribbon hanging from a drone hovering in front of him).

Malaysian Open Data User Group (MODUG) 2017 Malaysian Open Data User Group (MODUG) 2017
Minister Joseph Entulu Belaun cutting a ribbon from a drone, and Dr Yusminar of MAMPU presenting the current status of Malaysian open data efforts. (both images (c) MAMPU)

Dr Yusminar, who is the team lead with MAMPU for open data, and our direct counter part in our work with MAMPU, provided a frank overview of efforts so far, and things that still need to be tackled. This helped set the scene for the rest of the day by providing a shared understanding of where things currently stand.

Then we got to work with the participants, in two rounds of a plenary panel followed by roundtable discussions. The first round, after data holders and users in a panel discussed the current general situation, government and non-government groups discussed separately, looking at which data they see demand for, the challenges they encounter in publishing or using the data, and the suggestions they have overcoming those. The second round started with a panel bringing some international experiences and good practice examples, during which I got a new title, that of ‘open data psychologist’ because of stressing the importance of the social aspects, behaviour and attitude involved in making open data work. The panel was followed with round table conversations that mixed both data custodians and users. Conversations centered on finding a collective agenda to move open data forward. After each round the results from each table were briefly presented, and the output attached to the walls. Participants clearly appreciated having the time and space to thoroughly discuss the open data aspects they find important, and be heard by their colleagues and peers. They indicated wanting to do this more often, which is great to hear as creating the room for such conversations is exactly what the MODUG is meant for!

Malaysia Open Data User Group Malaysia Open Data User Group
Malaysia Open Data User Group Malaysia Open Data User Group
Roundtable discussions on a shared open data agenda for MODUG

The day(s) after the event we discussed the output and how moving forward into 2018 and 2019 we can further support MAMPU and the Malaysian open data efforts. This meant diving much deeper into the detailed actions that need to be taken. I’m very much looking forward to staying involved.

Malaysia Open Data User Group
Working with the MAMPU team on next steps

Kuala Lumpur Kuala Lumpur
After work catching up with Baden and enjoying the sights

New Dutch Government Agreement: What It Says About Open Data

A new government has formed in the Netherlands, after a record 7 months of negotiations following elections last spring. I read the coalition agreement (pdf in Dutch) between the four parties involved to see what, if any, it means for digitisation, transparency, and the use and availability of data in the coming years.

Starting from the principle that openness and transparency in the public sector are important, the agreement states that digitisation is more than a necessity for that, and an opportunity for better public services as well. (p9 Public governance) This translates into plans to further digitise public services, an ‘ambitious’ national digital agenda also for lower level public entities, more findable and accessible open data, and a new look at the stalled Open Government Law with the aim to balance mandatory openness against implementation costs. In addition the agreement calls for more digital access to the collections of museums and archives (p21, Culture), and promises to publish all transport and mobility related government data so it can be reused by vehicles, apps and planners (p41 Transport and Mobility).

It’s good to see that data governance is getting attention, and that it seems to look at data governance from a holistic perspective, taking into account openness, privacy and information security together.
Citizens will have more control over their own personal data that government holds. (p9, Public governance)
The usage and ownership of travel data (think of GPS trackers, RFID travel cards, (autonomous) car sensor data) will be regulated to maintain privacy while also allowing (general) re-use of that data (p41 Transport and Mobility)
Internet of Things is getting attention in terms of aiming for standards, as part of an ‘ambitious’ national cyber-security agenda (p5 Justice and security)

This means a first few steps towards PDM will be taken, and that the ethics of Internet of Things and the role of regulation in acquiring and using sensor data in the public space are on the radar of this government both in maintaining safeguards and enabling new socio-economic value. That is a welcome development.

That socio-economic value however only becomes reality if citizens and companies are able to use the opportunities that open data and digital infrastructure provide. The government agreement promises money in this regard to enable a conducive investment climate as well as a European digital market (p4, p35 Economy). It also allocates funding to increase digital literacy (p11, Education), including for cyber-security awareness (p5 Justice and security), and to stimulate more investigative journalism (p22, Media). The agreement also proposes a new task for the Competition Authority in digital markets, to prevent dominant internet companies blocking new entrants.

Interestingly the agreement makes several references to competition law, or more precisely to strengthening the regulation against government activities competing with private enterprise in areas that are not deemed ‘public interest’. (p9. Public Governance and p36, Economy). This may have consequences for data holding agencies like the Cadastral Office (real estate ownership and transaction data) and Chamber of Commerce (companies register, beneficial ownership data) that currently provide paid for services on top of data they have free access to themselves but charge others for. For a long time already there’s been debate on opening that data up, but maintaining revenue streams for these public bodies has proven more important until now. Should competition law change, that may indeed tip the balance. Until now political will was lacking here.

In summary it looks like this government agreement will result in more open data, and more pressure on local and regional government entities to play their part. It also seems that openness, privacy and security are more seen as one issue of data governance, not as separate or mutually exclusive issues. Thirdly the agreement shows will to also help create the conditions in which that can result in societal value.

Binnenhof
The prime minister’s office, called ‘the little tower’, by Inyucho

How a Small Municipality Shows the Way with Open Data

In 2014/2015 my colleague Frank and I worked with the Province of North-Holland and 9 municipalities in that province to position open data as a policy instrument: around specific local issues we would publish data, and reach out to potential re-users. Part of this process was to make open data a normal part of every day work on public tasks. Hollands Kroon, a rural municipality in the very north of the Province was one of the participants that succeeded in bringing open data into line management.

Now they have launched a new municipal website, following the so-called ‘top tasks’ model. In this model the most prominent information shown is the information citizens most need or want. I have interacted with many municipalities that because of moving to a ‘top-tasks’ website refused to publish data or the answers to the FOIA requests they received. They said “we’re in the process of limiting the information in our sites to the most sought after, so we’re not going to publish any data etc, that would be confusing.”

Not so in Hollands Kroon. This is how their new site looks, with open data a very prominent menu option.

HK Website

With this step, Hollands Kroon shows how they have embraced open data. Already after the program with the Province, called North-Holland Smarter, they had formed a data team, working to raise internal awareness for open data and data driven work, and working to raise interest in re-use. Now they’ve gone a step further in making open data a significant part of their external communications.

To me this is all the more remarkable, as when we started in 2014 Hollands Kroon as a small rural municipality doubted whether open data could be a useful tool to them, and assumed it would only make sense in urban environments, such as in Amsterdam, the biggest city in the Province of North-Holland. They then quickly realized there is potential for their own local context and policy issues as well, especially if you work together with neighbouring municipalities in the region, in collaboration with the Province.