Today I heard the EU High Value Data list in its first iteration is finally decided upon. In September 2020 we submitted our advice on what data to include in the thematic areas of geographic data, statistics, mobility, company information, meteorology, earth observation and environment. Last week the Member States submitted their final yes/no vote, and the final text was approved. The EC will now finalise the text for publication, and it should be published before the end of the year. It will enter into force 20 days after publication and government data holders have 16 months until April/May 2024 to ensure compliance. It’s been a long path, and this first list could have been better concerning company information. Yet, when it comes to geographic data (addresses, buildings, land parcels, topography), meteorology and that same company information, it draws a line under two decades of discussion, court cases and studies to help dismantle the revenue model of charging at the point of use. Such charges are a threshold to market entry, and are generally lower than the tax revenue otherwise gained from the activities it’s a threshold to.

It’s easy to just move ahead and think about how this is not enough, what still needs doing, how to implement this etc. But it’s good to acknowledge that when I first started working on open government data in 2008 I heard the stories of those who had been at it for many years since well before the first PSI Directive was agreed in 2003. Some of those people have by now been retired for quite some time already, and I worked on it standing on their shoulders. The implementation act for EU high value data sets is a big step, even if in the field we thought it a no-brainer for decades already.

Today a colleague at the Netherlands Space Office showed me a new Copernicus service, the ground motion service (EGMS). Quite an amazing data service to explore. Earlier I wrote about the European forest fire information service (EFFIS), and its use as a proxy for the fighting going on due to the Russian invasion of Ukraine. EGMS is another service based on satellite remote sensing, here radar telemetry tracking the subsidence or rising of the ground. As far as I understand it can’t ‘see’ soft materials (peat land subsiding e.g.), only sees hard materials (solid ground, or buildings on softer grounds).
The images are quite amazing, and the data is provided right alongside it.

First an overview of northern Europe. Blue is rising ground, red is sinking ground. Sweden and Finland show rising ground, this is still the bounce back of the earth since the last ice age ended when the tremendous weight of glaciers was removed. At the tip of the arrow you see subsiding ground, this is the result of gas extraction in Groningen province.

Zooming in on Groningen province, here’s the data for a single house, subsiding 4 centimeters in the past 6 years. No wonder many homes are getting damaged in that area, both from subsidence as well as from the earthquakes that accompany it.

For comparison, here’s the data from the street I live on. It shows a subsidence of 6 millimeters in the past 6 years.

And here’s the same data as in the graph in the image above, but exported from the Copernicus services as an SVG, and pasted here as text.

-14-12-10-8-6-4-202468101214Displacement mm2016011120160428201608142016113020170318201707042017102020180211201805302018091520190101201904192019080520191121202003082020062420201010Measurement dateORTHO Vertical: 20dXRnBSzzDataset: Point ID: Position: Mean velocity: RMSE: ORTHO Vertical20dXRnBSzz3242050.00 N 4007550.00 E -0.60 m-1.10 mm/year0.40 mm

This week the draft implementation act (PDF) and annex listing the first batch of European High Value Data sets (PDF) has finally been published. In the first half of 2020 I was involved in preparatory research to advise on what data, spread across six predetermined themes, should be put on this mandatory list. It’s the first time open data policy makes the publication of certain data mandatory through an API. Until now European open data policy built upon the freedom of information measures of each EU Member State (MS), and added mandatory conditions to what MS published voluntarily, and to how to respond to public data re-use requests. This new law arranges for the pro-active publication of certain data sets.

In the 2020 research I was responsible for the sections about earth observation, environmental, and meteorological data. We submitted our final report in September 2020, and since then there had been total silence w.r.t. the progress in negotiating the list with the MS, and putting together the implementation act. I knew that at least the earth observation and environmental data would largely be included the way I suggested, when last summer I got a sneak preview of the adaptation of the INSPIRE portal where such data is made available.

The Implementation Act

In the Open Data Directive there’s a provision that the European Commission can, through a separate implementation act, set mandatory open data requirements for data belonging to themes listed in the Directive’s Annex. At launch in 2019, 6 such themes were listed: Geo-information, statistics, mobility, company information, earth observation / environment, and meteorology.
The list of themes can also be amended, through another separate implementation act, and a process to determine the second set of themes is currently underway.

The draft implementation law (PDF) states that government-held datasets mentioned in its Annex must be published through APIs, under an open license such as Creative Commons Zero, By Attribution or equivalent / less restrictive. Governments must publish the terms of use for such APIs and these terms may not be used to discourage re-use. APIs must also be fully publicly documented, and a point of contact must be provided.

MS can temporarily exempt some of the high value datasets, a decision that must be made public, but limited to two years after entry into force of this implementation act. Additional usage restrictions are allowed for personal data within the data sets concerned, but only to the extent needed to protect personal data of individuals (so not as an excuse to disallow re-use and access to the data as a whole).

MS must report on their implementation actions every two years, in which they need to list the actual data sets opened, the links to licenses, API and documentation, and exemptions still in place. The implementation is immediately binding for all MS (no need to first transpose into national law to be enforcable), will apply 20 days after publication in the EU Journal, and MS have 6 months to comply.

The Data Sets Per Theme

In this first batch of mandatory open data, 6 themes are covered (PDF). Some brief remarks on all of them.

Mobility

This is, contrary to what you’d expect, the smallest theme of the six covered. Because everything that is already covered in the Intelligent Transport (ITS) Directive is out of scope, which is most of everything concerning land based mobility. What remains for the High Value Data list is data on transport networks contained in the INSPIRE Annex I theme Transport Networks, and static and dynamic data about inland waterways, as well as the electronic navigational charts (ENC) for inland waterways. This is much in line with the 2020 study report. There was some concern with national hydrographical services about ENCs for seas being included (making it harder to force sea going vessels to use the latest version), but my reassurances that it would be unlikely held true.

Geospatial data

Geospatial data is I would say the ‘original’ high value government data, and has been for centuries. The data sets from the four INSPIRE Annex I themes Administrative Units, Geographical Names, Addresses, Buildings and Cadastral Parcels are within scope. Additionally reference parcels and agricultural parcels as described in the 1306/2013 and 1307/2013 Regulations on the Common Agricultural Policy (CAP) are on the list.

Earth Observation and Environment

This was a theme I was responsible for in the 2020 study. It is an extremely broad category, covering a very wide spectrum of types of data. It was basically impossible to choose something from this list, not in the least because re-use value usually comes from combinations of data, not from any single source used. Therefore my proposed solution was to not choose, and advise to treat it as a coherent whole needed in addressing the EU goals concerning environment/nature, climate adaptation, and pollution. The High Value Data list adopts this approach and puts 19 INSPIRE themes within scope. These are:

  • Annex I: Hydrography, and Protected Sites
  • Annex II in full: Elevation, Geology, Land Cover, and Ortho-imagery
  • Annex III: Area management, Bio-geographical regions, Energy resources, Environmental monitoring facilities, Habitats and biotopes, Land use, Mineral resources, Natural risk zones, Oceanographic geographical features, Production and industrial facilities, Sea regions, Soil, and Species distribution

Additionally all environmental information as covered by the 2003/4/EC Directive on public access to such information is added to the list, and all data originating in the context of a wide range of EU Regulations and Directives on air, climate, emissions, nature preservation and biodiversity, noise, waste and water. I miss soil in this environmental list, but perhaps the Annex III INSPIRE theme is seen as sufficiently covering it. I still need to follow up on the precise formulations w.r.t. data in 31 additionally referenced regulations and directives.

What to me is a surprising phrasing is that earth observation is defined here including satellite based data. Not surprising in terms of earth observation itself, but because satellite data was specifically excluded from the scope of our 2020 study. First because the EU level satellite data is already open. Second because this list deals with data from MS, and not many MS have their own satellite data. When they do it is usually the result of public private collaborative investment, and such private investment may dry up if there are no longer temporary exclusive access arrangements possible, which would have resulted in considerable political objections. Perhaps adding space based data collection is currently being well enough watered down by defining the INSPIRE themes as its scope, while at the same time future proofing the definition for when satellite data does become part of INSPIRE themes.

Together these first three, mobility, geospatial, and EO/environment, place a full 24 out of 34 INSPIRE themes on the list for mandatory open data. This basically amounts to adding an open data requirement to INSPIRE. It places MS’ INSPIRE compliance very much in the focus of attention, which now often is limited, and further positions INSPIRE as a key building block in the coming Green Deal dataspace. It will be of high interest to see what the coming new version of the INSPIRE directive, currently under review, makes of all that.

Statistics

This topic is more widely covered in the High Value Data list, than it was in the 2020 study, both in the types of statistics included, and in the demands made of those types of statistics. Still there are lots of statistics that MS hold, that aren’t included here (while some MS do publish most of their statistics already btw): the selection is based on European reporting obligations that follow from a list of various European laws.
Topics for which statistics must be published as open data in a specified way:

  • Industrial production
  • Industrial producer price index, by activity
  • Volume of sales by activity
  • EU international trade in goods
  • Tourism flows in Europe
  • Harmonised consumer prices indices
  • National accounts: GDP, key indicators on corporations and households
  • Government expenditure and revenue, government gross debt
  • Population, fertility, mortality
  • Current healthcare expenditure
  • Poverty
  • Inequality
  • Employment, unemployment, potential labour force

Data for these reporting obligations should be available from the moment the law creating them has been in force. That means for instance that healthcare expenditure should be available from at least 2008, whereas employment statistics must be available from at least 2019, because of the different years in which these laws were enacted.

Company information

Company information from the start has been the most controversial theme of the six covered by this implementation act. I assume this theme has also been the prime political reason for the long delay in the proposal being published. In my perception because this is the only data set that actually might end up challenging the status quo in society (as it involves ownership and power structures, and touches tax evasion). In the 2020 study four aspects were considered, the basic company information, company documents and accounts, ownership information, and insolvency status. Two ended up in the draft law: basic company information and company documents. Opening ownership information, not even the ultimate beneficial ownership (UBO) information, from the start drew vehement objections (including from the Dutch government). Many stakeholders (including the NGO I chair) are disappointed with the current outcome. (Here’s an old blogpost where I explain UBO, and here’s SF writer Brin on what transparent UBO might mean to our societies.) The data that will become open data still may be 2 years in the future: the Open Data Directive allows a 2 year exemption, and this is the data where that exemption will be used I think.
That said, mandatory open company data and documents, even with the delay through exemptions, is already a step forward that puts an end to literally decades of court cases, obstruction, and lobbying for more openness. The very first PSI Directive in 2003 was already an expression of a broad demand for this data, now 20 years on it finally becomes mandatory across the EU. Some people I know have been after this for their entire professional careers and already retired. It’s easy to loose sight of that win when we only focus on not having (ultimate) ownership data included.

Meteorological data

This is the other theme I was responsible for in the 2020 study. Like with company information this is an area where the discussion about making it available for re-use is decades old and precedes digitisation becoming ubiquitous. When I started my open data work in 2008, most of the existing documentation and argumentation for the value of and need for open data concerned meteorological data. A range of EU countries already have this as open data, others not at all. While progress has been made in the past decades, the High Value Data list provides a blanket obligation for all EU MS, a result that would otherwise still be a very long time away if entirely voluntary for the MS involved.
Data included here includes all weather observation data, validated observations / climate data, radar data (useful for things like cloud heights, precipitation and wind), and numerical weather prediction data (these are the outputs of the combined models used for predictions).

The implementation act is up for public feedback until 21 June, but likely will retain its current form. I think it’s a pretty good result, and I am happy that I have been able to contribute to it.

It’s finally here, published today: the proposal for the EU High Value Data list. The list for the first time makes open data publication mandatory for government concerning (for now) 6 themes (geographic information, meteorology, mobility, statistics, earth observation and environment, and company information). Already in September 2020 an impact assessment and advice on policy options was delivered to the European Commission. I was part of that assessment team, and responsible for the themes Meteorology, Earth Observation and environment. Now we get to see what has been proposed to be implementend in law. I haven’t read it yet, will do that tomorrow first thing, but wanted to share the link here. There’s a window for feedback on the proposal open until 21 June 2022.

Gebookmarkt Schoon water is in Nederland nog ver weg (door Marcel aan de Brugh in NRC)

In mijn werk aan de kansen, gevolgen, implementatie en hobbels rond de nieuwe Europese digitale en data-wetgeving zijn links naar beleidsdossiers als de energietransitie, de omgevingswet en hier de kaderrichtlijn water onvermijdelijk, en zelfs de bedoeling: de Europese Green Deal verwijst expliciet naar de Europese datastrategie. Bijvoorbeeld als het om digitale rapportages gaat die rechtstreeks voortkomen uit (open) brondata i.p.v. zelfrapportage uit een Lidstaat. Dat maakt de Data Governance verordening en de Open Data richtlijn (nog altijd niet in Nederlandse wetgeving omgezet), met de bijbehorende verplichtingen rond de Europese High Value Data lijst van belang in dit beleidsthema, evenals de voorziene GreenData4All richtlijn die ook de regelgeving rondom milieugegevens zal moderniseren in het licht van die datastrategie.

De kwaliteit van beschermde wateren loopt zo sterk achter op Europese normen, dat het een grond kan vormen om economische activiteiten te blokkeren, zoals in de stikstofcrisis is gebeurd.

In de Europese Unie moeten vanaf 2027 alle aan Brussel gerapporteerde wateren in „een goede ecologische toestand” zijn. Dat is wettelijk bepaald in de Kaderrichtlijn Water, die sinds 2000 van kracht is. In 2019 was minder dan 1 procent in die vereiste goede ecologische toestand…

Marcel aan de Brugh in NRC


De Nieuwe Maas in Rotterdam, een van de ruim 700 Nederlandse wateren waar het om gaat. Foto Ton Zijlstra, licentie CC BY NC SA

Vandaag sluit de internetconsultatie voor de implementatiewet van de open data richtlijn. Ik schreef er met Kerst al over. Zojuist diende ik mijn zienswijze op het wetsvoorstel in. Die is samen met de andere openbare reacties beschikbaar online.

Als basis voor mijn reactie vergeleek ik de ontwerptekst (pdf), de memorie van toelichting (pdf), de Europese open data richtlijn, en ook de tekst van de huidige hergebruikswet. Want ook van wat er niet verandert kun je iets vinden tenslotte. Mijn/onze ervaringen in het veld sinds de allereerste open data richtlijn van kracht werd, vormen de bril die ik opzette.

Oorspronkelijk zou ik mijn reactie schrijven vanuit het perspectief van een klant. Maar na grondige beschouwing wordt die niet of nauwelijks geraakt door de wijzigingen, en dat maakte een reactie vanuit die organisatie niet heel nuttig. Tegelijkertijd is er wel genoeg te zeggen over het wetsvoortsel. Dus deed ik het vanuit mijn persoonlijke perspectief, en dat van ons bedrijf.

Het is prettig te zien dat ook andere organisaties hebben gereageerd. Dat maakte het mogelijk om zelf nog een paar dingen aan te voeren die anderen juist niet noemden, en er op te vertrouwen dat andere punten al voldoende door andere partijen waren belicht.

Nu is het wachten op het uiteindelijk voorstel, en de goedkeuring daarvan. Nederland is laat met deze wet, hij had vorig jaar juli al van kracht moeten zijn. De invoering van de Open Data richtlijn is één van de bouwstenen van de Europese datastrategie, en er staat veel te gebeuren. Dus hoog tijd dat Nederland hier stappen in zet. Dan kunnen we ook makkelijker met elkaar de kansen die er bij horen verzilveren.