This article is a good description of the Freedom of Information (#foia #opengov #opendata) situation in the Balkans. Due to my work in the region, I recognise lots of what is described here. My work in the region, such as in Serbia, has let me encounter various institutions willing to use evasive action to prevent the release of information.

In essence this is not all that different from what (decentral) government entities in other European countries do as well. Many of them still see increased transparency and access as a distraction absorbing work and time they’d rather spend elsewhere. Yet, there’s a qualitative difference in the level of obstruction. It’s the difference between acknowledging there is a duty to be transparant but being hesitant, and not believing that there’s such a duty in governance at all.

Secrecy, sometimes in combination with corruption, has a long and deep history. In Central Asia for instance I encountered an example that the number of agricultural machines wasn’t released, as a 1950’s Soviet law still on the books marked it as a state secret (because tractors could be mobilised in case of war). More disturbingly such state secrecy laws are abused to tackle political opponents in Central Asia as well. When a government official releases information based on a transparency regulation, or as part of policy implementation, political opponents might denounce them for giving away state secrets and take them to court risking jail time even.

There is a strong effort to increase transparency visible in the Balkan region as well. Both inside government, as well as in civil society. Excellent examples exist. But it’s an ongoing struggle between those seeing power as its own purpose and those seeking high quality governance. We’ll see steps forward, backwards, rear guard skirmishes and a mixed bag of results for a long time. Especially there where there are high levels of distrust amongst the wider population, not just towards government but towards each other.

One such excellent example is the work of the Serbian information commissioner Sabic. Clearly seeing his role as an ombudsman for the general population, he and his office led by example during the open data work I contributed to in the past years. By publishing statistics on information requests, complaints and answer times, and by publishing a full list of all Serbian institutions that fall under the remit of the Commission for Information of Public Importance and Personal Data Protection. This last thing is key, as some institutions will simply stall requests by stating transparency rules do not apply to them. Mr. Sabic’s term ended at the end of last year. A replacement for his position hasn’t been announced yet, which is both a testament to Mr Sabic’s independent role as information commissioner, and to the risk of less transparency inclined forces trying to get a much less independent successor.

Bookmarked Right to Know: A Beginner’s Guide to State Secrecy / Balkan Insight by Dusica Pavlovic (Balkan Insight)

Governments in the Balkans are chipping away at transparency laws to make it harder for journalists and activists to hold power to account.

There were several points made in the conversation after my presentation yesterday at Open Belgium 2019. This is a brief overview to capture them here.

1) One remark was about the balance between privacy and openness, and asking about (negative) privacy impacts.

The framework assumes government as the party being interested in measurement (given that that was the assignment for which it was created). Government held open data is by default not personal data as re-use rules are based on access regimes which in turn all exclude personal data (with a few separately regulated exceptions). What I took away from the remark is that, as we know new privacy and other ethical issues may arise from working with data combinations, it might be of interest if we can formulate indicators that try to track negative outcomes or spot unintended consequences, in the same way as we are trying to track positive signals.

2) One question was about if I had included all economic modelling work in academia etc.

I didn’t. This isn’t academic research either. It seeks to apply lessons already learned. What was included were existing documented cases, studies and research papers looking at various aspects of open data impact. Some of those are academic publications, some aren’t. What I took from those studies is two things: what exactly did they look at (and what did they find), and how did they assess a specific impact? The ‘what’ was used as potential indicator, the ‘how’ as the method. It is of interest to keep tracking new research as it gets published, to augment the framework.

3) Is this academic research?

No, its primary aim is as a practical instrument for data holders as well as national open data policy makers. It’s is not meant to establish scientific truth, and completely quantify impact once and for all. It’s meant to establish if there are signs the right steps are taken, and if that results in visible impact. The aim, and this connects to the previous question as well, is to avoid extensive modelling techniques, and favor indicators we know work, where the methods are straightforward. This to ensure that government data holders are capable to do these measurements themselves, and use it actively as an instrument.

4) Does it include citizen science (open data) efforts?

This is an interesting one (asked by Lukas of Luftdaten.info). The framework currently does include in a way the existence and emergence of citizen science projects, as that would come up in any stakeholder mapping attempts and in any emerging ecosystem tracking, and as examples of using government open data (as context and background for citizen science measurements). But the framework doesn’t look at the impact of such efforts, not in terms of socio-economic impact and not in terms of government being a potential user of citizen science data. Again the framework is to make visible the impact of government opening up data. But I think it’s not very difficult to adapt the framework to track citizen science project’s impact. Adding citizen science projects in a more direct way, as indicators for the framework itself is harder I think, as it needs more clarification of how it ties into the impact of open government data.

5) Is this based only on papers, or also on approaching groups, and people ‘feeling’ the impact?

This was connected to the citizen science bit. Yes, the framework is based on existing documented material only. And although a range of those base themselves on interviewing or surveying various stakeholders, that is not a default or deliberate part of how the framework was created. I do however recognise the value of for instance participatory narrative inquiry that makes the real experiences of people visible, and the patterns across those experiences. Including that sort of measurements would be useful especially on the social and societal impacts of open data. But currently none of the studies that were re-used in the framework took that approach. It does make me think about how one could set-up something like that to monitor impact e.g. of local government open data initiatives.

Today I gave a brief presentation of the framework for measuring open data impact I created for UNDP Serbia last year, at the Open Belgium 2019 Conference.

The framework is meant to be relatable and usable for individual organisations by themselves, and based on how existing cases, papers and research in the past have tried to establish such impact.

Here are the slides.

This is the full transcript of my presentation:

Last Friday, when Pieter Colpaert tweeted the talks he intended to visit (Hi Pieter!), he said two things. First he said after the coffee it starts to get difficult, and that’s true. Measuring impact is a difficult topic. And he asked about measuring impact: How can you possibly do that? He’s right to be cautious.

Because our everyday perception of impact and how to detect it is often too simplistic. Where’s the next Google the EC asked years ago. but it’s the wrong question. We will only know in 20 years when it is the new tech giant. But today it is likely a small start-up of four people with laptops and one idea, in Lithuania or Bulgaria somewhere, and we are by definition not be able to recognize it, framed this way. Asking for the killer app for open data is a similarly wrong question.

When it comes to impact, we seem to want one straightforward big thing. Hundreds of billions of euro impact in the EU as a whole, made up of a handful of wildly successful things. But what does that actually mean for you, a local government? And while you’re looking for that big impact you are missing all the smaller craters in this same picture, and also the bigger ones if they don’t translate easily into money.

Over the years however, there have been a range of studies, cases and research papers documenting specific impacts and effects. Me and my colleagues started collecting those a long time ago. And I used them to help contextualise potential impacts. First for the Flemish government, and last year for the Serbian government. To show what observed impact in for instance a Spanish sector would mean in the corresponding Belgian context. How a global prediction correlates to the Serbian economy and government strategies.

The UNDP in Serbia, asked me to extend that with a proposal for indicators to measure impact as they move forward with new open data action plans in follow up of the national readiness assessment I did for them earlier. I took the existing studies and looked at what they had tried to measure, what the common patterns are, and what they had looked at precisely. I turned that into a framework for impact measurement.

In the following minutes I will address three things. First what makes measuring impact so hard. Second what the common patterns are across existing research. Third how, avoiding the pitfalls, and using the commonalities we can build a framework, that then in itself is an indicator.Let’s first talk about the things that make measuring impact hard.

Judging by the available studies and cases there are several issues that make any easy answers to the question of open data impact impossible.There are a range of reasons measurement is hard. I’ll highlight a few.
Number 3, context is key. If you don’t know what you’re looking at, or why, no measurement makes much sense. And you can only know that in specific contexts. But specifying contexts takes effort. It asks the question: Where do you WANT impact.

Another issue is showing the impact of many small increments. Like how every Dutch person looks at this most used open data app every morning, the rain radar. How often has it changed a decision from taking the car to taking a bike? What does it mean in terms of congestion reduction, or emission reduction? Can you meaningfully quantify that at all?

Also important is who is asking for measurement. In one of my first jobs, my employer didn’t have email for all yet, so I asked for it. In response the MD asked me to put together the business case for email. This is a classic response when you don’t want to change anything. Often asking for measurement is meant to block change. Because they know you cannot predict the future. Motives shape measurements. The contextualisation of impact elsewhere to Flanders and Serbia in part took place because of this. Use existing answers against such a tactic.

Maturity and completeness of both the provision side, government, as well as the demand side, re-users, determine in equal measures what is possible at all, in terms of open data impact. If there is no mature provision side, in the end nothing will happen. If provision is perfect but demand side isn’t mature, it still doesn’t matter. Impact demands similar levels of maturity on both sides. It demands acknowledging interdependencies. And where that maturity is lacking, tracking impact means looking at different sets of indicators.

Measurements often motivate people to game the system. Especially single measurements. When number of datasets was still a metric for national portals the French opened with over 350k datasets. But really it was just a few dozen, which they had split according to departments and municipalities. So a balance is needed, with multiple indicators that point in different directions.

Open data, especially open core government registers, can be seen as infrastructure. But we actually don’t know how infrastructure creates impact. We know that building roads usually has a certain impact (investment correlates to a certain % rise in GDP), but we don’t know how it does so. Seeing open data as infrastructure is a logical approach (the consensus seems that the potential impact is about 2% of GDP), but it doesn’t help us much to measure impact or see how it creates that.

Network effects exist, but they are very costly to track. First order, second order, third order, higher order effects. We’re doing case studies for ESA on how satellite data gets used. We can establish network effects for instance how ice breakers in the Botnian gulf use satellite data in ways that ultimately reduce super market prices, but doing 24 such cases is a multi year effort.

E puor si muove! Galileo said Yet still it moves. The same is true for open data. Most measurements are proxies. They show something moving, without necessarily showing the thing that is doing the moving. Open data often is a silent actor, or a long range one. Yet still it moves.

Yet still it moves. And if we look at the patterns of established studies, that is what we indeed see. There are communalities in what movement we see. In the list on the slide the last point, that open data is a policy instrument is key. We know publishing data enables other stakeholders to act. When you do that on purpose you turn open data into a policy instrument. The cheapest one you have next to regulation and financing.

We all know the story of the drunk that lost his keys. He was searching under the light of a street lamp. Someone who helped him else asked if he lost the keys there. No, the drunk said, but at least there is light here. The same is true for open data. If you know what you published it for, at least you will be able to recognise relevant impact, if not all the impact it creates. Using it as policy instrument is like switching on the lights.

Dealing with lack of maturity means having different indicators for every step of the way. Not just seeing if impact occurs, but also if the right things are being done to make impact possible: Lead and lag indicators

The framework then is built from what has been used to establish impact in the past, and what we see in our projects as useful approaches. The point here is that we are not overly simplifying measurement, but adapt it to whatever is the context of a data provider or user. Also there’s never just one measurement, so a balanced approach is possible. You can’t game the system. It covers various levels of maturity from your first open dataset all the way to network effects. And you see that indicators that by themselves are too simple, still can be used.

Additionally the framework itself is a large scale sensor. If one indicator moves, you should see movement in other indicators over time as well. If you throw a stone in the pond, you should see ripples propagate. This means that if you start with data provision indicators only, you should see other measurements in other phases pick up. This allows you to both use a set of indicators across all phases, as well as move to more progressive ones when you outgrow the initial ones.finally some recommendations.

Some final thoughts. If you publish by default as integral part of processes, measuring impact, or building a business case is not needed as such. But measurement is very helpful in the transition to that end game. Core data and core policy elements, and their stakeholders are key. Measurement needs to be designed up front. Using open data as policy instrument lets you define the impact you are looking for at the least. The framework is the measurement: Only micro-economic studies really establish specific economic impact, but they only work in mature situations and cost a lot of effort, so you need to know when you are ready for them. But measurement can start wherever you are, with indicators that reflect the overall open data maturity level you are at, while looking both back and forwards. And because measurement can be done, as a data holder you should be doing it.

US Congress just before leaving for Christmas has voted to approve a new law, that mandates two key elements: public information is open by default and needs to be made actively available in machine readable format, as well as that policy making should be evidence based. In order for agencies to comply they will need to appoint a Chief Data Officer.

I think while of those two the first one (open data) is the more immediately visible, the second one, about evidence based policy making, is much more significant long term. Government, especially politics, often is willingly disinterested in policy impact evaluation. It’s much more status enhancing to announce new plans than admitting previous plans didn’t come to anything. Evidence based policy will help save money. Additionally government agencies will soon realise that doing evidence based policy making is made a lot easier if you already do open data well. The evidence you need is in that open data, and it being open allows all of us to go look for that evidence or its absence.

There’s one caveat to evidence based policy making: it runs the risk of killing any will to experiment. After all, by definition there’s no evidence for something new. So a way is needed in which new policies can be tried out as probes. To see if there’s emerging evidence of impact. Again, that evidence should become visible in existing open data streams. If evidence is found the experimental policy can be rolled out more widely. Evidence based policies need experiments to help create an evidence base, not just of what works but also of what doesn’t.

A great result for the USA’s open government activists. This basically codifies the initiatives of the Obama Presidency, which were the trigger for much of the global open data effort these last 10 years, into US federal law.

Recently I have been named the new chairman of the board of the Open State Foundation. This is a new role I am tremendously looking forward to take up. The Open State Foundation is the leading Dutch NGO concerning government transparency, and over the past years they’ve both persistently and in a very principled way pursued open data and government transparency, as well as constructively worked with government bodies to help them do better. Stef van Grieken, the chairman stepping down, has led the Open State Foundation board since it came into existence. The Open State Foundation is the merger of two earlier NGO’s, The New Voting (Het Nieuwe Stemmen) foundation of which Stef was the founder, and the Hack the Government (Hack de Overheid) collective.

Hack de Overheid emerged from the very first Dutch open government barcamp James Burke, Peter Robinett and I organised in the spring of 2008. The second edition in 2009 was the first Hack de Overheid event. My first open data project that same spring was together with James Burke and Alper Çuğun, both part of Hack de Overheid then and providing the tech savvy, and me being the interlocutor with the Ministry for the Interior, to guide the process and interpret the civil servant speak to the tech guys and vice versa. At the time Elsevier (a conservative weekly) published an article naming me one of the founders of Hack de Overheid, which was true in spirit, if technically incorrect.

In the past year and a half I had more direct involvement with the Open State Foundation than in the years between. Last year I did an in-depth evaluation of the effectiveness and lasting impact of the Open State Foundation in the period 2013-2017 and facilitated a discussion about their future, at the request of their director and one of their major funders. That made me appreciate their work in much richer detail than before. My company The Green Land and Open State Foundation also encounter each other on various client projects, giving me a perspective on the quality of their work and their team.

When Stef, as he’s been working in the USA for the past years, indicated he thought it time to leave the board, it coincided with me having signalled to the Open State Foundation that, if there ever was a need, I’d be happy to volunteer for the board. That moment thus came sooner than I expected. A few weeks ago Stef and I met up to discuss it, and then the most recent board meeting made it official.

Day to day the Open State Foundation is run by a very capable team and director. The board is an all volunteer ‘hands-off’ board, that helps the Open State Foundation guard its mission and maintain its status as a recognised charity in the Netherlands. I’m happy that I can help the Open State Foundation to stay committed to their goals of increasing government transparency and as a consequence the agency of citizens. I’m grateful to Stef, and the others that in the past decade have helped Open State Foundation become what it is now, from its humble beginnings at that barcamp in the run-down pseudo-squat of the former Volkskrant offices, now the hipster Volkshotel. I’m also thankful that I now have the renewed opportunity to meaningfully contribute to something I in a tiny way helped start a decade ago.

Last week I presented to a provincial procurement team about how to better support open data efforts. Below is what I presented and discussed.

Open data as policy instrument and the legal framework demands better procurement

Publishing open data creates new activity. It does so in two ways. It allows existing stakeholders to do more themselves or do things differently. It also allows people who could not participate before become active as well. We’ve seen for instance how opening up provincial and national geographic data increases the independent usage of that data by local governments. We’ve also seen how for instance the Dutch hiking association started using national geographic data to create and better document routes. To the surprise of the Cadastre a whole new area of usage appeared as well, by cultural organisations who before had never requested such data. So open data is an enabler for agency.

If as a government data holder you know this effect takes place, you can also try and achieve it deliberately. For policy domains and groups of stakeholders where you would like to see more activity, publishing data then is an instrument in for instance achieving your own policy goals. Next to regulation and financing, publishing open data is a new third policy instrument. It also happens to be the cheapest of those three to deploy.

Open data in the EU has a legal framework where over time more things are mandated. There is a right to re-use. Upon request dataholders must be able to provide machine readable data formats. In the Netherlands open standards are compulsory for government entities since 2008. Exclusive access to government data for re-use is, except for a few very strictly regulated situations, illegal.

To be able to comply with the legal framework, and to be able to actively use open data as a policy instrument, public sector bodies must pay more attention to how they acquire data, and as a consequence must pay more attention to what happens during procurement processes. If you don’t the government entity’s data sovereignty is strongly diminished, which carries costs.

Procurement awareness needed on multiple levels

The goal is to ensure full data sovereignty. This means paying real attention to various things on different levels of abstraction around procurement.

  • Ensuring data is received in open standards and regular domain specific standards
  • Ensure when reports are received that the data used, such as for graphs and tables, are also received
  • Ensure when information products are received (maps, visualisations) the data used for them are also received
  • Ensure procurement and collaboration contracts do not preclude sharing data with third parties, apart from on grounds already mentioned as exceptions in the law on freedom of information and re-use
  • Ensure that when raw data is provided to service providers, that data is still available to the government entity
  • Ensure that when data is collected by external entities who in turn outsource the collection, all parties involved know the data falls under the decision making power of the government entity
  • Ensure in collaborations you do not sign away decision power over the data you contribute, you have rights to the data you collectively create, and have as little restriction as possible on the data others contribute.

What could go wrong?

Unless you always pay attention to these points, you run the risk of losing your data sovereignty. This can lead to situations where a government entity is no longer able to comply with its own legal obligations concerning data provision and transparency.

A few existing examples from what can go wrong.

  • A province is counting bicycle traffic through a network of sensors they deployed themselves. The data is directly transmitted to a service provider in a different country. The province can see dashboards and download reports, but has no access to the sensor data itself, and cannot download the sensor data. While any citizen requesting the data could not be provided with that data, the service provider itself does base commercial services on that and other data it receives, having de facto exclusive access to it.
  • Another province is outsourcing bird inventory counting to nature preservation organisations, who in turn rely on volunteers to do the bird watching. The province pays for the effort. When it comes to sharing the data publicly, the nature preservation organisations say their volunteers actually own the data, so nothing can be publicly shared. This is untrue for multiple reasons (database rights do not apply, it is a paid for effort so procurement terms that unequivocally transfer such rights should they exist to the province etc), but as the province doesn’t want to waste time on this, nor wants to get into a fight, it leaves it be, resulting in the data not being made available.
  • An energy network provider pools a lot of different data sources concerning energy usage in their service area from a network of collaborating entities, both private and public. They also publish a lot of open data already. As part of the national effort towards energy transition they receive many data requests from local governments, housing associations and other entities. They would like to provide data, as they see it as a way of contributing to an essential public task (energy transition), but still say no to data requests in 60% of all cases. Because they can’t figure out which contractual obligations apply to which parts of the data, or cannot reconcile conflicting or ambiguous contract clauses concerning the data.
  • All provinces pool data concerning economic activity and the labor market in a private foundation in which also private entities participate. That foundation sells data subscriptions. Currently they also publish some open data, but if any of the provinces would like to do more, they would have to wait for full agreement. The slowest in the group would determine the actual level of transparency.
  • A province has outsourced the creation of a ‘heat transition atlas’, in which the potential for moving away from natural gas burning heating systems in homes using various alternatives is mapped. The resulting interactive website contains different data layers, but those data layers are themselves unavailable. Although there is a general list of which data sources have been used, it is not precisely stating its sources and not providing details on how the data has been transformed for the website.

In all cases the public sector data holder has put itself in a position that could have been prevented had they paid more attention at the time of procurement or at the time of entering into collaboration. All these situations can be fixed later on, but they require additional effort, time and costs to arrange, which are unnecessary if dealt with during procurement.

But we have procurement regulations already!

What about procurement regulations. We have those, so don’t they cover all this? Mostly not it turns out.

Terms of procurement talk about rights transfer of all deliverables, but in many cases the data involved isn’t listed as a deliverable, so not covered by those terms.
The terms talk about transfer of database rights, but those hardly ever apply as usually the scale of data collection and structuring into a database is limited.
Concerning research there is some talk about also transferring the data concerned, but a lot of reports aren’t research but consultancy services.

In the general regulations that apply to provincial procurement, the word data only is used in the context of personal data protection, as the dutch plural for date, and in the context of data carriers (hard drives etc). The word standards never occurs, nor does it contain references to data formats (even though legal obligations exist for government entities concerning standards and data formats)

The procurement terms are neither broad enough, nor detailed enough.

How to improve the situation

So what needs to be arranged to ensure government entities arrange their data needs correctly during procurement? How to plug the holes? A few things at the very least:

  • Likely, when it comes to standards and formats (which may differ per domain), the only viable place is in the mandatory technical requirements in a call for tender / request for proposals.
  • To get the data behind graphs, tables, info products and reports, including a list of resources and transformations applied, it needs to be specified in the list of deliverables.
  • Collaboration contracts entered into should always have articles on sharing the data you contribute, being able to share the data resulting from the collaboration, and rules about data that others contribute.

It is important to realise that you cannot through contracts do away with any mandatory transparency, open data, or data governance aspects. Any resulting issues will mean time consuming and likely costly repair activities.

Who needs to be involved

In order to prevent the costs of repair or mitigation of consequences, there are a number of questions concerning who should be doing what, inside a government entity.

  • What needs to be arranged at the point of tender, who will check it?
  • What needs to be part of all project starts (e.g. Checklists, data paragraphs), is the project manager aware of this, and who will check it?
  • Who at the writing and signing of any contract will check data aspects?
  • Who at the time of delivery will check if data requirements are met?
  • What part of this is more about awareness and operatios, what needs to be done through regulation?

Our work in the next steps

We intend to assist the province involved in making sure procurement better enables data sharing from now on. Steps we are currently taking to move this forward are:

  • We’ve put data sovereignty into the organisations strategy document, and tied it into overall data governance improvement.
  • With the information management department we’ll visit all main procurers to discuss and propose actions
  • We’ll likely build one or more checklists for different aspects
  • We’ll work with a 3 person team from the procurement department to more deeply embed data awareness and amend procurement processes

All this is basically a preventative step to ensure the province has its house in order concerning data.