Probably the top left gives the most realistic information. Image by Brooke Novak, license CC BY

An organisation that says it wants to work data driven as well as sees ethics as a key design ingredient, needs to take a very close look imho at how they set KPI’s and other indicators. I recently came across an organisation that says those first two things, but whose process of setting indicators looks to have been left as a naive exercise to internal teams.

To begin with, indicators easily become their own goals, and people will start gaming the measurement system to attain the set targets. (Think of call centers picking up the phone and then disconnecting, because they are scored on the number of calls answered within 3 rings, but the length of calls isn’t checked for those picked up)

Measurement also isn’t neutral. It’s an expression of values, regardless of whether you articulated your values. When you measure the number of traffic deaths for instance as an indicator for road safety, but not wounded or accidents as such, nor their location, you’ll end up minimising traffic deaths but not maximising road safety. Because the absence of deaths isn’t the presence of road safety. Deaths is just one, albeit the most irreparable one, expression of the consequences of unsafety. Different measurements lead to different real life actions and outcomes.

‘Gauges’ by Adam Kent, license CC BY

When you set indicators it is needed to evaluate what they cover, and more importantly what they don’t cover. To check if the overall set of indicators is balanced, where some indicators by definition deteriorate when others improve (so balance needs to be sought). To check if assumptions behind indicators have been expressed and when needed dealt with.

Otherwise you are bound to end up with blind spots, lack of balance, and potential injustices. Defined indicators also determine what data gets collected, and thus what your playing field is when you have a ‘data driven’ way of working. That way any blind spot, lack of balance and injustice will end up even more profoundly in your decisions. Because where indicators mostly look back in time at output, data driven use of the data underlying those indicators actively determines actions and thus (part of) future output, turning your indicators in a much more direct and sometimes even automated feedback loop.

Only if you’ve deliberately defined your true north, can you use your measurements to determine direction of your next steps. ‘Compass’ by Anthony, license CC BY ND