May. 04, 2012

Why some, but not all, of "Open Data Movement" is a joke

Tom Slee struck a nerve writing a blog post titled Why the "Open Data Movement" is a joke. Only problem with it was that he should have been more specific.  Readers who weren't well versed in Canadian politics didn't pick up on the localized nuances they were supposed to, and instead reacted to their very own reflections in the mirror.

His complaint is spot on when it addresses the range between datawashing and its directly related "open data" businesses that are sprouting like mushrooms after a rain. Though the real open data movement issue has nothing to do with data. It has to do with the old saying "You can lead a horse to water, but you can't make it drink."  Specifically, mismanagement.

Proper Use of Data

Tina Rosenberg from NYT wrote a recent damning piece called "Armed With Data, Fighting More Than Crime." The piece is a good example of appropriate open data as it addresses crime, data, data collections and especially decisions made with that data in the new (sometimes unspoken) sphere of open data movement.

But, this process is nothing new in the world. We've always made informed decisions based on analysis.

Census tallying used to take decades, then years, then months. Now that this generic data collection asymptote approaches zero, it's triggered a series of events that people, administrations and businesses are reacting to.  Same technological shift that made this possible is allowing for us to have greater sample sizes, to bypass survey vendors and sample directly, and perform  instantaneous analysis as the data comes in. Faster. More accurate.

We now have to make faster decisions

Message that open data movement carries puts pressure on decision makers, especially in the public sector. If they find their organization ill-equipped to handle this accelerated process, they can cop out and embrace an open data vendor and, paraphrasing a set of PR releases, let "citizen journalists engage with open government data and collaborate on solving problems while increasing transparency and improving level of trust between people and their government."  As noble as that sounds, most directives requiring data sets be released don't ever address a level of significance for that data.

If all you do is make a quota of compulsorily collected data sets available, you're doing it wrong:

  • Vendor's catalog of all 'available' data implies that it's a finite set, and there is nothing else to be had
  • No compendium exists of hidden data sets, or reasons why they are unlisted
  • Data limits are vague, if even provided: no collection timing lag, related costs
  • No context is provided on how original data provider uses the data sets

Putting finding gotchas on citizens evades responsibility, especially when it's inane data sets whitelisted by an anonymous employee.

It's a disconnected chain: Vendors can shrug their shoulders. That's all the data they got. Officials can shrug their shoulders. They released data, and in a budget-strapped year, it was John Q. Citizen who failed to spot an indicator of a major issue. And where was the newspaper? They always complain about not getting data, and now we handed it all to them. Why are you calling about data? It's all listed on our site.

Management Shortfall

Intentional, or unintentional, this is datawashing. Even Baltimore, a fairly progressive city in this data realm, has been caught under-reporting rape incidents to the tune of 80% over the past decade.  It took an investigation by the Baltimore Sun for the city to admit they've been gaming statistics.

As Tina expands her article, this data governance movement is merely a cosmetic change for some organizations. An intelligent process is being visually mimicked, but there is no improved accountability or responsibility, and there are no reactive follow ups.  Worst yet, there is no evidence that management is using any of the data to make decisions.

Instead, the allegedly open data is put out there and indirectly placates the public. While I understand that open data movement figureheads can't bite the hand that feeds them, their collective cheerleading minimizes actual issues and limits they incur.

Shirking from responsibility is a joke.

Find this interesting, or useful? Consider sharing the post.

Leave a Reply

Your email address will not be published. Required fields are marked *

Posts on this blog solely represent my personal opinions and technical experience.

© 2009-2019 Edin (Dino) Beslagic