Forget Data Mining, We Have Social Software
Malcolm Gladwell’s article in last week’s New Yorker, Open Secrets, is well worth the read.
The piece is build on a distinction between puzzles and mysteries:
Osama bin Laden’s whereabouts are a puzzle. We can’t find him because we don’t have enough information. The key to the puzzle will probably come from someone close to bin Laden, and until we can find that source bin Laden will remain at large.
The problem of what would happen in Iraq after the toppling of Saddam Hussein was, by contrast, a mystery. It wasn’t a question that had a simple, factual answer. Mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have too little information but that we have too much.
Using anecdotes from business, national security and medicine, Gladwell shows mysteries and analysis becoming increasingly prevalent in today’s world. This is a challenging conclusion for news operations designed to publish information and leave analysis up to readers.
Michal Migurski says this is why news organizations should focus more on analysis than data creation: “I don’t think the value of a newspaper is in its ability to populate a SQL table of obits or mayoral appearances,” he says. (Like many good things, I found Migurski’s post on The Scoop.)
No question, publishers need to get better at sorting and analyzing. But that doesn’t necessarily mean diving into data mining, network analysis and other traditional techniques that Migurski mentions. The same technology that Gladwell worries is creating a surfeit of information is enabling new methods of analysis.
Consider the social analysis techniques of successful web services like Google, Flickr and Delicious. Each of these sites uses the collective intelligence of web users to solve analytical problems that would be intractable for any single individual.
News sites can do the same thing. They can populate their SQL tables, publish them, let users tear through them, keep track of what interests users, and bubble those bits to the surface. The Sunlight Foundation’s Exposing Earmarks campaign is a great example of how this can be done.
Migurski and Gladwell seem to view analysis as the domain of Berkeley Ph.D’s. We’re now seeing that doesn’t have to be the case. By publishing data and structuring the interface in the right way, publishers can use the richness of their community to solve mysteries.
