Big Data Beat Up: Backlash Attacks on the Technology

After considerable positive hype, the skirmishes and attacks against Big Data technology have begun.

On the one hand there are those that either say we don’t need Big Data or that Big Data isn’t really anything different than standard data management and analysis.  For example, Jim Goodnight, SAS CEO, said that “the term big data is being used today because computer analysts and journalists got tired of writing about cloud computing.  Before cloud computing it was data warehousing or ‘software as a service’. There’s a new buzzword every two years and the computer analysts come out with these things so that they will have something to consult about.”

On the other hand, there has also been backlash against what people see as a misguided use of technology because they’ve tried to use Big Data without fully understanding the tools.

David Lazer, professor at NorthEastern University, points out issues when technology is applied to solve problems without having sufficient understanding for how to use or apply the technology.  “‘Big data hubris’ is the often implicit assumption that big data is a substitute for, rather than a supplement to, traditional data collection and analysis. We have asserted that there are enormous scientific possibilities in big data. However, quantity of data does not mean that one can ignore foundational issues of measurement, construct validity and reliability, and dependencies among data. The core challenge is that most big data that have received popular attention are not the output of instruments designed to produce valid and reliable data amenable for scientific analysis.”

Mike Loukides wrote for O’Reilly Radar that  “Ignore the hype.  Learn to be a data skeptic. That doesn’t mean becoming skeptical about the value of data; it means asking the hard questions that anyone claiming to be a data scientist should ask. Think carefully about the questions you’re asking, the data you have to work with, and the results that you’re getting. And learn that data is about enabling intelligent discussions, not about turning a crank and having the right answer pop out.”

A recent article in the New York Times identifies problems with Big Data which include:

• While Big Data can identify correlations in data, it can’t help us put those relationships into the context of the problem being solved or even whether we should treat the correlations as meaningful or as bogus.
• Using Big Data as a black box by someone with limited skills in analysis or scientific inquiry is sure to fail
• Without a scientific approach to problem solving, Big Data may be applied to solving ‘imprecise questions’, achieving results of limited value