Big Data/Hadoop: Death to the Data Warehouse?

There’s a war on now between traditional enterprise data warehousing (EDW) versus Big Data analytics, particularly tools powered by Hadoop.  Data warehousing is losing out as businesses become overwhelmed with data.  Pulling all data into a single monolithic database and then massaging it to fit into a structured data model becomes an ever more difficult problem.  Hadoop and Big Data offer  alternative approaches that spans many servers, are capable of managing huge data sets, and which are significantly more tolerant to unstructured and semi-structured data.

Ron Bodkin, President of Think Big, said that “I think that the industry has started to come along with the notion of a data reservoir where you can hold a wider variety of data, breaking down silos, and do things with data that’s not in as a structured or simple format, to allow connecting of a lot of that information, allowing you to do deep analytics, and allowing you to work with raw data in quite a comprehensive way. It [Big Data Technologies] has a lot of synergy compared to the downstream data warehousing world.”

Michael C. Daconta, author and technologist, predicts that “Hadoop (and the advent of NoSQL databases) will auger the demise of data warehousing appliances and the traditional single database implementation of a data warehouse. ”

Numbers seem to be supporting the prediction.  For example, a recent Wikibon Big Data survey found that:

• 65 percent of enterprises doing Big Data say that they’re actively shifting resources from data warehouses to instead use Hadoop and other Big Data technologies.  The other 35 percent say that they’re ready to to begin a shift of their data warehousing projects to Hadoop before the end of the year too.

While the businesses polled in the Wikibon survey are likely already Big Data and Hadoop adaptors, it seems significant that 100% of them indicate that they’ll be shifting at least some part of their data analysis resources away from data warehouses to Hadoop.

But there are some naysayers to the idea that data warehousing is going away any time soon.  Alisdair Anderson, HSBC’s Global Head of HSS IT Architecture, for example, said that  “there’s no relationship between EDW and Hadoop right now — they are going to be complementary. It’s NOT about rip and replace: we’re not going to get rid of RDBMS or MPP, but instead use the right tool for right job — and that will very much be driven by price.”