Big Data: Businesses Find Big Data Success Stories Possible, but Hard to Replicate

Big Data has made big promises, and that has been a problem.  While Big Data has had many successes, many companies trying to achieve similar successes have struggled.  Using technologies like Big Data and Analytics is getting easier, but still, being successful with big data often takes a skillset that many companies have not yet grown or recruited.

Leonard D’Avolio, assistant professor at Harvard Medical School and CEO of health IT vendor Cyft, said that “the excitement and anticipation of the potential of Big Data technologies and approaches is probably equally matched by the ambiguity, confusion and hype surrounding what these technologies are. What we learned from that, after the necessary growing pains of wasting billions of dollars trying to jam what is in effect new ways of doing business into existing companies without a very exact focus on the problem you are trying to solve and careful consideration of the existing constraints and workflows, is that you are very likely to end up basically running around with a hammer assuming every problem is a nail.”

For some time Big Data was synonymous with Hadoop.  More recently newer technologies like Apache Spark have improved usability, but it’s still not easy. Many of the newer Big Data technologies still are built around or have dependencies on Hadoop. Hadoop has benefits, but it is also complex and has comparatively slow performance.

Bob Muglia, CEO of Snowflake Computing, told Datanami that “I can’t find a happy Hadoop customer. It’s sort of as simple as that. It’s very clear to me, technologically, that it’s not the technology base the world will be built on going forward. The number of customers who have actually successfully tamed Hadoop is probably less than 20, and it might be less than 10. That’s just nuts given how long that product, that technology has been in the market and how much general industry energy has gone into it.”

Bobby Johnson, co-founder of Interana, said of Hadoop that “there’s a bunch of things that people have been trying to do with it for a long time that it’s just not well suited for. It’s never really broken out of the developer world. People have this idea of, ‘Oh there’s all these legacy data warehouses and the new cool way to do it is Hadoop.’ But when you try to do that, you realize it’s actually a lot worse. It’s better than a data warehouse in that have all the raw data there, but it’s a lot worse in that it’s so slow.”