Big Data: Early Adopters Struggle with Today's Technology – Longer Term, Tools Look Promising

Every day 2.5 quintillion bytes of data are being created — 12 terabytes of which are Twitter tweets alone.  IBM estimates that 90 percent of all data in the world was created in just the last two years; similarly, Gartner is predicting that the global volume of stored data will grow annually by 59 percent.  Big Data has been positioned as the solution that will help analyze and interpret the meaning of all this data.

Clearly not all companies have yet attempted to deal with Big Data.  Research from Iron Mountain found that half of large organizations don’t know what to do about Big Data and 21 percent say that they don’t plan on trying.  But then there are still quite a few businesses in the category of early adopters of Big Data technology.  These are companies motivated by the excitement of the potential for business insights which Big Data analytics can provide.

But as more businesses set up Big Data projects and dig into the technology, some of the rosy glow around its capabilities are fading.  Svetlana Sicular, Research Director at Gartner, wrote a blog saying that “the peak of inflated expectations was reached some time ago.  As an analyst, I see that big data starts falling off this peak and at the end of the peak, negative press and adopter complaints begin.”  In the short term, using today’s tools for Big Data projects can be challenging.  Longer term, the technology still appears very promising.

Hadoop and the MapReduce algorithm have been core to many of the vendor products being introduced to handle Big Data.  Gartner predicts that by 2015 65 percent of “packaged analytic applications with advanced analytics” will have Hadoop embedded in them.  Big Data tools continue to slowly advance and become easier to use.

But Sicular wrote that “my most-advanced-with-Hadoop clients [using today’s technology] are also getting disillusioned.  They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions.  Their disappointment applies to more advanced cases of sentiment analysis, which go beyond traditional vendor offerings.  Difficulties are also abundant when organizations work on new ideas, which depend on factors that have been traditionally outside of their industry competence, e.g. linking a variety of unstructured data sources.”