Hadoop and Big Data: Businesses Frustrated by Lack of Reliable, User-Friendly and Secure Solutions

While Big Data technologies made big headlines in the tech press in 2013, by the end of the year, the adoption of Big Data technologies across many businesses was only relatively modest.

A survey by the SandHill group of a cross-section of global businesses found that:

• 44 percent said that they are still in the exploration and education phase of Big Data technologies
• 16.3 percent are conducting proof of concept trials
• 11.1 percent are developing their first Hadoop-based applications

Not only has adoption been relatively slow, the types of solutions created to date have tended to be relatively mundane — Matt Assay comments that “by far the biggest use of Hadoop to date has been as a poor person’s ETL”.

While there is a strong interest in the potential of Big Data from the business side of organizations, many are frustrated by usability issues of the technology.  A survey of business executives taken by 1010data, for example,  found that a majority feel that they are locked out of Big Data technologies like Hadoop.  53 percent said that Big Data solutions aren’t addressing the needs of business users.  49 percent said that current Hadoop-based solutions are too complex and that they’d like to see solutions for Big Data that have “fewer moving parts”, and 62 percent said that in order for them to more effectively use Big Data technologies like Hadoop to solve business problems that they need to be better educated about how the technology works.

Specifically, the 1010data survey considered how the latest Hadoop 2.0 release can improve business solutions.  Hadoop 2.0 includes YARN which allows Hadoop resources to be managed for multiple jobs running across an array of servers.  But business executives aren’t yet convinced that the technology is ready for prime time.  45 percent of them are worried about reliability.  42 percent say that they have major concerns about costs, and 35 percent say that there is still too much low-level coding required to apply the technology to business solutions.  41 percent say that Hadoop is still too new and not yet as stable as other existing technologies.

But software developers and integrators are well aware of the problems.  It’s likely that 2014 will see a surge in Big Data product offerings and add-ons that specifically try to create an improved experience for users.