Big Data: Businesses Still Rely on SQL while they try to Bridge the Gap in Big Data Skills

We keep hearing that Big Data analytics skills are in short supply.   A report byMcKinsey warned of the shortage of skilled workers in May 2011.   The report found that “A significant constraint on realizing value from Big Data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from Big Data.  We project a need for 1.5 million additional managers and analysts in the United States who can ask the right questions and consume the results of the analysis of Big Data effectively.”

Similarly, a report by Deloitte also warned of a potential problem.  The report said that “Internet companies have led the way in exploring big data, but the sectors that are likely to follow include the public sector, financial services retail, entertainment and media. This could trigger a talent shortage, with up to 190,000 skilled professionals needed to cope with demand in the US alone over the next five years.”

Now more than a year later, we’re seeing increased evidence of a lack of Big Data skills.

James Murray, vice president and general manager EMEA at Splunk, said that “Data analysis is a skill set that the world is short on.  As people spend time buying expertise and experience we need smart people to ask smart questions…  Data exists but you need someone to manage it.”

Mike Merritt-Holmes, Co-Founder & CEO of Big Data Partnership, said that “The end-user customers and vendor community are all competing for the most talented big data developers and data scientists.  To gain real competitive edge, you have to have the right talent and the right teams with the relevant skill set. Apart from a minority of early-adopter companies, this does not exist today across many of the enterprises.

But companies are beginning to bridge the gap.  Alice Hill, director at on-line IT recruiting site Dice.com, said that “There are a lot of perceived gaps in available skills in the new technology just because the technology is new and there hasn’t been time for people with related skills to become trained in the new one.  If a gap continues for a long time it could put upward pressure on salaries, but what usually happens is that companies end up training existing staff and only hire a few people with specialized skills.”

Companies are not only training.  For good or for bad, they’re making do with their existing expertise and skill sets.  For example, yesterday’s Formtek blog reported results from a JasperSoft survey that found many businesses continue to use SQL as their main tool for analyzing their data, even when data sets grow to the size of Big Data.  The same findings were echoed by a report from Rainstor which found that 85.7 percent of companies still consider standard enterprise-tested SQL to be their principle tool for analyzing data.

Migrating data tools to something other than standard SQL will take time.  SQL is ingrained in many businesses.  After all, many companies have used SQL for more than twenty or thirty years as their primary data analysis tool.  It’s expected that newer technologies like Hadoop/MapReduce, HPCC, Hive, and Pig will be increasingly be used for large data analytics projects but that SQL will remain with us for quite some time.  And who’s to say that relational SQL vendors won’t try to put up a fight — something similar to how they’ve resisted object-oriented databases in the past?