Big Data: Can Next Generation Software Replace the need for Data Scientists?

Virginia Backaitis, Big Data recruiter, earlier this month wrote a piece for CMSWire called ‘the End of Data Scientists’ where she collects comments from executives of companies building analytics tools.  These businesses say that the next generation of software analytics tools to be unveiled in 2014 will be powerful enough to begin making the role of Data Scientist less relevant.

Can that be right?  It was only less than 12 months ago that everyone thought we’d never be able to get enough Data Scientists to quench the demand when it comes to analyzing Big Data.

But now, Big Data analytics companies like Tableau say that new software will allow non-technical business users the ability to generate insights from their data without the help of data scientists.  Alteryx says that “analysts will matter more than data scientists… Empowering analysts in business departments with big data and analytics will become more important than filling the perceived need for millions of data scientists.”  Andy Savitz, VP of Marketing at SAP’s KXEN, says too that their software will enable analysts to by-pass the data scientist and automate “much of their hard, messy and time-consuming work.”

But not everyone agrees.

Jordan Novet, tech journalist, quotes Pete Skomoroch, former principal data scientist at LinkedIn, saying that “startups might try hard to automate the process of discovering trends and anomalies in large volumes of data, but software most likely will not be able to replicate the capabilities of humans who are trained in data science principles and best practices…  Startups such as Trifacta could slim down the time it takes to get data ready for analysis, but later work still could use a skilled person’s input. Creativity, intuition — ‘the human element’ — still matter a great deal. ‘This is knowledge work.'”

Ted Cuzzillo, business intelligence analyst, writes for Information Management that “you can’t replace a data scientist’s judgment and know-how with software — you wouldn’t dare rely on it for truly important decisions —it is possible to set up self-service that moves some functions down the ladder. The data scientist then works on the really difficult, complex problems.”

So, if you’ve signed up to train to become a Data Scientist working with  Big Data analytics, it’s probably still worth it to keep on plugging — there are still jobs out there, at least still in 2014.