Big Data: The Role of the Data Engineer in Businesses

As businesses begin to take advantage of big data and associated data-driven technologies, there is an increasing demand to fill roles within companies that are focused on managing and analyzing data.  Specifically, the roles that seem to have seen the most demand are those of data scientist and data engineer.

But what’s really the difference between these two roles?  We asked Sean Kandel, CTO of Trifacta, a startup building data and visualization tools that are designed to improve accessibility of data analysis, for his view on the roles of data scientist and data engineer.

Sean told us that “a data scientist often uses a combination of skills, like programming and statistics, to build models that allow experiments with the data… [They can build] algorithms to discover some insight that has an impact on the business.”

“It’s the data engineer that is involved across the organization in understanding the needs of many different people in the company, like data analysts, business analysts or data scientists.  And they’re responsible for architecting the system that enables multiple consumers of data to both effectively and efficiently get the data that they need for whatever task that they’re performing.”

“And once you do discover an insight, a data engineer is the one who is often responsible for taking that insight and embedding it in the actual business process so that you can then use it day to day to improve operations.”

“What we’ve seen as data scientists have popped up within companies is that companies are seeing what value they can get out of these projects that heavily utilize data, and as that happens, they get hungrier to use more and more data…  Every time a new use case pops up, companies are running into problems where they’re finding that it’s really hard to find the data that they need, to prepare it for analysis, and to integrate data across different sources.  So there is more and more demand for someone like a data engineer who can really help an organization manage that process as it scales out the number of use cases around it.”

“A great example of [how the data scientist and data engineer work together] might be a data scientist who develops a new recommendation algorithm.  The data engineer might then be responsible for taking that recommendation algorithm and putting it into production so that day in and day out that algorithm is actually powering recommendations on a live web site.  The engineer then monitors those data algorithms and collects whatever information is necessary so that you might then go back and refine that data algorithm over time.”