Metanautix: Applying Interactive SQL-Based Data Analytics to Massive Data Sets

By Dick Weisinger

Dremel is a big-data analytics tool developed by Google that lets analysts interactively explore data using standard SQL. While Dremel has already been used at Google for eight years in production and while it’s been four years since Google first described the concept publicly in a research paper, Dremel hasn’t achieved anything like the hype that MapReduce has garnered, another technology out of Google used for processing large data sets.

That may be changing. Dremel is used now as the core of Google’s publicly available cloud-based service called BigQuery, the incubator project Apache Drill has an open source implementation of Dremel, and Cloudera’s Impala has also tried to clone the idea. And most recently, in August of this year, the startup Metanautix has come out of stealth mode and launched the Quest data compute engine, a tool that enables fast and easy analysis of any type of data, from any source, and at any scale.

Metanautix was founded by Theo Vassilakis and Toli Lerios in 2012. In August 2014, Metanautix received $7 million in a first round of venture funding led by the Sequoia group. Metanautix has a handful of early customers, including HP and Shutterfly, and they have also recently partnered with Tableau to enable a front-end experience for their back-end analytics engine.

Prior to Metanautix, Vassilakis had been Engineering Director at Google and managed the team of 75 engineers that developed Dremel as well as other data warehousing, visualization and analysis projects. Vassilakis teamed with Lerios, an infrastructure software engineer who developed video and image algorithms at Facebook, to found Metanautix.

Vassilakis said that their objective in founding Metanautix was to “create an enterprise-ready product that makes it possible for a traditional non-Google-type enterprise that doesn’t have a large engineering workforce or lots of machines and data centers to be able to do analytics from end to end.”

The 2010 paper from Google describes Dremel as an enabler for “interactive analysis of very large datasets over shared clusters of commodity machines. Unlike traditional databases, it is capable of operating on in-situ nested data… and it can execute many queries over such data that would ordinarily require a sequence of MapReduce jobs, but at a fraction of the execution time.”

A benchmark run by Google in 2012 found that Dremel could perform a complex regular expression text-matching problem that involved searching over 35 billion rows in 20 TB of data in about ten seconds. In fact, since Dremel searches over petabytes of data are often performed in seconds and because queries can be expressed simply in standard SQL format, Dremel is a useful tool for interactively analyzing massive amounts of data. Think trillions of records running on many thousands of servers.

The secret to Dremel’s lighting performance can be attributed to two core technologies:

Columnar storage This allows very high compression of data and scan throughput.
Tree architecture Queries can be dispatched across thousands of nodes and then aggregated into a final result

Armando Fox, a professor of computer science at the University of California at Berkeley, commented that “if you had told me beforehand me what Dremel claims to do, I wouldn’t have believed you could build it.”

While Metanautix Quest was inspired by Dremel, Quest goes significantly beyond Dremel to include unstructured data like media, text and office-type documents. They’ve also added connectors or ‘pipes’ that can reach out into datastores like Amazon S3, Hadoop and Oracle database.

Vassilakis said that “in general, we view the format of the data as a coincidental thing. We can see if the user is trying to query a JPEG file, for example… One way you might look at that, is as a table of metadata. The picture might have GPS coordinates for where it was taken, information about the device and also the time the picture was taken, and those values would make up a table of information. Or, you might look at it as a collection of pixels. At position (x, y) of the photo, the color is (r, g, b). We give mechanisms and user-defined functions so that you can take any piece of data and make it look like a table or multiple versions of a table, and then just query it, as though it had been imported into a database…”

Vassilakis continued, saying that “looking at the JPEG image that way, you might say ‘wow! there’s a lot of pixels out there’… But basically our observation is that every insurance company has a bunch of photos from every claim, and every retailer has a bunch of video from every store, and every telco has a bunch of recordings from all their calls. To date, you pretty much have to do custom software development for all those things. Even for simple things… But with technology like Quest, a lot of that could be automated.”