{"id":4958,"date":"2014-10-16T07:00:23","date_gmt":"2014-10-16T15:00:23","guid":{"rendered":"http:\/\/formtek.com\/blog\/?p=4958"},"modified":"2014-10-27T13:11:46","modified_gmt":"2014-10-27T21:11:46","slug":"big-datadremel-applying-interactive-sql-based-data-analytics-to-massive-data-sets","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/big-datadremel-applying-interactive-sql-based-data-analytics-to-massive-data-sets\/","title":{"rendered":"Big Data\/Dremel\/Metanautix: Applying Interactive SQL-Based Data Analytics to Massive Data Sets"},"content":{"rendered":"

Dremel <\/a>is a big-data analytics tool developed by Google that lets analysts interactively explore data using standard SQL. \u00a0While Dremel has\u00a0already been used at Google for eight years in production and while it’s been four years since Google first described the concept publicly in a research paper, Dremel hasn’t achieved anything like the hype that\u00a0MapReduce<\/a>\u00a0has garnered, another technology out of Google used for processing large data sets.<\/p>\n

That may be changing. \u00a0Dremel is used now as the core of Google’s publicly available cloud-based service called\u00a0BigQuery<\/a>, the incubator project Apache Drill<\/a>\u00a0has an open source implementation of Dremel, and\u00a0Cloudera\u2019s Impala<\/a> has also tried to clone the idea. \u00a0And most recently, in August of this year, the startup Metanautix <\/a>has come out of stealth mode and launched the Quest<\/a>\u00a0data compute engine, a tool that enables fast and easy analysis of any type of data, from any source, and at any scale.<\/p>\n

Metanautix was founded <\/a>by\u00a0Theo Vassilakis<\/a> and Toli Lerios<\/a>\u00a0in 2012. \u00a0In August 2014, Metanautix received $7 million <\/a>in a first round of venture funding led by the Sequoia group. \u00a0Metanautix has a handful of early customers, including HP and Shutterfly, and they have also recently partnered with Tableau<\/a> to enable a front-end experience for their back-end analytics engine.<\/p>\n

Prior to Metanautix, Vassilakis had been Engineering Director at Google and managed the team of 75 engineers that developed Dremel as well as other data warehousing, visualization and analysis projects. \u00a0Vassilakis teamed with Lerios, an infrastructure software engineer who developed\u00a0video and image algorithms at Facebook, to found Metanautix<\/a>.<\/p>\n

Vassilakis said that their objective in founding Metanautix was to “create an enterprise-ready product that makes it possible for a traditional non-Google-type enterprise that doesn’t have a large engineering workforce or lots of machines and data centers to be able to do analytics from end to end.”<\/p>\n

The 2010 paper from Google describes Dremel<\/a>\u00a0as an enabler for\u00a0“interactive analysis of very large datasets over shared clusters of commodity machines. \u00a0Unlike traditional databases, it is capable of operating on in-situ nested data… and it can execute many queries over such data that would ordinarily require a sequence of MapReduce jobs, but at a fraction of the execution time.”<\/p>\n

A benchmark run by Google in 2012 found that Dremel could perform a complex regular expression text-matching problem that involved searching over 35 billion rows\u00a0in 20 TB of data in about ten seconds. \u00a0In fact, since Dremel searches over petabytes of data are often performed in seconds and because queries can be expressed simply in standard SQL format, Dremel is a useful tool for interactively analyzing massive amounts of data. \u00a0Think trillions of records running on many thousands of servers.<\/p>\n

The secret to Dremel’s lighting performance<\/a> can be attributed to two core technologies:<\/p>\n