The most popular and comprehensive Open Source ECM platform
Let us know more
Cloud Computing: Cloudera's Impala Seeks to Replace Hive for Hadoop
Impala, a new open-source Apache-licensed tool has recently been added by Cloudera to the Hadoop toolset ecosystem. Inspired by Google’s Dremel, Impala is a real-time query engine for Hadoop.
Hadoop already has a tool called Hive which provides some amount of structured-data access to Hadoop data sets. While Impala is similar to Hive and is API-compatible with it, the implementation details are quite a bit different. Hive processes and compiles SQL queries into Java and then runs those queries as batch-mode MapReduce jobs. Hive effectively adds an extra step of SQL statement compiling in addtion to standard MapReduce Hadoop processing. Instead, Impala more closely resembles a relational database and directly processes Hadoop data, eliminating the MapReduce step. Another issue with Hive is that while the query language is SQL-like, it really only supports about 30 percent of the capabilities of SQL available in standard relational data bases.
The performance of Impala compared to Hive is expected to be significantly better. Cloudera is claiming performance gains ranging from 3x to 30x when using Impala compared to similar jobs run with Hive.
Mike Olson, CEO of Cloudera, told InformationWeek that “Our view is that, long-term, this will supplant Hive, but right now Hive has a huge installed base and lots of applications running against it, so I don’t imaging that [Impala] will replace Hive quickly. Because it’s real time, Impala is going to be much more attractive.”
Curt Monash, a database and analytics expert and principal at Monash Research, pointed out that a major weakness of Hadoop is that it is poor at interactive and ad hoc queries across a large data set. “Hadoop has trouble with interactive responses. If you can stand latencies of a few seconds, Hadoop is fine. But Hadoop MapReduce is never going to be useful for sub-second latencies.”
Charles Zedlewski, vice president of products at Cloudera, said that “The reason why people are attracted to it [Hadoop] is because it’s scalable, extremely flexible, easy to get started with, and inexpensive compared to what else is out there… MapReduce is by definition a batch system. There’s no getting around that. And, despite that, Hadoop made a lot of hard things easy. But it also made a lot of easy things hard. We want to make the easy things easy again [with Impala].”
Tony Baer, principal analyst for Ovum, said that “Mainstream enterprise adoption of Hadoop will inevitably raise expectations. Enterprises have grown accustomed to interactive querying and on-the-spot analytics with their existing data warehousing and BI infrastructures and will expect no less of Hadoop. With a real-time query capability powered by its new Impala engine, Cloudera is striving to level the playing field in performance and accessibility with massively parallel SQL platforms.”
