Big Data: Speeding up Big Data Processing with Apache Spark

Apache Spark is quickly gaining a strong following among Big Data users.  The big advantage of using Spark is that it allows in-memory processing which greatly speeds up the ingestion and processing of data.  Spark is also a bit more straight forward in terms of builds and the creation of data job workflows.  Apache Spark’s tagline is “Lightening Fast Clustered Computing”.

Ion Stoica, Co-Founder and CEO of Databricks, told Forbes that  “Spark is a parallel execution engine that is better than Hadoop MapReduce in three dimensions: First, it’s faster, because it is optimized to work efficiently with data stored both in memory and on disk. Spark holds the terabyte sort benchmark record, by beating the time of the previous record by 3x using 10x fewer machines.  The second advantage is that it provides a more powerful and flexible API than MapReduce, which makes it much easier for developers to write sophisticated applications. Typically, it takes between two to five times fewer lines of code to write the same application in Spark than in Hadoop MapReduce. Finally, Spark unifies a variety of computation models. It goes far beyond batch computation, and through a set of libraries, it supports many other workloads, including streaming, interactive queries, machine learning, and graph processing.  What makes all of these possible is a very flexible and powerful core engine which can execute large scale jobs in subseconds.”

Stoica added that “Spark also has a more general and easy-to-use API. So when you write applications in Spark, you don’t need to cast them as a bunch of maps and reduces. You can almost write arbitrary applications.  When we do regular surveys and ask people why they like Spark, half say it’s speed and half say it’s ease of use.”

Databricks, a startup founded by the creators of Spark, has recently announced the availability of a cloud platform based around Apache Spark and a Databricks workspace.  It’s a way to simplify the interaction a user has with Big Data.  There’s no need to have to directly interact with an Hadoop cluster.  Once users upload data to the a project in the Databrick platform, they’re able to start interacting with it and begin creating visuals, like charts and dashboards.  It’s possible for the user to schedule jobs via the job launcher to ensure that Spark jobs get run at specific times.