Big Data: Real-Time Stream Processing with DataTorrent

Traditional computing techniques are challenged when it comes to handling large volumes of real-time data.  Stream processing is an evolving programming paradigm that attempts to take on the challenge of processing large real-time data.  While Hadoop is one technology that’s been used for working with large data sets, it hasn’t been until the recent introduction of Apache Hadoop YARN and Apache Storm that big data projects have been able to begin to do anything other than static batch processing.

Stream processing is particularly suitable for applications that have these kinds of requirements:

• Compute intensive [high ratio of operations to I/O]
• Data Parallelism
• Data Pipelining where data moves continuously from producers to downstream consumers

One company focused on building a platform for enabling large real-time applications is DataTorrent.  DataTorrent is a startup company that has created a data-stream processing platform built with Apache Hadoop and YARN.  The founders of the company come out of Yahoo! and were involved with the original development of Hadoop.  DataTorrent addresses markets that need to handle large volumes of real-time data, like the financial services industry, manufacturing, and the Internet of Things (IoT).

The 1.0 version of the DataTorrent product focused on building a high performance real-time streaming platform.  With a 35-node cluster, DataTorrent was able benchmark 1.6 billion events/second processing with sub-millisecond response times. The platform is designed to be robust with stateful fault tolerance.  The focus of the 1.0 DataTorrent product was to drive the core technical aspects of the platform and to make the software available in as many ways, shapes and forms as possible.

Barely now three months after the 1.0 release, DataTorrent is making available today the 2.0 release of their product.

John Fanelli, VP of Marketing at DataTorrent, said that “the story around our 2.0 release is really the customer and how to make it more accessible for them and to take it to the next level, allowing non-developers to participate in the creating of real-time streaming applications… When you talk to a customer, they’ll say ‘Can you do my data? Coming from this source.  It looks like this size. And coming in at this speed.’  And the speed can be millions of events per second to every five seconds for getting the data.  The data doesn’t always have to be coming in real time. What we really drive is faster insight and real time action…  this release also adds additional operators into our open source community…  You write your streaming app by connecting a number of operators that the data streams through.  We have over 500 of these operators in an open-source GitHub repository, and that really provides our customers a great starting block.”

Coincident with the 2.0 DataTorrent release is also an announcement of private-beta availability of a graphical assembly application called DaVinci and a real-time visual dashboard designer, code-named Michelangelo.