Next Generation Databases: Real-Time Scale-Out Relational Processing

As databases grow, the most common way to stay pace with more data, more transactions, and more load is to scale-up the database hardware. Scaling up involves adding more CPU power, more memory and more and faster disks. But as data sizes continue to grow beyond what even single node servers are able to handle, the need then increases for scaling out hardware to run on multiple clustered nodes. And when the need further expands to be able to process large amounts of data in real time, even newer techniques have to be employed.

Relational databases have broken down when trying to process very large volumes of real-time data. Instead, real-time data processing has been an area where NoSQL solutions are increasingly being applied. But No-SQL databases haven’t yet been able to match the complexity and richness of the types of data queries that are possible with traditional relational databases, and because of that, some companies are working on developing creative ways to accelerate standard relational database processing.

Clustrix, for example, has pioneered techniques that goes beyond clustering. Their patented technique partitions data among server nodes in a cluster and then collects and aggregates the results. The concept of their approach is similar to how the MapReduce algorithm breaks down a problem into smaller data sets, calculates the result sets in parallel, and then aggregates and sorts the results. Compared to standard database clustering, calculations can be performed significantly faster.

Robin Purohit, CEO of Clustrix, said that the techniques that “we’re using are similar to many analytic solutions. But what’s different about our solution is that we’re doing it completely in a relational database. The patent that we got granted basically describes the data distribution, and a method for slicing and reslicing of data, but you always maintain the ACID and relational properties of the database. That’s what differentiates us from the majority of the NoSQL and data analytics tools today that have given up on that in order to get scale.”

George Gilbert, GigaOM Pro analyst, said that “Transactions must be able to drive decisions in real time. In-memory databases typically have the fastest performance but often trade off capacity of a large data set and analytic sophistication. DBMS that aren’t constrained just to memory can achieve the analytic sophistication and performance that still support transaction processing, although less than a data warehouse.”

Clustrix features include:

• Multi-version concurrency control (MVCC) to ensure that reads and writes don’t interfere with each other
• Aggregation is distributed and partially computed at nodes, making analytics scale better
• SQL query fan-out that allows multiple nodes and multiple cores per node to be added to the calculation
• Use of FLASH-based in-memory analytics

Purohit said that “people use our product for very high-volume transactions that are business critical and have a lot of peak demands. That’s pretty typical of e-commerce companies like Rakuten, and we have about nine customers in the e-commerce segment today using Clustrix in production as their primary database. We’ve also been seeing in the last year that once people scale out that they want to use their data and combine it in a real-time fashion, so they start running very complex queries on that on-line data, for example, seven-way joins or sixteen-way joins, on very large table sets. That is where we really improved the product over the last year and one half, to make sure that you can do complex analytics that you would typically do on a data warehouse, but using live operational data, so that you can get the response to questions in seconds. ”

How does Clustrix compare to a more traditional relational product like Oracle RAC solutions? Purohit said that “the difference with Oracle RAC is their use of shared-disk architecture. What that means is that it has very poor performance as you add more nodes. RAC primarily uses high availability and small cluster configurations, and generally with scale-out servers that are banded together, there is a lot of locking contention when you use shared disk. Our architecture is a completely distributed no-single-master solution, and the associated storage is all distributed in the server, so it is a true cloud architecture. To be able to do scale-up relational on that architecture with minimal locking contention is really the big differentiator. And we do all that at less than one-tenth the cost of Oracle standard database, and if you add in all the Oracle enterprise and RAC licenses, it may even be more like one-thirtieth of the cost.”