Cloud Data Management: SnowFlake: A Relational Database Built for the Cloud

In a world where Big Data technologies like Hadoop, HPCC and SAP HANA dominate the attention among most of the data analytics crowd, it’s almost an anomaly to hear anything about new relational database technology.  But today is different.  Today Snowflake Computing is announcing the introduction of a new relational database that’s totally optimized for the cloud.

Relational dataset technology has been around for more than thirty years, and it’s been more than a decade since the last architected-from-scratch relational database was built.  Do we really need an alternative to SQL Server, Oracle DB, PostgreSQL, or MySQL?

The founders of Snowflake thought relational databases needed a new design when then started the company in 2012.  Their goal was to create a relational database that fully separates data from compute, to design a database engine that was built to easily comply with the fundamental principle of the cloud that scale up and down should be easy and fast, to be fully elastic.

Existing cloud databases like SQL Azure have attempted to provide elastic scale-out capabilities with modest success.  Scale out with SQL Azure, for example, is possible using techniques like sharding, but successful scale out is only possible after architecting a data model that is amenable to sharding and modifying your data to fit that structure, a task that may not be very easy.

The new SnowFlake database is at the heart of their cloud service offering:  a product that is targeting data warehousing and analytics, something that they’re calling the SnowFlake Elastic Data Warehouse.  The new SnowFlake database is fully SQL compliant and fully compatible with Oracle’s flavor of SQL.  It’s been benchmarked against a 128 node system running the TPC-DS benchmark and found to be fully compliant.

While still in stealth mode, SnoweFlake approached 300 different potential customers, ranging from small startups to large established enterprises.  Currently 50 trials of the technology are under way, there are a number of paying customers, and there are five referenceable customers that include Adobe, Conde Nast, and Accordant Media.  SnowFlake has received venture capital from Sutter Hill Ventures, Redpoint Ventures and Wing Venture Capital.

Bob Muglia, former president of Microsoft’s $16 billion Server and Tools Business and now CEO at SnowFlake, commented that “we realized that you couldn’t really do any of these things with any of the existing solutions. We looked closely at what they could do with PostGreSQL, what could we do with MySQL from a data warehouse side, and does Hadoop make any sense. And we came to the conclusion that all those systems, while they had value to them, they were born in a different era before the cloud was created, and in order to build something new to take advantage of cloud capabilities, we had to start from scratch.”

Muglia continued, said that they asked “what if we made it really possible for people to leverage that cheap storage in the cloud, but still had the flexibility to get the compute when they needed it?  That was really a fundamental insight that led us to creating an architecture that hadn’t been deployed in data warehousing and definitely not in the cloud. And that is this notion that we can separate out storage, compute, and the metadata management. So in the cloud, we store our data in our internal format on cheap scalable services, and in the Amazon case, on Amazon S3. To give people the ability to store that data in a very very cheap form so that they don’t have to spend time worrying about whether they’re going to be able to afford to keep all of their data or to spend time figuring out what to throw away.”

Three fundamental attributes of the SnowFlake Elastic Data Warehouse are:

• Delivers data warehousing as software as a service
• Separation of Data and Compute
• One system that supports all data, both structured and semi-structured

Jon Bock, Vice President of Product Marketing at SnowFlake, explained the design principles of the SnowFlake data warehousing service saying that ” by separating the processing and the storage, we make it possible for people to really ‘right-size’ their systems on an on-going basis, which is really one of the fundamental properties of the cloud. So that was really a key insight in how we architected the system differently. Another thing was that we talked to a lot of these people who had this machine-generated data and they were trying to figure out how to process that and do analytics on that, given that that really doesn’t fit with traditional data warehousing.  And the same time, SQL analytics really doesn’t fit on things like Hadoop. And so we saw a lot of people who were saying that their use case was to use Hadoop, take this machine-generated semi-structured data and to transform it into something a database could understand and then to load it into a database in order to get fast analysis….  So we said why shouldn’t we actually redesign the database to make it possible to avoid having to do two different steps and to have a database that could understand the machine-generated semi-structured data?”

Bock continued, talking about the importance of elasticity, saying that “elasticity is a word that people automatically associate with the cloud, but that doesn’t really match what data warehouses today can do or what the current offerings in the cloud can do. So one aspect of this is certainly to be able to scale to whatever scale of data people have. Everyone likes to focus on that big end of that spectrum. Like what’s the biggest system that I can build, but the reality is that people have widely varying amounts of data. It’s really not a question for most people as to how big you can build the system as a question of how easy is it to scale up or down depending on what your needs are.  So in our case, because we have a separation of the storage and the compute, people can scale up, or occasionally if they need, to scale down data,  and they only have to pay for what they use. Data storage in the cloud is extremely cheap and you only have to pay for what you use. That’s a key property of the cloud. But no data warehouse at this point has been built to take advantage of that.”