The most popular and comprehensive Open Source ECM platform
Let us know more
Technology: Efficient Data Formats for Storing and Analyzing Data
Twitter tweets are restricted to only 140 characters, but multiply that by the trillions of tweets that are out in the wild and you have a lot of data that needs to be stored. Every day, at the current rate, there are about 12 terabytes of tweets being created. Twitter has looked for ways to store those huge amounts of data.
The Twitter data team looked at and rejected XML and JSON formats as being too wordy and unwieldy for managing such large amounts of data. At the opposite extreme, they also looked at CSV format which simply list data separated by commas. They found CSV too brittle and unable to support complex data structures that might require, for example, the nesting of data elements into subfields. They were looking for something that can hold the vast number of tweets being created but which is flexible enough to allow searching and quick retrieval.
To do that, Twitter has turned to a data format known as Protocol Buffers. Protocol Buffers were introduced by Google a few years back and the technology behind them were contributed as an Open Source project. Each tweet itself is stored along with 17 other data fields representing information about the tweet. Compared to XML, Protocol Buffers can represent data in a format that is 3 to 10 times smaller than XML and can serialize structured data 20 to 100 times faster.
Google uses Protocol Buffers extensively in the storage of their data and the recreation of the data structures needed to consume the data which is stored is built into the capabilities of the Protocol Buffer method. Twitter is combining the Protocol Buffer with big data management and analysis tools like Hadoop to be able to perform analytics on the data.
