{"id":1659,"date":"2010-11-09T07:00:35","date_gmt":"2010-11-09T15:00:35","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=1659"},"modified":"2010-11-09T07:00:35","modified_gmt":"2010-11-09T15:00:35","slug":"technology-efficient-data-formats-for-storing-and-analyzing-data","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/technology-efficient-data-formats-for-storing-and-analyzing-data\/","title":{"rendered":"Technology: Efficient Data Formats for Storing and Analyzing Data"},"content":{"rendered":"

Twitter tweets are restricted to only 140 characters, but multiply that by the trillions of tweets that are out in the wild and you have a lot of data that needs to be stored. \u00a0Every day, at the current rate, there are about 12 terabytes of tweets being created. \u00a0Twitter has looked for ways to store those huge amounts of data.<\/p>\n

The Twitter data team looked at and rejected XML and JSON formats as being too wordy and unwieldy for managing such large amounts of data. \u00a0At the opposite extreme, they also looked at CSV format which simply list data separated by commas. \u00a0They found CSV too brittle and unable to support complex data structures that might require, for example, the nesting of data elements into subfields. \u00a0They were looking for something that can hold the vast number of tweets being created but which is flexible enough to allow searching and quick retrieval.<\/p>\n

To do that, Twitter has turned to a data format known as Protocol Buffers. \u00a0Protocol Buffers were\u00a0 <\/a>introduced by Google a few years back and the technology behind them were contributed as an Open Source project. \u00a0Each tweet itself is stored along with 17 other data fields representing information about the tweet. \u00a0Compared to XML, Protocol Buffers can represent data in a format that is 3 to 10 times smaller than XML and can serialize structured data 20 to 100 times faster.<\/p>\n

Google uses Protocol Buffers extensively in the storage of their data and the recreation of the data structures needed to consume the data which is stored is built into the capabilities of the Protocol Buffer method. \u00a0Twitter is combining the Protocol Buffer with big data management and analysis tools like Hadoop to be able to perform analytics on the data.<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

Twitter tweets are restricted to only 140 characters, but multiply that by the trillions of tweets that are out in the wild and you have a lot of data that needs to be stored. \u00a0Every day, at the current rate,…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1659","post","type-post","status-publish","format-standard","hentry","category-big-data"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/1659","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=1659"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/1659\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=1659"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=1659"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=1659"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}