{"id":6166,"date":"2016-03-18T07:00:44","date_gmt":"2016-03-18T15:00:44","guid":{"rendered":"http:\/\/formtek.com\/blog\/?p=6166"},"modified":"2016-02-17T14:27:35","modified_gmt":"2016-02-17T22:27:35","slug":"big-data-apache-arrow-for-faster-and-more-efficient-management-of-columnar-data","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/big-data-apache-arrow-for-faster-and-more-efficient-management-of-columnar-data\/","title":{"rendered":"Big Data: Apache Arrow for Faster and More Efficient Management of Columnar Data"},"content":{"rendered":"

The Apache Software Foundation<\/a> recently released Arrow <\/a>as a new top-level project. \u00a0Arrow is embeddable software that enables columnar in-memory processing. \u00a0Structured data is usually managed with SQL operating on rows of a database. \u00a0Column operations are often tricky to set up, but done right, manipulating columns allows\u00a0larger data sets to be processed, and certain columnar operations are as much as 100 times faster than doing the same operation\u00a0in a row-based way.<\/p>\n

Jacques Nadeau<\/a>, chairman of the Apache Drill project, described\u00a0Apache Arrow <\/a>as “an accelerator for processing and storage systems. \u00a0It\u2019s a set of data representations that are much more CPU-efficient… \u00a0Doing in-memory columnar is hard.\u00a0Doing rows is easy.\u201d<\/p>\n

Arrow provides a common\u00a0data format that enables multiple systems to share, exchange and\u00a0communicate data. \u00a0Nadeau said that using Arrow can eliminate frequent serialization and deserialization of data that often can consume 70 to 80 percent of the processing cycles for some workloads.<\/p>\n

Nadeau said that “Cache locality, pipelining and superword operations frequently provide 10-100x faster execution performance. Since many analytical workloads are CPU bound, these benefits translate into performance gains, or more plainly, the potential for faster answers and higher levels of user concurrency.”<\/p>\n

Ted Dunning<\/a>, Vice President of the Apache Incubator, said that<\/a> “an industry-standard columnar in-memory data layer enables users to combine multiple systems, applications and programming languages in a single workload without the usual overhead.”<\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

The Apache Software Foundation recently released Arrow as a new top-level project. \u00a0Arrow is embeddable software that enables columnar in-memory processing. \u00a0Structured data is usually managed with SQL operating on rows of a database. \u00a0Column operations are often tricky to…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,35],"tags":[],"class_list":["post-6166","post","type-post","status-publish","format-standard","hentry","category-big-data","category-structured-data"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/6166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=6166"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/6166\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=6166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=6166"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=6166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}