{"id":2708,"date":"2012-01-19T07:00:52","date_gmt":"2012-01-19T15:00:52","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=2708"},"modified":"2012-01-19T07:00:52","modified_gmt":"2012-01-19T15:00:52","slug":"big-data-potential-for-bandwidth-to-choke-on-massive-data-analysis","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/big-data-potential-for-bandwidth-to-choke-on-massive-data-analysis\/","title":{"rendered":"Big Data: Potential for Bandwidth to Choke on Massive Data Analysis"},"content":{"rendered":"

While physicists are studying the possibility that they’ve discovered neutrinos<\/a> that can travel faster than the speed of light, data scientists studying Big Data are finding that data growth is blowing past the growth estimates of Moore’s law which has been successfully applied to model growth of hardware and software capabilities in the past.<\/p>\n

Anand Rajaraman<\/a>, senior vice president of Walmart Global E-Commerce and head of @WalmartLabs, said that<\/a> “data growth is already faster than both Moore’s Law and … network growth…\u00a0The benefits of big data stretch beyond business to earth sciences, biology, psychology and other fields…\u00a0Science has become more and more about collecting large amounts of data and doing analysis”.<\/div>\n

But the speed with which data is being collected is straining resources for managing and containing it. \u00a0The four areas of Big Data that are proving challenging include capture, storage, bandwidth\/infrastructure and analysis.<\/p>\n

One worry has been storage capacity. \u00a0While storage technologies are improving and costs per storage unit are dropping dramatically quickly, just keeping up with new technologies and the acquisition of adequate amounts of storage has been challenging for many companies.<\/p>\n

The analytics side has been equally challenging. \u00a0The technology to process massive amounts of data is relatively new and rapidly changing. \u00a0Keeping up to speed with the most current technologies is difficult. \u00a0Numerous studies have shown that most organizations are not adequately trained to be able to properly analyze Big Data sets<\/a>.<\/p>\n

Ping Li<\/a>, head of the Big Data Fund at venture capital company Accel Partners, said that\u00a0“a lot of the applications that ride on top of these new data platforms have yet to be invented Traditional business intelligence and ERP (enterprise resource planning) platforms are being adapted to deal with big data, but what’s needed are native applications developed specifically for the new world.”<\/p>\n

Yet another technical challenge with Big Data is bandwidth. \u00a0A recent study by Infineta System<\/a>s showed that Big Data can swamp networks. \u00a0Adoption of Big Data architectures will mean that increasingly data will be accessed and aggregated across enterprises and between data centers. \u00a0The load of processing of petabytes of information on a regular basis across these networks can easily become a bottleneck. \u00a0 Arthur Cole<\/a> of ITBusinessEdge wrote that<\/a> “most current WAN environments are not tailored for extremely large data sets, leading to bottlenecks and performance deterioration.”<\/p>\n

The Infineta Systems report<\/a> concludes that “Big Data in multi-site enterprises generates Big Traffic: the movement of large\u00a0datasets over WANs needed to support a Hadoop application before, during and\u00a0after execution. This factor can complicate the running of a Hadoop cluster and\u00a0may be unaccounted for in the prototype or proof of concept phases, resulting in\u00a0Hadoop applications that, when put in production, fall short of the performance\u00a0and scalability expected.”<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

While physicists are studying the possibility that they’ve discovered neutrinos that can travel faster than the speed of light, data scientists studying Big Data are finding that data growth is blowing past the growth estimates of Moore’s law which has…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-2708","post","type-post","status-publish","format-standard","hentry","category-big-data"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/2708","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=2708"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/2708\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=2708"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=2708"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=2708"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}