The most popular and comprehensive Open Source ECM platform
Let us know more
Big Data: Potential for Bandwidth to Choke on Massive Data Analysis
While physicists are studying the possibility that they’ve discovered neutrinos that can travel faster than the speed of light, data scientists studying Big Data are finding that data growth is blowing past the growth estimates of Moore’s law which has been successfully applied to model growth of hardware and software capabilities in the past.
Anand Rajaraman, senior vice president of Walmart Global E-Commerce and head of @WalmartLabs, said that “data growth is already faster than both Moore’s Law and … network growth… The benefits of big data stretch beyond business to earth sciences, biology, psychology and other fields… Science has become more and more about collecting large amounts of data and doing analysis”.
But the speed with which data is being collected is straining resources for managing and containing it. The four areas of Big Data that are proving challenging include capture, storage, bandwidth/infrastructure and analysis.
One worry has been storage capacity. While storage technologies are improving and costs per storage unit are dropping dramatically quickly, just keeping up with new technologies and the acquisition of adequate amounts of storage has been challenging for many companies.
The analytics side has been equally challenging. The technology to process massive amounts of data is relatively new and rapidly changing. Keeping up to speed with the most current technologies is difficult. Numerous studies have shown that most organizations are not adequately trained to be able to properly analyze Big Data sets.
Ping Li, head of the Big Data Fund at venture capital company Accel Partners, said that “a lot of the applications that ride on top of these new data platforms have yet to be invented Traditional business intelligence and ERP (enterprise resource planning) platforms are being adapted to deal with big data, but what’s needed are native applications developed specifically for the new world.”
Yet another technical challenge with Big Data is bandwidth. A recent study by Infineta Systems showed that Big Data can swamp networks. Adoption of Big Data architectures will mean that increasingly data will be accessed and aggregated across enterprises and between data centers. The load of processing of petabytes of information on a regular basis across these networks can easily become a bottleneck. Arthur Cole of ITBusinessEdge wrote that “most current WAN environments are not tailored for extremely large data sets, leading to bottlenecks and performance deterioration.”
The Infineta Systems report concludes that “Big Data in multi-site enterprises generates Big Traffic: the movement of large datasets over WANs needed to support a Hadoop application before, during and after execution. This factor can complicate the running of a Hadoop cluster and may be unaccounted for in the prototype or proof of concept phases, resulting in Hadoop applications that, when put in production, fall short of the performance and scalability expected.”
