The most popular and comprehensive Open Source ECM platform
Let us know more
Big Data: Using Data Sketches for Analyzing Massive Real-Time Data Streams
Big Data usually means massive amounts of data must be stored for analysis and processing. For many scenarios, trying to save all data simply isn’t feasible because of the costs involved in storing so much data. Trying to keep all data for all time can quickly become very expensive because of the large amounts of storage that are necessary to retain the data. The benefits derived from any data analysis may not be enough to outweigh the storage costs involved.
A technique called Data Sketching provides an alternative to either retaining all data or making no attempt to save any of it. A data sketch is a summary created by examining, processing or sampling the data of a real-time data feed.
Graham Cormode, a professor of Computer Science at the University of Warwick, wrote that “telecommunications companies, for example, want to keep track of the activity on their networks, to identify overall network health and spot anomalies or changes in behavior. Yet, the scale of events occurring is huge: many millions of network events per hour, per network element. While new technologies allow the scale and granularity of events being monitored to increase by orders of magnitude, the capacity of computing elements (processors, memory, and disks) to make sense of these is barely increasing. Even on a small scale, the amount of information may be too large to store in an impoverished setting (say, an embedded device) or to keep conveniently in fast storage.”
The Yahoo DataSketch library says that “if approximate results are acceptable… streaming algorithms, or sketches can produce results orders-of magnitude faster and with mathematically proven error bounds. For interactive queries there may not be other viable alternatives, and in the case of real-time analysis, sketches are the only known solution.”
