The most popular and comprehensive Open Source ECM platform
Let us know more
Big Data: Don’t Bother Rolling Out Big Data Tools Unless Your Application Really Warrants It
When to use Hadoop? It really depends on your application. And, as we will see, in some cases it doesn’t take big data sets for it to make sense to use Big Data.
Many, and probably most, applications don’t really have a need to use Big Data tools. Some projects have jumped on the Hadoop bandwagon to apply it to project where it might not bring much advantage. For example, a report by Microsoft found that the data set sizes for most applications and projects are well under 100 GB.
Vincent de Lagabbe, CTO of Kaiko, said that “considering the volume that we are dealing with, Hadoop would be overkill. It was fancy technology for the time and people started using it for everything, but most things you can do without it. From experience, it’s better to try to do without Hadoop – I mean use it when you need it but when you don’t, don’t bother. If you’ve got less than a petabyte of data, Hadoop is probably overkill.”
On the other end of the spectrum, in a ZDNet article by Toby Wolpe, the example of the company 23andMe, a personal genetic-profiling service, is used. It’s the example of an application where it makes sense to use Hadoop but where the actual data set size being analyzed is not that Big.
When processing a genome sample, the amount of data for one person’s DNA is only about 800 MB of data. Mike Gualtieri, principal analyst at Forrester, said “that’s not a lot. Would you call that big data? If I said 800MB is big data, I’d get laughed out of the room. But within that, there are four billion pieces of information and lots of patterns. So it’s a big processing challenge, it’s a big compute challenge. You don’t have to have petabytes of data to have a big-data opportunity or issue.”
