Big Data: Best Practices for Building Competency with Hadoop

Big Data platforms based on Hadoop and MapReduce technology are becoming increasingly popular.  But how can this organizations most effectively use them?

James G. Kobielus, senior analyst at Forrester Research, recommends in the report “Enterprise Hadoop Best Practices: Concrete Guidelines From Early Adopters in Online Services” the following strategy when building an Hadoop framework of best practices in the enterprise:

Align Hadoop with your Big Data business priorities. Define your organization’s strategy for Big Data.  Prioritize those projects and look to see how Hadoop could benefit them.  Kobielus said that “Enterprises should have their big data priorities straight before deciding to address them with Hadoop, an EDW, some hybrid of the two, or another approach entirely… Start with a well-scoped Hadoop project with a clear impact and near-term payoff on your core big data imperative. ”

Integrate Hadoop with Enterprise software like Enterprise Data Warehouse (EDW). Kobielus said that “Hadoop has considerable promise in cloud EDW  to support extremely scalable analytics, which is big data’s core application. ”

Build Hadoop implementations with enterprise-grade platforms. There are many available Hadoop products  now — evaluate them carefully before making a decision on what is right for your organization.  But Kobielus cautioned against standardizing on any one vendor or product — things are currently changing too quickly, and locking yourself into any one now could be a mistake.

Don’t build your Hadoop cluster any bigger than you need to. Kobielus commented that organizations should “explore a big data approach like Hadoop only if your data volumes are likely to scale into the high terabytes or even petabytes.  If you overinvest in data storage, computing and networking capacity, you’ll add to your cost overhead without any concomitant business benefit”

Architect Hadoop projects in a way that will allow them later to be combined. Kobielus said that “Hadoop clusters implement a common stack of Hadoop subprojects, from the storage layer on up. This architectural approach facilitates subsequent convergence of silos as well as easy promotion of MapReduce and other jobs between the silos… Be sure to align your tactical Hadoop deployments so you can integrate and federate them as needed into a shared-service utility.”

Build up Hadoop expertise among staff members. Reach out to the community.  Build expertise in MapReduce technology.  Seek out guidance frmo consultants and cloud/SaaS providers like Amazon and Appistry.