{"id":1048,"date":"2010-02-22T07:00:54","date_gmt":"2010-02-22T15:00:54","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=1048"},"modified":"2010-02-22T07:00:54","modified_gmt":"2010-02-22T15:00:54","slug":"technology-dealing-with-big-data","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/technology-dealing-with-big-data\/","title":{"rendered":"Technology: Dealing with 'Big Data'"},"content":{"rendered":"

The amount of data that companies and individuals are creating is growing incredibly quickly.\u00a0 Some organizations have managed to collect massive data sets.\u00a0 O’Reilly<\/a> refers to these huge data sets as ‘Big Data’ — data sets that are in the hundreds of gigabytes, terabytes, or much much more.\u00a0 Big data is being driven by data collection coming from web crowds, sensor networks, informatics and geodata.\u00a0 O’Reilly says that companies that are able to tap into and effectively analyze data will be at a competitive advantage to other companies, and uses Google as an example.\u00a0 Google’s success has been all about their ability of being able to make sense out of massive amounts of data.<\/p>\n

Big data analysis techniques will especially benefit data analysis projects like predictive models and ratings engines.\u00a0 When large amounts of data can be incorporated into the decision making process, better decisions will be made. And analysis of very large data sets<\/a> will lead to revolutionary breakthroughs in commerce, science and society.<\/p>\n

In the realm of business, profitability comes through customers, and the performance of business is controlled by these three dimensions<\/a>:
\n– Increasing the number of custoemrs
\n– Increasing the profitability per customer
\n– Retaining customers for longer periods
\nBeing able to better use and analyze customer data will let organizations improve performance — and analysis of Big Data collected from customers will help.\u00a0 Forbes <\/a>says that “‘Big Data’ is going to be ‘Big Business'”.<\/p>\n

Businesses have grabbed onto the idea and are scrambling to get their hands on better tools for data analysis.\u00a0 Hadoop<\/a>, an open-source application, is one tool that is creating a lot of excitement. Hadoop is a framework for enabling the connection and scale-up of cheap hardware and tools for analyzing unstructured and structured data.\u00a0 While Relational Databases are good at analyzing small subsets of data, Hadoop tries to examine and interpret all available data.\u00a0 It uses an algorithm called MapReduce, and works by being able to distribute data for analysis to a large number of servers that can all work in parallel, collating the results and then presenting the final result.\u00a0 The architecture allows Hadoop to operate at very high data availability and also to be able to process and analyze enormous amounts of information quickly.<\/p>\n

Hadoop is one technique that has gained popularity, and there has been a cottage industry of tools and businesses that have been built on top of Hadoop.\u00a0 But this field is in its infancy.\u00a0 Expect to see a lot of new ideas and activity around collecting and analyzing ‘Big Data’.<\/p>\n

<\/p>\n
\n
\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

The amount of data that companies and individuals are creating is growing incredibly quickly.\u00a0 Some organizations have managed to collect massive data sets.\u00a0 O’Reilly refers to these huge data sets as ‘Big Data’ — data sets that are in the…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,34,35,36],"tags":[],"class_list":["post-1048","post","type-post","status-publish","format-standard","hentry","category-content-management","category-storage","category-structured-data","category-technology"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/1048","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=1048"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/1048\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=1048"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=1048"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=1048"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}