{"id":3524,"date":"2013-02-08T07:00:48","date_gmt":"2013-02-08T15:00:48","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=3524"},"modified":"2013-02-08T07:00:48","modified_gmt":"2013-02-08T15:00:48","slug":"big-data-are-expectations-too-high","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/big-data-are-expectations-too-high\/","title":{"rendered":"Big Data: Are Expectations Too High?"},"content":{"rendered":"

2013 is expected to be a big year for Big Data. \u00a0About a quarter of $1+ billion companies are actively working on Big Data projects, and a third are in the evaluation and planning phases, according to a study by Actuate\/King Research<\/a>. \u00a0Of the roughly remaining 40 percent of companies, a lack of staff expertise and concern over the cost of the technology is holding them back from going forward with Big Data initiatives.<\/p>\n

An IDC report<\/a> (courtesy of EMC) found that businesses will increasingly fund their IT infrastructure to stay abreast of increasing amounts of data. \u00a0“The investment in spending on IT hardware, software, services, telecommunications and staff\u00a0that could be considered the ‘infrastructure’ of the digital universe and telecommunications will\u00a0grow by 40% between 2012 and 2020… Of course, investment in targeted areas like storage\u00a0management, security, Big Data, and cloud computing will grow considerably faster.”<\/p>\n

There’s a lot of data out there waiting to be analyzed. \u00a0IDC says that less than .5 percent of the thousands of exabytes of data which exist are analyzed. \u00a0But they also estimate that data analytics may be useful on only about a third of all stored data. \u00a0The .5 to 33 percent gap means that there is a lot of untapped data, which would imply the potential for Big Data analytics is high.<\/p>\n

But some question if Big Data is ready for prime time yet. A universal opinion<\/a> is that there is a tremendous shortage of the analytic skills required to do good work with Big Data. \u00a0And, as a result, there is a worry that because of \u00a0the huge interest in Big Data, it is likely that there will be a large influx of unqualified workers into the world of data analytics, tainting the quality of work.<\/p>\n

There’s also the question if the expectations for Big Data run too high. \u00a0Nicholas Carr<\/a> points out<\/a> an interesting section from\u00a0Nassim Nicholas Taleb<\/a>‘s recent book Antifragile <\/a>that runs counter to benefits that many perceive in analyzing huge amounts of data. \u00a0Taleb argues that analyzing large amounts of data over very short intervals, such as hourly or daily can often lead to very misleading conclusions. \u00a0Taleb\u00a0writes that “The more frequently you look at data, the more noise you are disproportionately likely to get (rather than the valuable part called the signal); hence the higher the noise to signal ratio… \u00a0The best solution is to only look at very large changes in data or conditions, never small ones.”<\/p>\n

And there is a worry that human intuition is being by-passed in favor of black-box-derived Big Data solutions. \u00a0Steve Lohr<\/a> writes in a New York Times article<\/a> that “Listening to the data is important, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?”<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

2013 is expected to be a big year for Big Data. \u00a0About a quarter of $1+ billion companies are actively working on Big Data projects, and a third are in the evaluation and planning phases, according to a study by…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-3524","post","type-post","status-publish","format-standard","hentry","category-big-data"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/3524","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=3524"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/3524\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=3524"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=3524"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=3524"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}