{"id":5160,"date":"2015-01-29T07:00:14","date_gmt":"2015-01-29T15:00:14","guid":{"rendered":"http:\/\/formtek.com\/blog\/?p=5160"},"modified":"2014-12-26T06:59:45","modified_gmt":"2014-12-26T14:59:45","slug":"big-data-help-wanted-data-janitor-positions","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/big-data-help-wanted-data-janitor-positions\/","title":{"rendered":"Big Data: Help Wanted: Data Janitor Positions"},"content":{"rendered":"

Garbage In. \u00a0Garbage out. \u00a0As much as big data leads the hype headlines for technology, data quality remains an achilles heel of the technology.<\/p>\n

Eva Ho<\/a>, a partner with the early-stage venture capital firm Susa Ventures , said that<\/a>\u00a0“the little secret of big data is that a lot of it isn’t clean.” \u00a020 percent of big data developers say that that managing the quality of their data is a major issue for them, as reported by Evans Data<\/a>.<\/p>\n

Jeffrey Heer<\/a>, a professor of computer science at the University of Washington, said that<\/a> “it\u2019s an absolute myth that you can send an algorithm over raw data and have insights pop up.”<\/p>\n

‘Data cleaning’ is the task of taking raw data and making it usable.<\/p>\n

Erhard Rahm<\/a> and\u00a0Hong Hai Do<\/a>, researchers at the University of Leipzig, explained that<\/a> “Data cleaning<\/em>, also called data cleansing<\/em> or scrubbing<\/em>, deals with detecting and removing errors and\u00a0inconsistencies from data in order to improve the quality of data. Data quality problems are present in single\u00a0data collections, such as files and databases, e.g., due to misspellings during data entry, missing information\u00a0or other invalid data. When multiple data sources need to be integrated, for example, in data warehouses, federated\u00a0database systems or global web-based information systems, the need for data cleaning increases\u00a0significantly.”<\/p>\n

50 to 80 percent of time spent by data scientists is estimated to be allocated to collecting and cleaning up data sets. \u00a0Data scientists tasked with data cleaning have been nicknamed ‘Data Janitors’ and ‘Data Wranglers’.<\/p>\n

Monica Rogati<\/a>, vice president for data science at Jawbone, told the New York Times<\/a> that “data wrangling is a huge \u2014 and surprisingly so \u2014 part of the job. \u00a0It\u2019s something that is not appreciated by data civilians. At times, it feels like everything we do.”<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

Garbage In. \u00a0Garbage out. \u00a0As much as big data leads the hype headlines for technology, data quality remains an achilles heel of the technology. Eva Ho, a partner with the early-stage venture capital firm Susa Ventures , said that\u00a0“the little…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-5160","post","type-post","status-publish","format-standard","hentry","category-big-data"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/5160","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=5160"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/5160\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=5160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=5160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=5160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}