{"id":8300,"date":"2018-05-07T07:00:45","date_gmt":"2018-05-07T15:00:45","guid":{"rendered":"http:\/\/formtek.com\/blog\/?p=8300"},"modified":"2018-03-19T20:09:11","modified_gmt":"2018-03-20T04:09:11","slug":"machine-learning-lack-of-reproducibility-threatens-credibility","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/machine-learning-lack-of-reproducibility-threatens-credibility\/","title":{"rendered":"Machine Learning: Lack of Reproducibility Threatens Credibility"},"content":{"rendered":"

Reproducibility\u00a0 and repeatability form the foundation of scientific research. Science works best when researchers have enough information and understanding of the parameters of research that has been done previously so they can reproduce those results and work to build on established and proven ideas.<\/p>\n

Unfortunately, some areas like Machine Learning are moving so fast that these basics are often overlooked.\u00a0 After all, new frameworks and tools for machine learning are being introduced monthly, if not daily.<\/p>\n

Peter Warden<\/a>, Machine Learning researcher, said that “ML frameworks trade off exact numeric determinism for performance, so if by a miracle somebody did manage to copy the steps exactly, there would still be tiny differences in the end results! In many real-world cases, the researcher won\u2019t have made notes or remember exactly what she did, so even she won\u2019t be able to reproduce the model. Even if she can, the frameworks the model code depend on can change over time, sometimes radically, so she\u2019d need to also snapshot the whole system she was using to ensure that things work.”<\/p>\n

Denny Britz<\/a>, Deep Learning Researcher,\u00a0\u00a0wrote that<\/a> “in practice, as everyone re-implements techniques using different frameworks and pipelines, comparisons become meaningless. In almost every Deep Learning model implementation there exist a huge number ‘hidden variables’ that can affect results.”<\/p>\n

Researchers need to take advice from David Donoho<\/a>, Stanford Professor, who wrote<\/a> “computational reproducibility is not an afterthought \u2014 it is something that must be designed into a project from the beginning.”<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

Reproducibility\u00a0 and repeatability form the foundation of scientific research. Science works best when researchers have enough information and understanding of the parameters of research that has been done previously so they can reproduce those results and work to build on…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[65,62],"tags":[],"class_list":["post-8300","post","type-post","status-publish","format-standard","hentry","category-deep-learning","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/8300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=8300"}],"version-history":[{"count":4,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/8300\/revisions"}],"predecessor-version":[{"id":8304,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/8300\/revisions\/8304"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=8300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=8300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=8300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}