{"id":11160,"date":"2021-12-30T07:00:00","date_gmt":"2021-12-30T15:00:00","guid":{"rendered":"https:\/\/formtek.com\/blog\/?p=11160"},"modified":"2021-07-17T11:46:11","modified_gmt":"2021-07-17T19:46:11","slug":"crisis-of-reproducibility-the-need-for-a-reproducibility-checklist","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/crisis-of-reproducibility-the-need-for-a-reproducibility-checklist\/","title":{"rendered":"Crisis of Reproducibility: The Need for a Reproducibility Checklist"},"content":{"rendered":"\n

‘Not reproducible’, or as some Google researchers are calling it, ‘underspecification<\/a>‘. AI results reported in papers might have spectacular and glowing results, but when other researchers attempt to replicate those same results when following the methodology specified in the research paper often fail.<\/p>\n\n\n\n

Why? There are a multitude of reasons, and intentional cheating by manipulating data to achieve the desired result, potentially for professional gain, is likely not a major reason, although it can’t be entirely dismissed.<\/p>\n\n\n\n

One of the big problems is that a lot of research results are based on a very small data set with all of the data coming from the same location. This lack of data diversity can often lead to impressive results base on that narrow slice of data used but which then fails when applied to anything outside of that domain.<\/p>\n\n\n\n

On the opposite side of the spectrum are results reported from extremely large data sets, for example data available only to large tech companies, and then processed using extremely powerful computers. In this case, the results can not even be attempted for reproducibility by others because the cost of acquiring the resources to try.<\/p>\n\n\n\n

Anna Rogers<\/a>, a machine-learning researcher at the University of Massachusetts, asked in an article by Wired<\/a> \u201cis that even research anymore? It\u2019s not clear if you\u2019re demonstrating the superiority of your model or your budget.\u201d<\/p>\n\n\n\n

Ben Dickson<\/a>, founder of TechTalks<\/a>, quoted some comments from users of the Reddit r\/MachineLearning<\/a> community in his article on thenextweb.com<\/a>: \u201cProbably 50%-75% of all papers are unreproducible. It\u2019s sad, but it\u2019s true,” and \u201cThink about it, most papers are \u2018optimized\u2019 to get into a conference. More often than not the authors know that a paper they\u2019re trying to get into a conference isn\u2019t very good! So they don\u2019t have to worry about reproducibility because nobody will try to reproduce them,” and “its’ easier to compile a list of reproducible papers than nonreproducible ones.”<\/p>\n\n\n\n

To help combat the problem of reproducibility, Joelle Pineau<\/a>, a computer science professor at McGill, has developed what she is calling a “reproducibility checklist”<\/a>. She suggests the checklist be required for papers that are accepted to provide additional guidance about how the results were achieved. The checklist includes things like the data, the number of models trained, the compute resources used, and any additional assumptions made when arriving at the results. Ideally, public availability of the code and data sets used in the research would be a major step towards enabling reproducibility.<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

‘Not reproducible’, or as some Google researchers are calling it, ‘underspecification‘. AI results reported in papers might have spectacular and glowing results, but when other researchers attempt to replicate those same results when following the methodology specified in the research…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11160","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/11160","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=11160"}],"version-history":[{"count":4,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/11160\/revisions"}],"predecessor-version":[{"id":11165,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/11160\/revisions\/11165"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=11160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=11160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=11160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}