Open Data: Rapid Data Decay of Scientific Data

While governments have been pressured over the last few years to increasingly make data they’ve collected publicly accessible, the same level of pressure hasn’t yet been applied to data that’s been collected within the scientific community.

A paper published in December 2013 in Current Biology reports that about 80 percent of raw scientific data collected for research is typically lost within 20 years after the publication of the research.  The authors of the paper found that most data could be recovered from research conducted within the last two years, but after that accessibility to research data drops off to as low as 20 percent for research performed in the early 1990’s.

The culprits for the loss of data include obsolete storage technology, lack of processes for data archival, and frequently changing contact information for researchers like email addresses.

Tim Vines, author of the paper, said that “Publicly funded science generates an extraordinary amount of data each year.  Much of these data are unique to a time and place, and is thus irreplaceable, and many other datasets are expensive to regenerate…  Some of the time, for instance, data was saved on three-and-a-half inch floppy disks, so no one could access it…  The current system of leaving data with authors means that almost all of it is lost over time… I don’t think anybody expects to easily obtain data from a 50-year-old paper, but to find that almost all the datasets are gone at 20 years was a bit of a surprise…  Losing data is a waste of research funds and it limits how we can do science.  Concerted action is needed to ensure it is saved for future research.”

The study was made in response after a researcher at the University of British Columbia in global climate change, Lizzie Wolkovich, tried to contact Melvin McCarty at the US Department of Agriculture to gain access to a 15-year span of data collected by the department.  She’d found that McCarty had since died and that none of his colleagues knew how to retrieve the data from the original research.  Wolkovich said that “I think there’s been an issue of people not recognizing the value of data — especially back then.  If only we had known, and people were trying harder to protect this data we’d have a lot more information on what’s going on with global climate change.”

Peter Murray-Rust, reader at the department of chemistry at the University of Cambridge in the UK, said that “I’m surprised the numbers are not higher…  For example, in computational chemistry and materials science, essentially no primary data is published.”

Some journals, like Molecular Ecology where Vines is a managing editor, now require that raw data be submitted along with research papers, and the journal then archives the data along with the article.