{"id":1080,"date":"2010-03-31T07:00:56","date_gmt":"2010-03-31T15:00:56","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=1080"},"modified":"2010-03-31T07:00:56","modified_gmt":"2010-03-31T15:00:56","slug":"storage-preserving-the-web","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/storage-preserving-the-web\/","title":{"rendered":"Storage: Preserving the Web"},"content":{"rendered":"

Information on the web comes and goes.\u00a0 How many times have you clicked on a link of a web page only to find a non-existent page?\u00a0 Actually the average life of a web page is only around 44 to 77 days, or about a couple of months.\u00a0 Every six months, 10 percent of web pages are lost or recycled.\u00a0 Seems like not a big deal, but to people trying to chronicle the history and life of today, it is very disconcerting.<\/p>\n

Google and internet search tools typically maintain a cache of old now defunct web pages.\u00a0 And the Internet Archive organization<\/a>s\u00a0 keeps old stash of old web pages on their “Wayback Machine<\/a>“.<\/p>\n

In the UK, the British Library is teaming up with IBM to archive web content originating from the UK — web pages from the UK domain which currently has about 8 million sites.\u00a0 The project is being called BigSheets <\/a>and is built using Open Source components like Hadoop<\/a>, Nutch<\/a>, Lucene and Pig<\/a>.<\/p>\n

The goal of BigSheets is not only to archive web content as it is created, it also will attempt to analyze, classify and provide tools that will allow research to be performed with the information. The British Library has been collecting data since 2004 and now has about 5 terabytes of data from 6000 web sites.\u00a0 Prior to BigSheets, the effort required 10 people to manually archive and maintain the data.<\/p>\n

The British Library Chief Executive, Dame Lynne Brindley, said <\/a>the website would aim to create a record of the \u201cmajor cultural and social issues being discussed online\u201d, and \u201cavoid the creation of a digital black hole in the nation\u2019s memory\u201d.<\/p>\n

\n
\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

Information on the web comes and goes.\u00a0 How many times have you clicked on a link of a web page only to find a non-existent page?\u00a0 Actually the average life of a web page is only around 44 to 77…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,34,36],"tags":[],"class_list":["post-1080","post","type-post","status-publish","format-standard","hentry","category-enterprise-content-management","category-storage","category-technology"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/1080","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=1080"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/1080\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=1080"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=1080"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=1080"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}