{"id":203,"date":"2007-02-07T14:08:52","date_gmt":"2007-02-07T22:08:52","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=203"},"modified":"2007-02-07T14:08:52","modified_gmt":"2007-02-07T22:08:52","slug":"storage-smart-storage-products-save-big-space","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/storage-smart-storage-products-save-big-space\/","title":{"rendered":"Storage: Smart storage products save big space"},"content":{"rendered":"

Storage Magazine<\/a> has announced their list<\/a> of Best Storage Products from 2006.\u00a0 Storage hardware is definitely getting smarter.\u00a0\u00a0Two similar\u00a0features that propagated through vendor products in 2006\u00a0are deduplication\u00a0and single-instance storage (SIS).\u00a0 Both of these technologies try to identify data redundencies and to reduce the storage of multiple copies of data.<\/p>\n

Compliance is significantly driving up the amount of electronic data that is being stored, particularly storage of emails and instant messages.\u00a0 Consider an email message sent to 15 staff members containing a 3MB attachment — archival of all 15 copies of the same email\u00a0would be\u00a0a huge waste of storage.\u00a0 Email archive software typically will use a single-instance storage (SIS) algorithm so that only one copy is stored and references or pointers are created that point to the shared copy.<\/p>\n

Deduplication technology looks beyond file-level storage and tries to identify redundancies on a subfile or block level.\u00a0 Similar to the way SIS works, when replicated data is identified, one copy is retained and referenced from multiple places.<\/p>\n

Deduplication is particularly useful\u00a0for backups because only unique data block differences need to be backed up.\u00a0 The deduplication process can be broken down<\/a> into three steps:<\/p>\n

    \n
  1. Analysis<\/strong>\u00a0— An image of the backup data is analyzed.\u00a0 The analysis may use hints from information like metadata, and file and path names.<\/li>\n
  2. Redundancy Identification<\/strong> — Based on the analysis, identify which data pieces are redundant.\u00a0 Rather than bit-by-bit comparisons, a hash algorithm is typically applied to a block of data which creates a unique signature for that block.\u00a0 Redundant blocks are identified when their hash signatures match.\u00a0 Common hash algorithms used might be SHA-1, MD5 or some proprietary method.
    \nBit-by-bit comparisons are sometimes done too, and that produces the most efficient identification, but the method is very I\/O and compute intensive.<\/li>\n
  3. Redundancy Elimination<\/strong> — Reference pointers need to be created.\u00a0 Depending on the product vendor, either forward or reverse referencing is used.\u00a0 Reverse references point to the first occurrence of the data.\u00a0 Forward references writes out the current data and updates the previous occurence to be a pointer to the new area.<\/li>\n<\/ol>\n

    Another consideration<\/a> is something that is called in-band and out-of-band deduplication.\u00a0 In-band means that deduplication is performed as the data is collected and written.\u00a0 Out-of-band is a secondary asynchronous process that happens later.<\/p>\n

    In-band processing can slow down the backup process because of the deduplication processing.\u00a0 Out-of-band dumps a complete backup to disk and then applies deduplication.<\/p>\n

    Out-of-band processing has the advantage that deduplication could be run across parallel processes.\u00a0 But it also has the disadvantage that during the duplication process the data may need to be read more times and if multiple processes are used, there could be disk contention problems.\u00a0\u00a0Out-of-band also requires that an original data image be captured and then shrunk during the deduplication process which means that more disk space needs to be available.<\/p>\n

    \"Digg<\/a><\/div>
    \"Reddit<\/a><\/div>
    \"Stumble<\/a><\/div>
    \"Buzz<\/a><\/div>
    \"Vote<\/a><\/div>
    \"Share<\/a><\/div>
    \"Bookmark<\/a><\/div>
    \"Kick<\/a><\/div>
    \"Shout<\/a><\/div>
    \"Share<\/a><\/div>
    \"Bookmark<\/a><\/div>
    \"Post<\/a><\/div>
    \"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

    Storage Magazine has announced their list of Best Storage Products from 2006.\u00a0 Storage hardware is definitely getting smarter.\u00a0\u00a0Two similar\u00a0features that propagated through vendor products in 2006\u00a0are deduplication\u00a0and single-instance storage (SIS).\u00a0 Both of these technologies try to identify data redundencies and…<\/span><\/p>\n

    Read more ›<\/a><\/div>\n

    <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,34],"tags":[],"class_list":["post-203","post","type-post","status-publish","format-standard","hentry","category-content-management","category-storage"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/203","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=203"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/203\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=203"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=203"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=203"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}