{"id":27,"date":"2006-06-09T07:57:43","date_gmt":"2006-06-09T15:57:43","guid":{"rendered":"http:\/\/www.formtek.com\/blog\/?p=27"},"modified":"2006-06-09T07:57:43","modified_gmt":"2006-06-09T15:57:43","slug":"ilm-tiered-storage-management","status":"publish","type":"post","link":"https:\/\/formtek.com\/blog\/ilm-tiered-storage-management\/","title":{"rendered":"ILM Tiered Storage Management"},"content":{"rendered":"

Broadband, fast computers, cheap storage and multi-megapixel color cameras and scanners have resulted in an explosion of large-content files.\u00a0<\/p>\n

10 years ago, the type of file data managed by document and content management systems was dominated by black-and-white TIFF images.\u00a0 The rule-of-thumb for bitonal TIFF images was that the size of one compressed scanned page\u00a0was\u00a0about 30K.\u00a0 In that scernario,\u00a0a 100-Gig disk\u00a0was probably more than adequate for\u00a0most bitonal imaging applications.\u00a0<\/p>\n

But while black-and-white TIFF<\/a> images are still pretty common, content like high-resolution color images, video, audio and business documents with multi-media embeddings\u00a0are in\u00a0even more demand now.\u00a0 Enterprise Content Management<\/a> (ECM) systems provide secure storage for any kind of content, and\u00a0large-size “multi-media” content is no exception.<\/p>\n

A single high-resolution color image will typically be multi-megabyte and even modestly-sized videos\u00a0can occupy tens to hundreds of megabytes of storage.\u00a0 That kind of data could eat up 100-Gig disk pretty fast.\u00a0<\/p>\n

As data ages, it typically loses importance and is referenced less and less frequently.\u00a0 Keeping\u00a0large amounts of\u00a0old data on-line at minimum complicates full-system backups and also has a good chance of eventually slowing down or gumming up overall system performance.<\/p>\n

That’s where the idea of Information Lifecycle Management<\/a> (ILM) comes in.\u00a0 Much of the ILM concept centers on Records Management<\/a> (RM), how long to retain documents, and when and how to dispose or archive documents.\u00a0<\/p>\n

But in terms of storage and cost efficiency,\u00a0ILM also has\u00a0the notion that a piece of content might gradually migrate its storage location through different tiers of storage media. When content is new, very relevant, and accessed frequently, it would be kept on the highest-performance storage media.\u00a0 And as\u00a0the content ages and becomes less relevant, it would be migrated and stored on media that is cheaper but with lower performance.\u00a0 Data could still be searchable and available on-line, but it may take longer to retrieve the content.<\/p>\n

To realize a multi-tiered storage strategy in an ECM system, the system needs to be designed to be able to support distributed storage across multiple volumes.\u00a0 This sounds like a simple requirement, but there are many systems that can’t handle it.<\/p>\n

Many ECM systems are built around the concept of storing large binary content as database BLOBs<\/a>.\u00a0 But relying on a central database to store all content will lead to problems when the total size of the content being managed starts to get big.\u00a0 The concept of tiered storage is very hard to support in this architecture — being able to identify and offload BLOBs for storage on another media may not even be possible.<\/p>\n

In the Formtek system, we did not use BLOBs as a primary storage method for content.\u00a0 Formtek | Orion<\/a>\u00a0by default stores binary content external from the document metadata housed in the database, although it doesn’t completely exclude the use of BLOBs.\u00a0 In some situations it may make sense, for example, for storing frequently displayed small thumbnail images.\u00a0<\/p>\n

The advantage of storing binary content external to the database is many.\u00a0 First, the support\u00a0for an ILM tiered storage policy is greatly simplified.\u00a0 Transaction processing is simplified.\u00a0 The system will scale well and can be administered much more easily.\u00a0 And finally, random access within a file and file streaming, especially for video and audio, is much easier to accomplish.<\/p>\n

Formtek can simultaneously manage data distributed across many devices by using the Formtek Storage Server component.\u00a0 The Storage Server\u00a0is an agent that runs on the machine where the data is stored and that can receive and transmit data with\u00a0the main Orion server and\/or directly with user machines accessing the content system applications.\u00a0 With this architecture, content files can be distributed across many devices, each of which is managed by a Storage Server agent.<\/p>\n

\"Digg<\/a><\/div>
\"Reddit<\/a><\/div>
\"Stumble<\/a><\/div>
\"Buzz<\/a><\/div>
\"Vote<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Kick<\/a><\/div>
\"Shout<\/a><\/div>
\"Share<\/a><\/div>
\"Bookmark<\/a><\/div>
\"Post<\/a><\/div>
\"Google<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"

Broadband, fast computers, cheap storage and multi-megapixel color cameras and scanners have resulted in an explosion of large-content files.\u00a0 10 years ago, the type of file data managed by document and content management systems was dominated by black-and-white TIFF images.\u00a0…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,31,34],"tags":[],"class_list":["post-27","post","type-post","status-publish","format-standard","hentry","category-content-management","category-security","category-storage"],"_links":{"self":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/27","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":0,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/posts\/27\/revisions"}],"wp:attachment":[{"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/media?parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/categories?post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/formtek.com\/blog\/wp-json\/wp\/v2\/tags?post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}