ECM: Government Challenged by Massive Growth in Data

Think your corporate content management requirements are daunting? Think about the government. Tax returns, email, court records, legislation, correspondence, military records… Agency after agency, at all levels of government, are cranking out incredible amounts of unstructured data. And it is all complicated by the maze of retention and record management requirements and regulations stemming from scores of legislation.

At the federal level, it is the responsibility of the US National Archive and Records Administration (NARA) to manage the complete scope of documents and records. Most federal documents eventually are archived into the Electronic Records Archive. But some historical documents need to be retained in their original format — consider just the Historical Records of Congress, documents that total more than 380 million pages and require 33 miles of shelving to house.

The National Archives works along side of 300 different federal agencies. File and data formats are constantly changing and evolving. A major challenge for historical retention is format obsolescence. For example, Microsoft Word is the current de facto standard for office documents — but will there continue to be software that can read and render today’s documents 100 years from now?

The amount of data being generated each and every day is incredible and continually increasing. To help individual federal agencies to put in places good data management processes, a Federal Enterprise Architecture was created as a framework for agencies to use when capture records management requirements and setting up their individual data management systems.

But data retention is ultimately the responsibility of the individual agencies, and as a result, compliance and the degree of sophistication in systems across the agencies varies widely. Because many agencies don’t view records management as a critical responsibility for their agency, the systems can be lacking. Getting agency people trained on the systems and technology and then to follow through and to enforce the system use and policies are big challenges.

The National Archives will require all federal agencies in September 2009 to document how they use records schedules. The agencies will need to show how they can store, search for and manage all their records.

Ultimately all federal documents are to be archived within the new Electronic Records Archive which is scheduled to go on line this September. The first agencies to test the new system include the Patent and Trademark Office, Bureau of Labor Statistics, Navy Oceanographic Office and National Nuclear Security Administration.