Right data, right place, right time

High-profile cases of fraud and financial mismanagement have unleashed a torrent of further data retention rules and regulations

Day by day, the average volume of data that organisations have to store and manage is rising. Meanwhile, the laws, regulations and industry guidelines governing the retention of that information are expanding in scope, and the penalties for non-conformance are becoming more stringent.

Easily one of the biggest headaches for today’s IT leaders is how to handle the ever-expanding pools of unstructured data ­ – email, instant messaging, word processing documents, web page content, audio and video files. Unlike structured information, which is housed and indexed in set formats such as database files, this data has tended to be tricky to trace and manage.

Numerous tools and methodologies to help search, classify, label and store unstructured data have now emerged, but decoding the various labels that different vendors apply to the technology, and identifying the individual products that may or may not prove useful in particular environments can be difficult.

Many of the tools required for efficient information management are now encompassed in enterprise content management (ECM) software. Analyst Gartner predicts that spending on ECM software will rise by an average of 12 per cent a year between 2008 and 2012.

Today, most enterprise IT vendors include ECM products within their software portfolios, and there remain a few smaller ECM specialists such as Open Text and FileNet. All offer systems designed to handle both structured and unstructured data stored in different storage repositories ­ – content management systems, databases and data warehouses, for example – ­ spread across distributed organisations, and linked to workflow and business process management tools.

Among the various products that fall within the ECM category are records management tools. These are not designed exclusively for handling digital media and are often delivered alongside optical (OCR) and intelligent (ICR) character recognition technologies required to accurately scan written or printed information. On top of this, these tools focus on file indexing, data retention and deletion schedules, data protection and accurate metadata storage.

The need for records management tools has been intensified by the increase in rules, regulations and standards dictating how companies store information and in what format. In the UK, The National Archives has defined a standard for how different record types should be managed, how metadata should be created and security applied, and also offers implementation guidelines.

High-profile cases of fraud and financial mismanagement have unleashed a torrent of further data retention rules and regulations. Many vendors have revamped their archiving and search tools with increased emphasis on their corporate governance attributes as a result.

One of the fastest-growing areas of the ECM market is electronic discovery (e-discovery). These tools perform searches of unstructured data ­ – usually email ­ – to identify and retrieve specific references or content within compliance-dictated timeframes for legal purposes.

E-discovery is usually included as one element of information lifecycle management (ILM), an umbrella term used by many storage companies to define a process for the management of specific data or information throughout its useful life, from creation through storage to deletion.

ILM is closely linked to hierarchical storage management, a technique that sees storage software move data from one type of media to another according to either the frequency with which it is accessed, or preset compliance rules that dictate how long the information should be retained, and how quickly it can be retrieved, before its eventual deletion.

Archiving strategies within ILM frameworks dictate how long data is kept on one type of media before being moved to another. The need to keep down costs is one of the overriding themes here. Companies are increasingly finding it prohibitively expensive to keep high volumes of data on hard disks for even a short period of time, given the rising costs associated with housing, powering and cooling servers.

As a result, many firms move data on to less-expensive storage mediums such as tape or lower cost, but slower, Serial ATA drives as soon as possible, and try to keep the information stored across the organisation to a minimum.

At global foreign exchange company CMC Markets, the introduction of an information management strategy has helped manage the 10TB of data that the firm backs up regularly.

CMC retains data on disk for about five weeks, so that the retention period captures whole calendar months. Any information that users request to be restored from that five-week period can be retrieved within two hours, but beyond that the data is stored on tape and takes longer to obtain.

“If you could keep everything forever, everyone would be happy, but there has to be some trade-off,” says Greg Gawthorpe, technical operations team leader at CMC Markets. “It is very rare that people request backed-up information older than a month, and if it is older than that, they are generally happy to wait for us to get the tape back within 24 hours.”