Order out of chaos

One of the biggest challenges organisations face in the next few years is how to manage their information effectively.

The rationale is that if businesses see information as the lifeblood of their organisation, managing it is – or must become – a strategic issue.

Unfortunately, most organisations are struggling, says Neil Macehiter, a partner at analyst MWD Advisors.

The historical raison d’être of most IT departments was to manage structured data held in databases to support transaction processing. But, says Macehiter, we have moved from a world of purely structured information to one where most information is unstructured, in the form of email, web sites and word processing documents. ‘The proliferation of unstructured data has brought about a shift in the way organisations use information,’ he says.

Rather than relying purely on tightly coupled applications that operate in rigid functional silos, most companies now require access to both structured and unstructured information from across the organisation and beyond, to support collaborative, ad hoc business processes and ways of working.

Addressing such a change is a significant challenge, and one that is likely to accelerate as the amount of information generated increases and the requirement to access it quickly and effectively grows, not least so that organisations can meet regulatory compliance obligations.

‘The industry has a strong heritage of managing structured data, but if you look at the way unstructured data is used there is much less heritage,’ says Macehiter.

‘Issues such as where it is located and how to back it up and manage it consistently are top of many people’s minds, given that organisations are increasingly differentiating themselves in terms of how well they use the information they have.’

Another key issue is that many IT departments concentrate more on technology than on information.

‘Historically, information has been seen as second-class and as something that applications manage,’ says Macehiter. ‘But one of the problems is that information management is such a broad term. It is really about managing information throughout its lifecycle, but to do that effectively requires a significant change in attitude.’

Writing policies, for everything from creation and distribution to retention and destruction, is an organisational concern, and one that requires a true chief information officer (CIO), says Mike Davis, a senior analyst at Ovum.

‘This is where a CIO is really needed to work with the rest of the organisation. They have to have lifted themselves away from the operational side of IT to sit on the board and be a genuine strategist,’ he says.

‘The one thing that underpins the whole information management debate – and it is the hardest thing – is to identify the value of corporate information and understand the organisation’s information needs.’

But such awareness involves taking a holistic view of the organisation’s assets and creating an information architecture to make it easier to find them.

In the past, the concept of creating enterprise data models failed because these models were not linked to business processes and proved difficult to keep up to date. But methodologies such as the Zachman Framework have advanced the management of information.

‘These methodologies help you to picture and understand where your information assets are and how to secure them,’ says Macehiter.

‘It is about bringing information up a level and being able to think of it in the same way that people thought about systems architectures.’

Once such an architecture and related information management policies and processes have been established, hierarchical storage technologies can prove helpful. The aim is to store different types of data in the most appropriate and cost-effective medium, depending on how it is used and how often users need to access it. Frequently used data, for example, might be kept in a disk array, while rarely used data might be archived on tape.

Although some organisations have applied information lifecycle management concepts already, the focus to date has been on tackling only a subset of information, rather than addressing the issue across the board.

One organisation that found such an approach useful, however, is Ordnance Survey.

Among its geographical data, the organisation collects high-resolution aerial photographs from which it extracts topographical information to form the basis of mapping products and related services.

The information needs to be retained for 50 years or more for comparison purposes, whereas other data that Ordnance Survey holds is subject to thousands of changes every day. A snapshot is taken annually, however, and stored for posterity.

Dave Lipsey, information systems infrastructure manager at Ordnance Survey, explains the challenge. ‘We had so much data that our backup window was extending into the working day and interfering with our daily operations,’ he says.

‘But the cost of collecting aerial data is huge, so we needed to ensure that it was safe. Backing up onto linear tape open (LTO) tapes was less reliable than we wanted it to be, however, and it was also quite expensive.’

As a result, the organisation introduced an archiving system from BridgeHead Software and Plasmon to store its static files, and is now considering widening its use to also include the annual snapshot data.

Another crucial concern for organisations trying to manage their information effectively is ensuring that it is clean and provides a single version of the truth, regardless of where it is located. Because information acts as the foundation for corporate decision-making, operational data from many different sources not only has to be accessible, increasingly in real-time, but also consistent and accurate.

Bill Swanton, vice president of research at analyst AMR Research, believes that the need for operational data integration is pushing many companies to try to tackle the problem of master data management.

‘They want to avoid duplication of core information about customers, partners and suppliers, and ensure that it is consistent and accurate both inside the company and outside,’ he says.

‘But it is a big issue, and a key problem is that most companies have never had a formal definition of their key data.’

Agreeing such definitions is a governance issue and a political minefield that cannot be underestimated. It can potentially be addressed using specialised repository products that are starting to emerge from vendors such as SAP, Oracle and Tibco.

But uptake of such applications is embryonic, with most organisations simply cobbling together a hotchpotch of tools, including metadata repositories, discovery tools and workflow systems. ‘The problem is that it is a higher-level business and architectural issue,’ says Swanton.

‘You have to manage the process by which information is created and handled, but getting this process right takes years. People are taking the first steps towards control, but it is like pulling a thread. You find that the same data is used in more and more places, which means there are more places to mess up the hand-offs.’

Because information management is such a complex issue, it is not something that can be tackled in one fell swoop.

‘Start to try to get a handle on your important information assets, but don’t try to boil the ocean,’ says MWD’s Macehiter.

‘It is about focusing on the issues that are highest up the corporate agenda, thinking about what information pertains to what business processes, and dealing with it piece by piece.’

Islington Council improves information management

Yell Group boosts data competence

What the experts say about information management

Civil servants struggle with edata