Bridget Worth asks the pIT stop panel:

How can I measure the success of my delivered IT applications from a business perspective?

The pIT stop panel's replies:

I always worry when I see a question like this. My usual answer is with the question: “What did IT and the business agree that the measurable metrics were going to be, before the project was started?”. If the question is only asked after the application has gone live then something is wrong in the relationship between business and IT.

To avoid getting into this situation I recommend that a “benefits contract” is entered into when the project is initiated. This lays out what the key expectations are between the IT department and the business sponsors, including agreeing in advance the metrics that will be used to measure success. Some of the metrics are likely to be simple project management metrics, such as ensuring that the project delivers its agreed functional scope, to time and to budget. Others are likely to be service-level related, such as system response time, system availability, and so on. However, the most valuable metrics are those that relate to the business outcomes that are delivered as a result of the application being delivered, such as a reduction in debtor days outstanding.

My other worry with this type of question is that it implicitly treats applications as something separate, whereas they should really be part of a wider business change project – and it is the business change project that should be the real focus of measurement. Applications are only a small part of a business change project, with training, process change, behavioural change and many other activities rounding out the plan. I’d much prefer to see a set of business impact metrics agreed for a business change project, with the technical and project IT metrics mapped against these, to show which IT specific metrics are needed to support them.

By David Mitchell, senior vice president, IT research, Ovum

Measuring the benefits of particular IT projects to the business - the good old “return on investment” (ROI) - remains a tough if not impracticable challenge. The irony here is that other parts of the business are often seemingly held to less aggressive targets than IT is. The old comment about advertising – “We know 10 per cent of it works, we just don’t know which 10 per cent” is salutary. Even mergers and acquisitions activity, the heart of so much recent corporate “growth” often relies on estimates, fudges and gut instincts.

Perhaps the biggest challenge is to understand which elements of a “new application” are actually driving a business change. After all, very few organisations today have extensively modelled their business services, end to end, to be correlated to IT infrastructure. If you really want to know the ROI of a new application you need to be delivering IT as a measurable service – having adopted a framework such as the IT Infrastructure Library (ITIL). First model the business service and the various cost components involved. It should be far easier to subsequently calculate application impacts.

Usually a new application is introduced in the context of a business process change; it’s difficult to identify which benefits came from the new infrastructure, and which came from the new business process.

There are some core metrics any business can understand. Deploying an application enabled the company to cut a certain number of jobs say, or reduce its telecoms expenses by a third. However, even the job cutting benefit can be overstated. You may have moved jobs off the payroll and onto a new outside consulting budget. IT should of course use numbers, but don’t treat it as a science project – use the financials you have to support the business case. Don’t be afraid of spin. IT needs to use PR just as the rest of the business does.

By James Governor, co-founder and principal analyst, RedMonk