Top 10 big IT outsourcing and consulting disasters
When outsourcing goes wrong, it can go very expensively wrong...
No-one ever got fired for buying IBM, according to the old saying. But what about for outsourcing to IBM. Or, indeed, bringing them or any other major IT services consultancy, such as Atos, Capita, or EDS/HPE, in to help out with any major project?
When global engineering giant Aecom announced in January that it was to outsource its IT, pretty much lock, stock and barrel, to IBM the big question was, do they know what they might be letting themselves in for?
It's one thing to outsource a particular activity, such as payroll or printer maintenance, or to bring in specialists to help with a project, but outsourcing the whole lot is effectively betting the firm on another company's competence, not just now, but in three, five and even ten years time too.
Furthermore, should an organisation want to bring its IT back in-house in the future, it may not have the in-house competence to do so - see, for example, the challenges that HMRC has had in trying to move on from its monolithic Aspire outsourcing contacts with Accenture and Capgemini.
Although the majority of outsourcing and consulting contracts no doubt prove successful in terms of the metrics agreed for them (more or less), when they go wrong, it's normally in everyone's best interests to keep schtum. But when an outsourcing deal goes very wrong, it can end in acrimony and end up in court.
So, if only to stimulate some minds in Aecom's executive suite, here's some of the biggest IT outsourcing and consulting failures of all-time. Good luck, guys! I'm sure everything will be just fine…
10) Cambridge University Hospitals NHS Foundation Trust and Epic: £200m
For many people, patient record systems are something that should be filed under ‘How hard can it be?' After all, it's just a database that needs to be delivered to a limited number of users - mostly just GPs.
Somehow, though, the NHS both nationally and at local level has spilled billions of taxpayers' pounds on patient record systems that have either flat-out not worked or expensively failed to deliver what was promised.
But at Cambridge University Hospitals NHS Foundation Trust, a botched implementation of an online patient records system provided by US software company Epic almost caused financial meltdown. Indeed, the Trust ended up having to put into ‘special measures' partly as a result of the financial problems it caused.
The system went live on 26 October 2014 and was intended to enable medical staff to access patient records on handheld devices instead of waiting for notes. But instead of improving performance, a report in November last year by the county's clinical commissioning group (CCG) found a 20 per cent drop in A&E performance, along with other problems.
The conclusions of a report into the mess were scathing: "The Trust underestimated the scale and challenges of implementing its new electronic patient record system, e-Hospital, and the impact this would have on its provision of healthcare for its patients. These issues led to significant cost increases and a failure to realise the benefits the system could provide."
The Care Quality Commission watchdog, meanwhile, found that the new system was met with confusion by staff, while locum cover proved harder and more expensive to achieve as it was difficult for new locums who were not familiar with the Trust's systems to use it.
Worst of all, though, the system produced inaccurate discharge information, leading to a risk that patients would not receive appropriate follow-up care. There were also issues with timely access to information to provide patients and other healthcare professionals with up-to-date information because of the introduction of the new system.
Happily, the whole affair didn't ruin the career of Professor Keith McNeil, former CEO of Addenbrooke's Hospital, a part of Cambridge University Hospitals NHS Foundation Trust, who was appointed NHS chief clinical information officer last year.
Nor has it harmed Epic, which has just sealed a deal to supply its excellent software to University College London Hospitals NHS Foundation Trust as part of a new ‘technology partnership' with Atos.
Next: The US Navy and EDS; and, Cornwall Council and BT
Top 10 big IT outsourcing and consulting disasters
When outsourcing goes wrong, it can go very expensively wrong...
9) US Navy and EDS: $6.9bn
Normally, it's the outsourcing company that comes out smelling of roses (or, more specifically, cash) even if a deal falls short (to put it politely).
The five-year contract with the US Navy and Marine Corps, signed in October 2000, was valued at $4.1bn with an option to extend bringing the total value up to an eye-watering $6.9bn. Yes, $6.9 billion and, yes, on an intranet (hey, it was a global one!).
The aim of the project was to consolidate networks, manage desktops and deliver intranet communications to around 360,000 seats in about 300 US Navy and Marine Corps bases around the world.
Whoever negotiated the contract on the US Navy side knew what they were doing: Payment was cunningly based on performance and working alongside EDS would be Raytheon, WorldCom (RIP), Dell and WAM!NET (now Harris ITS).
In a surprisingly prescient research note published when the contract was signed, Gartner analyst Christopher Ambrose described the deal as a "huge win for EDS", but added: "The company and its partners will find it difficult to make the implementation both successful and profitable."
He wasn't wrong. EDS had failed to appreciate the full scope of the contract, under-estimated the risks of non-compliance and was left picking up a bill for more than $500m when it was unable to fulfil its contractual obligations in full. Indeed, for once it wasn't taxpayers that ended up picking up the bill, but EDS.
Ten years on, the system is nevertheless used by 700,000 sailors, Marines and civvies, delivering everything from email to online training.
Hence, the deal is one of the rare examples in which the client ultimately ground-out a good deal from its outsourcing partner, and didn't foot the bill when the project started to go awry - which makes a change.
8) Cornwall Council and BT: £300m
Normally when an outsourcing relationship breaks down, both sides like to keep it hush-hush. But not out in the West Country where the 10-year, £300m outsourcing deal to BT never was fully accepted across the board at a political level when it was signed in 2013.
First, was the "extraordinary secrecy" with which the deal was struck, which came after a bigger deal was rejected following scrutiny in the Council debating chamber.
That's often not a problem after the ink has dried on the contract, but BT struggled to come close to some of the very modest benchmarks laid out in the contract, amplifying opposition among a hardcore of councillors.
BT's response didn't help: if you have to file an injunction to keep an outsourcing contract on the road it's fair to say that the relationship has already irretrievably broken down.
So, before you know it, the two sides were in court: the Council arguing that it shouldn't be required to continue a long outsourcing contract with a company that couldn't meet its targets, and BT arguing that a ten-year deal is a ten-year deal.
The High Court, though, found in favour of the Council, with Justice Knowles ruling that BT "faced problems of its own making" and hadn't provided "the service it had promised to the standard it had promised".
Ouch.
Next: The State of Texas and IBM; and, the UK Border Agency and Raytheon
Top 10 big IT outsourcing and consulting disasters
When outsourcing goes wrong, it can go very expensively wrong...
7) The State of Texas and IBM: $863m
If you thought BT and Cornwall was bad - not to mention all the other IT outsourcing fails in the UK public sector - then take a look at the US, where they do things bigger, better, and more disastrously.
One of the key selling points for the State of Texas with this $863m outsourcing deal was that, in the first two years, IBM was going to consolidate data centre operations for 27 State agencies into just two data centres.
By the time of the first contract suspension in 2008, IBM had migrated just five agencies' systems and was only working on another five. Worse still, the State revealed that IBM wasn't performing the nightly backups required under the contract - a charge that IBM admitted, promising to fix the problems.
But by 2010, the problems had grown, according to The Outsource Blog, and encompassed disaster recovery, staffing, security, service-level failures, asset management, system management and monitoring, efficient use of resources, procurement, capacity planning, project analysis and reporting, technology refreshes, and reports and forecasts - so more than just a couple of nit-picky little things.
Oh, and IBM still hadn't finished the data-centre migration that it had originally promised to do in the first two years and - surprise, surprise - costs had gone up. These higher costs were the result of the consumption-based pricing agreed between the two organisations back in 2006.
"IBM promised an investment in people, processes and technology to bring the benefits of data centre consolidation to the state of Texas. We have had continual problems with basic service delivery and IBM has failed to deliver on their promises," said State CIO Karen Robinson in a statement in 2010, released to coincide with an open letter outlining IBM's many failings.
By 2012, the State had had enough and IBM was out the door - only to be replaced by a combination of Xerox (yes, you read that correctly) and Capgemini in an eight-year $1.1bn deal.
IBM, for its part, claimed that stroppy State staff were to blame for the problems, although if they're causing problems for Capgemini and Xerox the two companies appear to be keeping a lid on it better than IBM did.
6) UK Border Agency and Raytheon: £1bn
Immigration is a perennial political issue, so you would think that the government would have developed a top-notch computer system to manage it, would you?
Well, you'd be wrong.
The idea of the e-borders project was to control illegal immigration after border exit checks were scrapped. As much as £1bn has been tipped into the near-bottomless pit of this project, conceived back in 2003, and finally cancelled in 2014.
US defence contractor Raytheon won the bidding to implement the programme and ran it from 2007 to 2010 without delivering too much of note or value.
However, the-then new home secretary Theresa May made the decision to terminate the contract with Raytheon when the coalition government was elected in 2010, not unjustifiably citing the "mess" she inherited due to "well-documented issues with the delivery and management of the programme", she said.
A backlog of applications was building up, 50,000 rejected asylum seekers were unaccounted for by the system, and the Public Accounts Committee damned it as a system in chaos.
May continued: "Key milestones had been missed and parts of the programme were running at least a year late. Raytheon Systems Ltd had been in breach of contract since 2009. Prolonged negotiations had taken place under the previous government, which had led nowhere."
That wasn't necessarily the whole story. With a politically charged subject such as immigration comes more acute than ever political and bureaucratic interference. At least, that was Raytheon's justification and it stuck to it.
Indeed, despite the manifold problems with the contract, the company came out smelling, not of roses, but of money: it was awarded £224m in compo for the incorrect termination of the contract following a process of binding arbitration in August 2014.
Next: BSkyB and EDS; and, The State of Indiana and IBM
Top 10 big IT outsourcing and consulting disasters
When outsourcing goes wrong, it can go very expensively wrong...
5) BSkyB and EDS: £318m
When an outsourcing contract ends up in court, it's fair to say that the relationship between vendor and client has irretrievably broken down. But possibly the only reason why more disputes don't end-up in court is not just the mutual fear of embarrassment, but also the sheer cost.
The BSkyB outsourcing disaster was one of the many gifts that EDS bequeathed to HP when the systems giant had splashed out $13.9bn to buy one of the world's biggest consulting companies in May 2008.
EDS, of course, became HP Services, then HPE Services and is now DXC Technologies - all in the space of less than a decade.
At issue was a failed £50m customer relationship management (CRM) system project announced by the satellite television company to the London Stock Exchange in February 2000.
This was supposed to integrate BSkyB's various customer databases and enable it to more easily up-sell, cross-sell, and offer self-service. And it was going to be "operational within 12 months".
The only trouble was, when it announced the project to shareholders, it hadn't even started it.
A bidding process was hastily convened and EDS emerged the lucky winner, with some of the work sub-contracted to third parties, including the consulting arm of Enron's excellent auditors, Arthur Andersen.
By the time the contract was signed in November 2000, however, it would've been impossible for EDS to meet the time-scales that the company had promised its shareholders.
"To implement the new system as rapidly as possible, and therefore make up some of the lost time, the project was to use a rapid application development (RAD) methodology.
"This is unlike the traditional waterfall development - which follows a strict sequence of events from definition of requirements, to design, to build and, finally, to test," according to law firm Linklaters, which analysed the case at the time.
"Instead, rapid application development allows these sequences to work in parallel and parts of the projects to be delivered in different phases. While this allows a faster delivery, it also entails greater risk.
"Furthermore, EDS only had an understanding of Sky's high-level business requirements. There was no functional specification or technical architecture design. These were to be developed post-contract," it continued.
The project ran into difficulties within six months, and an amendment was signed in July 2001 - but the problems of a compressed and highly parallelised implementation of a complex system remained.
"Arthur Andersen, to whom EDS had subcontracted the requirements work, and Sky were [still] unable to produce suitably detailed requirements.
"This meant EDS could not adequately develop the architectural design, which held up the detailed design work and ultimately compromised the development of deliverable code," according to Linklaters.
In March 2002, BSkyB threw out EDS and decided to do it itself.
By now, both sides were cheesed off with the whole process, to say the least, each blaming the other for the failure of the project and threatening legal action.
BSkyB, however, had the better lawyers. EDS had "failed properly to resource the project", wrote the judge overseeing the case, leading to delays. When BSkyB had re-asserted control, according to the judge, EDS had done "little work" and had "failed to exercise reasonable skill and care, or conform to good industry practice".
The high point of the court case came when EDS employee Joe Galloway was found to have not-entirely-kosher qualifications. BSkyB's brief, Mark Howard QC, demonstrated the point by brandishing in court similar qualifications from the same institution that he had acquired, by post, for his dog, Lulu*.
That pretty much sealed the case in favour of BSkyB, who trousered £270m in damages and a staggering £48m in costs. The bill accounted for one-fifth of all EDS's net income from services that year.
* Unfortunately, Mark Howard QC couldn't provide us with an image of Lulu, the cleverest dog in Britain
4) The State of Indiana and IBM: $1.6bn
If you thought the State of Texas-IBM outsourcing saga was bad enough, then try IBM's $1.6bn outsourcing deal with the State of Indiana for size: even bigger, even more disastrous and even more acrimony when it all fell apart.
The aim of the outsourcing deal was to help the State streamline its shonky social services systems, enabling people to claim online, in person or over the phone.
Unfortunately, error rates went through the roof, while applicants complained of long waits for their claims to be processed. The State blamed IBM for flaws in the system, while IBM blamed the spike in claims caused by the recession in 2008-09, and inadequate staffing levels to handle the claims.
Happily for the lawyers, the State decided to sue IBM for breach of contract. IBM, meanwhile, was equally miffed at being thrown overboard, and counter-sued.
The judge in the first case declared a pox on both houses. "Overall, both parties are to blame and Indiana's taxpayers are left as apparent losers," wrote Marion County Judge David Dreyer. He attributed the expensive disaster to "misguided government policy and over-zealous corporate ambition".
The case (and its appeals) continued putting the finest quality bread on the table for the lawyers of both sides well into 2016, when a judge ruled that IBM was in breach of contract, but still entitled to $50m in fees that the State had refused to pay. Equally, however, the State was entitled to seek up to $175m in damages, which was a long way short of the $1.3bn it was seeking.
Ultimately, therefore, only the lawyers ended up laughing all the way to the bank.
Next: The Child Support Agency and EDS; and, Queensland Health (Australia) and IBM
Top 10 big IT outsourcing and consulting disasters
When outsourcing goes wrong, it can go very expensively wrong...
3) Child Support Agency and EDS: £756m
The Child Support Agency was one of the most long-running public sector flops in the UK - and that's up against some pretty stiff opposition.
It was pretty good hunting for EDS, though. Indeed, the contract to build the IT that was supposed to support the Child Support Agency trousered EDS, as it was then called, a cool £456m. But the project was, according to the Public Accounts Committee of the House of Commons, "a turkey from day one".
The system was launched late, and was delivered with no less than 14 critical faults. This led to 250,000 non-compliance issues not being dealt with, meaning that £3.5bn in child support didn't get paid.
Throwing good money after bad - it was taxpayers' cash, after all - the government lobbed a further £300m at EDS in 2007 in the hope that either:
- All the faults would miraculously get fixed, which was highly unlikely; or,
- It would go away, at least until the other side of the next election.
It did neither, with staff completely unable to access case files because the system - for reasons best known to what had then become known as HP Services - had been reduced to a crawl.
The Work and Pensions Committee, in February 2010, said that some progress had been made, but that there remained a very large number of IT problems which had "no workaround and are causing cases to get stuck".
In EDS's defence, it's probably fair to say that meddling by politicians didn't help (does it ever?) with the Department for Work and Pensions (DWP) deciding to reform the CSA at the same time that EDS was supposed to be delivering the system.
Edward Leigh, the-then chairman of the Public Accounts Committee, summed up the carnival of incompetence:
"Ignoring ample warnings, the DWP, the CSA and IT contractor EDS introduced a large, complex IT system at the same time as restructuring the agency. The new system was brought in and, as night follows day, stumbled and now has enormous operational difficulties."
2) Queensland Health, Australia and IBM: AU$1.2bn
How much should it cost to implement a simple payroll system?
For a company as large as IBM, it's a surprise it even bothered to bid for what started out as a modest AU$6m contract with the Queensland Health in 2007 to implement SAP and Workbrain under the State's 2002 Shared Services Initiative.
What's even more surprising, though, is that it ended up costing taxpayers AU$1.2bn before the lackadaisical civil servants and politicians decided to pull the plug - an overspend of 16,000 per cent. Dream clients, or what?
The project was supposed to be done and dusted by mid-2008.
But before the mayor or equivalent dignitary could even think about ordering the red ribbon to declare the new system ‘open', IBM confessed that it had uncovered a number of "unforeseen" technical challenges, pushing the cost up more than four times to $27m and then, by the end of the year, to AU$180m.
That didn't set alarm bells ringing loudly enough and so, from there, it seems as if it was open season on what could arguably be described as the biggest rubes in the world. For execs at IBM in Australia, it must have seemed like all their bonuses had come at once.
The project had been intended to migrate payroll from an obsolete system that couldn't even perform many payroll functions without manual intervention. But what with one thing or another, IBM managed to drag the project out until September 2010 before the State finally terminated the agreement in a peculiar deal that miraculously released IBM from all potential claims for compensation.
The official report into the disaster found an impressive combination of incompetence and mendacity all round.
Lovers of The Simpsons, though, will be tickled to learn that one of the characters central to the unfolding disaster went by the name of Mr Burns (pictured, above).
Brought in to aid the State in its selection process, Mr Burns proved to be highly partial to IBM's bid, according to Richard Chesterman QC, the lawyer brought-in to examine the whole disaster.
"Mr Burns had a relevant and impressive CV. He was, however, unknown to anyone in government, new to the country... and the Shared Services Initiative," wrote Chesterman in his riveting 264-page report.
IBM was subsequently banned by the State, which finally found the cojones to file suit against IBM, regardless of the earlier agreement not to - and lost, being forced to pay IBM's legal costs in the process.
Next: National Programme for IT/NHS Connecting for Health
Top 10 big IT outsourcing and consulting disasters
When outsourcing goes wrong, it can go very expensively wrong...
1) National Programme for IT/NHS Connecting for Health: £12.7bn
Really big, multi-billion pound IT disasters can't be achieved alone. They require years of effort, political commitment, dedication, lots of money, of course, and genuine teamwork.
And the National Programme for IT really is up there with the world's biggest and most expensive IT consulting and outsourcing disasters: a grand, politically led initiative that would transform the management of the sacred NHS with the magic of modern IT. What could possibly go wrong?
The idea of the Programme was to build a single, UK-wide centrally-mandated electronic care record for patients, which would connect 30,000 local GP practices with 300 hospitals and all healthcare in between. How hard could that be?
As it turned out, very hard (and expensive) indeed.
The project was split into one ‘spine' and five clusters, with most of the development outsourced to various big-name services companies.
Patients would be able to access their records online, using the new, fangled internet (this was the early 2000s) via what was billed as "the world's biggest civil information-technology programme", to quote the Programme's Cholmondley-Warner-ish pitch.
Obviously, such an ambitious idea couldn't be run by a civil servant or one of the NHS's many managers, so some top-class British management consulting talent was brought in to oversee the whole project in the form of ex-Accenture man Richard Granger.
Appointed Director General of Information at the National Health Service in 2002, the whole IT system was budgeted at £12.7bn, and a bunch of IT consulting firms were brought in to work on various different parts of the Programme.
By 2009, however, the Programme had provided only "little clinical functionality", according to the Public Accounts Committee. It was also slammed for inadequate security and patient privacy, which were treated as afterthoughts. Indeed, the only awards the Programme ever won were two Big Brother Awards from Privacy International.
The NHS claims that the NPfIT provided a foundation for IT in the NHS, including the Spine, N3 Network, NHSmail, Choose and Book, Secondary Uses Service and Picture Archiving and Communications Service, but for £12.7bn taxpayers have a right to expect a lot more than that. Roll-out was patchy and the hospitals that did implement it blamed it for a series of failures.
In the meantime, many local surgeries still have their patient record databases running on the same Microsoft DOS systems that they were using way back in 2002 when the Programme was conceived. As IT consulting and outsourcing disasters go, that's quite an impressive failure.
Granger, though, trousered a reported £280,000 per year for heading up the Programme, although his credentials for the role were questioned by his own mother, who described the scheme as "a gross waste of money".
She also revealed that her son had initially failed his computer studies degree at Bristol University, and was only allowed to resit his final exams following an appeal to Princess Anne.
Computing's IT Leaders Forum 2017 is coming on 24 May 2017.
The theme this year is "Going Digital: Why your most difficult customer is your best friend".
Attendence is free, but strictly limited to IT Leaders. To find out more and to apply for your place, check out the IT Leaders Forum website.