Comic Relief replaces 'knocked together' payment services

Charity engages Braintree to move away from previous payment system which was in danger of failing and losing millions of pounds of donations, admits CTO

Comic Relief has raised over a billion pounds for a variety of charities over its 30-year existence. A substantial part of its annual donations are given over a few weeks of intense campaigning every year, culminating in red nose day, and sport relief.

And until fairly recently, those donations were processed over a creaking, jury-rigged platform which was seriously in danger of collapsing.

The man tasked with bringing the system into the 21st century is Comic Relief's chief technology officer Zenon Hannick. He tells Computing how the previous makeshift system came together.

"For the 2011 red nose day we had an old Java-based platform which was just knocked together," says Hannick. "Various companies had leant us servers, we had them shipped to our data centre in London, then about a dozen of our partners came together and assembled it all. It was a proper snowflake."

Comic Relief had on average been doubling the amount of donations it received in every campaign at that point, and 2011 was no exception. The volume of payments they received came very close to breaking the platform and flooding the charity's internal network.

"I knew that if we experienced the same increase in 2012 we basically wouldn't be able to take the money," he admits.

So Comic Relief decided to rebuild its platform, aiming for multiple cloud services and redundancy at every layer, including at both the cloud provider and service provider levels.

"We'd been with WorldPay for around ten years at that point," explains Hannick. "And we wanted to look around the industry. We ended up approaching 25 to 30 payment providers, but many of them said they couldn't cope with the level of donations. We need to process around 500 donations per second in real-time."

Being such a high-profile event, many providers didn't want to take the risk of experiencing a failure and making the wrong sort of headlines the next day. Further complicating requirements, Hannick also wanted donations processed synchronously all the way to the banks and back, so givers would know that their attempts had succeeded or failed.

"On the old platform someone would enter their credit card details, and that would be held in the platform then dripped through to the service provider, which could take up to eight hours to reach us," says Hannick.

That was a particular problem for Comic Relief, since the BBC expected to be able to announce live totals regularly throughout its various event broadcasts.

"We have to tell the BBC the totals we're raising, and we used to do it based on money we didn't quite have yet, so we could have been in breach of compliance. Some providers said they could work with that requirement, so we did a lot of testing with them."

[Turn to page 2]

Comic Relief replaces 'knocked together' payment services

Charity engages Braintree to move away from previous payment system which was in danger of failing and losing millions of pounds of donations, admits CTO

Originally the charity purely used WorldPay, but examined the payment provider market each year to see how the situation had evolved, and in particular if any new companies had sprung up with new capabilities. Eventually they found Braintree, whose payment platform Comic Relief has now integrated and uses actively.

"The great thing about Braintree is the implementation and integration was extremely simple," explained Hannick. "They're a younger company, so they have no legacy problems and can move at a rapid pace."

He adds that being essentially a very ‘bursty' business, with activity very much centred around the annual televised events, Comic Relief likes to regularly load test on its platforms, with automatic tests carried out every night in some cases.

"But for many payment providers that just wasn't possible," Hannick explains. "Many of them had low capacity test platforms which just couldn't carry the load, but Braintree was much more flexible."

He identifies what he terms a sea change in the payment services industry, with a flood of new companies appearing with high-performance platforms and well-functioning APIs all able to move far more rapidly than previous incumbents.

"With these new payment mechanisms appearing, we know that Braintree will be at the forefront, integrating new technology as it starts to gain traction, and keeping us informed," says Hannick.

But there are risks as well as benefits when choosing a young, innovative company. Was Hannick ever concerned about trusting a vast wealth of financial transactions to a young firm without a long track record of success and list of satisfied blue chip clients?

"We gained confidence from their technical ability and clear communication," he responds. "When you're a new company and you don't have years of reputation, you prove yourself by being better. Their knowledge, and the way they backed it up with technical know-how was what convinced us."

And as for the security risks, Hannick explains that they too were minimal. Each year NCC Group provides penetration testing services, and tries to hack the charity's various platforms, including its payment services.

"They found no critical vulnerabilities in the donation application," he says. "They found some small Javascript issues, and recommended improvements, but that was all."

Comic Relief's Red Nose Day is set to return in March 2017.