The power of open source: From Postgres to Cassandra at First Utility

First Utility CIO Bill Wilkins talks to Graeme Burton about how open-source technology helped the company become the biggest independent energy utility in the UK

When Bill Wilkins joined energy provider First Utility, he faced a peculiar IT challenge based on architectural decisions that had been made soon after the company was founded - and which were now holding it back.

Although its provisioning system - one of the most important elements of any gas and electricity company - was technically sophisticated, it was unable to handle more than about 200 concurrent new customers at any one time. Furthermore, because provisioning could take 28 days that limitation severely crimped the company's growth potential.

"There were two basic problems. The first was that the provisioning system wasn't scalable, so we could only grow at a fairly rudimentary pace. This was because of the way it had been built, and the complexities that the implementation choice had created. It was built using a business process engine called Yet Another Workflow Language (YAWL), which had been developed at Queensland University," says Wilkins.

He continues: "That provisioning system, although it was functional and automated a lot of the on-boarding 'journey', just didn't scale. At the time it was quite innovative in its own right because it was so highly automated, it meant we didn't need to hire lots of people to send EDI messages off to various parties when we acquired a customer, because a lot of that was automated.

"We required a server with 64GB of memory just to provision 200 customers. It was crazy because these processes last many weeks and all of those process dates were held in memory. It wasn't a very scalable architecture because it was based around this open-source workflow engine.

"It had hard disk backup for the memory state, but didn't take advantage of that in terms of freeing up RAM... So everything lived in memory, as well as on disk."

To-do list

Updating the provisioning system was one of four key items Wilkins identified when he was brought in as an interim CIO with a brief to examine the organisation's IT and evaluate its proposed technology roadmap.

At the time, he says, the plan was to shift software and hardware over to a single platform from a single software, hardware and services vendor, all backed on to an expensive commercial database - with the bill no doubt picked up by First Utility's backers, who would have had to dig deep in order to pay for it all.

But Wilkins, who had spent most of his career at Sun Microsystems, had other ideas. "I came in from Sun, where we'd had a much more open attitude towards open-source technology. I looked at what they were doing and I said: you can use a free database [and] a free app server," says Wilkins, who proposed using Postgres instead.

"I had to convince people [the First Utility board and its backers] that there was nothing that First Utility was going to encounter in the next three-to-five years that was atypical of any organisation that was using open source: there were big players out there using open-source technology to far higher levels of scale than First Utility would ever be likely to over the next three-to-five years, so it was a fairly safe bet.

"And I had to remove that fear, uncertainty and doubt from the executive team members and the board team that it was a risky strategy, because commercially it would be a much better strategy. We would be removing a huge amount of 'corporate taxes' from a relatively small start-up and, therefore, we could invest that money in building out your services and technology function," says Wilkins.

Then, having persuaded them to take a very different - but much cheaper - technology approach, the company turned round and asked Wilkins to stay on and implement it.

"I'd never worked in a high-scale consumer business before, so it was a new challenge," says Wilkins.

The power of open source: From Postgres to Cassandra at First Utility

First Utility CIO Bill Wilkins talks to Graeme Burton about how open-source technology helped the company become the biggest independent energy utility in the UK

Here to stay

The first project, the provisioning system, took the best part of a year to implement and phase in - a botched provisioning system would have pole-axed the company, so Wilkins had to move cautiously. Written in Java and backed on to a Postgres SQL database, without it the company would struggle to grow to any significant mass.

"The first project we kicked off when I joined was building a new provisioning system. It would be event-based and based around a 'restful' model, which removed the barriers to scale.

"In the old system, you had states about customers, their meter technology details and so on living in memory on the server for 28 days, assuming that they were all successfully completed, which they never did. The big engineering benefit of the new system is that it didn't manage that state in memory, it was event-based, so every time an event came in, it looked up all the customer details and worked out what it had to do only at that point, do it, and all the process state was released at that point.

"If I had to point at one of the key design characteristics that enabled it to scale it was that: remove state. Well, you can't remove state, but don't create more states in your workflow engines, use the states in your transactional systems. The life-span of the transactional boundary was around an event, rather than around a two- or three-week process."

Stuck with the old workflow-based system, First Utility had struggled to grow with sufficient pace, even though the system was highly automated.

"They could sell, because that's just a function of lowering your price point [but] when that kind of thing happened in the old days, we struggled to keep the service level to those customers high enough because the provisioning engine couldn't cope with the on-boarding volumes," says Wilkins.

In other words, one sought-after recommendation from Martin Lewis, the self-styled "money-saving expert" could overload the system.

"During the last few years since the new system has been on, we've never had a problem in terms of on-boarding performance. At one point, about 18 months ago, we took around 40 per cent of the market for new customers - of all the switches across the UK, we took just under half of them - and our systems had no problem with that. That's because we were using modern approaches to integration, and it doesn't involve workflow, it's event-based integration," he adds.

Billing

While no utility can even exist without a good provisioning system, nor can they last long - especially in gas and electricity - without a robust billing system.

"The most complex thing in an energy business, certainly in the UK market, and the thing that most CIOs in energy companies are concerned with, are the billing systems," says Wilkins.

The billing systems aren't just about taking money from customers, but matching it with genuine usage by customers, on the one hand, while keeping track of the gas and electricity that the utility is purchasing from the market on the other - while, as a company, making sure all that is done profitably and consistent with positive cashflow.

And, while the billing system that Wilkins inherited (the third that the company had built in two years) when he joined was functional, it wasn't geared to handle the kind of growth that the company needed. Indeed, for certain customer-service functions, requests had to be forwarded to IT for SQL queries to be made direct to the database because parts of the user interface hadn't been completed - which was far from ideal.

"The billing engine was home-grown and, because it was bare-bones, it had two problems: the first one was that it could only support a sub-set of the profiles that energy consumers have in the UK. When you're an energy consumer, it is the meter attached to your house that supplies your energy, particularly on the electricity side, and dictates how the industry sees you as a 'profile'. Essentially, there are one-rate, two-rate and three-rate.

"But the billing engine at the time could only deal with single-rate customers. So not only were we limited in terms of how fast we could scale, but we were also limited in terms of the market we could take," says Wilkins.

The power of open source: From Postgres to Cassandra at First Utility

First Utility CIO Bill Wilkins talks to Graeme Burton about how open-source technology helped the company become the biggest independent energy utility in the UK

The trouble with energy market billing, he adds, is that there aren't many software companies with much experience of the UK market - just SAP. "We found a company in Israel, called LogNet Systems, that actually had a multi-product billing engine and could prove that it could do anything from video-on-demand to mobile and road tolling all in one billing system.

"They had some experience in utilities in Portugal and other European countries. So we partnered with them, took their core engine, called MaxBill, and essentially jointly worked on the UK energy component," says Wilkins.

"It can support smart meters and traditional consumers. We bill all of our customers monthly so, from an industry perspective, it's quite advanced. It's the component on which we've recently re-platformed all of the management side of it on to Cassandra. So, again, we're probably the only energy company in the UK using a NoSQL database approach to handle a mission-critical transactional system," says Wilkins.

The reason?

"Performance and scalability. It's raw performance and scalability that we were looking for. We have now realised the level of performance that we couldn't achieve through the traditional design on a relational database.

"I'm not saying it couldn't be achieved by some very bright Oracle technicians but, for us, moving the database technology gave us a significant increase in performance because at the same time we were able to re-model all of the data structures in it to work with that NoSQL approach.

"It's obviously got a clustered capability so we can throw more processing nodes into that cluster as we grow, so we feel like we've de-risked our future scalability as part of that," says Wilkins.

He adds: "We only completed that upgrade in the past month. There were a few butt-clenching moments, but it's running and we're seeing the performance gains that we wanted."

Technical challenge

Much of the technical challenge in upgrading such core systems wasn't necessarily finding staff skilled in Java, Postgres or even billing systems, but in ensuring that the newly hired engineering team understood the energy utility industry, and its peculiarities and idiosyncrasies.

"The energy industry has a reasonable amount of complexity; it's quite unique to the UK, so therefore the domain of people that understand it is very small. So when we were building that system, we had to hire new people with new technology skills - Java, the Spring Framework - but also teach them about a relatively complicated and unusual industry," says Wilkins.

"It took about a year before we were willing to switch it on and make it the main provisioning system. We obviously went through a period of transition where we parallel-ran both of them."

Unusually, perhaps, for a relatively young company handling sometimes unpredictable volumes of new customers, First Utility has largely eschewed cloud computing for supporting its core systems. Where First Utility has gone cloud is primarily with Salesforce.com for customer relationship management (CRM) and New Voice Media instead of splashing out on a PBX and the specialist staff - or service contract - to run it.

The success of Wilkins' tech strategy at First Utility is demonstrated by the fact that the company is not only still here, but still growing, having surmounted many of the hurdles that have confounded other independent energy utilities.

"There are various scale thresholds. Some of them are real, tangible thresholds where government regulations mean that once you have crossed a certain number of consumers, more regulation applies to you, more operating taxes apply to you.

"Then there are more myth-type barriers. At the time, there was no independent provider that had gone beyond 50,000 customers and survived," says Wilkins.

Today, though, the company supplies some 800,000 households, competing against the giants such as British Gas in a market that is driven by a combination of price and customer inertia - customers who can't be bothered to switch even though it could be in their best financial interests.

However, First Utility may not be stopping at energy - there is, no doubt, a good reason why the company selected a billing system that could not only handle gas and electricity in the UK's idiosyncratic energy market, but much more besides.