Interview: Bet365.com chief technology officer Martin Davies

Online gambling specialist Bet365.com employs up to 160 in-house software developers at its Stoke-on-Trent headquarters at any one time, as well as 300 IT staff to keep its web systems up and running.

With a growing customer base that can see up to one million people connected to a site, generating roughly 2,000 updates per second, the sheer volume of web traffic and transactions places huge pressures on the systems that support it, and Bet365.com has to stay at the leading edge of web software development to handle them.

Computing: What's unique about the Bet365.com IT set-up?

Davies: Bet365.com is very much a technology company and one that is doing something very difficult in terms of scale that not many others are.

The company is very open to new technology and we are trying to give our software developers the ability to start solving problems by approaching things in different ways. Traditional technologies such as relational databases are starting to show problems with systems on the scale of ours, so we are looking at NoSQL custom written systems that we develop ourselves and try to innovate rather than use what is already available.

What systems do you use currently and how have they evolved to give the punters faster access?

We have multiple systems presenting to the internet, the largest of which are the advertising platform and banner systems, which use the same live feed at the in-play systems [which allow punters to place bets during live sporting events] updating odds in real time with around a two second look-up latency.

We initially used an Ajax platform to get snapshots of the data but as we grew the in-play parts of the site we found we needed more scalability on the back end, which was having to support a larger amount of load.

So we put in a custom caching layer that we developed ourselves – there was no other technology we could exploit to do the same thing, it’s all our own mechanism and we don’t use memcache or anything like that.

Then we started looking at alternative ways of delivering content using Comet [a web application model that allows web servers to push data out to browsers].

We effectively have the same high volume of data replication going across multiple database servers, which we push into a virtualised distribution system that sends it out to all the clients. This runs on about 200 VMware appliances that sit in our main datacentre in London.

We don’t outsource any of our IT – all the infrastructure we have we have built ourselves, including designing and building our own network.

How do you deliver all that data to online gamblers efficiently and with minimum time lapse?

We use about 18 ISPs to minimise the risk of downtime, but also to make sure we deliver content to our customer base in 240 countries as efficiently as possible; having upstream peering is vital for us.

We don’t use a content delivery network (CDN) but try to route the data more effectively to get the data to the end-user as quickly as possible by sending it out in small packets from the same location – you cannot use a CDN to do that.

How much money would you lose if your datacentre went down?

Business continuity is a very big part of what we are looking at strategically now, and we are bringing another datacentre on line in Manchester built around the Cisco Nexus platform.

The infrastructure is growing every week and we will try to deploy stuff into a virtual environment where we can, though that is not always possible – some systems are worked so hard that there is no advantage in virtualising them because of the CPU load and performance.

We are still in the early stages of getting the first systems live at the moment and have some disaster recovery systems in place, but the aim is that in a couple of years we will be doing bi-directional replication so that if one datacentre goes down we can switch to the other.

Where do most faults tend to occur?

Historically we have had good network infrastructure based across multiple sites and high speed redundant communications in place as well.

The area where we see most issues for performance is in the back end databases because our transactional throughput is growing all the time and scaling them is more difficult.

We are now looking at alternative ways of delivering data where we will probably use traditional relational databases but in a slightly different way.

We are using Gemfire [SQLFabric] to help us with the data replication stuff and in some areas where there is high throughput, like the banner ad system. Basically, all the business logic sits within a Gemfire cache and that takes an enormous load off the relational databases.

Do you struggle to find developers with appropriate skills?

We struggle to find experienced people with systems middleware and Java skills, especially those with experience of latency systems, particularly because we are based in Stoke-on-Trent and those people tend not to be based in the Midlands area.

We are primarily a Microsoft house though, using the .Net software stacks, SQL Server and C++ in some areas, as well as open source tools on the Java side, particularly where we have virtualised systems and are using different flavours of Linux.