How DevOps allows a 14-person team to manage one of the world's largest e-commerce sites

'If you're doing something more than twice a day automate it' says MercadoLibre while Pivotal explains why it is open-sourcing all its software

MercadoLibre is often referred to as "Latin America's Ebay". The eighth-largest e-commerce platform in the world, it operates in 14 countries and has 115 million users. It also runs the regions first and largest private cloud. Based on the OpenStack cloud platform since 2011, this cloud operates across three data centres on a total of 2,000 physical servers and 15,000 virtual machines.

These are big numbers, but astonishingly the infrastructure is managed by a very small team.

"We have 14 people running the whole infrastructure for the MercadoLibre family of sites," says Max Tkach, technical leader cloud services and operations.

"We have a really small team because we automate as much as we can," adds Dario Nievas, cloud infrastructure technical leader, sharing his rule of thumb: "If you're doing something more than twice a day, automate it."

As well as being responsible for the infrastructure, their team functions like an "in-house Amazon", Nievas continues, providing cloud services for MercadoLibre's team of developers to enable them to make calls to the OpenStack API and to provision resources for themselves.

A DevOps approach to is used to break down functional silos, explains Tkach. While team members have core responsibilities such as storage, networking and operations, everyone is expected to have skills in other areas too.

"For us DevOps is the way to go. You basically turn yourself into a one-man army. You can handle practically any situation you have to deal with, and this makes for a more adaptable, flexible and dynamic team," he says, adding that this frees up Nievas and himself to focus more on research and development.

DevOps is also much more fun, he says.

"You can hop from project to project, and adapt yourself to the needs. You know where to find resources such as the specifications or drivers for a piece of hardware, or a new language or technology."

However, there are some disadvantages to the approach, including finding qualified staff.

"It's quite a wide role," says Nievas. "It's hard to find some who's a good fit because you need to know something about development, infrastructure, sysadmin and other things. On the storage and networking side many people are used to managing appliances using a web UI; they seldom use code or APIs, and they are not used to thinking of infrastructure as code."

"You need experience of being being hands-on but you also need the theoretical side, you need to read a lot, get a good theoretical knowledge, and that's hard with the pace of change in technology," adds Tkach.

DevOps and open source advancing together

The DevOps methodology would be almost impossible to deliver with the proprietary model, and Nievas credits the company's decision choose open source software wherever possible for the success of the approach at MercadoLibre.

"Open source and the DevOps role are growing up together," he says.

"Software and networking vendors are standardising the way they expose services, thanks to projects like OpenStack. If you use the Cinder block storage service for OpenStack you can have multiple storage vendors, which is very cost-effective, with all the different drivers being transparently available through Cinder as a single service."

This means that the storage guys can start "programming and automating things", Nievas goes on: the DevOps approach, in other words.

"We love open source. The open source philosophy is in our DNA, co-operating to make things better by creating communities and joining forces, being able to give something back," says Tkach.

However, there are caveats: "You have to be clever if you choose open source solutions. They sometimes fail or get replaced so you need to be able to cope with that."

That said, with infrastructure on the scale and complexity of MercadoLibre's, hardly anything - be it open source or proprietary - tends to work "out of the box", he explains, and at least open source allows knowledgeable people to adapt the software to make it fit.

As an example he mentions Galera Cluster, the open source synchronous clustering software for MySQL developed by Codership that his team uses to ensure high availability of all the databases held on OpenStack, including "all the metadata used to run our sites".

"We are not the database guys so we needed a robust, reliable high availability solution on the database side," he says. Given the complexity he asked Codership to perform an initial consultancy task to help tune Galera to the firm's architecture. Since then he has rarely had to call on them, but if he needs further assistance he knows there is always a "huge community" there to help. "We are very happy with the support we've received," he says.

A vendor's view

"We're making all our products open source," says Leo Spiegel, senior vice president of strategy and corporate development at EMC/VMware spin-off Pivotal, mentioning that this comprises a total investment in acquisitions and intellectual property of "more than a billion dollars" in big data database products such as Greenplum, HAWQ and GemFire.

He continues: "Open source is the new standard in the enterprise. The old standards bodies such as IEEE are not what they used to be, really I believe they been supplanted by open source. Enterprises are demanding that technology companies support open source. And it's almost becoming table stakes if you're a software vendor."

As well as enabling the firm's software to run natively on other vendor's platforms, something he insists Pivotal's customers are demanding, it also makes good business sense, Spiegel says.

"There's this perception that open source really means free, but in the enterprise there are very few that download the free version and just use that."

However, like many other open-source firms, Pivotal has yet to turn a profit. But Spiegel says revenues and usage are up and it's just a matter of time. Referring to Hadoop vendor Hortonworks, with whom Pivotal, IBM and other vendors have created a standard Hadoop core called Open Data Platform (ODP), Spiegel says:

"We are losing money, they're losing money, many companies are losing money but their revenues are growing beautifully and they've taken the company public. Wall Street believes is a rational business model and that's the point."

The other big change is cloud, allowing the subscription model to become more viable.

"The cloud has changed the economics and the architecture of computing forever," he says.

"The 'Oracle model' traditionally has been to sell you a big perpetual license with a 20 per cent maintenance contract. So that perpetual license becomes revenue the moment you sign the contract. In a subscription model you're basically taking a percent of that every year over many years and so it lowers your revenues, but interestingly I believe that's a more valuable business because Wall Street really likes predictable business models.

"The problem with the perpetual model is that every quarter you're out hunting for elephants. With a subscription model, they can look at ARR [annual recurring revenue] which is a new important term in the subscription world, and they can get a good sense of where revenues are going to be two or three years from now. It's all about predictability."

Pivotal is also the company behind the Cloud Foundry collective and is the driving force for ODP. More about this here.