Software flies off the production line as Orbitz adopts DevOps and Jenkins

Travel giant cuts release times 75 per cent by moving towards continuous delivery and shifting some Ops staff to development

Travel company Orbitz Worldwide, which is currently in the midst of a takeover by rival Expedia, is perhaps best known in the UK for its online subsidiaries ebookers, HotelClub and CheapTickets. The firm also sells travel software to airlines and agencies, loyalty schemes to banks and offers managed travel services for businesses.

Underpinning this diverse set of businesses are 180 different applications that are maintained by 350 developers in 35 teams around the world.

In order to ensure that these applications are all kept up to date, Orbitz uses continuous integration (CI), whereby developers merge their individual copies of the code in a central repository several times a day. Recently, however, the firm decided it needed to move to the continuous delivery (CD) methodology, which has the goal of ensuring that the code is always ready for release. This allows minor updates and security fixes to be made quickly and pushed to the user, reducing the need for time-consuming, inefficient and sometimes unreliable patching.

Jacob Tomaw, senior principal engineer at Orbitz Worldwide, explained that the decision to move towards CD was prompted by the fact the testing process had become burdensome, taking up much valuable time that could otherwise be used to innovate new features and put them into production.

"Our feedback loop on innovation was about 18 days, and most of that was consumed by manual tests and other checks that we had put in place in reaction to previous incidents," he said.

In moving towards a CD approach, Orbitz has combined its CI and application release systems on a single platform. With everything in one place much of the testing process can be automated and consistency improved.

The open-source orchestration and automation software delivery suite Jenkins has been used for CI for a few years, but with the move towards CD the company has adopted the CloudBees Jenkins Enterprise package to improve the production pipeline. This includes role-based access control and high availability plug-ins, templates to standardise builds, as well as features allowing pre-production snapshots to be automatically deployed to the quality assurance environment.

Already the changes have borne fruit with release cycles cut by more than 75 per cent from the 18 days that were required to test and deploy new features to just a few now.

"We have cut our release times in half, and then cut them in half again," said Tomaw, adding: "We now ensure that each one of the artifacts we build for our applications is produced consistently and in a repeatable way, a capability that is essential for CD."

The next stage will be to allow the developers to assume complete control of their software, from coding to release.

"We're seeking to have the systems in place that will allow our development teams to deploy their own software and be fully responsible for the outcome," Tomaw told Computing.

"Right now our service operation centre [SOC] does the actual deployment, which is mostly just pushing a button and monitoring the outcome."

Handing this responsibility over to the developers "will enable teams to adopt the best practices and cadences that work for them", he said.

These changes have meant structural alterations to IT teams in line with the DevOps approach. At the operational level some staff have seen their roles absorbed into the development function. Tomaw omitted to mention whether any redundancies have been necessary, but he did say that some Ops positions have been affected more than others.

"The changes are most pronounced with the teams closest to the development teams like our SOC, site reliability engineering team and system engineering. They are changing the most," Tomaw explained.

"Tasks have been taken from them to be owned by development teams; this sometimes leads to their staff taking roles on development teams."

Other Ops staff, such as those working in network engineering or the newer cloud platform team, have seen fewer changes to their roles.

To smooth the process by which operations staff become more involved with the code, Tomaw said he would like to see improvements to Jenkins in the shape of better reporting tools and also usability features to lower the learning curve.

"Jenkins is clearly a developer tool and lacks a lot of polish that can lower the barrier of entry and improve a novice's ability to use the tool as an expert," he said. "Much of this lack of polish comes from the great liberty that plug-ins have to change the tool.

"As far as functionality, I would like to see some kind of failure analysis that would classify similar failures and perhaps predict failure. Also cloud-based executers are going to become much more important as the industry moves away from on-prem computing."

Moving to the new ways of working has inevitably thrown up a few problems on the way, but Tomaw said that the process of dealing with them has been a positive in itself.

"These issues are the healthy cost of delivering higher value sooner to the business. We have encountered many, from poor development practices to hidden operations changes," he said.

"I had someone explain this using the analogy of lowering the water level needed to navigate a channel. As the water goes down there are stones that emerge that were not a problem before. You have to prioritise the stones and clear the channel. Eventually we might be dredging silt, but we are still finding a few stones."