Mercedes-Benz Tech Innovation: From 'open source not allowed' to 'FOSS preferred'

It's good PR for large organisations to send teams of developers to present at conferences, but things are not always as they seem. Sometimes these are unrepresentative internal startups, experimenting at the cutting edge while the vast majority of their colleagues carry on as normal.

This is decidedly not the case for Mercedes-Benz Tech Innovation, a 1,200-strong fully owned subsidiary of the German car giant that develops its software - but it was a few years back. At that time, said lead expert on Peter Mueller, speaking at the recent KubeCon & CloudNativeCon Europe event, the organisation was "optimised for cost and not for speed, for governance and not for developer experience or innovation."

In practice, this meant that ordering a new firewall took weeks, and that developers were tightly restricted as to the software tools, frameworks and middleware they were permitted to use, most of it closed source. Open source was forbidden unless explicitly allowed.

This was not unusual back in 2014, but Mercedes-Benz Tech Innovation had an advantage over many in that the vast majority of its employees are engineers, and most of the management come from the same background. They understood the frustrations and the bottlenecks in the software pipeline, and so when small teams started tracking what the likes of Google were doing with open source tools and DevOps and suggested they could follow suit, there was less resistance than might otherwise have been the case.

"The management understand tech, and they said, 'Hey, cool, we support you'," explained team lead Joerg Schueler, who, like Mueller, works on the platforms side. "And that was very helpful. The combination of good technical engineers and a good supporting management is key."

Stuck in the slow lane

In 2014 Mercedes-Benz Tech Innovation (then called Daimler TSS) was stuck with the classic dev and ops divide. Application teams complained that they were being asked to deliver features to impossible deadlines, which often failed to run because of deployment problems. Meanwhile, the ops teams, who were having to keep several plates spinning at once, protested that each deployment was different. Plus, the manual ticketing system was a pain for everyone.

"We had this configuration drift and the ops team often did not know in detail what they were doing because they were responsible for several projects. If something went wrong, or if we did not describe everything they had to do, then there were problems," Mueller, told Computing.

Mercedes-Benz Tech Innovation operates a system of competence groups. It formed a development and operations competence group to talk through the problems.

"We came together regularly to talk about our pain, So the dev teams and the ops team together, and also some evangelists who were reading all the new stuff and who go to conferences," said Mueller, adding that these evangelists would often take members of the devs and ops teams with them.

They were particularly enthused with what was happening in open source at Google and Pivotal, he explained.

"Google were also talking about what they are doing with SRE best practice, and there was Pivotal with their 12 factor apps and how to be cloud native we really got inspiration from them. Also Red Hat with OpenShift and the whole Docker movement, of course."

"Our software engineers and the software engineers from for example Google and others have similar goals, a similar mindset," added Schueler.

At the time, the microservices architecture was becoming popular as a way of managing tasks between teams, and Kubernetes, which began life at Google, was on version 0.9 and still rough around the edges. Nevertheless, they decided to go ahead and set up a single shared Kubernetes cluster with a CI/CD pipeline, and immediately saw its potential for increasing release velocity.

However, they quickly realised that the single shared cluster would not fit their needs, and because Kubernetes fleet management solutions were virtually non-existent at the time, they built out the platform basing it on OpenStack private cloud and "100% free and open source software".

Despite the fact that the company consists mainly of software engineers, there was some confusion at first with the new approach. Developers continued to turn up with an image and ask the platform teams to run it for them on the new stack, only to be told that self-service was now the ethos, and they had to deploy themselves.

"One of the absolute mantras in the early days was, 'you build it, you run it for yourselves. We will not operate it for them'," Mueller explained. "And if they're not fine with that, they cannot go to our platform."

To overcome such hurdles, the team added education to its role.

"They were not resistant, they just were not that far on the road to cloud native, they had a different maturity level," Mueller went on. "So they had to learn. We had community channels where we helped them, so we were doing much more than just the basic platform support, we were also coaching them."

Both sides of the dev and ops divide could see the potential for faster delivery with fewer headaches. They were aware of DevOps, having been introduced to the concept at conferences, and were keen to push ahead, said Schueler, although this has meant some changes to their roles.

"The platform teams develop the platform and operate the platform on their own and are responsible for being on call. And the application teams have the responsibility to develop applications and also to operate them, and to go on call when something's broken. That was one of the biggest mind shifts."

Into the fast lane

From small, experimental beginnings, the engineers' early experiments with open source, microservices, containers, cloud native and Kubernetes have completely transformed the way software is produced at Mercedes-Benz Tech Innovation.

There are now five platform teams running more than 900 Kubernetes clusters, serving their customers at Mercedes-Benz facilities worldwide. Mueller and Schueler said they don't know much about all the software that's tested and deployed on the platform they've helped to develop, because it's entirely self-service, and they only get involved if something goes wrong.

Over the years that have added additional tools to the platform such as logging, monitoring and databases to "reduce the cognitive load on the DevOps teams", and the applications teams around the world tend to help each other rather than leaning on the platform teams. The platform remains entirely on-premises, although they plan to offer a public cloud offering shortly.

Mercedes-Benz is now one of the top code contributors to Kubernetes and OpenStack and has published its own FOSS Manifesto, which includes the principle: "Prefer open source: An engineer shall look for Open and Inner Source [internally developed] alternatives before writing custom code or using proprietary alternatives."

"So from open source not allowed to prefer open source is pretty good. And it also has a clear commitment to contribute to open source and to play an active role in open source communities. I think it's pretty cool," said Mueller.