The Linux Foundation: How to fix the internet

Jim Zemlin, executive director of the Linux Foundation, also says the future is basically a software-defined data centre, and concludes by saying open source developers are 'like poets'

The Linux Foundation, the organisation designed to promote Linux, and open source software development practices, plans to improve internet security by co-ordinating teams of dedicated coders, and large firms with the financial power to fund them.

Speaking at technology conference IP EXPO in London today, Jim Zemlin, executive director of the Linux Foundation, began by outlining the ubiquity of Linux, the open source operating system originally developed by Linus Torvalds.

"Over 850,000 mobile devices running Linux activate every day," said Zemlin. "There are 700,000 TVs sold per day, most running Linux. Google, Twitter, Facebook and Amazon are all powered by Linux. Nine of the world's top 10 supercomputers run Linux."

Since 2005 over 8,000 developers from around 800 firms have contributed to the Linux kernel (the fundamental part of the operating system that translates user or other types of requests into instructions for the device's CPU). Zemlin said that a major new kernel comes out every two to three months, which is a far more regular update than other operating systems, like Microsoft's Windows platform, which usually only sees new revisions every five or more years.

The development process for Linux is extremely collaborative, and Zemlin highlighted this as a major strength of open source software in general, which he said firms are now seeking to turn to their advantage.

"Everyone's now talking about how to leverage open source. It's simply a better, faster, cheaper way to create software and innovate. Things like Hadoop create billions of dollars in value. Hortonworks had a $1bn IPO, and it was totally built on open source.

"Even Microsoft and Apple are embracing open source. Microsoft is using Linux to create SDN [software-defined networking] infrastructure in its Azure cloud."

He went on to list other examples of open source proving itself to be big business, like $1bn open source development platform Docker, and electric car manufacturer Tesla, which is "creating next-gen automotive cockpit experiences using Linux".

"In almost any technology product, around 80 per cent of its code is created by the open source community," said Zemlin. "The internet is almost totally reliant on open source software."

His sales pitch for open source over, Zemlin turned to the challenges.

"We've reached a golden age of open source. Almost every technology company is reliant on open source. Everything in open source is good. Or not."

Zemlin said that one of biggest challenges in computing today is security.

This is as true for open source software as it is for proprietary, with vulnerabilities like 2014's Heartbleed bug - an issue with the internet's OpenSSL cryptography. When Zemlin asked the audience who had experienced problems at work due to Heartbleed, most raised their hands.

Zemlin then referred to a famous Linus Torvald's quote: "Given enough eyeballs, all bugs are shallow."

"If that makes for more secure software, why are we seeing security vulnerabilities in open source?" he asked.

He answered his own question by referring to Harlan Stenn, who runs the Network Time Foundation (NTF), and until recently was earning $25,000 per year for maintaining a crucial internet system.

"The open source software we all rely on every day in some cases is maintained by a small group of people, or even a single person. NTPd [Network Time Protocol daemon] keeps time on the internet, every major system uses this technology to synchronise time. He [Stenn] was essentially working part time by himself. The guy who keeps time on the internet was employed part time!"

Zemlin continued: "GnuPG [an open source encryption system], one of the key ways to secure packages, email and the wider internet is going broke.

"OpenSSL, for a long time was maintained by two guys named Steve. That means that the internet for a long period of time was secured by those two guys. OpenSSH, the way to have secure communications between servers, was maintained by one guy working part time.

"The Bash [a command language interpreter] maintainer is working on it nearly alone."

Zemlin's point is that over a long period of time, the world unwittingly became dependent on open source - and a few overworked, under-resourced open source experts - for the security and integrity of the internet.

"In most cases, like Linux they had a robust community of developers and companies. But other projects get very little attention or resources, and it's completely out of proportion with the importance those systems play in our society."

The Linux Foundation's solution is to bring developers and firms together to raise and apportion funds and resources to those projects deemed crucial to the safe running of the internet.

"Because the software we depend on is open source, we have access to the code, we know who writes it, and we can leverage shared testing tools," explained Zemlin. "And because everyone in the world depends on it, we think we can get lots of people together to fix it.

"The future is basically a software-defined data centre, everything is being abstracted from hardware into software. So the problem is not going away, secure software is very difficult to write and maintain."

But the Foundation is acutely aware of the risks of bringing large corporates into the open source communities which had for so long existed without their involvement.

"The value of open source is its rich community and system of peer reviews, and we don't want to screw up that organic innovation by trying to put some construct on top of that."

So the organisation created a new programme called the Core Infrastructure Initiative (CII) in April last year.

"The idea is to go fix potholes on the information superhighway," said Zemlin.

And there are 20 other major technology firms involved in the programme, including HP, Intel, Rackspace and Dell.

"The idea is to go out and help improve the security and stability of the internet," said Zemlin, before acknowledging, in an extremely understated way, the scale of the challenge.

"It's hard," he admitted. "The strategy will run over a long period of time, with no immediate results. But over many months we'll be in good shape."

Zemlin concluded by describing the three-part strategy of the CII.

1. Finding projects that need help

"We raised a multi-million dollar fund, and we're using that to provide grants to projects to help them out. Like paying for developers to work on OpenSSL, so they can respond to bug reports, and reduce the backlog of patches and make it more secure and stable.

"We've created a new open source project to identify open source ecosystems which are critical to the security of the internet that are at risk. The key is to quickly identify critical projects with small teams and limited time for strong security-aware coding. Now Harlan Stenn works full-time on NTPd, and we brought on other coders to help out. Together we created a more secure version called NTPd-sec."

2. Best practices

"We're creating a set of secure best practices for coding. In 2002, Bill Gates wrote a memo to all at Microsoft about security. 'We're gonna stop development on new product functionality at Microsoft until we fix our security problems', he said. They built a software assurance team, and had a meaningful impact on the security and integrity of Microsoft code, which included better testing and threat modelling. It took a long time, but the security of Microsoft products got significantly better.

"The problem for us in the open source community is we don't have the same command and control mechanisms that Gates had in 2002. No one can write a letter to everyone who works in open source and say 'you're going to write better code'. So how do we create a culture of secure coding practices in open source and software in general?

"The answer is we have a Badge programme, where we're designing set of criteria for secure best practices in open source. Developers, maintainers and other contributors can go in, look at the requirements and see meaningful things that open source projects can do to improve their practices. It includes things like responsible disclosure processes and better testing.

"This will indicate that the teams care about security. Open source developers are motivated by people using their code. They're like poets and artists. They write software and get gratification by the broad adoption of that software, and by the sharing of ideas. By indicating that their software is secure, more people will be motivated to use it."

3. Shared tools and resources

"We're going to provide auditing, test suites, analysis and fuzzing tools that open source developers could leverage but through lack of resources have not to date. We're going to provide those tools for free to open source projects so they can better write their code.

"We're doing a wholesale audit of the entire SSL codebase, that's 500,000 lines of code, looking for vulnerabilities. If we do this well, we will produce more secure software over time. That will reduce the time that IT people spend remediating things like Heartbleed, which cost something like $500m in terms of lost time, not counting the costs of data loss.

"The cost of insecure code in a modern interconnected world is extraordinarily high. The Linux Foundation and the largest companies in the world believe there's a long-term path to creating more secure code. The security, privacy and stability of the internet matters to all of us. Those things depend on open source and we have an opportunity to act.

"We believe that creating a more secure and robust internet is good for all of us," concluded Zemlin.