Supply chain attacks: Chainguard VP on why software needs a manufacturer mindset

Eliminate vulnerabilities before software is shipped not after the fact, says VP engineering Dustin Kirkland

During a talk at a recent Computing cybersecurity event, former hacker and CISO Greg van der Gaast drew an interesting comparison between the aviation and software industries.

An airline doesn’t check if a wing is going to fall off every time a plane flies, it relies on the manufacturer to build better bolts. This proactive approach has led to a fall in the rate of crashes over time. Meanwhile, in software the pattern has been to patch after a hack has happened, with the responsibility falling on the user rather than the vendor. The rate of attacks has been rising year-on-year.

“We operationally risk manage everything. It makes no sense,” van der Gaast said. "We’re not stopping the introduction of vulnerabilities; we’re only ramping up the mitigation.”

Speaking of the aviation industry, last week Qantas announced that its systems had been breached, with the likely theft of millions of customer records from a third-party contact centre. Full details of the cyberattack are yet to emerge, but what is clear, not least to hackers, is that complex supply chains are extremely difficult to defend. And that goes for software supply chains as well as physical ones.

Almost all modern applications, including proprietary software, are constructed from hundreds of open source components pulled from numerous repositories. Each of these components will likely have known vulnerabilities, serious or otherwise, some of which may have existed for months or even years. Tracking these components so that vulnerabilities can be quickly fixed is the purpose of SBOMs (software bills of materials), component manifests delivered with each new version of the application, but even this is an after-the-horse-has-bolted measure, and SBOMs are far from universally adopted.

Removing vulnerabilities before release

For years, cyber experts like van der Gaast have argued that software vendors themselves should take more responsibility for ensuring the security of their code, just as an aircraft manufacturer is liable (in theory at least) for the quality of their planes, and car manufacturers are required to recall vehicles and fix dangerous faults, rather than relying on customers to do it.

It's a philosophy shared by Dustin Kirkland, VP of engineering at Chainguard. Chainguard produces a hardened operating system, container images, VMs and libraries, mainly for customers who need to comply with FedRAMP and similar security standards. "We take security vulnerability remediation off of their plate so it's no longer their responsibility, it's ours," Kirkland told Computing.

Chainguard OS is a rolling enterprise Linux distribution that's based on the open source, community-maintained Wolfi OS. But that's where the community involvement ends. From that point in, everything is controlled by the company, which patches all known vulnerabilities before releasing its software to customers and while their subscription lasts.

The rolling OS is used to create hardened container images. Each time a new vulnerability is fixed - which could be several times a day - a fresh container image is created, together with a new SBOM, and either pushed to subscribing customers or made available for them to download. These images, in turn are used by customers to build their own software, secure in the knowledge that vulnerabilities that could find their way into their code have been eliminated. Currently there are around 1,400 images in Chainguard's directory ranging from minimal base containers to those with specialised libraries and applications.

For perpetrators of software supply chain attacks, a common way in is via open source libraries for Python, Node.js and Java in repositories like PyPi, npm and Maven, which host hundreds of thousands of individual packages. With depressing regularity malicious actors have succeeded in inserting their own nefarious code into packages, or creating spoof libraries with similar names to legitimate ones. To ensure they are bug-free, Chainguard builds all libraries from source before added them to the images.

They kicked off with Java, a language widely used in the enterprise.

"We tracked down the source code to the top 20,000 Java libraries [JARs], and rebuilt those from source, every version that we could get our hands on,” Kirkland explained. “Now it's about 300,000 JARs that we've built from source. Those JARs are resilient against malware, we're able to scan and ensure that those JARs don't include deliberately malicious code. Since we're building from source, we're able to remediate CVEs [Common Vulnerabilities and Exposures] in those JARs as well."

Next up was Python, with the company rebuilding the top libraries on PyPi and working on from there; Node is to follow, Kirkland explained. "We're not yet talking loudly about that, but it's the obvious next one now we've got the automation cranking that's able to do Java and Python."

Reducing the attack surface

All software contains vulnerabilities, and new ones are constantly emerging. It's impossible to make code that's 100% secure, but minimising the number of known vulnerabilities before putting software into production or using it to build more software is surely a positive move. While Chainguard retains complete control over its own rebuilt software, Kirkland says it is "deeply committed to the open source community. When we have fixes to bugs in open source projects, we routinely contribute those upstream."

In addition, the company contributes to foundational projects like Sigstore and SLSA and recently launched Container Hardening Priorities (CHPs), which provides concrete criteria for assessing the security of container images.

Chainguard claims an 80% reduction in attack surface by creating minimalistic, hardened container images that exclude unnecessary packages, tools and components that could be exploited, and a 97% reduction in known vulnerabilities. It guarantees remediating critical vulnerabilities within seven days and all others within 14 days. In March, the company received $356 million in Series D funding, which it is putting into expanding the range of containers and libraries on offer, Kirkland said.