Fixing Log4Shell: how a university patched all its endpoints over a weekend
It's all about knowing what you have, how the software is interconnected and then getting boots on the ground, says SNHU's endpoint team
The Log4Shell vulnerability in the open source Java component Log4J has caused huge consternation among IT security professionals for two principal reasons. First, the flaw is trivial to exploit, requiring no special expertise on the part of a malicious actor to establish a foothold on an affected network, steal data or execute code. Second, Log4J is embedded in a huge number of applications, including Apache web server which is used in a third of the websites in the world, and many, many applications written in Java. In short, it's a security nightmare.
The bug was made public on 17th December - a Friday - which for many security professionals meant a lost weekend.
"Saturday morning we immediately jumped into meetings, everyone was boots on the ground. And, of course, we were important to that because we manage endpoints strictly it's the single largest attack surface in the organisation," said Jordan LaFontaine, endpoint operations team lead at Southern New Hampshire University (SNHU).
LaFontaine and his team are responsible for 7,500 endpoints at the university, ranging from laptops to virtual machines in the cloud. The majority of these endpoints run the Java virtual machine (JVM), Apache or other Java-based software, and so were vulnerable to Log4Shell.
The task of protecting them was not made any easier by the fact that a total of three versions of Log4J were released in the days following the initial disclosure, requiring the endpoints to be upgraded three times over.
"It was one version after another," LaFontaine sighed. "They released a new one and hey, the next day we'd find out we have to replace that one too."
But in the immediate aftermath, the main challenge was locating all the Log4J instances in the system, whether the versions of Java they were running were vulnerable, and finding how the component interacted with other software.
Fortunately, the University had recently deployed Lakeside Software's digital experience platform - although for asset management duties rather than for security. Nevertheless, because Lakeside software installs agents on the devices it monitors, including all of the endpoints, SNHU's endpoint team were able to quickly pull thousands of data points from those endpoints and create a dashboard to show exactly where Log4J instances were located and how an attack might affect other connected systems.
This enabled LaFontaine's team to track down and automatically patch vulnerable systems much more quickly than would have been the case had they relied on their other monitoring tools, which, he said, could not match Lakeside for depth of view.
The component problem
Log4Shell is just the latest in a series of cases where vulnerabilities emerge in widely used but poorly supported open source components. Infamous examples include the Heartbleed and Spectre flaws resulting from an OpenSSL glitch, but there are many more - including the unusual case this week of a developer apparently sabotaging his own libraries as a protest.
These libraries have become an unpredictable source of risk, and for many years calls for action to support developers, who often work for free, have largely gone unheard. Huge corporations benefit from their efforts while giving nothing back except for blame when things go wrong. And it's not just open source software that's affected, LaFontaine commented.
"It's also a problem that open source software gets used to support solutions that are not open source. I think that something does need to happen, where there's accountability and checkpoints along the way to make sure that this sort of thing doesn't happen again."
There are signs that things may be starting to change, however. US national security adviser Jake Sullivan this week summoned software companies to discuss ways to beef up cyber security, and a Google pledged $1 million to help developers adhere to security guidelines, in addition to other funds it has already allocated to securing open source. And following another call from the authorities, this one after the SolarWinds breach, the Open Source Security Foundation raised $10 million to help secure software supply chains.
So, there is some movement, but change can't happen soon enough, said IT endpoint operations associate director Will Collard.
"It's a systemic issue. You see these freeware pieces of software and no one's managing it. It's like a Wild West, it's a big problem."
Given the problem, universities and other organisations need to keep a handle on what's running where, so that when zero-days emerge they can jump on them quick, Collard added.
"With Lakeside we were able to see those interactions between Log4J and how they tied in with Java and other applications, so we could mitigate it. Everyone needs to dig in and use whatever tools they have to mitigate the risks."
There have been numerous reports of successful exploits of Log4Shell around the world, including by state actors, but the SNHU's endpoints are not among them, nor are they likely to be thanks to the team's rapid response. A lost weekend well spent.