Log4Shell, a remote code execution vulnerability in Log4j 2, was disclosed in December 2021. Its remediation effort extended through the first quarter of 2022. This incident produced a blueprint for how organisations should respond to critical supply chain vulnerabilities.
The severity of Log4Shell came from its attack vector. An attacker submits a string to an application that logs user input. This string contains a JNDI lookup that Log4j evaluates when it logs the message. The JNDI lookup connects to an attacker-controlled server and loads an arbitrary Java class. One logged string results in full code execution.
Organisations immediately faced an inventory problem. They had to answer: where is Log4j in our environment? The answer required a software bill of materials for all applications and infrastructure. Most organisations did not have one. Finding Log4j required scanning running containers, installed JARs, JARs nested inside WARs and EARs, and dependencies in cloud services. This scan took days in many organisations.
Organisations used various tools to scan their environments. For example, they used Snyk, Black Duck, or DependencyCheck to identify vulnerable Log4j versions. Some organisations wrote custom scripts using tools like Python or Bash to parse logs and identify potential attack vectors. These scripts helped to prioritise which systems needed immediate attention.
The remediation timeline involved several steps. Organisations had to identify all affected instances, test that the upgraded version did not break application behaviour, deploy the update to production, and verify the upgrade was successful. For external vendors' software, organisations were dependent on the vendor's timeline. For applications they controlled, the upgrade itself was often straightforward; the discovery and testing were not.
In one organisation, the testing phase took over a week. They had to test over 500 applications and services, and about 20% required additional testing due to custom integrations. This experience led them to implement automated testing for future upgrades, reducing the testing phase from weeks to days.
Organisations that experienced Log4Shell remediation difficulties invested afterwards in automatic software bill of materials generation as part of their build pipeline. They also implemented dependency vulnerability scanning in their CI/CD process. Additionally, they created a software asset inventory that could be queried for specific package versions.
The preparation for the next Log4Shell incident involves having a software bill of materials and vulnerability management practice. This is not about having a faster response capability. It's about being prepared.
The Log4Shell incident showed that organisations need to take proactive steps to manage their software supply chain. This includes having a complete inventory of their software assets and being able to quickly identify and remediate vulnerabilities.
Organisations that did not have a software bill of materials had to scramble to create one. This was a difficult and time-consuming process. It highlighted the importance of having a complete and up-to-date inventory of software assets.
The remediation effort for Log4Shell was complex and time-consuming. It required significant resources and effort from organisations. The incident highlighted the need for organisations to be better prepared for supply chain vulnerabilities.