After months of seeking to understand the key lessons learned from the Log4j flaw, one big takeaway for the Cyber Safety Review Board (CSRB) has been the ongoing need for more centralized resources and security assistance structures that could help “uplift” the open source community, said board members during a panel at Black Hat USA this week in Las Vegas.

The CSRB - a panel of private and public sector industry leaders launched earlier this year by the Department of Homeland Security with the task of better understanding what happened during significant cybersecurity events - in July released a number of non-binding recommendations that in part took into account underlying issues in the open source ecosystem and how these challenges played into the discovery, disclosure and fixing of the bug. While talking to 80 organizations and security researchers, the board looked at whether someone “could have caught this functionality before it made it into production,” said Heather Adkins, deputy chair of the CSRB and vice president of Security Engineering at Google.

“Our finding was that only with the right kind of expertise during a code audit would you have been able to flag that this was a problem, and the communities just don’t have those resources available to them all the time when they write code,” said Adkins on Wednesday. “So we know these organizations need to have the resources for things like that but also, how do you get your code fuzzed at scale, how do you find bugs at scale?”

Overall, Log4j highlighted the significance of the open source ecosystem and the important part it plays in the Internet as a whole - but despite this criticality, security risks continue to exist across the open source software ecosystem, mainly stemming from tight resources. When speaking to the open source community, the board found a “very diverse set of rules” existed among different open source foundations in running projects, said Adkins.

“They have a particular role in this ecosystem, they’re volunteer based, and a lot of the projects are governed according to how they want to run the projects, so there’s a lot of variety there, and they also have a lot of strong opinions about what should and shouldn’t be happening within their communities,” said Adkins. “We can learn alot about the benefits of the open source community having the freedom to run software the way they run software, but we also learned there are millions if not billions of people relying on them getting it right.”

Adkins said that many times the fixes in the open source community are being done “in the open,” meaning that anyone keeping an eye on the Apache Software Foundation’s (ASF) git pulls or looking at their release candidates may have noticed they were fixing the code in the JNDI functionality, even if they didn’t mark it specifically as a patch. Log4j was first reported on Nov. 24, 2021 by a security engineer from the People’s Republic of China (PRC)-based Alibaba Cloud Security team - but while ASF was working to devise a fix for the flaw, another PRC-based cybersecurity company, BoundaryX, disclosed the flaw on WeChat before ASF made a publicly available update. The board hypothesized that someone noticed Apache's efforts to develop the patch before it could release an official fix and begin that mass patching phase, leading to mass exploitation, said Adkins.

“It was a moment for us to sit back and think, well, as we think about the software ecosystem and getting patches out quickly and that surge that had to happen… This is an opportunity for us to think about how we build a software ecosystem where we can all move very quickly because we know that things happen in the open, that bugs get discovered and they get exploited before they are disclosed,” said Adkins. “That was a really important finding for us in terms of how we want to shape the ecosystem.”

The inherent security issues that exist in the open source software ecosystem itself pose long-term problems. These have already been discussed by the U.S. government and tech sector on the heels of the Log4j flaw’s disclosure, which have discussed a proposal to set up an independent clearing house to offer support and match volunteers with open source projects that need help, and by open source foundations themselves, including the Open Source Security Foundation's Alpha-Omega project to help the maintainers of thousands of critical open source software projects find and fix security vulnerabilities in their code.

The CSRB expanded on these proposed measures in its recommendations with a call for the U.S. government to “play a role in driving security enhancements” for the overall ecosystem as a significant consumer of open-source software. That role would include the OMB taking the reins in directing federal agency IT staff members to assist in securing and maintaining the open source software on which they rely. The CSRB also recommended that CISA invest more heavily in open source software security by creating a Software Security Risk Assessment Center of Excellence in order to develop and manage a central inventory of all software across federal agencies, with the end goal of facilitating vulnerability notification and response. Private-sector organizations that use open-source libraries in order to build commercial software should also offer up financial resources for the open-source projects that they deploy, said the board.

“We’re skiing on top of a pretty good avalanche of support... that is looking at how we help uplift the open source community,” said Adkins.

One overarching takeaway from the report is that the Log4j flaw will continue to pose a risk to organizations long after the dust has settled, with exploitation levels continuing to persist and evolve. When taking on Log4j, “the feeling was that this was a fresh event that had a lot to unpack for virtually every kind of organization out there… Every organization was impacted because Log4j is sprinkled across the ecosystem and it had serious ramifications for the open source community,” said Rob Silvers, chair of the CSRB and DHS undersecretary for policy, during the Black Hat session.

“This was a really high impact event that affected everyone,” Silvers said.