Copyright Ion Channel 2020

Maintenance: The Biggest Risks Are Boring Journalists on the cyber beat have a hard row to hoe. They really stretch the narrative with dramatic stories of shadowy villains pursued by wily security researchers whose security tools are big advertisers on the cyber beat. But it’s hard to create dramatic tension when the conflict boils down to easy exploitation of bad maintenance. Maintenance. I can feel your eyes glazing already. When COVID-19 hit, Ion Channel analyzed dozens of software capabilities used in public health, epidemiology and for citizen access to federal healthcare and financial systems. There wasn’t a single one that did not have critical vulnerabilities, including malicious packages and viruses along with the regular assortment of published software vulnerabilities. These vulnerabilities had been present for months, because the software wasn’t being maintained. Software that enables data exchange in public health and millions of citizen interactions with federal systems hadn’t been updated in weeks. Some hadn’t been touched in half a year. This is where vendors usually get on their high horse and start scolding the market to inspire fear and shame: “Developers build crap software. Shame, shame, shame, bad, bad, buy our expensive product!” Scolding is a cheap shot. And it’s misdirected, because those developers, app teams and contractors are rewarded, promoted and paid for a lot of things, but security isn’t one of them. Managers and customers want new features and functionality. They want AI/ML, mobile. Throw in some blockchain and you’ve got yourself a multi-million dollar program. No-one’s pitching great maintenance, low levels of technical debt and speedy remediation time. That doesn’t win the work. That’s not going to justify another full-time equivalent on the project. No-one gets promoted for great maintenance. Maintenance suffers because it’s boring. But known vulnerabilities accrete like barnacles on the hulls of enterprise software, and they’re allowed to just sit there. Lapses in maintenance lower the bar for sophistication and cost to the adversary. It’s not an art heist – it’s fish in a barrel. Ion Channel’s software supply chain assurance addresses maintenance risk by solving two business process problems: 1) SecDevOps creates a false sense of security for software that’s not being actively developed and maintained. In-band monitoring – security review when software is built – is better than one-time approval. But it leaves a big blind spot

Copyright Ion Channel 2020

for capabilities that are not being built daily. Software projects have a life cycle: they start with “Hello World” and flurry of activity. They get a lot of love as they grow. They get tested. Bugs get fixed. And then, at a certain level of maturity, someone sticks a fork in the project and calls it done, and activity tails off. At that point, vulnerabilities are still emerging against components of that project. But it’s not being built all that often. So if all you’ve got to check it is pipeline SAST, there’s no situational awareness of those emerging risks. Ion Channel analyzes software as it’s built. But more importantly, the platform analyzes all previously built capabilities out of band, whether or not they’re being built. Assurance is continuous, even if delivery is not. This applies to internal development, but also to contractor deliverables and vendor products – anything coming into an enterprise from the outside. To enable more comprehensive automated audit and faster remediation of internal capabilities, Ion Channel logistics service securely delivers software updates – open source components, assured binaries and software containers – with full chain of custody, so that when updates are required, those updates have already been vetted and securely delivered and their maintenance history has been updated. 2) When maintenance isn’t measured, it isn’t managed. “Fire drill” maintenance on projects laden with technical debt is costlier, and more stressful and demoralizing than regular maintenance. But there aren’t a lot of good ways to measure software maintenance, a prerequisite for managing it well. Ion Channel’s maintenance dashboards and maintenance audit capabilities allow customers to automatically measure and track software maintenance across a large portfolio that includes open source components, vendor products, contractor deliverables and internal development.

Every product and component and Ion Channel’s supply chain assurance inventory has a history page that visualizes the project’s pass/fail compliance and maintenance record, based on governance rules established by the customer. When a project goes out of compliance (or starts its history out of compliance), an audit clock starts to measure the elapsed time before the project is updated and reanalyzed in a passing state. Changes from passing to failing state, and from failing back into passing state, are recorded and visualized to give project and security owners an instant sense of maintenance history over the preceding year.

These insights support leadership decisions to incentivize and reward good maintenance (e.g. days in a passing state, rapid MTTR) and preferentially select components and teams with good maintenance records. Automated and auditable maintenance records can be used to define and enforce contract provisions and SLAs for remediation time and performance awards for value-added maintenance and buy-down of technical debt.

Copyright Ion Channel 2020

Conversely, leadership can review a portfolio of poorly maintained projects to decide whether additional resources are required to bring projects into compliance or whether those capabilities should be replaced and deprecated or archived. This maintenance and compliance history is updated daily or on a customer-defined schedule, regardless of whether suppliers are building, maintaining or delivering updates to their products and capabilities. It establishes the kind of transparency that’s necessary to secure new and legacy systems on an ongoing basis. It identifies teams that build maintainable code that’s easier to update and secure, and quantifies the impact of maintenance on enterprise security posture. Case Study: CDC GeneFlow

Of the federal enterprise development teams that received security findings from Ion Channel, the GDIT Scientific Computing and Bioinformatics Support (SCBS) team for the Office of Advanced Molecular Detection (OAMD) at the U.S. Center for Disease Control was the closest to a NASCAR pit crew in their approach to maintenance. CDC’s Advanced Molecular Detection group is charged with developing high-performance computational capabilities to identify, investigate and track diseases, including COVID-19. In the first months of COVID-19, its technical personnel were not under-tasked.

Within hours of unexpectedly receiving security findings in May 2020, the CDC GeneFlow development team e-mailed to acknowledge the disclosure. Responses to security disclosures typically range from attack-the-messenger to passive-aggressive dismissal or de-prioritization of the message. No-one likes pop-up findings. We get it.

The first word of the GeneFlow team’s e-mail reply was “Thanks.” The team

outlined steps to replace outdated dependencies with up-to-date versions that were not vulnerable. Research capabilities, often deployed in academic environments, cannot rely on the presence of perimeter defense, intrusion detection or role-based access to minimize attack surface created by vulnerabilities in the codebase. Eliminating known vulnerabilities is particularly important for capabilities that are released to end-users who may or may not have solid system security plans.

Within 24 hours, almost a third of GeneFlow’s software vulnerabilities were

eliminated in the publicly available release. The project was refactored, tested and released with no High or Critical vulnerabilities the following week and has been free of Highs and Criticals for a month and counting.

So what enables this kind of quick tire-change on a relatively sophisticated

software capability? One answer is that good maintenance was established before it was necessary to remediate a software vulnerability. Of the project’s direct dependencies prior to the disclosure, only three were more than one major version out

Copyright Ion Channel 2020

of date. A low level of technical debt reduced the risk that a security-relevant component update would break the system.

A second reason is that even though this project is not in active development –

its commit frequency peaked in 2019 – there is more than one core committer active on the project. From a supply chain security perspective, the dwindling of support to just one person (whose participation may be nominal) is a risk indicator. If there are two people who consider themselves responsible for a project, even one in a steady state, it’s more likely to be remediated in a timely fashion (or at all).

For this reason, Ion Channel’s software supply chain governance criteria include

single-committer projects as a risk metric that can be used to gauge supplier risk, either as a stand-alone rule or in combination with other cyber hygiene and ecosystem risk metrics.

Ultimately, our position – and value proposition – is to combine continuous

awareness of software vulnerabilities with deeper analysis of supply chain risks that software scanners don’t detect. Maintenance – lack of maintenance or risky patterns of maintenance – isn’t a risk that pipeline or endpoint detectors are going to flag. But it’s as fundamental to third party risk management. Transparency on the maintenance of software components, applications and products isn’t glamorous. But it may be the most effective way to get a handle on risk in your software inventory.

To learn more about using Ion Channel to risk-manage your software supply chain for open source components, contractor deliverables and vendor products please email: info@ionchannel.io.

www.ionchannel.io