1

May 2020

AFCEA International Cyber Committee

OPEN SOURCE SOFTWARE AND MISSION-CRITICAL APPLICATIONS:

A CAUTIONARY TALE

2

INTRODUCTIONThe way the United States develops, tests and deploys software, including for mission-critical applications, has changed. Broad interconnectivity among developers enabled by ubiquitous Internet access, readily available repositories, and both policy and culture that encourage reuse

of software has given rise to an environment where integration of existing components rather than redevelopment has become the main vehicle for bringing software and embedded software systems into operational use. Development and Operations (DevOps) is all the rage.

Government policy has endorsed and commercial pressures have accelerated this trend. The efficiencies of using and embedding open source software (OSS) imply a number of risks. The AFCEA Cyber Committee has examined the origins of this OSS trend, its motivation and some of these risks with a view to inform AFCEA companies and members as to the benefits and risks of this approach. The committee members draw an analogy to previous quality control experiences. The integrity of the supply chain delivery mechanism is as important as the delivered components.

The committee members conclude with some suggestions for mitigating risks when building systems they intend to trust. Those systems are now developed principally by integrating existing untrusted components having known vulnerabilities. Often those components are themselves subject to continual modification, improvement and correction.

3

The notion of sharing software with others is hardly new, but most would argue that the present open source movement began with Richard Stallman, a computer science researcher at MIT in 1984. He argued that software should be freely available for others to examine, understand, modify, improve upon and redistribute for the

benefit of others. This meant, naturally, that the human-readable source code needed to be readily available to others for code inspection and modification.

Stallman formed the Free Software Foundation and founded the GNU project. He wrote the GNU C compiler and GNU Emacs, both pivotal to software development at the time. He created the GNU General Public License (GPL), which allows for the copying, modification and redistribution of software licensed under the GPL. None of those actions require explicit permission of the original owner; the only obligation is that modifications be public and therefore visible to the originator.

The GPL permits use of the software on any number of target platforms without restriction as to purpose. It must allow for the distribution of source code via the Internet in a form that enables understanding, modification and redistribution. Distributors may charge for distributing software under the GPL but are not required to do so and may not require others to charge for derivative works. The GPL license must accompany the source code. Any derived works must

FREE AS IN FREE SPEECH, NOT AS IN FREE BEER

4

be distributed under the same terms as the GPL. Any derived work may receive patent protection, but the patent must provide for free license of the derivative work for all under the GPL or not be licensed at all. Other common licenses today, all variants in some way of the GPL, are from MIT, LGPL, Apache, Mozilla, Python, Eclipse and BSD.

Stallman also coined the term “copyleft.” Stallman’s approach was to copyright his software and then distribute it via the GPL. His efforts motivated others to do the same and share software under the GPL. His world was the university where computers were research tools, software was the enabler and people shared ideas and software freely. Researchers were paid to research, not to program; programming was a research tool.

As computers became more widely used in industry, people were paid specifically to create more commercially applicable programs, such as finance and accounting systems, that had inherent commercial value. Their employers restricted usage and rights to their work, generally by keeping the source code private and licensing, not selling, software binaries and certainly not distributing the source code under the GPL. Thus, most of the early experience with software was either through research or government labs or with commercial licensing through companies such as Microsoft.

Commercial licenses were the antithesis of the GPL, restricting use to individual, specified platforms and prohibiting modification or redistribution without permission and compensation. Commercial interest in open source software as a business opportunity rather than a challenge to revenue did not emerge for many years.

It took “The Cathedral & the Bazaar,” an essay by Eric Raymond in 1999, to move what had been a hero culture of individual contributors like Stallman on their programs to a community where others were welcomed to participate as developers, document writers, testers or online Q&A advisers. The “How Many Eyeballs Tame Complexity” section of this essay not only inspired others to join the open source movement but also formed the fundamental advantage open source software has in implementing mission-critical software.

5

Netscape released its browser source code into the public domain in 1998, developing the Mozilla Public License and inviting programmers to voluntarily improve the product. Netscape’s clear intention was to increase its browser market share. Similarly, IBM supported Apache, believing the stable and already widely deployed server software would increase its server hardware market share.

But it was the advent of Linux and the decision by Red Hat in 1995 to invest in packaging, documentation, training and field support in freely available open source Linux kernel and related software that established open source software as an integral component in modern commercial software development. IBM bought Red Hat in 2018 for $34 billion following Red Hat’s purchase of Jboss much earlier. Microsoft bought GitHub, a hosting platform enabling broad sharing of software source code and documentation, for $7.5 billion in June of the same year.

Android, the world’s most popular mobile OS, is based on open source Linux, as is the Amazon Echo, the Nest suite of home management products and many automotive applications. Google Chrome, one of the world’s most popular desktop browsers, is based on open-sourced Chromium. Firefox, the runner-up desktop browser, also is open source and descended from Netscape Mozilla. AT&T, Google, Amazon and Microsoft have all released open source artificial intelligence software frameworks enabling developers to rapidly integrate existing and publicly available components to create AI applications. Docker, an open source container technology, enables developers to package applications to include only the necessary supporting software, simplifying development and dependency management and enabling the rapidly expanding DevOps movement. Kubernetes, open-sourced by Google in 2014, is a platform that is based on Google’s internal container cluster manager Borg.

GitHub provides for software development version control using Git, which is open source software. It provides access control and several collaboration features such as bug tracking, feature requests, task management and wikis for every project. Free GitHub accounts are commonly used to host open source projects. GitHub reports having more than 37 million users and more than 100 million repositories, including at least 28 million public repositories, making it the largest host of source code in the world. GitHub is but one repository for ideas software fragments, much of which is open source.

6

Launchpad, hosted by Canonical Software, is a platform for open source Ubuntu Linux, the most popular Linux distribution and Linux applications.

SourceForge is the repository for 430,000 open source projects and has over 3.7 million registered users. It enables more than 4.5 million software downloads a day. It is based on the subversion code versioning software, which also is open source.

The Apache Maven Central Repository hosted by Sonatype, a developer of software repository management tools, holds over 304,000 Java components from 2,700 suppliers and receives more than 21,000 component updates every day. They report that the number of download requests for Java components exceeded 125 billion in 2018 and is exponentially increasing. NPM, hosting the JavaScript repository, reports an annualized rate in excess of 500 billion downloads of packages this year.

Additionally, development tools and platforms are increasingly open source. In 2006, Sun released much of its Java virtual machine (JVM) as open source software under the GPL. JavaScript, Python, C, C++, Go, Ruby and others all have open source implementations. The open source Eclipse development environment supports most modern programming languages.

While Arduino is normally thought of as inexpensive open source hardware, the open source software development environment of the same name has inspired countless young people to engage in STEM projects. The Raspberry Pi Foundation, creator of a series of low-cost but powerful single-board computers, has adopted an open source variant of Linux as its official software distribution. At the other extreme, Linux is the operating system of choice of all 500 of the world’s fastest computers.

The net result is that open software is everywhere. Sonatype estimates that 85 percent of software content in applications delivered today comes from outsourced sources. Today, if an end user software remains proprietary, it was probably developed, tested, stored or distributed using some form of open source software.

7

It is seductive to think of open source as “free,” as in without direct initial cost to the acquirer, just a download and maybe an install. It follows, therefore, that the U.S. government, particularly under the Obama administration’s Open Government Initiative, mandated, via the Office of Management and Budget, the sharing with the American people of software developed at public expense.

The U.S. Defense Department issued clarifying guidance on open source software use in 2009. More significantly, the department determined that it was commercial software and thus became an option under a number of make-versus-buy provisions of the Defense Federal Acquisition Regulation Supplement. By regulation, contractors and their

subcontractors were required to use existing commercial software where possible to meet mission requirements.

The department also has enabled open source sharing via a SourceForge proxy, SourceForge.mil, protected by the Defense Information Systems Agency firewall. ProjectForge provides the same application life-cycle management tools for the department’s projects and programs as Software Forge but for projects and programs that need to restrict access to specific project members.

Clearly, the re-use open source software is widely encouraged and accepted as a best practice in contemporary software and embedded systems development. Sometimes, open source software is referred to as free open source software (FOSS). The committee does not use the FOSS term in this paper to reinforce that while OSS may be freely obtained, its use is not without life-cycle cost.

WHY DOES IT MATTER TO US NOW?

8

It should come as no surprise that in the advent of free repositories and literally millions of open source projects, many of which depend on one another and billions of downloads annually, the notion of any reasonable centralized authentication as to the origin or any assurance as to correctness is virtually impossible. Sonatype studies the characteristics of the Java Central Repository components downloaded each year as part of its annual “State of the Software Supply Chain” report. Of those components, in 2018, 8.8 percent had known common vulnerabilities and exposures associated with them, yet they were downloaded anyway. Sometimes, to ensure dependency integrity, older versions with known vulnerabilities were accessed even when newer, patched versions were available.

The very advantage of open source, that the value of a network increases asymptotically with the number of nodes (Metcalf’s law), denotes that an active review and commit culture, software developer Eric Raymond’s “many eyeballs,” is vital. But there is the downside variant in Fred Brooks’ law that comes from having too many people involved if the project is loosely structured. According to Brooks, integration risks develop at component interfaces and the more interfaces (contributors), the more likely it is that integration problems will ensue.

EXPLOSIVE GROWTH: BOTH SUPPLY AND DEMAND—THE LURE OF OPEN SOURCE

9

Even as open source has found acceptance in the world, it still faces challenges. While Raymond’s observation that many eyes lead to the earlier detection of vulnerabilities—maliciously inspired or not—in readily inspected source code, it does not necessarily follow that those eyes are

available, competent or interested. Furthermore, and more significantly, it does not presume the migration of upstream fixes in open source software into downstream releases or end products. Some of the fundamental open software components, such as OpenSSH, which have existed for many years and are widely used, actually have few such eyes, enabling latent vulnerabilities to perpetuate unnoticed. Two high-severity problems that had existed for many years were noted as recently as 2016.

The strength of community, openness and trust in open source software unfortunately also is a weakness. A cultural system organized to solicit and accept patches, fixes and improvements from a variety of somewhat vetted but fundamentally untrusted sources will inevitably allow some malicious code to be introduced. While the number of detected intrusions of this nature compared to the number of legitimate updates is minuscule, the practice continues. In early 2019 a patch to a popular server-side Ruby gem was uploaded to the official repository. The malicious version became a dependency on at least 1,600 other GitHub repositories with an unknown number of downloads among them. While the bug was found and fixed within a day, because of the nature of open source software, there is no way to determine how many systems remain vulnerable.

WHAT CAN GO WRONG?

10

A 2016 IEEE Security and Privacy paper by Abbas Acar and colleagues found that the quality of Android applications code was directly affected by the sources from which their code was obtained. They tested grad students and professional developers under time constraints assigned to one of four conditions: free choice of resources; Stack Overflow, a popular code fragment sharing repository, only; Android official documentation only; or other software books only. Those only using Stack Overflow produced significantly fewer secure code than any other group. The tradeoff was that those using Stack Overflow produced far more functional code under constrained conditions than any other group.

To measure the relevance of these findings in the broader culture, the study looked at 200,000 apps on Google Play, finding that 94 percent of those apps used at least one API call employed by the subjects during the coding exercise. Many of the security flaws found in the apps written for this study propagated security flaws from software found on Stack Overflow. The researchers concluded that API documentation resulted in more secure solutions but took longer to achieve when compared to just downloading snippets from Stack Overflow and integrating them.

Another risk to the open source movement comes from a strength—the volunteer culture itself. People want to be involved doing what they enjoy: creating products for others to use. Not only are employees who participate in open source projects more satisfied with their work, but their companies also make more money. Thus, having a strong volunteer culture ensuring a continuing number of feature improvement and maintenance software commits is essential. While it may seem to be an extraneous factor, maintaining cordial and professional relationships within the community is necessary. Therefore, the unpredictable and in some cases unwarranted behavior of some leaders of the open source movement is a problem.

While Stallman was perhaps the first champion of open source software to have impassioned views and to ensure others knew them, he has not been the last. Stallman resigned from the Computer Science and Artificial Intelligence Laboratory (CSAIL) in September 2019 following a long history of strained relations with the MIT administration. Linus Torvalds—the creator of the open

11

source operating system Linux and also found at the heart of Android, the Amazon Echo, Tesla displays and many more products—apologized in 2018 for years of unprofessional behavior. He announced that the project would adopt a code of conduct to help the project’s contributors decide how to behave toward one another.

The irony of adopting a patching policy that enables the automatic downloading and installation of patches from reputable sources is that it enables yet another way to propagate malware onto previously secured machines. The Russian hackers initiated the now-famous NotPetya attack by attacking the servers of the Linkos Group, creators of finance and accounting

software widely used in Ukraine. When end users downloaded and installed updates from what they believed to be a reputable supplier, the virus spread. The payload enabled the recovery of passwords and credentials of other totally patched systems that were stored on the infected machines. Thus, the seemingly responsible act of keeping software current through updates from the vendor is actually not without risk.

A final risk to the success of open source software stems from the very freedom of the software licensing arrangements themselves: the freedom not to update software even when vulnerabilities have been discovered, publicly disclosed and remedial patches created. There are some experts, reinforced by the NotPetya experience, who judge the risk of attempting to patch otherwise operational software to be greater than the risk that malicious software will exploit a known vulnerability.

Some proprietary software now makes it very difficult not to update, however inconveniently, their software. While thoughtful practitioners can debate the criteria and details of a patching policy and strategy, not having one at all, or having one that just reacts to bad news, seems like a flawed long-term approach.

12

If the creation, distribution and use of open source software is viewed as fundamentally a manufacturing and distribution supply chain problem, there is more than 70 years of quality engineering theory and experience in those areas. The work of W. Edwards Deming in developing the Plan-Do-Study-Act (PDSA) cycle at Toyota, illustrates the idea of deductive and inductive learning built into the learning and improvement cycle and is germane to OSS development and maintenance. The software supply chain is analogous to hardware manufacturing in that there are suppliers (OSS projects), distributors (online component repositories), manufacturers (software development teams) and finished inventory (user applications.)

Deming emphasized in-process participatory improvements and statistical sampling rather than exhaustive finished product testing. He believed in developing a small number of suppliers, building trust relationships with them and sharing as-built final results to help suppliers identify and fix their upstream problems quickly. He also believed that keeping track of what components were integrated in which products enabled manufacturers to tactically recall flawed inventory, thus minimizing end users’ overall risk and enhancing the manufacturer’s reputation. These Deming principles apply not only within individual open source software projects but also throughout the supply chain.

There is no evidence that open source software is initially of higher quality or has fewer vulnerabilities than proprietary software. But by the very nature of “many eyeballs,” poor coding and documentation practices are replaced, and a larger percentage of the vulnerabilities become known and publicly reported. For software with active commit bases, flaws get patched faster. For projects with a more rapid release policy, those fixes are available for operational use faster. Sonatype found that projects that release more frequently—more commits—and update their dependencies more frequently—observing the commits of suppliers—are more secure. They also found that having more dependencies did not necessarily decrease security. But they also discovered that simply being a popular project—more downloads—does not make a project safer. The eyeballs need to be on the source code not on the download package.

NOW WHAT DO WE DO?

13

• Open source software is here to stay. The usage is increasing dramatically. It is everywhere, including in places people do not realize.

• Open source software enables the speed and efficiency of the DevOps culture, which is great until quality suffers.

• Open source software is not free. There is a total cost of ownership over the product life cycle. Be prepared to invest in the update and maintenance tail.

• Users should cultivate trust relations with a few suppliers and make it their job to keep their own suppliers, in turn, up to date. They must know their suppliers’ dependencies.

• Employers should support their employees’ participation in open source projects that interest them that come from suppliers company owners depend upon.

• A convenient but not very useful policy would be “only download and use trusted software.” A far more pragmatic approach would be to use software only from sources that have an established public bug tracking and resolution, known update cycle policies and an active and extensive commit culture.

• Users should know where their products have gone so that updates can be applied easily and securely and a flawed product can be recalled.

• Organizations should ensure the integrity of their distribution system and local repositories. They should not allow their employees or suppliers to download source or binaries from the wild, which means investing in both people and technology to manage their internal repositories. It means having a corporate policy on the criteria for importation and use of open source software and providing resources so that following the policy is both efficient and effective. It means using well-developed and understood repository management software. Given that virtually all applications developed today either include open source software in the final product or are developed or tested with open source tools, a policy of “no” is a bad idea.

• Employers should integrate your security team with your developers. It is seductive to let others find bugs or worry about correctness or assurance when you are late, and they are in a separate group.

CONCLUSIONS

14

• Have your security team stay current on the known, reported vulnerabilities in components your products use. Have them work with developers earlier in the process so alternative sources for components with long-standing open common vulnerabilities and exposures reports or infrequent or irregular release policies can be researched. Begin the risk analysis as far forward in the development cycle and as far away from the deployment decision as possible. If the security analysis concludes there are no flaws, get a new security team.

• Most importantly, apply supplier updates often. Accept the risk and cost of integration problems when applying update patches as they should be an expected part of product maintenance. But if supplier-induced or perpetuated problems persist, select another supplier. A “few” suppliers means more than one. With more than 100 million open source software projects available, finding a reputable fork with active development of a component you are now depending upon should not be difficult.

15

The AFCEA International Cyber Committee White Paper Serieswww.afcea.org/committees/cyber

Copyright 2020 AFCEA International. All rights reserved.

All distribution must include www.afcea.org.