T10 Concurrent Session Thursday 10/25/2007 11:15 AM JUMP TO:

Biographical Information

The Presentation

Even Cavemen Can Do It: Find 1,000 Defects in 1,000,000 Lines of Code

in 30 Days

Presented by:

Gregory Pope and William Oliver, Lawrence Livermore National Laboratory

Presented at: The International Conference on Software Testing Analysis and Review

October 22-26, 2007; Anaheim, CA, USA

330 Corporate Way, Suite 300 , Orange Park, FL 32043 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com

Gregory Pope Gregory Pope has over 35 years of experience applying common sense to developing software in the commercial and government sectors. Greg has held positions from programmer to CEO. He has been an invited keynote speaker for numerous international symposiums related to Software Quality (STAR, Software Testing Automation, Quality Week, ITEA-DoD) and remains active in presenting papers and articles. His articles have been featured in Better Software, Computer Design, Application Development Trends, Industry Week, Computer World, Signal, Electronic Defense News, San Jose Mercury, and Software Maintenance News. He has presented over 300 seminars internationally on Software Testing, Software Quality Assurance, and Software Project Management. He also holds patents for automated software testing systems (Ferret®) and has developed a number of methodologies used for test design and execution. Greg was selected to the software program manager's network to audit major DoD projects and has been on the Board of Directors, International Internet and Quality Week since 1992. He is a member of the DOE's Software Quality Assurance subcommittee and 2003 winner of the DOE Merrin Award. Currently Greg works for the University of California at the Lawrence Livermore National Laboratory where he is Software Quality Engineering Group Leader and V&V Project Leader for Advanced Scientific Computation on the world’s largest computers. Prior to this, in 1993, he founded and ran Azor, Inc., a software testing company, until it was purchased by Tescom in 2000. Greg has also held a number of management and technical positions involved with mission critical testing of cruise missiles and military communication systems, system engineering and development of software code for real time sorting and searching algorithms for electronic countermeasures, telemetry and data acquisition systems for flight testing helicopters, and experimental testing of jet engines. Greg has a BS Degree from the Connecticut Sate University and an MBA from The University of Phoenix. He is also an ASQ certified Software Quality Engineer.

William Oliver William Oliver received a BS in Mathematics from the University of Utah in 1974. He served as a Nuclear Submarine Officer until 1979 and worked in the Aerospace Industry as a Test Engineer on solid fuel rocket engines for 3 years. He worked in the process control industry for approximately 15 years before joining Lawrence Livermore National Laboratory in February of 1996. Since then he has completed all course work for a Masters Degree in Computer Science and has developed requirement specifications for control systems, been a software developer for scientific applications that run on the worlds largest supercomputers, and is currently a Software Quality Engineer supporting Software Quality Assurance practices for scientific codes. He is currently the project administrator for a static analysis tool and assists various code teams in performing software defect analysis

Kimberly Ferrari Kimberly Ferrari has over 20 years of software development experience in both the commercial and government sectors. Kim has held positions from programmer to Software Engineering Lead for a team of 12 developers. Currently Kim works for the University of California at the Lawrence Livermore National Laboratory where she is the Security and Protection Group Leader and Software Engineering Lead for the Security and Protection Group. The Security and Protection group is primarily responsible for a product named Argus, which is the DOE standard physical access control and intrusion detection software. This software is deployed at multiple DOE sites and runs in a 24X7 environment with high quality standards. She programs in many languages including C#, C++ and Ada. Prior to this, she worked for Lockheed Martin. The main project she participated in was a project for NASA STGT (Second TDRSS (Tracking Data Relay Satellite System) Ground Terminal). Kim has a BS Degree in Computer Science from Shippensburg State University in Pennsylvania.

Finding 1,000 Defects Finding 1,000 Defects In 30 Days In 30 Days

In 1,000,000 Lines of In 1,000,000 Lines of CodeCode

ByByGreg PopeGreg Pope--Lawrence Livermore National LabsLawrence Livermore National LabsBill OliverBill Oliver--Lawrence Livermore National LabsLawrence Livermore National Labs

Kim FerrariKim Ferrari--Lawrence Livermore National LabsLawrence Livermore National Labs

STARWestSTARWest 20072007

UCRL-PRES-233206 This work was performed under the auspices of the U.S. Department of Energy by the University of California, Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48

Static AnalysisStatic Analysis

Done on the source code (C, Done on the source code (C, C++, Java), ideally before C++, Java), ideally before testing (or after)testing (or after)Finds problems compilers missFinds problems compilers missUnit or system levelUnit or system levelSeemed like a good Seemed like a good technology to pursuetechnology to pursueTraditional tools like Traditional tools like FlexilintFlexilinthad very high false positive had very high false positive rates.rates.

UsesUses

Newly developed codeNewly developed codeModified codeModified codeLegacy codeLegacy codeOpen source codeOpen source codeLibrary codeLibrary codeFeeder codesFeeder codesCode generatorsCode generators

How we Picked Our ToolHow we Picked Our Tool

Research the tool industry (Research the tool industry (stickymindsstickyminds))Write down the tool requirementsWrite down the tool requirementsWeight the requirementsWeight the requirementsScore the candidate toolsScore the candidate toolsRun a feasibility demonstration on the top Run a feasibility demonstration on the top scoring tools to meet specific needsscoring tools to meet specific needs

Used Used KepnerKepner Trego AnalysisTrego Analysis

Feasibility StudyFeasibility Study

Picked an open source astrophysics code Picked an open source astrophysics code with C++ and Python.with C++ and Python.

This code was similar to the actual codes This code was similar to the actual codes we plan to run the static analysis onwe plan to run the static analysis on

And the Winner WasAnd the Winner Was

A dead tie, both tools found the same number of A dead tie, both tools found the same number of defectsdefectsBoth tools met our requirementsBoth tools met our requirementsLooked at other criteria:Looked at other criteria:

Reporting capabilityReporting capabilityRefactoring and architectureRefactoring and architecturePricePriceSupportSupportCorporate structureCorporate structure

Decided to Go With the Decided to Go With the KlocworkKlocwork KK--7 Tool7 Tool

First Pilot on a real codeFirst Pilot on a real codeTypes of errors foundTypes of errors found

Memory management problemsMemory management problemsSecurity vulnerabilitiesSecurity vulnerabilitiesCoding styleCoding styleGeneral defectsGeneral defectsothersothers

Training ClassTraining Class

Users (2 days)Users (2 days)Administrators (5 days)Administrators (5 days)Used classroom with desktop computersUsed classroom with desktop computers

How to Use the ToolHow to Use the Tool

Build the Source Code Build the Source Code The tool learns how to build the codeThe tool learns how to build the codeRun the tool on the specially built codeRun the tool on the specially built code

Resources RequiredResources Required

P4 2GHz or faster CPUP4 2GHz or faster CPU2GB RAM2GB RAM100 GB Disk100 GB DiskAccess to Source CodeAccess to Source Code

Case Study 1Case Study 1

137 KSLOC C137 KSLOC CPortable Software Development APIPortable Software Development APIUsed across multiple platforms in Scientific Used across multiple platforms in Scientific Computing ApplicationsComputing ApplicationsFound 82 ABR defects Found 82 ABR defects (array bounds overflow)(array bounds overflow)

All ABR defects fixed in a few days All ABR defects fixed in a few days compared to multiple weekscompared to multiple weeks

Categories of ErrorsCategories of Errors

DefectsDefectsHeader File ProblemsHeader File ProblemsLowLow--Level Interface ProblemsLevel Interface ProblemsSecurity VulnerabilitiesSecurity Vulnerabilities

Defect types IncludedDefect types Included

Coding StyleCoding StyleConcurrencyConcurrencyMemory Management ProblemsMemory Management ProblemsNull Pointer DereferenceNull Pointer DereferenceUse of Uninitialized DataUse of Uninitialized Data

Header File ProblemsHeader File Problems

Cycle in Include FilesCycle in Include FilesMissing Include FilesMissing Include FilesUnnecessary Include FilesUnnecessary Include FilesMissing Direct Include FilesMissing Direct Include Files

LowLow--Level Interface ProblemsLevel Interface Problems

Object defined in header and declared in Object defined in header and declared in headerheaderExtern object defined in headerExtern object defined in headerStatic object defined in headerStatic object defined in headerDuplicated headerDuplicated headerMany OthersMany Others

Security VulnerabilitiesSecurity Vulnerabilities

Access ProblemsAccess ProblemsBuffer OverflowBuffer OverflowDNS SpoofingDNS SpoofingIgnored Return ValuesIgnored Return ValuesInjection FlawsInjection FlawsInsecure StorageInsecure StorageUnvalidated User InputUnvalidated User Input

Coding Style ExampleCoding Style ExampleAssignment in Condition (boolean)

1 class A {2 void foo(bool,bool);3 };

4 void A::foo(bool flag,bool OK)5 {6 If (flag = OK) { //boolean variable is assigned7 }8 }

Klocwork produces a defect report like the following:

test6.cxx:6:Warning:assignment in boolean condition

Concurrency ExampleConcurrency ExampleMissing Unlock for Variable

1 void no_unlock(pthread_mutex_t *mutex) {2 if (cond())3 pthread_mutex_lock(mutex);4 else {5 pthread_mutex_lock(mutex);6 pthread_mutex_unlock(mutex);7 }

8 printf(\"missing unlock\");

9 }

Klocwork produces a defect report like the following:

no_unlock.cc:9:Error:Variable '*mutex' locked on line 3 was notunlocked

Memory Management ExampleMemory Management ExampleMemory Leak (mlk.must)

1 class A {2 void foo();3 };4 void A::foo()5 {6 int *ptr = new int;7 *ptr = 25;8 ptr = new int;9 *ptr = 35;10 }

Klocwork produces a defect report like the following:

mlk.must.cc:8:Error:Memory leak. Dynamic memory stored in 'ptr' allocatedthrough function 'new' at line 6 is lost at line 8mlk.must.cc:10:Error:Memory leak. Dynamic memory stored in 'ptr'allocated through function 'new' at line 8 is lost at line 10

Memory Management ExampleMemory Management ExampleMemory Leak (mlk.might)

1 void foobar(int i)2 {3 char *p = (char*)malloc(12);4 if(i) {5 p = NULL;6 }7 return;8 }

Klocwork produces a defect report like the following:

mlk.might.c:7:Error:Possible memory leak. Dynamic memory stored in 'p'allocated through function 'malloc' at line 3 can be lost at line 7

Null Pointer DereferenceNull Pointer Dereference

1 void npd_gen_must() {

2 int *p = 0;

3 *p = 1;

4 }

Klocwork produces a defect report like the following:

npd.gen.must-ret-expl.c:3:3: Error(1):NPD.GEN.MUST: Null pointer 'p' that comes from line 2 will be dereferenced at line 3

Might be a DereferenceMight be a Dereference1 void npd_gen_might(int flag, char *arg) {

2 char *p = &arg;

3 if (flag) p = getNull();

4 if (arg) {p = arg;}

5 xstrcpy(p,"Hello");

6 }

7 Void xstrcpy(char *dst, char *src) {

8 if(!src) return;

9 dst[0] = src[0];

10 }

Klocwork produces a defect report like the following:npd.gen.might-ret-call.c:5:8: Error(1):NPD.GEN.MIGHT: Nullpointer 'p' that comes from call to function 'getNull' at line3 may be dereferenced by passing argument 1 to function'xstrcpy' at line 5.

Array Bounds OverflowArray Bounds Overflow

1 int main() {

2 char fixed_buf[10];

3 sprintf(fixed_buf,"Very long format string\n"); return 0;

4 }

Klocwork produces a defect report like the following:4:Critical:Buffer overflow, array index of 'fixed_buf' may be outside the

bounds. Array 'fixed_buf' of size 10 declared at line 3 may use index values 0..24

Case Study 2 Case Study 2 -- OverviewOverview345 345 KSLOCsKSLOCs Borland C++ Borland C++ Version 6 Version 6 Software controlling user Software controlling user interfaces and gateways for interfaces and gateways for Physical security systemPhysical security systemC++ Code has been running:C++ Code has been running:

24/7 for four years24/7 for four yearsVariety of Applications Variety of Applications

Alarm Display Alarm Display –– Very criticalVery criticalConfiguration Editors Configuration Editors –– less criticalless critical

At multiple sitesAt multiple sitesCode very stable with rare Code very stable with rare unexplained errorsunexplained errors

Case Study 1 Case Study 1 –– Bugs foundBugs foundFound 285 bugsFound 285 bugs

Critical: 56 (Null Pointer dereference, Buffer overflows) Critical: 56 (Null Pointer dereference, Buffer overflows) Severe: 14 (use of Free Memory) Severe: 14 (use of Free Memory) Error: 193 (Memory leaks and Uninitialized variablesError: 193 (Memory leaks and Uninitialized variablesWarning: 21 (Inconsistent case labels)Warning: 21 (Inconsistent case labels)

Able to fix/resolve 50+ of these bugs in a few Able to fix/resolve 50+ of these bugs in a few hours.hours.

Some Borland generated unused code being flaggedSome Borland generated unused code being flaggedGoal is to resolve all 263 bugs before next release with minimalGoal is to resolve all 263 bugs before next release with minimal effort. effort.

Cost / Savings is Huge vs. finding these bugs the old fashioned Cost / Savings is Huge vs. finding these bugs the old fashioned way.way.Expect to be able to fix all 263 bugs with 40 hours developer tiExpect to be able to fix all 263 bugs with 40 hours developer time. me. Traditionally would take weeks / months to fix these.Traditionally would take weeks / months to fix these.

SeveritiesSeverities

CriticalErrorSevereUnexpectedInvestigateWarningSuggestionStyleReviewInfo

StatusStatus

AnalyzeAnalyzeIgnoreIgnoreNot a ProblemNot a ProblemFixFixFix in Next ReleaseFix in Next ReleaseFix in Later ReleaseFix in Later ReleaseDeferDeferFilterFilter

False PositivesFalse Positives

MistakesMistakesContext AssumptionsContext AssumptionsMissing InformationMissing InformationDoes not have the benefit of dataDoes not have the benefit of data

PrePre--FilteringFiltering

Do not want to overwhelm developersDo not want to overwhelm developersSystem constraints and context matterSystem constraints and context matterFilters can be tailoredFilters can be tailored

MetricsMetrics

117 metrics117 metricsCyclomatic complexityCyclomatic complexityHalstead scienceHalstead scienceCode MetricsCode Metrics

Lines of codeLines of codeComments densityComments densityStatements per moduleStatements per moduleOO metricsOO metrics

ArchitectureArchitecture

Documents designDocuments designFinds cyclic clustersFinds cyclic clustersHelp fix cyclic clustersHelp fix cyclic clusters

EnforceEnforce

Command line version of static analyzerCommand line version of static analyzerFinds defects on a per file basisFinds defects on a per file basisDevelopers can use before putting code in Developers can use before putting code in repositoryrepository

Getting Developer Buy InGetting Developer Buy In

Easy to getEasy to getSpeeds up trouble shootingSpeeds up trouble shootingFinds tricky problemsFinds tricky problems

Tunable OptionsTunable Options

Specify defects to look forSpecify defects to look forChange defect severity levelsChange defect severity levelsCreate rules that show up as defectsCreate rules that show up as defectsEvolutionary changesEvolutionary changesIncremental code improvementIncremental code improvement

SurprisesSurprises

The tool worked out of the boxThe tool worked out of the boxOur code did not break the toolOur code did not break the toolFeature richness was beyond expectationsFeature richness was beyond expectationsDevelopers and Users like the toolDevelopers and Users like the toolEasy to installEasy to installGood training and technical supportGood training and technical support

Lessons LearnedLessons Learned

Assign primary gurus for the toolAssign primary gurus for the toolPrePre--filtering takes some timefiltering takes some timeDeveloper experience needed to use toolDeveloper experience needed to use toolOther projects will want to use the toolOther projects will want to use the tool

SummarySummary

Even a caveman can find the defectsEven a caveman can find the defectsBut it takes a developer to analyze and fix But it takes a developer to analyze and fix themthem