1 how to apply static and dynamic analysis in practice © software quality week ‘97 how to apply...

1How to Apply Static and Dynamic Analysis in Practice©

Software Quality Week ‘97

How to Apply Static and Dynamic Analysis in Practice

-

Otto Vinter

Manager Software Technology and Process Improvement

Brüel & Kjaer Sound & Vibration Measurement

email: [email protected]


Company Profile

Brüel & KjaerSkodsborgvej 307, DK-2850 Naerum, Denmark

Tel: +45 4580 0500, Fax: +45 4580 1405

High-Precision Electronic Instrumentation for

• Sound

• Vibration

• Condition Monitoring

• Gas Measurements


The PET Process Improvement Experiment

The Prevention of Defects through

Experience-Driven Test Efforts

(PET)

PET Objectives• Extract knowledge on frequently occurring problems in

the development process for embedded software

• Change the development process by defining the optimum set of methods and tools available to prevent these problems reappearing

• Measure the impact of the changes in a real-life development project

Funded by the CEC ESSI Programme (Project no. 10438)


Defect Analysis from Error Logs

Results of the Analysis of Error Reports• no special bug class dominates embedded software

development

• requirements problems, and requirements related problems, are the prime bug cause (36%)

• problems due to lack of systematic unit testing is the second largest bug cause (22%)


The PET Experiment

Actions to Improve Unit Testing • introduction of static and dynamic analysis

• host/target tools

• basic set of metrics

Comparative Analysis• assess a trial-release of a product

• remove all static analysis anomalies

• increase test coverage to industry best practice

(branch coverage > 85%)

• measure the effect after production release


Results of the PET Experiment

Applying Static and Dynamic Analysis:• 75% reduction in production-release bugs

- compared to trial-release

• 70% requirements bugs in production-release

- because bugs in other categories dropped

• 46% increase in testing efficiency

- bugs found per hour of testing

• app. 75% payback on tools and training

- reduced maintenance effort


Where to Apply Static and Dynamic Analysis

Test Activities• Requirements Verification

• Design Verification

• Unit Test

• Integration Test

• Acceptance Test

• Beta Test

} Static and Dynamic Analysis


Static Analysis in Unit Testing

• Static analysis is performed

- before unit testing starts

- and after any modification to unit

• Complexity Metrics

- use McCabe complexity metric as an indication

- take loop nesting into account

- use visual inspection of data flow graphs• for perceived complexity

• focused code inspection

• Analysis reports are reviewed / inspected

- determine special actions

- determine test coverage profile

- test coverage levels


Static Flowgraph (McCabe = 10)


Static Flowgraph (McCabe = 20)


McCabe = 46


McCabe = 55


Visualisation Theme

If it looks right, it could be wrong,

but if it looks wrong, it can’t be right.

Uffa Fox, yacht designer



Data Flow Analysis• Undefined but referenced variables

- must be removed from the code

• Variables defined but not used in scope

- should be documented in the code

• Variables redefined with no use in between

- should be documented, if not close to

• Suspicious casting (loss of information, mismatches)

- if unavoidable, use explicit casts

- should be documented

• Global variable anomalies (local overrides global)

- should be removed



• Establish project standards for code

- no goto’s, breaks in all cases, procedure size, etc.

• Remove unused items

- unreachable code (incl. procedures)

- declared variables that are not used

• Address design architecture problems

- use visual inspection of control flow graphs

- procedure parameter anomalies

(referenced only, defined only, not used)


Static Analysis Results

Data Flow Anomalies

Type of Bug Distribution

• Use of uninitialised variable: 15 %

• Variable defined but not used in scope: 24 %

• Variable redefined with no use in between: 36 %

• Parameter mismatch: 6 %

• Unreferenced procedure: 18 %

• Declared but not used variable: 0 %

• Other types of static bugs: 0 %


Dynamic Analysis in Unit Testing

Test coverage strategy • based on information from static analysis

• determined on a unit by unit basis

Unit test coverage must take into consideration• perceived complexity

- based on a visual inspection

• degree and type of reuse

• criticality

- product and project issues

• developer experience

- domain, product, and unit

• bug and change request history


Dynamic Analysis in Unit Testing

• Dynamic analysis reports are reviewed / inspected

- after unit testing (and a static analysis)

- and after any modification to unit

• Test coverage achieved

- test coverage level (e.g. 85% of branches)

- visual inspection of dynamic flow graphs

- determine special actions

• Static and dynamic analysis reports are saved in the configuration system along with the code


Problems when Introducing Static and Dynamic Analysis

Static analysis• an extreme amount of anomalies to evaluate

- more than 1 per 60 lines of code

Dynamic analysis• best suited for automatic testing

• documented test cases

• no automatic test case generation

- only structure based

• test cases are selected by normal unit testing techniques

- equivalence partitioning, boundary value analysis ...


Problems when Introducing Static and Dynamic Analysis

Tools break• code expands when instrumented (40%)

• large amount of tracing data is generated (Mbytes)

So• instrument only parts of system

- handle frequently used parts separately

• run the regression test suite several times

- with different parts instrumented


Conclusions on Static/Dynamic Analysis

Performance Improvement • an efficient way to remove bugs

• marginal delay on trial release date

• marginal increase in the testing resources required

• immediate payback on tools, training & implementation

• increased quality and reduced maintenance costs

• increased motivation

• applicable to the whole software development industry, incl. embedded software

1 how to apply static and dynamic analysis in practice © software quality week ‘97 how to apply...

Documents