ocp analysis and adaptation results rockwell collins atc iowa state university may, 2003

OCP Analysis and Adaptation ResultsOCP Analysis and Adaptation Results

Rockwell Collins ATC

Iowa State University

May, 2003

May 03 / SLIDE 2

Overview

Issues list

Pattern catalog

Tool support

OCP results

Plans

May 03 / SLIDE 3

Issues list

Identifies 83 issues related to software for high assurance systems based on:

— Handbook for Object-Oriented Technology in Aviation (OOTiA), Proceedings of FAA/NASA Workshops 1 & 2 on Object-Oriented Technology in Aviation, 2003

— Guide to the Certification of Systems with Embedded Object-Oriented Software , Aerospace Vehicle Systems Institute (AVSI), 2001

— Gary Daugherty, Certifiability of CORBA, Rockwell Collins Technical Report, 2001

— “Object-Oriented Technology (OOT) In Civil Aviation Projects: Certification Concerns”, CAST-4, January 2000, http://av-info.faa.gov/software

— Leanna K. Rierson. Object-Oriented Technology (OOT) in Civil Aviation Projects: Certification Concerns, Federal Aviation Administration, Washington, D.C.

— Ye Wu and Jeff Offutt. Maintaining Evolving Component-Based Software with UML, George Mason University Technical Report, 2002

— Barbara Liskov and Jeanette Wing. “A Behavioral Notion of Subtyping”, ACM Transactions on Programming Languages and Systems, 16(6): 1811-1841, November 1994.

May 03 / SLIDE 4

Issues list

Includes both design level and language level issues:

— Bertrand Meyer. “Applying design by contract.” IEEE Computer 25(10):40-51, October 1992.

— Guide to the use of the Ada Programming Language in high integrity systems , ISO/IEC TR 15942, 2000

— John Barnes. High Integrity Ada: The SPARK Approach, Addison-Wesley, Harlow, England, ISBN 0-201-17517-7, 1997

— Use of the C++ Programming Language, Certification Authorities Software Team (CAST), position paper, CAST-8, January, 2002

— David W. Binkley. C++ in Safety Critical Systems, NIST, November 1995

— Boeing, BCA Technical Standard for the Use of C++ in Airborne Software, D6-82801, 2002

— Scott Meyers. Effective C++, 2nd edition, Addison-Wesley, Reading, MA, 1998

— Scott Meyers. More Effective C++, Addison-Wesley, Reading, MA, 1996

— Ian Joyner. C++??: A critique of C++ and Programming Language Trends of the 1990s, 1996

— Marshall P. Cline and Greg A. Lamow. C++ FAQs, Addison-Wesley, Reading, MA, 1995.

May 03 / SLIDE 5

Issues list

And issues related to verification and testing:

— Jeff Offutt, et al. A Fault Model for Subtype Inheritance and Polymorphism, Twelfth IEEE International Symposium on Software Reliability Engineering (ISSEE ’01), 2001

— Robert T. Alexander et al. Fault Detection Capabilities of Coupling-based OO Testing, Colorado State University Technical Report, 2001

— Robert V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison-Wesley, Reading, MA, 2000.

— Robert T. Alexander and Jeff Offutt. Analysis Techniques for Testing Polymorphic Relationships, In Thirteenth International Conference on Technology of Object-Oriented Languages and Systems (TOOLS30), 1999.

May 03 / SLIDE 6

Pattern catalog

Forty two design patterns intended to address the most significant issues

Ten of these are abstract (define the problem)

Thirty two are concrete (provide specific/alternative solutions)

Specified in “Gang of Four” format

With formal specifications (to support automation) in XPSL/XML

And well defined relationships between patterns (extends, requires, excludes, etc.)

Developed incrementally

May 03 / SLIDE 7

Pattern catalog

Increment one

— Development of eXtensible Pattern Specification Language (XPSL/XML)

— Development of patterns for:

Control flow,

Data integrity,

Pointer integrity,

Unintended side effects,

Multiple inheritance

May 03 / SLIDE 8

Pattern catalog

Increment two

— Extension of XPSL (in terms of specific queries/views/transformations)

— Development of patterns for:

Contracts and subtyping,

Coupling and cohesion,

OO metrics,

Dynamic resource allocation,

Safety critical language subsets

May 03 / SLIDE 9

Tool support

For 8 patterns addressing 10 issues

Implementation based on ISU’s KCS toolset

Programs represented in eXtensible Common Intermediate Language (XCIL/XML)

Framework to support XPSL queries, views and transformations

Developed incrementally

May 03 / SLIDE 10

Tool support

Increment one

— Development of XCIL

— Translation of C++ to XCIL,

— Verification of translation,

— Analysis framework (queries and views),

— Analysis support for selected increment one patterns,

— Focus on “local” patterns, affecting individual OCP components

May 03 / SLIDE 11

Tool support

Increment two

— Adaptation framework (transformations),

— Adaptation support for selected increment one patterns,

— Analysis support for selected increment two patterns,

— Focus on “global” patterns, requiring an analysis of the entire OCP

May 03 / SLIDE 12

Technical issues

Size of the OCP

Manual nature of the review process

Lack of tool support for HA analysis and transformation

Lack of user control (user defined patterns)

Regression testing (first do no harm)

Need to integrate with other tools

May 03 / SLIDE 13

Technical Approach

Catalog

of patterns for high assurance software

KCStoolset

(for translation, analysis, and adaptation)

Modeling tools

(GME, Rose, etc.)

OCP and other software source (in C++, Java, other languages)

HA SW

XML/XMI models

Patterns in XML/XMI

Adapted OCP, and other software source (in C++, Java, other languages.)

Models

SW

HA Models

Adapted XML/XMI models

Developer

Review/editIssues List

T-VEC

Regression testing

T-VEC formal specs

May 03 / SLIDE 14

OCP results

OCP 2.2

http://dirac.ee.iastate.edu/ocp/

Issues

Patterns

Matches

Adaptations

May 03 / SLIDE 15

Classification of problems

Hard run time error— An error appears in the current code. A fault may occur when the erroneous code is

executed.

Lurking run time error— A hard run time error may result if additional correct code is introduced.

Potential run time error— A problem that may fall into one of the run time error categories – although we don’t

know which.

Obstacle to analysis— The code cannot be analyzed in some manner typically required for “certification”

(potentially masking/hiding run time errors).

Maintenance risk— The code is correct, but subsequent changes are likely to introduce errors.

May 03 / SLIDE 16

Increment one

Addresses issues related to

— Implicit type conversion

— Violations of strong typing

— Control flow

— Compiler generated constructors, operators

— Pointer integrity

May 03 / SLIDE 17

Issue 4: Implicit type conversion

Implicit type conversion may involve a potential loss of data or precision

— “Type conversions shall not be made from a longer to a shorter (number of bits) type”

“Type conversion shall not be made from an unsigned type to a signed type of the same size”

“Type casting shall be explicitly coded with use of the keywords const_cast, static_cast and reinterpret cast, and shall not be left as an operation only implied by the code (e.g., an assignment statement)”

Boeing. BCA Technical Standard for the Use of C++ in Airborne Software, D6-82801

— “Loss of information rule. To help ensure correctness, any conversions that may result in loss of data or data accuracy and precision should be explicit, should be conspicuously marked (identified) in the program source code, and should only be permitted after thorough review and analysis of potential adverse effects.” [OOTiA Position Paper on Type Conversion]

May 03 / SLIDE 18


Implicit type conversion between logically unrelated types (e.g., integer and boolean, character and integer) may violate the principles of strong typing

— “Unchecked conversion rule. To help ensure type safety and verification, all unchecked conversions should be explicit, be conspicuously marked (identified) in the program source code, and be permitted only after thorough review and analysis of potential adverse effects.” [OOTiA Position Paper on Type Conversion]

— “Scalar type rule. Implicit conversions between scalar types should only be permitted when the new type defines a set of operations that is logically a subset of those defined by the old type.” [OOTiA Position Paper on Type Conversion]

— “Supertype rule. To help ensure type safety and verification, all implicit type conversions involving references to class instances should only represent a conversion from a subtype to one of its supertypes.” [OOTiA Position Paper on Type Conversion]

May 03 / SLIDE 19


Implicit type conversion may incur a significant, hidden overhead (e.g., due to implicit constructor calls)

— “Low overhead rule. To ensure that the developer does not introduce significant hidden overhead and performance issues, implicit type conversions should only be used for predefined scalar types and references to structured types.” [OOTiA Position Paper on Type Conversion]

Very hot to Hot, rank = 3

May 03 / SLIDE 20

Pattern: Safe Type Conversion

Restricts implicit conversions to those that are type safe, low overhead, and guarantee no loss of data or precision, i.e.

— Conversion of a smaller integer type to larger integer type whose range subsumes the original, with sign extension of two’s complement values

— Conversion from a smaller IEEE 754 floating point type to a larger IEEE 754 floating point type

— Conversion from a reference type S to a reference type T, provided S is a subtype (subclass or subinterface) of T

— Conversion from an Array/Vector whose element type is a reference type SC to an Array/Vector whose element type is a reference type TC, provided that SC is a subtype of TC

— Conversion from a general collection type whose element type is a reference type to a collection of the same type whose element type is a reference type TC, provided that SC is a subtype of TC

May 03 / SLIDE 21

Pattern: Safe Type Conversion

Specifies that type conversions between logically unrelated types (e.g., integer and boolean, character and integer) must be explicit

Specifies that type conversions that may incur a significant overhead (e.g., that use constructors or conversion operators to perform type conversion) must be explicit

Specifies that supertype to subtype conversions (down casts) must either be proven to be correct (by means of analysis) or must be guarded by a run time check

May 03 / SLIDE 22

Matches

No implicit conversions between integer and float

58,675 implicit conversions of a larger value to a smaller value

352 implicit conversions of a signed value to an unsigned value

87 implicit conversions of an unsigned value to a signed value

Matches represent potential run time errors

May 03 / SLIDE 23

Adaptations

Subject to review

Need to consider which implicit conversions are “high overhead”

Need report on types converted from/to

Need report on all writes to variables involved in conversions

Need to consider whether the actual range of values of the source type fits within the range of values permitted by the destination type

Need to consider whether the converted values are treated as numeric values (whether signed/unsigned matters)

Introduce user defined types or typedefs

Use bool instead of integer type

Use single size, avoid conversions

Use explicit conversions

Introduce run-time range checks (when not performed by C++)

May 03 / SLIDE 24

To do

Identify “low overhead” vs. ”high overhead” implicit conversions

Provide the suggested reports as an aid to the review process

Provide for feedback (annotations) to feed the toolset information derived from reviews

Look for all conversions between logically unrelated types (not just integer/float)

Look for explicit conversions that involve a potential loss of data or precision

Look for down casts (supertype to subtype conversions)

Look for definitions of implicit conversion operations

See pattern for details

May 03 / SLIDE 25

Issue 17: Break in switch

Error prone, especially when cases are later added or removed

— “Each case in a switch, which contains any executable code, shall be terminated with a break.”

“This eliminates the dubious practice of falling through a case to use code in a succeeding case for common processing. It is too easy to lose track of the correct program flow in code maintenance.”

Mandatory for DO-178B software levels A through D


Hot, Rank = 4

May 03 / SLIDE 26

Pattern: Fall through switch case

Multiple cases may be associated with the same code

However, each block of code associated with a case must end in a ‘break’

This includes the last case of the switch statement (in case other cases are later added after it)

May 03 / SLIDE 27

Matches

461 fall through cases

Matches represent maintenance risks

May 03 / SLIDE 28

Adaptations

Refactor code to either:

— introduce a method called by both cases (when the shared code is cohesive)

— introduce an inlined call (when the shared code is short and cohesive)

— introduce a conditional (if statement or nested case) when the shared code is not cohesive

— introduce replicated code (when the shared code is short but not cohesive)

May 03 / SLIDE 29

To do

Support one or more of the suggested adaptations

Support user selection of the adaptation to be applied

May 03 / SLIDE 30

Issue 19: Use of goto’s

Complicates analysis

— The use of goto should be excluded because it complicates flow analysis (FA), symbolic analysis (SA), timing analysis (TA), and structural testing (ST).

The use of goto is regarded as both unnecessary, and undesirable compared to the use of other control structures.

It should not appear in well written code.

Guide for the Use of the Ada Programming Language in High Integrity Systems , ISO/IEC TR 15942

R. Chapman. SPARK – A state-of-the-practice approach to the Common Criteria implementation requirements, Praxis Critical Systems

Hot, Rank = 4

May 03 / SLIDE 31

Pattern: Goto-free execution

The pattern forbids all explicit goto’s

But does not forbid implicit goto’s (such as those implied by the use of if statements, switch statements, loops, or exception throws)

May 03 / SLIDE 32

Matches

1,744 explicit gotos

Matches represent false positives if the goto’s are generated by macros as a substitute for exception throws (as many apparently are)

These should be OK, assuming exception handling/error handling jumps are themselves allowed

Otherwise matches represent maintenance risks, obstacles to analysis, and potential run time errors

May 03 / SLIDE 33

Adaptations

Subject to review

Unless used as a substitute for exceptions, refactor code to:

— replace goto with structured control flow (conditional, loop, break, continue, error return, other common forms)

If used as a substitute for exceptions, apply exception/error handling patterns:

— Enforcing the disciplined use of exceptions (per Bertrand Meyer)

— Ensuring all reported errors (thrown exceptions) have handlers – as in Java

— Eliminate macros in favor of pattern substitution, allowing a variety of error reporting/error handling strategies

May 03 / SLIDE 34

To do

Provide support to for “macro recognition”

Develop/apply error handling patterns, replacing the use of macros for this purpose

May 03 / SLIDE 35

Other related patterns from the catalog

Jump-free execution

No user thrown exceptions

Top level handler for predefined exceptions

May 03 / SLIDE 36

Issue 25: Compiler generated operations

Compiler generated operations have default meanings that may be inappropriate for a given class.

— Since the default constructor, copy constructor, destructor, and the operators operator=, operator&, and operator, (i.e., operator<comma>) all have default meanings; they should be explicitly defined in every class. To avoid unwanted implicit calls to these constructors and operators, declare them private. – [Binkley, C++ in Safety Critical Systems]

— Law of The Big Three: “If a class needs a destructor, or a copy constructor, or an assignment operator, it needs them all” … “During code reviews and debugging sessions, we have traced many core dumps back to violations of The Law” [Cline and Lomow, C++ FAQs]

— “Declare a copy constructor and an assignment operator for classes with dynamically allocated memory” [Scott Meyers, Effective C++, 2nd edition, item 11]

— Declare a constructor for all classes with an user defined invariant – Based on the FAA/NASA OOTiA Handbook complete initialization rule, invariant rule and invariant assertion rule

Hot, Rank = 4

May 03 / SLIDE 37

Pattern: Compiler generated operations

Constructor rule

— If we have a user defined invariant, we need a user defined constructor

— All fields without initial values must be set by a user defined constructor

Copy constructor rule

— If the class has dynamically allocated components, we need a user defined copy constructor

Assignment rule

— If the class has dynamically allocated components, we need a user defined assignment operator

Destructor rule

— If the class has dynamically allocated components, we need a user defined destructor

Consistency rule

— If the class has a user overrides the default behavior of its destructor, copy constructor, or assignment operator, then it should override the behavior of all three

May 03 / SLIDE 38

Matches

1,451 classes with compiler generated default constructors

2,078 classes with compiler generated copy constructors

2,100 classes with compiler generated “=” operators

2,218 classes with compiler generated “&” operators

Matches are subject to application of the previous rules

Violations of the rules represent likely run time errors or lurking run time errors

May 03 / SLIDE 39

Adaptations

The user is responsible for introducing the necessary operations, while the tool is responsible for checking the user supplied code

In addition the tool may introduce run-times checks of constructor postconditions, to ensure the class invariant is established

Checks of user supplied constructor

— Run time check of invariant as postcondition

— Check that all fields are initialized

Checks of user supplied copy constructor

— Run time check of invariant as postcondition

— Check that all dynamically allocated components are replicated (deep copy)

Check of user supplied assignment operator

— Check that all dynamically allocated components are replicated (deep copy)

Check of user supplied destructor

— Check that all dynamically allocated components are deallocated

May 03 / SLIDE 40

To do

Provide tool support for associated rules

Support the selective insertion of code to check constructor postconditions (class invariants)

May 03 / SLIDE 41

Issue 30: Pointer arithmetic

Use of pointer arithmetic

Bounds checks and invalid pointer values

— “Operational software shall not include pointer arithmetic.”

“In general, there is little justification for going to the effort of designing code to do pointer arithmetic. The performance gained by doing pointer arithmetic is seldom worth the increased risk of error.”

Explicit pointer value calculation is prohibited for DO-178B software levels A through D


— “Direct manipulation of pointer values (pointer arithmetic) should be prohibited … Java and C# can be used as a guide. What cannot be done is these languages generally should not be allowed. ” [Daugherty, Certifiability of CORBA]

Obstacle to analysis

Hot, Rank = 5

May 03 / SLIDE 42

Pattern: No pointer arithmetic

A pointer can be used only as a substitute for an opaque reference, i.e., in XCIL:

— On the left hand side of an assignment

— In a reference to the value of an attribute

— In a reference to the value of an array element

— In a reference to the length of an array

— In reference to the run time type of an object

— As the target of a dispatching call

— As an argument to a call

— As a value returned by a method

— As the thrown value of an exception

— In a reference to an object to be locked/unlocked

May 03 / SLIDE 43

Matches

470 instances of pointer arithmetic

Matches represent potential run time errors and obstacles to analysis

May 03 / SLIDE 44

Adaptations

Replace each array element pointer with an array base pointer and index, performing all arithmetic on indices rather than pointer values

Selectively introduce run-time bounds checks (typically not performed by C++ compilers)

Restrict other uses of pointers in XCIL to the cases given (in which the use of a pointer is compatible with the use of an opaque reference), requiring the user to make associated changes

May 03 / SLIDE 45

To do

Complete tools-related work on transformations to eliminate arithmetic involving array pointers

Develop strategy for incremental introduction of changes

— For examnple, support a conversion from array pointer types to a type representing an array reference + index (class ArrayReference)

— Explicitly apply this conversion to all parameters passed into/out of the portion of the software to be changed

— Incrementally adapt the entire OCP and its support libraries

May 03 / SLIDE 46

Issue 47: Use of union (untagged variant records)

Can lead to errors that cannot be caught by the compiler and violations of strong typing

— “The type union should not be used”

“union is an untagged type. Its use can lead to data type errors that cannot be trapped by the compiler”

Prohibited for DO-178B software levels A through D


— “Use a class hierarchy with virtual functions in place of a union as it provides type checking of the data stored in the union” [Binkley, C++ in Safety Critical Systems]

Hot, Rank = 6

17 April 03 / SLIDE 46

May 03 / SLIDE 47

Pattern: Non-discriminated union

The pattern forbids all untagged (non-discriminated) unions

May 03 / SLIDE 48

Matches

9 declarations of untagged union types

Matches represent maintenance risks, obstacles to analysis, and potential run time errors

May 03 / SLIDE 49

Adaptations

Replace untagged union with

— class-subclass

— class-state = tagged union (discriminated record)

May 03 / SLIDE 50

To do

Add abstract pattern and two concrete subpatterns (for conversion to class-subclass and class-state) to catalog

Develop tool support for the associated transformations (giving the user a choice of which to apply)

May 03 / SLIDE 51

Issue 56: Implicit method return

Without an explicit return statement, the developer’s intent may be unclear, control flow may be more difficult to follow, and it may be unclear what values may be returned by a given method

— “Return from a function shall be accomplished only by an explicit return statement, i.e., the code shall not use a default or implicit return.”

“Not writing return in a function relies on the default return. Explicit return in every branch that terminates has two benefits. It simplifies the maintenance task of understanding the flow of control, and makes it completely clear the possible values that are returned.”

Mandatory for DO-178B software levels A through D


Other, Rank = 9

May 03 / SLIDE 52

Pattern: No implicit method value return

All execution paths associated with a method that returns a value must end with an explicit return

May 03 / SLIDE 53

Matches

None found

May 03 / SLIDE 54

Unimplemented local patterns

Enforce invariant (Issue 13)

Closed argument list (Issue 15)

No dangling pointers to local variables

No assignment statement arguments (Issue 40)

Loop termination (Issue 44)

No user thrown exceptions (Issue 49)

Top level handler for predefined exceptions (Issue 49)

May 03 / SLIDE 55

Unspecified local patterns

Undefined and implementation defined behavior (Issue 2)

Incomplete construction (Issue 12)

Use/overriding of default parameter values (Issue 16)

Accidental mismatch of software and processor (or compiler), e.g. with respect to desired precision and accuracy (Issue 27)

Overloading and implicit type conversion involving calls to overloaded methods (Issue 28)

Side effects in expressions and arguments (Issue 50)

Side effects in functions that return a result (other than an error code) (Issue 51)

Template verification (Issue 52)

Verification of inlined methods (Issue 53)

Class cohesion (Issue 66)

May 03 / SLIDE 56

Increment two

Addresses issues related to

— Lack of precise requirements/interface specifications

— Integration errors at class boundaries

— Supertype/subtype compatibility

— Use of dynamic dispatch

— Dynamic dispatch during object construction

— Safety critical language subsets

May 03 / SLIDE 57

Issue 1: Precise requirements/interface specifications

Lack of precise requirements/interface specifications

— “If we don’t state what a module should do, there is little likelihood that it will do it (The law of excluded miracles)” – Bertrand Meyer

— Programming by contract: “The relationship between a class and its clients [can be] viewed as a formal agreement [contract]. Only through such a precise definition of every module’s claims and responsibilities can we hope to attain a significant degree of trust in large software systems” [Bertrand Meyer, Object-oriented Software Construction]

Integration errors at class interface boundaries

— “The compositional relationships of inheritance and aggregation, especially when combined with polymorphism, introduce new kinds of integration faults” and “increase the difficulty of the detection of faults that result from the integration of components” – Roger T. Alexander

— In OO systems, “complexity is relocated to the connections among components” and “there is less static determinism (many faults can now only be detected at runtime)” - Offutt

Very hot, Rank = 1

May 03 / SLIDE 58

Issue 6: Supertype/subtype compatibility

“The single largest source of design errors is improper inheritance” [C++ FAQs, Issue 119]

Faulty intuition

— Intuition with regard to classification can lead to superclass-subclass relationships that violate the Liskov Substitution Principle (a.k.a. “Square peg in a round hole”) [Binder, Testing Object-Oriented Systems]

Failure to implement to inherited specifications

— A subclass may fail to follow the Liskov substitution principle, breaking the promises to clients made by superclasses (a.k.a. “Naughty children”) [Binder, Testing Object-Oriented Systems, p. 503]

— Subclass methods may fail to act on superclass attributes that are modified by methods they override (a.k.a. “State Definition Anomaly (SDA)”) or may assign them inconsistent values (a.k.a. “State Defined Incorrectly (SDI)”) – Offutt

— A method may call other overridden (possibly private or protected) methods that break the promises made by their parent methods (a.k.a. “Indirect Inconsistent State Definition (IISD)” or “State Visibility Anomaly (SVA)”) - Offutt

May 03 / SLIDE 59


Failure to implement to inherited specifications

— “Revoking an inherited public service is evil” [C++ FAQs, Issue 122]

— “It’s legal for an overridden method to do nothing if, and only if, the specification of the service in the base class tells users that it might do nothing. Without such an adaptable specification in the base class, doing nothing is like false advertising.” [C++ FAQs, Issue 127]

Misuse of inheritance to share code/data

— Inheritance can be misused to support code sharing without consideration of substitutability and the rules for behavioral subtyping (i.e., LSP)

Programmer specified optimizations

— Some languages make it easy to break the inheritance and subtyping guidelines in order to perform programmer specified optimizations (e.g., by omitting the C++ “virtual” keyword to avoid the overhead of dynamic dispatch).

May 03 / SLIDE 60


Accidental overriding/hiding

— It is important that the overriding of one operation by another always be intentional rather than accidental (a.k.a. “Accidental override”) [Binder, Testing Object-Oriented Systems]

— The overriding of an attribute in a subclass can hide a superclass attribute with the same name resulting in an inconsistent state when a sequence of superclass and subclass methods are called (a.k.a. “State Definition Inconsistency (SDIH)”) - Offutt

Missing override

— Failure to override a method, e.g., in order to maintain a stronger subclass invariant, will not be detected by compilers, leading to undetected errors (a.k.a. “Missing override”) [Binder, Testing Object-Oriented Systems, p. 502]

— As a result, superclass methods may put a subclass in a state inconsistent with the subclass invariant (a.k.a. “Inconsistent Type Use (ITU)”) – Offutt

Hot, Rank = 4

May 03 / SLIDE 61

Issue 23: Use of dynamic dispatch

Dynamic dispatch raises potential issues with respect to flow analysis, timing analysis, structural coverage, and the reuse of test results

— “Data flow analysis and control flow analysis are complicated by dynamic dispatch, because it may be unclear which method in the inheritance hierarchy is going to be called.” [FAA/NASA Handbook for Object-Oriented Technology in Aviation (OOTiA)]

— “Structural coverage analysis is complicated by dynamic dispatch because structural coverage changes when going from subclass to superclass. A closely related issue is that inheritance and polymorphism may cause difficulty in obtaining structural coverage (both modified condition/decision coverage (MC/DC) and decision coverage). ” [FAA/NASA Handbook for Object-Oriented Technology in Aviation (OOTiA)]

— “Requirements-based testing is complicated by dynamic dispatch, because it may be difficult to determine how much superclass verification activity can be reused for its subclasses.” [FAA/NASA Handbook for Object-Oriented Technology in Aviation (OOTiA)]

Hot, Rank = 4

May 03 / SLIDE 62

Issue 5: Dynamic dispatch during object construction

Need to ensure subclass methods are not called before subclass data has been initialized, establishing the subclass invariant

— “Dynamic dispatch can introduce problems related to initialization (i.e., especially with regard to deep class hierarchies)” [Binder]

— Use of dynamic dispatch by a superclass constructor (or by a method in the transitive closure of this constructor) can lead to the use of subclass attributes before they have been initialized (a.k.a. “Anomalous Construction Behavior (ACB1)”) - Offutt

— Use of dynamic dispatch by a superclass constructor (or by a method in the transitive closure of this constructor) can lead to the use of superclass attributes before they have been initialized (a.k.a. “Anomalous Construction Behavior (ACB2)”) - Offutt

Hot, Rank = 4

May 03 / SLIDE 63

Pattern: Formal interfaces

Addresses Issue 1: Lack of precise requirements/interface specifications

— “If we don’t state what a module should do, there is little likelihood that it will do it (The law of excluded miracles)” – Bertrand Meyer

— Programming by contract: “The relationship between a class and its clients [can be] viewed as a formal agreement [contract]. Only through such a precise definition of every module’s claims and responsibilities can we hope to attain a significant degree of trust in large software systems” [Bertrand Meyer, Object-oriented Software Construction]

— MC/DC test coverage criteria [DO-178B]

Matches

— All class interfaces

See SEC Catalog for details

Use of Daikon tool from MIT [Ernst] may allow pre/postconditions to be generated semi-automatically from code

May 03 / SLIDE 64

To do

Initially formal interfaces will be developed in terms of T-VEC pre/postconditions with no support from the KCS toolset

Subsequently pre/post specifications will be imported/exported by the KCS toolset in order to:

— Support the checks specified by the Formal Interface pattern

— Permit the selective use of pre/post and invariant specifications as the basis for the generation of run-time checks (as in Eiffel)

— Support the types of analysis/adaptation suggested by the Formal Subtyping pattern

Initially this will involve a conversion to/from the T-VEC file format

Eventually it should be possible to import/export this information from UML requirements and design models

And to provide assistance to the user in the generation of formal specifications from existing code (e.g., using Daikon or similar tools)

May 03 / SLIDE 65

The importing of T-VEC pre/post specifications

20 Mar 03 / SLIDE 65

UML

Action Semantics

JVM MS-ILXCIL (1.3g)

XMI

Extended UML

XCIL map

T-VEC specs

C++ (EDG AST)

XCIL map/

merge

XCIL map/

merge

May 03 / SLIDE 66

The exporting of pre/post specifications to T-VEC

20 Mar 03 / SLIDE 66

UML

Action Semantics

JVM MS-ILXCIL (1.3g)

XMI

Extended UML

XCIL map

T-VEC specsXCIL

extract/ map

XCIL extract/

map

C++ source line mapping

May 03 / SLIDE 67

Pattern: Inheritance with overriding

Simple overriding rule

Accidental override rule

Simple dispatch rule

Complete initialization rule

Initialization dispatch rule

May 03 / SLIDE 68

Pattern: Method extension

Method extension rule

Sole exception to the simple dispatch rule imposed by the Inheritance with overriding pattern

See SEC Catalog for details

May 03 / SLIDE 69

Patterns: Formal subtyping

Addresses — Issue 6: Supertype/subtype compatibility— Issue 23: Use of dynamic dispatch— Issue 5: Dynamic dispatch during object construction

In particular— “The single largest source of design errors is improper inheritance” [C++ FAQs, Issue

119]

Errors may result from— Faulty intuition— Failure to implement to inherited specifications— Misuse of inheritance to share code/data— Programmer specified optimizations— Accidental overriding/hiding— Missing override

May 03 / SLIDE 70

Pattern: Formal subtyping

Analysis results— 938 subtyping relationships

— 1,392 cases in which an operation is implemented by multiple methods (OMM pointcut)

— 3,112 instances of “true dynamic dispatch” (TDD pointcut)

— Interested in subsets (OMM incompatibilities) and intersections (OMM incompatibilities with TDD) to separate hard run-time errors from lurking errors

Review results (sample of ~40 subtyping relationships)— 24 violations of subtyping rules/LSP (including conflicting postconditions, less visible

subclass operations, disabled methods, null method implementations, and weaker postconditions)

Violations represent either lurking or hard run time errors

See SEC Catalog and OCP Subtyping Relationships for details/adaptations

May 03 / SLIDE 71

Matches

938 subtyping relationships

1,392 instances of operations implemented by more than one method (OMM)

3,112 instances of true dynamic dispatch (TDD)

May 03 / SLIDE 72

OMM checks

Check OMM for violations of LSP

— Tool can identify all instances of OMM

— Violations of LSP are associated with relationships (between an operation and an implementing method, or between a method and an overriding method)

— Tools can identify certain simple violations

— Others can be found in manual reviews

Represents either a hard run time error or a lurking error

OperationA

MethodA1

MethodA2

MethodA3

Relationships between overriding methods must comply with LSP, or be marked as an error

Relationships between the operation and its implementing

methods must comply with LSP, or be marked as an error

May 03 / SLIDE 73

Problematic call occurs when LSP violation in TDD operation-method set

— Considering only the operation and the methods that could be invoked as a result of the call (i.e., those defined by subtypes of the declared type of the target object)

— There is an LSP violation in this set (between the operation and one of the methods, or between a method and an overriding method)

Represents a hard run time error if the client depends on the erroneous relationship

Or a lurking error otherwise

TDD checks

OperationA MethodA2

MethodA3

Call to target.operationA ()

The call is OK as long as all these relationships comply with LSP

May 03 / SLIDE 74

Review results

24 violations of subtyping rules/LSP found in a manual review of a sample of ~40 subtyping relationships, including:

— conflicting postconditions (CPOST),

— less visible subclass operations (LVO),

— disabled methods (DISM),

— null method implementations (NMI),

— weaker postconditions (WPOST)

See the OCP Subtyping Relationships report for further details

May 03 / SLIDE 75

To do

Include checks for the most common LSP and specification errors that can be found by tools (LVO, DISM, INVI, NMI, UNERR, OVER, UNDER)

Allow user specification of additional LSP violations (CPOST, INVI, MINV, OVER, UNDER, WPOST, etc.) detected in reviews

Have the toolset track which LSP relationships have not been checked (all OMM relationships should be reviewed)

Determine which instances of TDD represent “problematic” calls

Introduce related transformations (as suggested in the slides that follow)

Provide tool support for checking the simple dispatch rule (Inheritance with overriding pattern) and the method extension rule (Method extension pattern)

Extend tool support to include all pointcuts, views, and transformations defined by the Formal Subtyping pattern and the “inheritance” of pre/postconditions

May 03 / SLIDE 76

Less visible operation (LVO)

An inherited operation is made less visible in a subclass

Revoking an inherited public service is evil” [C++ FAQs, Issue 122]

And a fundamental violation of LSP

And a violation of the rules defined by the Inheritance with overriding pattern

Easily detectable by toolset

One instance found in manual review of ~40 OCP subclasses

Can refactor the class to introduce a precondition and query to make the operation optional

Or refactor the class hierarchy

May 03 / SLIDE 77

Unexpected error (UNERR)

An overriding operation/method reports an error not specified in its supertype/superclass

A fundamental violation of LSP

And a violation of the rules defined by the Inheritance with overriding pattern

Potentially detectable by toolset, given a formally stated postcondition

Two instances found in manual review of ~40 OCP subclasses (one representing DISM)

Can respecify parent operation to allow reporting of the error

Or treat it as a case of NMI if the error corresponds to this

May 03 / SLIDE 78

Disabled method (DISM)

An existing method is overridden in a subclass by method that asserts that the method is disabled

A special case of UNERR and a fundamental violation of LSP

And a violation of the rules defined by the Formal subtyping pattern

Detectable by toolset, given the possible forms a disabled method may take


Can refactor the class to introduce a precondition and query to make the operation optional

Or refactor the class hierarchy

May 03 / SLIDE 79

Invalid or incomplete implementation (INVI)

The execution of the method does not deliver on the promises made by the operation/method’s description/postcondition (whether inherited or defined by the class)

A violation of LSP

And a violation of the rules defined by the Formal subtyping pattern

Most easily detected by review

But some cases can be caught by toolset given a formally stated postcondition

Or by looking at special cases (such as NMI)

14 instances found in manual review of ~40 OCP subclasses (12 of them instances of NMI)

May 03 / SLIDE 80

Null method implementation (NMI)

A method of a concrete class defines a null implementation of an operation that conflicts with its description (whether inherited or defined by the class).

A special case of INVI, and a violation of [C++ FAQs, Issue 127].

“It’s legal for an overridden method to do nothing if, and only if, the specification of the service in the base class tells users that it might do nothing. Without such an adaptable specification in the base class, doing nothing is like false advertising.” [C++ FAQs, Issue 127]

Detectable in most cases by toolset, given given the possible forms a null method may take

12 instances found in manual review of ~40 OCP subclasses

Can specify a precondition to make the operation optional, or qualify the postcondition to specify that the operation may do nothing

If all operations of the class are null, then it is preferable to address this as a case of NSCI

May 03 / SLIDE 81

Null subclass implementation (NSCI)

NMI for all methods of subclass

Suggested by NullObject pattern

Problematic in high assurance systems

— Unless it is a null implementation is always OK, for all clients under all circumstances

— And the class name/contract defined by the superclass makes this clear

Should be replaced by HA version of the pattern, which:

— Ensures any optional null behavior is defined by each superclass operation’s pre/post specification, i.e., the postcondition is of the form “true or optional-result”

— Names the superclass to make it clear that it’s implementation may be null

— Defines a subclass that represents the root of all non-null implementations (so that clients that require a non-null implementation can use strong typing to guarantee this)

Or (less desirable) refactor the base class to introduce postconditions and the query isNull to make all results explicitly optional

May 03 / SLIDE 82


For example, replace Lock, NullLock, LockImpA, LockImpB

With OptionalLock, NullLock, Lock, LockImpA, LockImpB

Lock

NullLock LockImpA LockImpB

OptionalLock

NullLock Lock

LockImpA LockImpB

postcondition indicates result is optional

postcondition indicates result is guaranteed

May 03 / SLIDE 83


Or (less desirable), refactor the Lock class to introduce postconditions and query isNull to make all results explicitly optional

Lock

NullLock LockImpA LockImpB

Refactor to introduce postconditions and query isNull to make all results explicitly optional

isNull returns true isNull returns falae

May 03 / SLIDE 84

Overspecified operation (OVER)

An operation is overspecified by its description

Usually as a consequence of specifying implementation details rather than just the result, from the client’s perspective

A violation of the OOTiA Coupling pattern

Can be detected by tools, given formally specified pre/postconditions

Neither the pre or post should contain references to features of the class that are less visible than the operation being specified


May 03 / SLIDE 85

Underspecified operation (UNDER)

An operation is underspecified by its description, e.g., by failing to specify how errors are reported (UNDER-ERR)

See the Formal interface pattern

Errors of omission are detectable by toolset, given a formally specified postcondition

The tool can, for example, determine when a change to a public property of the class fails to appear in the postcondition

7 instances found in manual review of ~40 OCP subclasses

May 03 / SLIDE 86

Other violations of LSP

To detect other violations of LSP we can structure the pre/postconditions to make the relationships between them obvious (as in Eiffel)

To ensure the precondition is weaker or the same in a redefined operation, for instance, we can assume the parent precondition is “inherited”, and only allow it to be extended by or’ing it with an additional clause

And to ensure the postcondition is stronger or the same in a redefined operation, we can assume the parent postcondition is “inherited”, and only allow it to be extended by and’ing it with an additional clause

Always specifying pre/postconditions in this way makes them easy to verify (using tools such as T-VEC that generate test cases from pre/postconditions)

See the Formal subtyping pattern in SEC Catalog

May 03 / SLIDE 87

Pattern: Subtyping tests generated from formal interfaces

Minimum compatibility rule

LSP compliance rule

Redefined/implementing precondition rule

Redefined/implementing postcondition rule

Subtype invariant rule

Inherited test case rule

Separate context rule

May 03 / SLIDE 88

Pattern: Subtyping tests generated from formal interfaces

Assumes contracts can be written after the fact

— “Contracts are inherent in library design; if not explicitly stated, they are lurking anyway under the cover, either suppressed or replaced by comments in the program, explanations in the documentation, exceptions and other ersatz techniques.” – Bertrand Meyer

Assumes tool support for test case generation from pre/postconditions— Using T-VEC

May 03 / SLIDE 89

To do

Generate T-VEC specs (by hand) for all modules we plan to change

Follow the Formal subtyping pattern with respect to the rules for the “inheritance” of pre/postconditions and compliance with LSP

Introduce KCS tool support for the importing/exporting of T-VEC specs

Provide tool support for the “inheritance” of pre/postconditions in accordance with the pattern

Explore the use of tools such as Daikon to generate pre/post specifications from code

May 03 / SLIDE 90

Pattern: Multiple implementation inheritance

Addresses issue 8: Ambiguity resulting from the use of multiple implementation inheritance

Multiple inheritance session of OOTiA workshop unattended because no one thought MII appropriate for high assurance applications

Review results (sample of ~40 subtyping relationships)

— 4 instances of multiple implementation inheritance

Matches represent maintenance risks and potential run time errors

Rules— Replace with delegation for DO-178B levels A through C

See SEC Catalog for details/adaptations

May 03 / SLIDE 91

To do

Provide tool support for the identification of cases of multiple implementation inheritance and the pointcuts associated with the pattern

Provide tool support for the replacement of multiple implementation inheritance with delegation

May 03 / SLIDE 92

Issue 71: Obstacles to level A static analysis

Need for level A static analysis, including data/control flow analysis

Need for formally defined semantics

Need for an analyzable, precisely defined language subset for safety-critical applications

Hot, Rank = 4

May 03 / SLIDE 93

Pattern: Safety critical formal language subset

Limit XCIL actions and predefined types to those that map to the UML Action Semantics or the JVM

While permitting the use of structural elements from UML (classes, interfaces, components, operations, methods, etc.) to organize them

Impose additional restrictions based on the ISO High Assurance Ada standard and SPARK, including those defined by other patterns in the catalog

See the SEC Catalog for details

May 03 / SLIDE 94

To do

Provide tool support to identify actions that do not map to the JVM instruction set

Provide tool support for the identification of functions with side effects and other restrictions mentioned by the pattern

Provide tool support for other SPARK-like patterns called for by this pattern (which have not already been implemented)

Formally define this pattern in XPSL

May 03 / SLIDE 95

Lessons learned

Tool support has been invaluable in identifying potential problems

— More effective in terms of hours spent, accuracy, and counts of problems found— Necessary due to size of OCP

Need to be able to interactively refine problem description, tune patterns

May 03 / SLIDE 96

Lessons learned

A broad, automated problem analysis is needed to find likely problems

— Given size of the OCP

— And number of issues

Results of analysis should dictate which patterns to develop further

— Need to consider both ranking and probability of occurrence

— Need to support interactive exploratory analysis

— Need to refine problem analysis to include discriminators

Need to support a progression from manual, to interactive, to fully automated transformation

— Most important transformations require developer in the loop

— Need to support use of tool as an assistant

— Need to support critics/review process

May 03 / SLIDE 97

Lessons learned

Need automated support for the evolution of meta-models (XCIL, XPSL)

— These change frequently

Need automated checking of translation results

— Well formedness rules

Need to support import/export from multiple tools (e.g., UML, C++, T-VEC)

— Need to merge inputs from different tools into common XCIL model

— Need to extract XCIL and map to multiple representations (XMI, T-VEC pre/post, C++)

Reliance on XML comes at a price

— Supports rapid tool development

— But need lots of processing power/space

— Can optimize toolset itself (using patterns) after the fact

Target language independence works

May 03 / SLIDE 98

Unimplemented global patterns

Synchronization (Issue 3)

Multiple interface inheritance (Issue 7)

Replacement of multiple implementation inheritance with delegation (Issue 8)

No dynamic allocation after initialization (Issue 9)

Initialization before use (Issue 12)

Class coupling (Issues 13, 67, 68)

Dynamic dispatch test coverage

Method extension

Deep hierarchy

May 03 / SLIDE 99

Unspecified global patterns

Checking of inputs and outputs at partition boundaries (Issue 1)

Concurrency (Issue 3)

Aliasing, involving pointers or arguments (Issue 11)

Enforcement of safety invariants (Issue 14)

Re-entrant function execution (Issue 18)

Enforcement of architectural layering/partitioning rules (Issue 24)

Initialization dependencies between classes/modules (Issue 46)

Unreachable deactivated code (Issue 57)

ocp analysis and adaptation results rockwell collins atc iowa state university may, 2003

Documents