12 coding and testing -...

CS 620 Spring 2014 Univ. of Massachusetts Copyright L. Osterweil, all rights reserved

Coding and Unit Testing!

Software Engineering!Computer Science 520-620!

Spring 2014!Prof. Leon Osterweil!

!


Requirements Spec.!

Test Plan!

Test Results must !match required behavior!

Design!

Characteristics of!System to be ! built must!match required!characteristics!

(high level)!Architecture!

(low level)!specification!

Code!

Code must!implement!design!

Hi level design must!show HOW requirements!can be met!

consistent!views!

Test plan!exercises!this code!


The Coding Phase!

• Goal: Produce executable code in a coding language!!• Gets down to very specific details:! --Procedures/algorithms! --Data structures! --The interactions between them! --THE DEVIL IS IN THE DETAILS!!• Coding is usually 10-15% of the effort on a software! !development project: We will spend little time on it in !

!this course!!• Coding should follow closely the specifications resulting! from the final phases of design! --Modular structure of the code! --Object specifications (data modules) (Data abstractions)!


Coding vs. Code!•  Here too more is know about the artifact than the

activity!•  Coding!

– How should we create code?!– Helps to have a detailed low-level design!– Helps to have some patterns and idioms!– But iteration/rework/refactoring will be needed!– When, and how do we know when to do what?!

•  Code!– Choice of notations here too!

» different coding languages!– But same criterion!

» What is best at meeting stakeholder needs?!


Coding!• Goal: Create code that can be executed on a computer!!• Developer writes source code!!• Object code emitted from a compiler!

!So, is source code really just another model?!!• Executable results from loading object code with libraries,! utilities, etc.!!• Important to keep all of these straight!!• Some languages are designed to support specific !

!design methodologies!!• Some languages are special-purpose, well adapted to !

!specific application domains!!!


Overall Coding Language Trends!

•  Less focus on being close to the machine!•  More focus on integrating with development

lifecycles!•  Different language approaches for different

lifecycle approaches!– Static languages for critical software!

» Emphasis on modularity, testing support, etc.!– Dynamic, high-level languages for applications

in highly competitive situations!» Dynamic, late-binding, powerful semantic

features!•  Increasing concern for security!


What makes a codinglanguage “good”?!


What makes a codinglanguage “good”?!

If it meets the needs of its stakeholders!


A “good” language is one that meets the needs of its stakeholders!

•  Different kinds of projects!–  Quality is super-important!–  Rapid deployment is key!–  Evolvability is paramount!–  Emphasis on user interface!–  Etc.!

•  Suggest languages with strengths like!–  Readability!–  Expressive power!–  Low level (close to the machine)!–  Dynamism and late binding!–  Etc.!


On Languages!•  Bad code can be written in any language!

– But some languages encourage bad practices!•  Good code can be written in any language!

– But some languages encourage it/make it easier!– And discourage bad practices!

•  Most modern languages try to encourage good practices!– Like those we have been advocating (in discussing

design)!» Modularity!» Information hiding!» Data abstraction!» Incorporation of design and requirements

specification into code!» Support for testing and analysis!» Concern for security!


Information Hiding in Implementation!•  Implementation units should hide internal details as

specified by a Modular design!– Superior procedure semantics support this better!

•  Implementation units should communicate through well-defined interfaces (not global variables).!– What is so bad about global variables?!– Some languages make global data easier than others!

•  Some languages make it hard to inspect internals of Modules. !– Others make it easier!

•  Different decisions are harder or easier to hide !– Algorithm!– Data representation!– Lower-level modules!– Policy!


Language Trend Towards Hiding!

•  Machine/assembly languages hide nothing!– C is essentially a quasi-machine-independent

assembly language!•  Early programming languages (Fortran, Cobol)

had the same problem!– Extensive use of global data!

•  Later languages (Ada) made it possible to hide internals !– And made global data very awkward!

•  Modern languages (Java) make it hard to expose internals!


Data Abstractions!• User's (client's)-eye view of the data types to be used!!• Essentially the same as Parnas notion of a "data module"! --and the notion of an "object"!!• Cluster of "accessing primitives" / "methods" whose! purpose is to provide the only mechanisms for! manipulating data of a given type!!• Problem: How to specify the semantics of these types! --without specifying their implementation! --internal part vs. external part!!• Being rigorous helps separate (even slightly) different! notions of an ADT from each other!!


Languages incorporating Assertions!

•  Program code is interspersed with assert statements !– Assert statements define intent!– Assertions defined by programmer!– Locations identified by programmer!– Reactions to violations defined by programmer!

•  Assert statements may use different language semantics!– Usually Boolean logic!

•  Sometimes have their own private data space!!


Assertions!•  Specifications of intended properties of execution state!

– e.g. relations among the values of program variables!•  Precise specifications of Required behavior!

– I.e. embodiments of requirements !•  Can be used to checking relations between code and!

– Design!– Requirements!

•  Acknowledge need for code to fit into overall development lifecycle!

•  But we will see other uses soon!


Y := current_time!!….!

<code sequence>!X := current_time;!run_time:= X - Y;!ASSERT run_time< 2.0;! <rest of code>!!!

Inserted by! Tester!

Automatically ! processed! into!

if ~(run_time > 2.0) Then!!Assertion_violation_handler;!


Assertion-Based Testing!•  Imbed assertions with program executable code!•  Assertions function as runtime checks!•  Supports zooming in on internal workings of the program!•  Examine behaviors at internal program locations while

the program is executing!– Augments examining only final outputs!

•  Comparison: Runtime evaluation of assertions!•  Facilities for programming reactions to violations!

– Supports looping (rework) in the development lifecycle!


Tool Suites and Environments!

•  Better tools make languages more useful!– Can mean success or failure of a language!

•  Some major types of tools!– Editors!– Error checkers!– Testing aids!– Powerful libraries!

•  But tool integration (into environments) is key!


Patterns!•  Higher level implementation constructs!•  Idioms (Rich and Waters, ~1985)!•  The “Gang of Four” book!

– Inspiration from “real” architects (C. Alexander)!•  Idioms in common use!•  Suggest ways that humans think/human esthetics!•  Continuing attempt to bring languages closer to how

humans think, and away from how machines work!•  Transcend specific languages!•  Some finding more direct support in newer languages!


Unit Testing!

•  Determining the characteristics and properties of a compilation unit of code!

•  In the case, the artifacts are less a focus than the activity!

•  How should unit testing be done? What process(es) should be employed?!

•  And how should the results be interpreted?!•  Very little attention paid to the notation in which

the results are expressed!


Requirements!Specification!

Architecting! Implementation!Designing! Coding!

System!Testing!

Integration! Testing!

System Test! Plan!

Integration !Test Plan! Unit!

Test !Plan!

Software!Sys Testing!

Software Sys. !Test Plan!

Flow of control edge (the ImmFol relation)!

Data Flow edge (artifacts flow from tail to head)!

The Full Development Lifecycle!

Unit Testing!


Requirements!Specification!

Architecting! Implementation!Designing! Coding!

System!Testing!

Integration! Testing! Unit Testing!

System Test! Plan!

Integration !Test Plan! Unit!

Test !Plan!

Software!Sys Testing!

Software Sys. !Test Plan!

Flow of control edge (the ImmFol relation)!

Data Flow edge (artifacts flow from tail to head)!

The Code-and-Test Part!


Testing!

•  Testing is the systematic (?) search of a program's execution space!

•  The essence is comparison of behavior to intent !•  Behavior determined by examining test execution results!•  Intent derived (somehow) from (various) specifications!•  Comparison typically has been done by text examination!

– Although much more “automatic” testing done now!


Testing!

Input Space!

Output Space!

Program !


Testing is Sampling the Input Space!

•  Key problem: What is the input space?!– What is the software intended to do?!

•  Subproblem: The input space is large!– One dimension for each program input!– Each dimension can have as many elements as

there are legal inputs (eg. 2**32 different integers)!– Each input really is different!– How different? Which differences matter?!

•  Key Problem: How to sample from this input space?!


Program sum_of_roots;!Real sum, x, r;!sum := 0;!Do forever!

!input x;!!if x = 9999.99 then exit!! !else !!! !r := sqrt(x);!! !sum := sum + r;!!end do;!

print sum;!end;!!

!!

What is the input space?!

sum_of_roots takes an arbitrarily!long sequence of real numbers,!and computes the sum of their!square roots. The real number!sequence must be ended with!the number 9999.99!

Specification!Implementation!


Computing the Input Space!

•  There are ~2**32 possible different values for each input!•  If n values are read in, then there are (2**32)**n different

points in the input space!•  The number of different input values read in is unlimited!•  There is no limit (theoretically) to the size of the input

space!


Some observations about the example program input space!

•  There is no real need to test every possible combination of input values!– Most executions behave “the same”!

•  But some input combinations are “different”!– Negative values will produce a failure!

•  There is a virtually limitless number of inputs that don’t cause the negative square root failure!

•  A sufficiently large sequence of input values will cause an overflow failure !

Effective selection of test cases requires thought and care!


The Testcase Selection Problem!•  Testing lets you put your program under a

microscope!– Can examine minutiae!– But only for current execution!

•  To find faults you need to select test data to cause failures!

•  Testing can demonstrate the presence of faults (when suitable test cases are selected)!

•  But demonstrating the absence of faults requires knowing the behaviors of all executions!– But there are (virtually) infinitely many possible

executions!– So how to sample the inputs representatively? !


Partitioning the Input Space!

•  Rationale: All points in the same subdomain are processed “equivalently” by the program!

•  But:!– How to determine the partition?!– How to know how far the equivalence holds?!– How to select the point(s) within each domain

to use as the actual input(s)?!•  The manual and/or requirements specification can

help!


Equivalence Partitioning!

•  The typical approach!•  A test of any value in a given class is equivalent

to a test of any other value in that class!•  If a test case in a class reveals a failure, then any

other test case in that class should reveal the failure!

•  Some approaches limit conclusions to some chosen class of faults and/or failures!


Testing!

Input Space!

Output Space!

Program !


Input Space Partitioning!

Input Space Divided into Domains!

Program !


Black Box vs. White Box Testing!

SELECTED INPUTS

RESULTANT OUTPUTS

INTERNAL BEHAVIOR

DESIRED OUTPUT

SOFTWARE DESIGN

“BLACK BOX” TESTING

“WHITE BOX” TESTING

SELECTED INPUTS

RESULTANT OUTPUTS

DESIRED OUTPUT

(CLEAR BOX TESTING)!


Structural (White Box) Testing!•  Testcase choices driven by program structure!•  Flowgraph is most commonly used structure:!

– Represent statements by nodes!– If a statement can execute immediately after another,

connect the nodes representing them by an edge!– Every program execution sequence is a path!

•  Criteria based on “coverage” of program constructs!– All statements (node coverage)!– All control transitions (edge coverage)!– All possible paths, loop iterations (path, loop coverage)!

•  How to generate input data to do this?!•  What exact data sets are used to force these coverages?!

– It matters!


Rigorously defined Flowgraph helps!

totalpay := 0.0;!for i = 1 to last_employee! if salary[i] < 50000.! then salary[i] := salary[i] * 1.05;! else salary[i] := salary[i] * 1.10;! totalpay := totalpay + salary[i];! end loop;!print totalpay;!

totalpay := 0.0;!

for i = 1 to last_employee!

if salary[i] < 50000.!

salary[i] := salary[i] * 1.05! salary[i] := salary[i] * 1.10!

totalpay := totalpay + salary[i]!

end loop!

print totalpay!

Different paths partition the execution space!


Using Flowgraphs to Partition Input Space!

•  Requires use of static analysis (described soon)!•  Use program branching conditions to create sets

of constraints!•  Solving constraints for each path yields the input

domain that forces execution down that path!•  Called Symbolic Execution!•  More on this soon!


How to assure testing is thorough?!•  Assure every node is covered by at least one test

case (node coverage/statement coverage)!•  Assure every edge is covered by at least one test

case (edge coverage/branch coverage)!•  Assure every loop is exercised!

– Both by iteration and fall-through!•  Assure all execution paths are covered!

– Practical impossibility!!

And there are many other criteria!


How to assure testing is thorough?!•  Assure every node is covered by at least one test

case (node coverage/statement coverage)!•  Assure every edge is covered by at least one test

case (edge coverage/branch coverage)!•  Assure every loop is exercised!

– Both by iteration and fall-through!•  Assure all execution paths are covered!

– Practical impossibility!!

And there are many other criteria!

Best to remember that:!!Testing is buying knowledge!


Functional (Black Box) Testing!•  Specification of Intent:!

– Expressed explicitly!– Increasingly completely!

» Functionality, timing, accuracy, robustness, ...!

– Increasingly rigorously!» Mathematics, FSA’s!

– Ideally arise directly from requirements and design specifications!

•  Specification of Behavior!– Tools to capture test outputs (inputs too)!

•  Comparison!– With automatic comparators!!


Examples of Black Box Testing Goals!

•  Does the software do what it is supposed to do?!•  When might it fail?!•  How fast does it run?!•  How accurate are the results?!•  What are its failure modes and characteristics?!•  What can I count on?!•  What should I worry about?!•  What are its strengths and weaknesses?!


Functional (Black Box)Testing Guidelines!

•  Result in many test cases!•  Some test cases may satisfy many objectives!•  Keep track of the goal(s) of each test case!•  Changes in specification will cause changes in

functional test cases!•  Need a way to organize, use, reuse, and monitor

functional testing!•  NOTE: many of the functional testing guidelines

can also be applied in structural testing (at a more detailed and formal level)!


Assertions!

•  Specifications of intended properties of execution state!– e.g. relations among the values of program

variables!•  Precise specifications of required behavior!•  Development of increasingly elaborate assertion

languages!•  Often used for checking relations between code

and design!•  But we will see other uses soon!


Y := current_time!!….!

<code sequence>!X := current_time;!run_time:= X - Y;!ASSERT run_time< 2.0;! <rest of code>!!!

Inserted by! Tester!

Automatically ! processed! into!

if ~(run_time > 2.0) Then!!Assertion_violation_handler;!


Assertion-Based Testing!•  Imbed assertions with program executable code!•  Assertions function as runtime checks!•  Supports zooming in on internal workings of the program!•  Examine behaviors at internal program locations while

the program is executing!– Augments examining only final outputs!

•  Comparison: Runtime evaluation of assertions!•  Facilities for programming reactions to violations!

– Supports looping (rework) in the development lifecycle!


Functional vs. Structural Testing!•  Cannot determine if software does what it is supposed to

do without considering the intent!– a special case not handled by the implementation will

not be tested unless the specification/ requirements are considered!

•  Cannot ignore what is actually done by the software!– program may treat one element in domain differently

than stated in the specification!– implementation often employs use an algorithmic

technique with special characteristics that are not highlighted in the specification!

– Both functional and structural testing must be done!

Should use all available information in testing!

12 coding and testing -...

Documents