system test planning inputs from requirements & design

System Test Planning

Inputs from Requirements & Design

November 10, 2001 System Test Inputs from Requirements & Design; R Dameron, CU Boulder

2

What is integration testing?• System testing

– Integration testing

– Acceptance testing• In-house – “alpha”

• By subset of selected customers – “beta”

• Single customer for one-customer product

• Integration testing

– Older texts will say techniques are• Bottom-up, top-down, or “sandwich”

• These generally presume what lifecycle model?

– Incremental development• Typically a “vertical slice” that widens with each increment


3

Test Perspective• Test Planning involves at least two topics

– What attributes to test– In what order

• Project manager should have system test representation early in development (req. spec.)– Tester’s perspective on requirements specs– Tester’s perspective on design– Sequence of feature development influences when

testing can begin and which testing can begin• Incremental – vertical slice gives realistic “access” to

features to be tested• When possible, production-code access to features

removes need to write drivers & stubs for integration


4

Integration strategy influence• Integration strategy determines order modules/objects need to

be available

• Exerts strong influence on order in which they are written, debugged, unit tested

• Can think of incremental development as an integration strategy

– Modules/objects need not be completely written, only the necessary features for this increment

• What about large enhancements to large systems?

– Determine enhancement sequence prior to coding, possibly prior to design of enhancements

– Can “deliver” incrementally enhanced large system to internal test group where “integration testing” is “testing successive enhancements at system level”


5

Which is better?

• Advantages to testing whole system with gradually increasing number of features

• ???• Contexts in which it is truly necessary to do older

notion of integration and integration testing– That is, build components of a system, test,

hook some components together, test, etc.?


6

Focus• We will focus on in-house acceptance testing –

what is commonly called system testing• Customer-acceptance testing can be thought of as

– Beta – selected “friendly” customers provide real environment situations that are hard to imagine or duplicate

Or– Single customer/contract

• What constitutes acceptance test is sometimes agreed upon after requirements

• Developer has opportunity to “study to the test”• Does not remove obligation to design robustness

that was not overtly specified


7

• In-house acceptance testing – must represent an imaginary tough but likely customer (will be real soon enough!)

• What are the sources for designing system tests?• First, review – what are the categories?

• What about the (F)RUPS or RAS categories?


8

What are various sources of information we have looked at so far?

What if they are not available? (more on this shortly)


9

Categories

• Functional

• Performance

• Stress

Sources

•

•

•

Match – hint: not one to one


10

What kind of test is it?• Note the progressive or fluid nature of certain

functional/performance/stress tests

• In Increment N (you can think of it as “release N”)

– Functional tests will execute features targeted for that increment

– Stress tests may include• Purposely omitted feature aspects, necessary for completeness,

represented or discovered in state transition diagrams, event diagrams, decision tables

• Must “not handle” in a graceful manner

• Become functional tests in another increment/release

– Some quality attributes that should be specified become stress tests if they were not specified


11

What tools are in the toolbox?

• Each tool discussed so far (diagrams, tables, checklists, even IEEE Standards outlines)– Should be used by development team

• To discover and specify requirements• To elaborate design

– If they are available, what does system test team do?

– If they are not?


12

Lest we forget …• Have not forgotten glass-box testing• We’ll address this in detail after design section• Overlaps unit testing and system testing• Examples

– Equivalence classes– Boundary conditions– Structural induction– Weakest precondition– Input-output assertions


13

Next: How do we focus limited resources – time, money, staff?

• Evaluating what’s important to have right – “it’s not a car”– Frequency of usage– Criticality– Operational profiles – we’ll look at this

• Evaluating fault-proneness– If we knew which units are likely to have the most

problems, we could emphasize those– It’s harder than you may think – we may not get to it in

this course, not because it isn’t important but because “what are fundamentals” is an arbitrary line.

system test planning inputs from requirements & design

Documents