1ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Déjà vu - Integration and Test Challenges

April 2014

Tom Wissink, Lockheed Martin Senior FellowDirector of Integration, Test & Evaluation

2ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

A little bit about Me – an Experiment

My Test Journey started at STARWEST 1998– That was 25 years into my Career

• 5 Companies and over 20 consulting assignments• Programmer, Manager, Tech Lead, Integrator, Tester, etc.

– 15 Years ... a never ending Journey

Resulted in a sabbatical from and then a return to Lockheed Martin

– Worked for a small Software Testing Company; three years of consulting, training and learning

• With large Fortune 500, mid-size & small companies

– One of my new goals - ensure no one has to leave the company to learn the right way to test software!

3ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

An Important Caveat

• I will make many general comments about software and software testing:– These comments will come from:

• My 40 years experience • Consulting/Working in the aerospace & commercial

industry• Testing research with Industry and Academia • Supporting Government & Industry Test Associations,

Committees and Conferences

– The facts and figures used are publically available

• These are my personal views and comments!

4ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

The Past

Day 1 in Software Testing at least from my careers perspective– I started before the first important

book on software testing was published• “The Art of Software Testing”

By Glenford J. Myers

5ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

The Space Shuttle Program

• First Shuttle Launch:– http://www.youtube.com/watch?v=KwLlzUSOW_E

• Last Shuttle Launch:– http://www.youtube.com/watch?v=2EFuLap5Pgg

6ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Software Testing Lessons from the Past

– Software systems getting larger and more complicated and test needs to increase methods and techniques• Not many specific software testing techniques utilized • But they were known at this point in time but not taught/used• Structured Programming methods/techniques

– Defects found during final testing • The need for multiple test phase became apparent

– Unit, functional, subsystem, system, user• The idea of a separate integration phase was not planned but quickly

became known as an area to specifically work– Was also talked in the literature by different terms, module

testing, interface testing, etc.

– Integration & troubleshooting skills realized as a key need

7ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

The Present

Software Testing Today– Many, many book on the market

and classes in Universities (typically at the graduate level)

– Many theses and dissertations on software testing exist

– Conferences, tools, consultants– Issues still exist in software testing

April2014

8ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

The F-35 Joint Strike Fighter

• The F-35 Flight Testing Video:– http://www.youtube.com/watch?v=8HvOFTIyQzc

9ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

F-4 (1960)

A-7 (1964)

F-111 (1970)

F-15A (1975)

F-16 (1982)

B-2 (1990)

F-22 (2000)

F-35 (2012)

8 10

20

35

45

65

8087

Percentage of Functionality in Software

The F-35 Joint Strike Fighter

60,000 LOC

2 mil LOC

8 mil + LOC

10ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Current State of the Practice

Toyota faulty accelerator Cost impact over a billion dollars

2002 NIST Study indicates poor software testing infrastructure costs

the US economy as much as

$59 billion a year

Healthcare.govSignificant technical issues, cost increase and very late

Ariane 5 Basically one bad line of code in a reused module from Ariane 4

11ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Is the Current State Chaos or Not?

12ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Recent Government Actions in T&E2006 – Testing of Defense Systems in an Evolutionary Acquisition Environment

- Results from several Hearing, Studies, etc

2008 – Defense Science Board on DT&E

2012 – Industrial Methods of the Effective Development and Testing of Defense Systems – Focus is on scientific testing approaches

and methods like DOE and combinatorial test design

National Defense Authorization Act 2012 includes:• New Government role: “Chief Developmental Tester”

- The Chief Developmental Tester drives T&E activities across the entire program life cycle, ensuring consistency with customer's T&E Strategy.

13ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

More Government Actions in T&E

National Defense Authorization Act 2013 includes:• Annual report to Congress on:

- Programs that don’t follow DT&E recommendations

- About technical maturity and integration risks of critical technologies

• Presidents budget shall include DT&E line item• Increase to DT&E staff to support

program oversight

14ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Progression of Software Systems

Simple• Easy to understand• Repeatable processes to

create• No Expertise needed to

follow process• Completely testable

Complicated• Large Number of

components• Large # of connections• Integration of different

technologies • Difficult to have

knowledge across all elements

• Process to create is difficult but not impossible

• Incomplete testing

Complex• System evolves • Very large # of

connections• Feedback changes

behavior• Adaptive• Emergent• Social Behavior• Unexpected results

Test is Easy• Test Everything

Can’t test everything• Knowing what to test is

more important then how much

Non-predictable results• How to test what you don’t

know?

XWe are Here

15ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Current Software Testing Lessons

– Déjà vu: Software systems getting much larger and more complicated and test needs to increase methods and techniques • Many specific software testing techniques known but not used • Object Oriented & Agile programming methods/techniques

– Déjà vu: Many defects found during final testing• Multiple test phase utilized but poor test coverage

– Unit, functional, subsystem, system, user• The idea of a separate integration phase is known but typically not

done until after the first release of software capability doesn’t work• No real improvement in the literature available on integration

– Déjà vu: Integration & troubleshooting skills realized as a key need

16ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

The Future

Testing Tomorrow – Ever increasing System and Software Complexity!

Model-based Test & Integration–Many if not most test cases

generated & executed automatically – less human execution time– However, Human testing will

become even more important– Integration still a human endeavor

with help from Models Unknown – Need Research

17ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Is there a Test Technology Gap?

Singularity

The test community need to close the gap

Test Technology

Gap

TechnologyTest Research

18ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Complicated Systems

System of Systems

Autonomous Platforms

Collaboration

Large number of connectionsFree-scale networks

Large set of sensors

Feedback loops modifying behavior

Non-predictable

Systems/Software Complexity

Evolving FutureToday

19ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Simple Emergent Behavior Example

https://www.youtube.com/watch?v=1X-gtr4pEBU

20ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Complex I&T Research Sources1. Brat, Guillaume, et al. "Verification of autonomous systems for space applications."

Aerospace Conference, 2006 IEEE. IEEE, 2006.2. Truszkowski, Walter F., et al. "Autonomous and autonomic systems: A paradigm for

future space exploration missions." Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 36.3 (2006): 279-291.

3. Dixon, Clare, Alan Winfield, and Michael Fisher. "Towards temporal verification of emergent behaviours in swarm robotic systems." Towards Autonomous Robotic Systems. Springer Berlin Heidelberg, 2011. 336-347.

4. Gleizes, Marie-Pierre, and Jorge J. Gomez-Sanz. "Agent-Oriented Software Engineering X." (2009).

5. Strunk, Elisabeth A., M. Anthony Aiello, and John C. Knight. "A survey of tools for model checking and model-based development." University of Virginia (2006).

6. De Lemos, Rogério, et al. "Software engineering for self-adaptive systems: A second research roadmap." Software Engineering for Self-Adaptive Systems II. Springer Berlin Heidelberg, 2013. 1-32.

7. Gjondrekaj, Edmond, et al. "Towards a formal verification methodology for collective robotic systems." Formal Methods and Software Engineering. Springer Berlin Heidelberg, 2012. 54-70.

8. Choi, Jiyoung. Model checking for decision making behaviour of heterogeneous multi-agent autonomous system. Diss. Cranfield University, 2012.

21ICST 2014 Keynote: Déjà vu - Integration and Test Challenges

Summary

• Academia and Industry must work more closely to solve the software complexity issues– e.g. Model-based test case generation at multiple

levels of testing

• Goal: Software testing evolve along with technology and methodology discovery

22ICST 2014 Keynote: Déjà vu - Integration and Test Challenges