how to break software: embedded edition

MQ PM Tutorial

9/30/2013 1:00:00 PM

"How to Break Software:

Embedded Edition"

Presented by:

Jon Hagar

Consultant

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Jon Hagar

Grand Software Testing

Jon Hagar is a systems-software engineer and tester consultant supporting software product

integrity and verification and validation, with a specialization in embedded and mobile software

systems. For more than thirty years Jon has worked in software engineering, particularly testing,

supporting projects including control system (avionics and auto), spacecraft, mobile-smart

devices, IT, and attack testing of smart phones.

8/20/2013

1

Jon Hagar Copy right 2013 1

How to Break Software:

Embedded Edition

Attacks to find common bugs quickly in embedded software systems

Jon D. Hagar

[email protected]

[email protected]

http://www.breakingembeddedsoftware.com/


Software Test Attacks To Break Mobile and Embedded Devices

Some Thoughts

What are our objectives?

◦ Definitions and introductions

◦ Understand Applicable Mobile and Embedded Test Concepts

◦ Practice some testing

8/20/2013

2



This is a Workshop Tutorial

A bit of a talking head (with charts) ◦ Based on my book

Attendees should be prepared to: ◦ Do some reading & thinking ◦ Use the reference material ◦ Talk & ask questions ◦ Share (lessons learned and retrospectives)

LET’S PLAY TEST… and try out some new things….. It might get LOUD



Agenda

Definitions and introductions

Risk-based concepts

Exploratory approaches

Attacking scenario(s)

Attacking the hardware-software interface

Wrap up and references

8/20/2013

3



Definitions Embedded Software Systems . . .

Interact with unique hardware/systems to solve specialized problems in the “real world”

◦ IT software runs with largely “generic” hardware

◦ Users are barely aware the device uses or has software

Usually have significant hardware interface issues and concerns

◦ Initialization, noise, power up/down, timers, sensors, etc.

Often are resource constrained

◦ RAM, ROM, stack, power, speed, time, etc.

Typically has a restricted or no Human User/Computer Interface (HCI) but is evolving rapidly

Often no way (or only a risky way) to update and/or change the software

Involves risks, hazards, safety, and/or some specialized domain knowledge and logic/algorithms usually controlling hardware



Close Cousins: Mobile, Smart, & Handheld

As the names imply, these are devices—small, held in the hand, often connected to communication networks, including:

◦ Cell and smart phones – apps (not covered today)

◦ Tablets

◦ Medical devices

Typically have:

◦ Many of the problems of classic “embedded” systems

◦ The power of PCs/IT

◦ More user interface (UI) than classic embedded systems

◦ Fast updates

Are getting more powerful, memory and features (software, e.g., apps)

The “hot” area of computers/software

◦ Testing rules are “evolving”

8/20/2013

4



What do these look like? Examples

– Avionics systems: planes, cars, rockets, military. . .

– Telecom: switch, routers, phones, cell devices

– Transportation: traffic control, railroad, trucking

– Industrial control: lighting, machines, HVAC, nuclear/power

– Medical: pacemaker, AEDs, defibrulators, dispensers, etc.

– Home and office systems: control, entertainment (TV box)

– And the list goes on • Now starting to include PDA’s and other items that “blur”

the lines



Fundamental Software Capabilities Defined

Dr. James Whittaker lists four capabilities: 1. Software accepts inputs from its environment 2. Software produces output and transmits it to its

environment 3. Software stores data internally in one or more

data structures 4. Software performs computations using input or

stored data

Embedded devices can be refined with ◦ Function in/with Time ◦ Use/control of unique hardware, OR ◦ Real world specialization of items 1 and 2

8/20/2013

5



Knowing the Bug (error) - Defined

• Handheld/Embedded software has similar defects to other software

• Requirements & Design

• Logic & Math

• Control Flow

• Data

• Initialization & Mode changes

• Interfaces

• Security

• Gaming

• etc. . .

But adds context defects/issues

• Software and hardware development cycles done in parallel, where aspects of the hardware may be unknown to the software development effort

• Hardware problems which are often fixed with software late in the project

• Small amounts of dense complex functions often in the control theory or safety/hazard domains

• (a big one) Very tight real-time performance issues (often in milli- or micro-second ranges)



“World” of Mobile-Smart/Embedded Software

Inputs and outputs involve hardware, software, and humans

Time dependent ◦ NOTE: most software has “time” (performance) issues but here

things are often “hard real time”

◦ Embedded and real-time “time” may be a requirement

Software

Stimulus-Inputs Response-Outputs

Expected

Unexpected

Wanted

Unwanted Hardware

8/20/2013

6



Exercise: Why do we test?

Handheld Mobile/Embedded Software (or any software)?



YAM* Lifecycle Embedded (*yet another model)

Software - Many builds, iterations and increments Test “circles” around schedule milestones

…………

…Prototype………

………Prototype n…..

But what about the hardware lifecycle?

start Lab drop end

Build 1

start Eng drop lab drop end

Build 2

Test Efforts

8/20/2013

7



Example High Level Embedded Lifecycle

Results: Software is “late”

Hardware Build

Hardware Build

System Creation

Hardware Build

Software Build

Software Build

Software Build

Software Build

Hw Issue

Software Build Test/V&V



My Assumptions…

This is not a “general” class on systems, software, and/or testing and

I assume the following knowledge: ◦ Test plans and planning ◦ Requirements testing ◦ Test labs and building labs ◦ Standards you operate under (yes, there are many) ◦ Tools you use ◦ Testing experience (a software system) ◦ Embedded design for testability is an accepted

practice

That you want something more . . .

8/20/2013

8



If What I Assume is False… (when you get home)

Reference list is available to do some reading

Other full classes are available

You are reading books

You will ask questions

You are looking to have an epiphany

You are ready to learn

Keep in mind that I do not have all the answers


Section 1: Testing Preliminary

8/20/2013

9



Exercise: Test the Embedded Game

Break into teams Load the 20Q app on smart phone (if you want) Define a test Define some rules: No destructive testing please

List of requirements ◦ This is a handheld game

◦ You think of something (say spinach) and it figures out what you are thinking by proposing 20 questions to you

◦ Questions begin with animal, vegetable, mineral and go from there

◦ Game has non-standard input keys, display screen, and embedded software

◦ Game knows things and will figure out what you are thinking of

Now . . . build a test for this device



What do you mean you do not have enough?

What is wrong?

What do you need to do testing?

Is this not the world many testers live in?

We should start simple in testing, but maybe this is not simple enough?

8/20/2013

10



So, Let’s Back up a Little

Let me give you some attack support concepts & techniques (in case you don’t know these)

You can apply these if you are a staff tester or a

“crowd source” contractor

This is a simulation, but in the real world, often you will just be given the software or a device to test ---- You CAN test…….



Risk and Exploratory-Attack Testing

You cannot test everything

Risk(s) based testing helps bound the test scope problem

Testing is about providing information and understanding

Exploration gets you started with whatever you have (or don’t have)

8/20/2013

11



Basic to Attacks: Risk-Based Testing

Address, mitigate, attack and retire product risks

Do you remember what a risk is? ◦ Potential problem - consequence and effect ◦ Occurrence – likelihood or chance of happening ◦ Impact – what happens

Do this from the beginning (proposal) to the end (retirement) of the product (Hw-Sw) lifecycle

Risks should feed the Attacks (more on that later)



Sample Product Risks Testers Should Consider Safety

Security

Hazard

Business impacts

Control (loss of. . . )

Computation

Functional elements

Non-functional

Data

Regulation (s) and legal factors

Output noise

Environment and input factors

System factors – complexity, interfaces, human/non-human

8/20/2013

12



How to Use Risk Analysis in Testing

Goal oriented testing (where to focus)

Priority of attack and scenario

◦ Never enough time to test everything

◦ Can define the “un attacked” (risks)

Minimization of risks by focusing on the scary or critical first

Provide information back to the team sooner



Risk-Based Testing Process (simple)

Identify the product

Find product supporting information

Identify risks associated with the product

Risk priority (what you will test first?)

The resulting risks by priority define the attacks

8/20/2013

13



Risk Analysis Throughout the Test Process

Many testers just think “requirements” in embedded, but…

Always be “thinking” risks, since it can drive and control your testing

Do this by team brainstorming (make lists)

Tests and analysis provide learning/data points/information

◦ Errors in an area of code?

◦ Hardware that doesn’t work?

◦ Piece of code from a vendor is more complex?

◦ Operations the system will/can do? Particularly off nominal and unusual (where bugs hide)



Exercise: Redeaux Back into teams

Conduct a risk exercise for your device

Risk Statement ( If x, then y happens) Priority

8/20/2013

14


Risks Should Define Exploration For mobile-embedded, exploratory testing can be important



Exploratory – Attack Testing

What is it? ◦ Scientific “methods”

◦ Engineering understanding

◦ May call it something else, but most of us do it

◦ Attacks “target” specific bugs using test techniques

How and when to apply? ◦ As early in a lifecycle as possible (with prototypes, models,

etc.)

◦ When you want to “learn” and test at the same time

◦ When being a little “informal” is OK

◦ All the time?

8/20/2013

15



Exploratory–Attack Testing Definition

Bach/Kaner: “Exploratory testing is simultaneous learning,

test design and test execution.”

Exploratory testing has rules and concepts

Underlying it is a “model” of human understanding of software and knowing how that fails

NOT AD HOC: Ad hoc has all too often been associated with sloppiness, carelessness, no documentation, non-repeatable, and so forth—but may have its place at times



In Embedded Exploratory testing is situational - Use it when…

Need rapid feedback

Learning

Upfront rapid learning

Attacking

There are risks

Need independent assessment

Targeting a defect

Prototyping

Need info

Testing beyond the requirements

8/20/2013

16



General Concept of Exploratory

Many authors define it as:

Time/Schedule (limited)

The Tester (your team)

A Testing Mission (also called “Charter”)

Results Usually in the form of opened Defects Sometimes an annotated Mission statement and

opened Defects list Maybe a “report” Retrospective (more on that in a minute)



Exploratory Critical Components

Test Design

Critical Thinking

Diverse Ideas

Rich Resources

Careful Observation

8/20/2013

17



Pattern for This Class (one of many)

Have an outline (top level plan and/or risk list)

Create a flip chart, notecard, state model, or some representation of each test task

◦ No “heavy” weight documentation of the “test case”

◦ See Exploratory Charter (test objective)

Have a Target concept or charter (Risk, Attack, Bug, Learning, etc.)

Have a schedule/time box (hours — not more than 1-2 days)

Do the test

◦ Design test

◦ Execute test

◦ Learn about the product: change the risk list, modify/add tests, and so on

Repeat the process as needed



Exploratory Test Card (Charter) Name of Test: Who is testing (test team)

What to Test: Success Criteria:

◦ Risk (s): 1.

◦ Attack 2.

◦ Other (requirements, …..) 3.

Support items needed:

Role (User you play during the test): Others Steps Actions:

◦ 1. ◦ 2. ◦ 3.

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)

Document all of these.

8/20/2013

18



Exercise: So Let’s Go Back to

the Embedded Game Device to Do Testing

Test of the Game App

Use the risks (chart 25) for the device to define test objectives

Apply Exploratory-Attack

◦ Do a Charter

Learn – do one cycle of exploration



Group Flip Chart Feedback - Retrospective

What did you accomplish?

◦ Did you find any bugs? If so, how many?

What did you think of?

What would you do differently?

8/20/2013

19


Section 2: So we have Risk Analysis & have done a first Exploratory Test

Basic and addressed in many books

What’s next?

Lets get to different levels of testing, embedded devices, and more fun



What is an Attack

Attacking your software–In part, the process of attempting

to demonstrate that a system (hardware, software, and

operations) does not meet requirements, functional and non-functional objectives

◦ Embedded/handheld software testing must include “the system”

(hardware, software, operations, users, etc.)

Attacks go after common modes of failure and bugs to demonstrate that “does not meet” exists

We go after our enemy with many approaches ◦ Tools

◦ Levels

◦ Attacks

◦ Techniques

◦ Etc.

8/20/2013

20



An Attack Is…

Based on a common mode of failure seen over and over ◦ Maybe seen as a negative, when it really is a positive ◦ Goes after the “bugs” that may be in the software ◦ Based on or using classic test techniques and test concepts Lee Copeland’s book on test design Many other good books

A Pattern (more than a process) which must be modified for the context at hand to do the testing

Testers learn these in a domain after years and form a mental model (most good testers attack)



Kinds of Attacks Whittaker offers a good starting point for software

attacks in general that can be applied to embedded: ◦ User Interface Attacks ◦ Data and Computation ◦ File System Interface ◦ Software/OS Interface

Whittaker’s “How to Break Software” lists 23 attacks

“Software Test Attacks to Break Mobile and

Embedded Devices” (my book) adds 32 attacks and 8 sub attacks

8/20/2013

21



Embedded Attack Classification

Developer Attacks (unit/code testing)

Control System Attacks

Hardware-Software Attacks

Mobile and Embedded Software Domain Attacks

Time Attacks (Performance)

Human User Interface Attacks

Smart and/or Mobile Phone Functional App Attacks

Mobile/Embedded Security Attacks

Generic Attacks ◦ Functional, mind mapping, and combinatorial tests



The Software You Test

Do you know how it fails?

Do you test for success or failure or both?

Will this workshop give you all the answers and all possible attacks?

◦ No, but you can start asking questions and thinking

8/20/2013

22



Introducing the Robots Requirements – in the class hand out for each robot grouping

Rules (this exercise takes some thinking and reading)

◦ NO destructive testing (Please BE CAREFUL with the robots)

◦ There are bugs to be found (record and report them)

◦ Each group defines an attack and gets “time” on the devices (but time in our environment is limited—just as it is in the real world)

Environment

◦ This room, but we have some “test tools”

◦ This is software testing, but within the hardware it is embedded in

This will be a simple testing process, but use the attack concepts and report experiences back in the debrief

1. Define risks based on hardware, software, requirements, and bugs (risk list)

2. Conduct each attack session using a charter

3. Define a test attack using provided attack pattern (handout)

4. Will do a debrief after each test session

5. Group will rotate different robot configurations



Additional Considerations

We will try to run at least 2 different attacks

Use the concepts we used on games

Ask questions (of each other & me)

No destructive testing and I am the “tester” (you must tell me what to do)

Use the tools at hand

But first we need to think about embedded users – next exercise

8/20/2013

23



More Test Time

We are going to do another attack on a different embedded device

Each group will be tasked with one of the two attacks

Each group should complete a charter and see the “test master” for access to the device

You’ll have a robot with software loaded

Practice identifying: Risks, Users, Exploration, and Attacks

Follow the “suggested patterns” of the attacks

We’ll go until no time left



Exercise: Understanding Users for Embedded Attacks

Let’s list some users of the games and robots because . . .

◦ Users play into risks, attacks, bugs, what to look for

◦ You should be able to do the same for the robots (or any software you test)

8/20/2013

24



Exercise Answer: List Users


Now you have a few basics:

• Risk thinking

• Exploration

• Software’s users

• Attack patterns are provided next

8/20/2013

25



Attack Group 15

Scenarios and actions over time



Attack Stories, Tours, and Scenarios

Call them what you will

There are subtle differences depending on whose material you have read

They are how the system gets used end-to-end

They combine use, users, information, techniques, tools, and (maybe) attacks

8/20/2013

26



Apply This Attack When… Time interacts with the software, events, inputs,

and outputs

Checklist of things to look for and consider (possible bugs)

◦ Order problems

◦ Too Long

◦ Too Fast

◦ Not at Right Time mark or point

◦ Late

◦ Late or early

◦ Early

◦ Deadlocked caused by a race condition(hard to find)

◦ Extra input or output events

◦ Missing events

◦ Wrong input/output within events



Attack Factors

What - Look for things not in the right order

Who – Test team

Where – Lab and/or field testing where hardware and software interact

◦ Tools may be important here

8/20/2013

27



How (a generic pattern)? Understand what the system does or is supposed to do

◦ a sequence of events or functions

◦ Look in: concepts of operations, user guides, use cases, models, & any other information that will detail functions and usage over time

◦ From these, organize a sequence or set of sequences

First attack case: Focus on a typical situation based on requirements and/or use cases

Second attack case: Consider the off–normal, non–failure modes

◦ Look for the failure modes and effects—does the software recover well?

◦ Review and understand system errors and failure history from the field

Build up histories of attacks based on outputs and log files

◦ Warning: log files can contain large amounts of detailed data and this can also adversely affect the performance (especially timing) of the software

Conduct risk analysis as the effort progresses

Final Attack cases: Build Extreme cases such as “Soap Opera” Tests

Warning: Watch becoming “script” bound



Exercise – Tell Me Stories Work in groups Key points ◦ Define a “story/scenario” with a name and outline ◦ Use the check list on chart 51 (note what is used) ◦ Follow pattern of chart 52 (note steps on charter) Can you build a Tour (combination of story patterns)

Products ◦ Risk list ◦ User list ◦ Test charter ◦ Note bugs (if any)

Complete the feedback retrospective Do several attacks (note what each one is) Expand to “extreme cases”

8/20/2013

28



Attack 7

Digitals v Analog Integration



Attack Hw-Sw Interface Group

Here we are attacking the hardware-to-software interface

Many bugs hide in the interfaces

- Developers often miss these

8/20/2013

29



Attack: Analog-Digital Hw-Sw Interfaces

When - The software is “controlling” the unique hardware

What – Look at the interface, hardware (as a user), and what the software is controlling

Who – Test team (independent)

Where – Lab where the hardware and software are both present

Bugs to look for (next page)



Taxonomy: A2D and D2A Bug Possibilities

Type Situation Impact Notes A2D A2D representation information is lost

because measurement is not precise Software computation is based on incorrect data

Number of bits used to store the analog converted data is not large enough or sampling rate to get bits is not correct.

A2D A2D information is contaminated with noise

Software computation use noise when it should not

The noise term may not be known, accounted for, or misrepresented.

A2D A2D information is calculated correctly

Computation has unknown error

Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point

D2A D2A conversion losses “least significant bits” (LSB) in conversions, but bits are, in fact, important because computer word sizes are too small

Output to analog device is wrong

Number of bits stored from the digital world to the analog world do not have enough precision, so analog data is incorrect.

D2A D2A information does not account for noise of the real world

Software computation does not include a factor for noise

The analog values are not correct given the noise of the real world (output data may be lost in the noise).

D2A D2A information is calculated correctly because of internal factors

Computation has unknown error

Sources of error can come from: calibrations used on variables, variables lacking initialization, or calculations are not done with enough accuracy (single versus double floating point

8/20/2013

30



How?

Up front data gathering and analysis are important beginnings – what do we know (or can ask about)

Identify input devices

Identify output devices

Define the input disturbances (unexpected system inputs)

Define possible output disturbances (unexpected system outputs)

Determine what is or is not possible in the test environment

Conduct a risk analysis (see likely bugs table)

Identify the users of the device and software

- Testers should be aware that embedded systems have resource constraints in memory, CPU usage, and time

Use the above information to define an exploratory chart attack

Go run that attack

Learn

Design

Repeat (until time runs out)

Think!



Questions to Ask with this Attack If the hardware is a prototype (not like what will be in the field), will

that impact testing or test results?

If a simulation is used, what bugs might be missed because actual hardware or software is not used?

If the test inputs and environment are not representative of the real world both in terms of expected and unexpected values, what risks will be acceptable?

If the hardware is not understood, will testing be weak?

If the major sources of “noise” is not defined, will the system be susceptible to impacts from unexpected inputs or outputs?

All of these questions will involve test tradeoffs, acceptable risk, and compromise

◦ 40% of this kind of attack should be “normal” situations

◦ Start with normal and move to off-normal and stress cases

8/20/2013

31



Exercise – A2D/D2A

Work in groups Key points ◦ Think how to look for bugs of chart 58 ◦ Use the questions on chart 60 (note what is used) ◦ Follow pattern of chart 59 (note steps on charter) Can you expand with stories/tours?

Products ◦ Risk list ◦ User list ◦ Test charter ◦ Note bugs (if any)

Complete the feedback retrospective Do several attacks (note what each one is) Expand to “extreme cases”



Feedback – Retrospective Session

What did you accomplish?

◦ Bugs?

◦ Tests?

What things did you think of?

◦ Wish I had a _____???? I need more time???

What favors or opposes an attack?

What would you do differently next cycle?

8/20/2013

32


Final Thoughts . . .



Wrap Up

This tutorial covers some basic introduction (key attacks) and sampling

◦ There are many more

Understanding your local context and error patterns is important (one size does NOT fit all)

Attacks are patterns…you still must THINK!

These attacks target Embedded and Mobile

8/20/2013

33



More Attacks (from my book and others)

Attack 1: Static Code Analysis

Attack 2: Finding White–Box Data Computation Bugs

Attack 3: White–Box Structural Logic Flow Coverage

Attack 4: Finding Hardware–System Unhandled Uses in Software

Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs

Attack 6: Long Duration Control Attack Runs

Attack 7: Breaking Software Logic and/or Control Laws

Attack 8: Forcing the Unusual Bug Cases

Attack 9 Breaking Software with Hardware and System Operations

9.1 Sub–Attack: Breaking Battery Power

Attack 10: Finding Bugs in Hardware–Software Communications

Attack 11: Breaking Software Error Recovery

Attack 12: Interface and Integration Testing

12.1 Sub–Attack: Configuration Integration Evaluation

Attack 13: Finding Problems in Software–System Fault Tolerance

Attack 14: Breaking Digital Software Communications

Attack 15: Finding Bugs in the Data

Attack 16: Bugs in System–Software Computation

Attack 17: Using Simulation and Stimulation to Drive Software Attacks

Attack 18: Bugs in Timing Interrupts and Priority Inversion

Attack 19: Finding Time Related Bugs

Attack 20: Time Related Scenarios, Stories and Tours

Attack 21: Performance Testing Introduction

Attack 22: Finding Supporting (User) Documentation Problems

Sub–Attack 22.1: Confirming Install–ability

Attack 23: Finding Missing or Wrong Alarms

Attack 24: Finding Bugs in Help Files

Attack 25: Finding Bugs in Apps

Attack 26: Testing Mobile and Embedded Games

Attack 27: Attacking App–Cloud Dependencies

Attack 28 Penetration Attack Test

Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack

Attack 28.2 Sub–Attack Fuzz Test

Attack 29: Information Theft—Stealing Device Data

Attack 29.1 Sub Attack –Identity Social Engineering

Attack 30: Spoofing Attacks

Attack 30.1 Location and/or User Profile Spoof Sub–Attack

Attack 30.2 GPS Spoof Sub–Attack

Attack 31: Attacking Viruses on the Run in Factories or PLCs

Attack 32: Using Combinatorial Tests

Attack 33: Attacking Functional Bugs



Summary: Thank You (ideas used from)

James Whittaker (attacks) Elisabeth Hendrickson (simulations) Lee Copeland (techniques) Brian Merrick (testing) James Bach (exploratory & tours) Cem Kaner (test thinking)

Many teachers Generations past and future Books, references, etc.

8/20/2013

34



Book List (my favorites)

“Software Test Attacks to Break Mobile and Embedded Devices” – Jon Duncan Hagar, due out late 2013 (http://www.crcpress.com/product/isbn/9781466575301)

“How to Break Software” James Whittaker, 2003 ◦ And his other “How To Break…” books

“Testing Embedded Software” Broeckman and Notenboom, 2003 “A Practitioner’s Guide to Software Test Design” Copeland, 2004 “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993 “Computer Related Risks”, Neumann, 1995 “Safeware: System Safety and Computers”, Leveson, 1995

Honorable mentions:

◦ “Embedded System and Software Validation” Roychoudhury, 2009 ◦ “Systems Testing with an Attitude” Petschenik 2005 ◦ “Software System Testing and Quality Assurance” Beizer, 1987 ◦ “Testing Computer Software” Kaner et. al., 1988 ◦ “Systematic Software Testing” Craig & Jaskiel, 2001 ◦ “Managing the Testing Process” Black, 2002



More Resources

• www.stickyminds.com – Collection of test info

• My Web site: www.breakingembeddedsoftware.com

• Association of Software Testing

– BBST Classes http://www.testingeducation.org/BBST/

• Your favorite search engine

http://www.crcpress.com/product/isbn/9781466575301



http://www.stickyminds.com/

http://www.breakingembeddedsoftware.com/

8/20/2013

35



Definitions (for this class)

Taxonomy – the practice and science of classification.

Test – the act of conducting experiments on something to determine the quality and provide information

Test case – one set of inputs, environmental set up, and results (expected and unexpected)

Attack – to set up, forcefully and attempt to “damage” the system or software, using tools, methods, and techniques

Bug (error) – results that depart from the expected (from requirements, design, standards, user, etc.)

Lifecycle – from beginning-to-end, the steps, stages, and activities to create (birth-to-death)

Procedure – a particular way of accomplishing tests, usually written (one or more test cases)

Tour – a journey to find information (tests) with a focus/direction (story)

Scenario – a sequence of events with a test plot or story

Script – see procedure, but normally uses automation

Users – someone/something that interacts with the system/software (can be human or machine, or?)

Quality – value to someone and that they will pay for




• What to Test: Success Criteria:

– Risk (s): 1.

– Attack 2.

– Other (requirements, …..) 3.

• Support items needed:

• Role (User you play during the test): Others Steps

• Actions:

– 1.

– 2.

– 3.


8/20/2013

36





– Risk (s): 1.

– Attack 2.



• Role (User you play during the test): Others Steps • Actions:

– 1. – 2. – 3.




Exploratory Test Card (Charter)

Name of Test: Who is testing (test team)

• What to Test: Success Criteria: – Risk (s): 1.

– Attack 2.




– 1. – 2. – 3.


8/20/2013

37





– Risk (s): 1.

– Attack 2.




– 1. – 2. – 3.






– Risk (s): 1.

– Attack 2.




– 1. – 2. – 3.


how to break software: embedded edition

Technology

jon hagar

support items

variables

embedded devices

embedded devices

8202013 exercise

embedded devices

risk analysis