use combinatorial testing for mobile device fragmentation
TRANSCRIPT
T4 Mobile Testing 5/5/16 9:45
Use Combinatorial Testing for Mobile Device Fragmentation
Presented by:
Jon Hagar
Grand Software Testing
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
Jon Hagar Grand Software Testing Jon Hagar is a systems software engineer and testing consultant, supporting software product integrity and verification and validation (V&V), with a specialization in mobile and embedded software system testing. For more than thirty years, Jon has worked in software testing and engineering projects. He authored Software Test Attacks to Break Mobile and Embedded Devices; consults, presents, teaches, and writes regularly in many forums on software testing and V&V; and is lead editor/author on committees including OMG UTP model-‐based test standard, IEEE 1012 V&V plans, and ISO/IEEE/IEC 29119 software test standard. Contact Jon at [email protected].
Use Combinatorial Testing for Mobile Device Fragmentation
Jon D. Hagar, Consultant, Grand Software Testing [email protected]
1 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
• Scary stories
• It only takes a few minutes of using an App before users like or hate it
• Worse than that. . . – Many users will post a poor social media review of the app or device
– You may be on the nightly news (bad press is not good)
– A question I get a lot, “how do we deal with fragmentation?”
• So You want to be
– Part of the billions of devices
» You want to be GREAT
2
The Mobile Opportunity
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
What We Will Cover
• Introduction and definitions
• A combinatorial test attack pattern
• Some Combinatorial (CT) Tools
– Demo
• Wrap up
3 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Basic Definitions
• Test – the act of conducting experiments on something to determine the quality (ies) and provide information
– Many methods, techniques, approaches, levels, context
– Considerations: input, environment, output, instrumentation
• Quality (ies) – Value to someone (that they will pay for)
– Functional
– Non-functional
– It “works”
– Does no harm
• Are there (critical) bugs?
4 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
The Mobile-IoT-Embedded Space
5
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
Cloud
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
• Embedded – Software contained in “specialized” hardware…
• Mobile and handheld devices—small, held in the hand, connected to communication networks, including
– Cell and smart phones – apps
– Tablets
– Medical devices
• IoT – Internet of Things are traditional devices with software and comms added
• Mobile, Handheld, IoT typically have:
– Many of the problems of classic embedded systems
– The power of PCs/IT
– More user interfaces than classic embedded systems
– Fast and frequent updates
• Devices are “evolving” with more power, resources, apps, etc.
• Mobile and IoT are (currently) the “hot” area of computers/software
You know what they are. . . Right? Embedded, IoT, Mobile and Handheld?
Test Brakes What’s this?
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
• Embedded – Software contained in “specialized” hardware…
– Minimal networking-communication
PLUS
• Mobile and handheld smart devices—small, held in the hand, highly connected (web, cloud, servers,….)
• IoT – Internet of Things are “traditional” embedded and new devices with software and communication added
What is a Mobile (and IoT) device?
Test Brakes
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Defining Software Capabilities
• James Whittaker defines 4 fundamental capabilities that all software possesses
1. Software accepts inputs from its environment
2. Software produces output and transmits it to its environment
3. Software stores data internally in one or more data structures
4. Software performs computations using input or stored data
• To this, we expand and refine based on an mobile context:
– Within time
– Using specialized hardware (as sub of items 1 and 2 above) and control
– Security and privacy
– Different development lifecycle constraints
8 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Attack-based Testing Patterns What is an attack?
• A pattern (of testing) based on a common mode of failure seen over and over – Some see this as a negative, when it is really a positive – Attacks seek the “bugs” that may be in the software – May include or use classic test techniques and test concepts
• Lee Copeland’s book on test design • Many other good books
• A Pattern (more than a process) which must be modified for the context at hand to do the testing
• Testers learn mental attack patterns when working over the years in a specific domain
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Example Attacks (from “Software Test Attacks to Break Mobile and Embedded Devices”)
• Attack 1: Static Code Analysis
• Attack 2: Finding White–Box Data Computation Bugs
• Attack 3: White–Box Structural Logic Flow Coverage
• Attack 4: Finding Hardware–System Unhandled Uses in Software
• Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs
• Attack 6: Long Duration Control Attack Runs
• Attack 7: Breaking Software Logic and/or Control Laws
• Attack 8: Forcing the Unusual Bug Cases
• Attack 9 Breaking Software with Hardware and System Operations
• 9.1 Sub–Attack: Breaking Battery Power
• Attack 10: Finding Bugs in Hardware–Software Communications
• Attack 11: Breaking Software Error Recovery
• Attack 12: Interface and Integration Testing
• 12.1 Sub–Attack: Configuration Integration Evaluation
• Attack 13: Finding Problems in Software–System Fault Tolerance
• Attack 14: Breaking Digital Software Communications
• Attack 15: Finding Bugs in the Data
• Attack 16: Bugs in System–Software Computation
• Attack 17: Using Simulation and Stimulation to Drive Software Attacks
• Attack 18: Bugs in Timing Interrupts and Priority Inversion
• Attack 19: Finding Time Related Bugs
• Attack 20: Time Related Scenarios, Stories and Tours
• Attack 21: Performance Testing Introduction • Attack 22: Finding Supporting (User) Documentation
Problems • Sub–Attack 22.1: Confirming Install–ability • Attack 23: Finding Missing or Wrong Alarms • Attack 24: Finding Bugs in Help Files • Attack 25: Finding Bugs in Apps • Attack 26: Testing Mobile and Embedded Games • Attack 27: Attacking App–Cloud Dependencies • Attack 28 Penetration Attack Test • Attack 28.1 Penetration Sub–Attacks: Authentication —
Password Attack • Attack 28.2 Sub–Attack Fuzz Test • Attack 29: Information Theft—Stealing Device Data
• Attack 29.1 Sub Attack –Identity Social Engineering
• Attack 30: Spoofing Attacks • Attack 30.1 Location and/or User Profile Spoof Sub–Attack • Attack 30.2 GPS Spoof Sub–Attack • Attack 31: Attacking Viruses on the Run in Factories or
PLCs • Attack 32: Using Combinatorial Tests • Attack 33: Attacking Functional Bugs
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
In Mobile and IoT Many Example Combinations: Standards, Interfaces, Protocols, Platforms, Software, and Data Patterns
11
Network-Comm
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Many of these Combinations Will Need Testing
Exercise: How should we test these? (How do you do it now?)
12
How many tests are needed?
Coverage of combinations?
How do we find errors?
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Combinatorial Testing (CT) Math Offers Solutions
• CT has long history of Usage
• CT uses many tools
• CT is still underused
• CT has some cool possibilities
• CT should be one of the attack techniques used
• Find out how CT can help your testing
13 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Math-based Testing
Testing is a sampling problem: How can Math aide testing?
• Test systematically the numbers of devices, configurations, networks, etc.
• Use sampling in environments and quality control
• Use sampling of data from the input domain space
• Help use Big Data Analytics to feed testing
14 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Pattern Attack 32: Combinatorial Tests
15
• When to apply this attack? – There are numerous related variables and variable values which
interact
– Validation Analysis Upfront
– Testing throughout the life cycle and in Maintenance Mode
• What faults make this attack successful? – Untested configuration combinations
– Data “bugs”
• Who conducts this attack? – Tester, analyst
• Where is this attack conducted? – Tool running in the lab or field
• How to determine if the attack exposes failures? – A test fails to meet success criteria
– Hard crash - NIST Data
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Attack 32: Combinatorial Test Patterns
16
• How to conduct this attack – basic pattern
– Identify combinatorial situation
– Identify combinatorial tool
– Identify variables
– Identify values
– Identify constraints on values
– Enter variables and values into tool with constraints
– Exercise resulting combinations in usage scenario tests or automated tests
– Look for failures
– Repeat and refine as needed
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
• Android or Other OS
Example Usage: Numbers of data choices, devices and configurations
17
• Hardware
• Connected devices
• Data
• Routers
• Home Protocols
How many Tests?
10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests!
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Using the ACTS Combinatorial Tool: Example
18
Parameters:
Andriod AppPlatform
[Device 1, Device 2, Device 3, Device 4, Device 6, Device 7, Device 8, Device 9, Device 10]
IoTProtocolHome [true, false]
IoT Devices
[Refrig, Stove, mircrowave, TV, front door, Garage door, Home gaurd, Stereo, Temp Control, Lights, Drapes, Water Heater, window openers]
Routers [0, 1, 2, 3, 4, 5]
Comm providers [Cell1, Broadband, cable, Cell 2, Space based, Vendor godzilla]
Data [1, 0, -1, 99999, -99999, 100, -200]
Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers Comm providers Data
0 Device 1 false Refrig 1 Broadband 0
1 Device 2 true Refrig 2 cable -1
2 Device 3 false Refrig 3 Cell 2 99999
3 Device 4 true Refrig 4
Space based -99999
4 Device 6 false Refrig 5
Vendor godzilla 100
5 Device 7 true Refrig 0 Cell1 -200
119 Tests
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Other Statistical Tools to Consider
General Technique Concept Tool Examples (Note 1)
Examples of where technique can be used
Specific sub- technique examples
Combinatorial Testing
ACT [4], Hexawise[5] rdExpert [6] PICT[7]
Medical, Automotive, Aerospace, Information Tech, avionics, controls, User interfaces
Pairwise, orthogonal arrays, 3-way, and up to 6 way pairing are now available
Design of Experiments (DOE)
DOE ProXL[8] DOE++ [9] JMP [10]
Hardware, systems, and software testing where there are "unknowns" needing to be evaluated
Taguchi [12] DOE
Random Testing
Random number generator feature used from most systems or languages
Chip makers, manufacturing quality control in hardware selection
Testing with randomly generated numbers includes: fuzzing and use in model-based simulations
Statistical Sampling SAS [10]
Most sciences, engineering experiments, hardware testing, and manufacturing
Numerous statistical methods are included with most statistical tools
Software Black box Domain Testing
Mostly used in manual test design, though some tools are now coming available [11]
All environments and types of software tests. These are “classic” test techniques, but still underused
Equivalence Class, Boundary Value Analysis, decision tables (Note 2)
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Many Variables and Choices
20
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
And the ACTS tool in real-time (be on the high wire)
Tool Demo
Link To ACTS Tools
21 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Expanding Combinatorial Testing
• For Dev-Ops – Sampling user data
– Model-based testing
– Advanced data selection
• Support domain testing
• Do test without an Oracle – NIST Study
– Combine with automation
– Run 4-to-6 way combo’s
– Look for Major Crashes
22 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
Summary
Common Mobile Problem
• Data selection
• Dealing with numbers of configurations – Hardware, Software, Protocol,
etc.
• Testing within time and budget
Overlooked Solutions
• Data analysis with sampling – Classic testing
• Combinatorial Testing with tools – Test Automation (not a
requirement)
• Reduce combinations to fit within budget and schedule
23 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
•
•
•
•
•
•
•
•
•
•
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Book List
• “Software Test Attacks to Break Mobile and Embedded Devices”
Jon D. Hagar, 2013
• “How to Break Software” James Whittaker, 2003
– And his other “How To Break…” books • “A Practitioner’s Guide to Software Test Design” Copeland, 2004
• “Introduction to Combinatorial Testing” D. Richard Kuhn Raghu N. Kacker Yu Lei , 2013
•
Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”