mdd test qa_test2014_bryan_bakker
DESCRIPTION
Presentation given at QA&Test 2014 in Bilbao, SpainTRANSCRIPT
MDD and its impact on
testing...
A nanotech case study
Bryan Bakker
@Bryan_Bakker
Contents
Intro
MDD
MDD and Testing
Vacuum redesign
Experiences
© Sioux 2014 | Confidential | 2
About Bryan Bakker
Test Expert
Certifications: ISTQB, TMap, Prince2
Member of ISTQB Expert Level on Test Automation
Tutor of several test related courses
Domains: medical systems, professional security
systems, semi-industry, electron microscopy
Specialties: test automation, integration testing, design
for testability, reliability testing
© Sioux 2014 | Confidential | 3
About Sioux
HERENTALS NEDERWEERT
EINDHOVEN
UTRECHT MOSCOW
© Sioux 2014 | Confidential | 4
DANANG
About FEI
World leader in electron
microscopes
Light microscope: 1000x
200nm (limited by the
wavelength of light)
Electron microscope: 4Mx
0.05 nm
Nm = a billionth of a meter (10-9
meter)
© Sioux 2014 | Confidential | 5
About FEI
© Sioux 2014 | Confidential | 6 2 nm2 nm 0. 5 nm0. 5 nm
Atomic structur of Ge
(Germanium). Distance
is 0.5 nm
Breast cancer cell.
Magnification 5.000x Salmonella bacteria.
Magnification 80.000x
What is a model?
These models are not meant:
© Sioux 2014 | Confidential | 7
What is a model?
Simplification of the reality
By definition wrong, but can be useful
Do not try to put everything in the model
© Sioux 2014 | Confidential | 8
Traditional model
© Sioux 2014 | Confidential | 9
Traditional
Model
Source
code
Is the model
correct?
Is the
source code
correct?
ASD (Analytical Software
Design)
Model Driven Development technology
Component Based Development
Models are verified mathematically at design time
Extensive model checker:
Deadlock
Live-lock
Starvation
Race conditions, etc
© Sioux 2014 | Confidential | 10
Model
© Sioux 2014 | Confidential | 11
Model:
- Precise
- Complete
- Correct
Model verification
© Sioux 2014 | Confidential | 12
Model:
- Precise
- Complete
- Correct
Formal model
and verification
Generate
formal model
Design
errors
Code generation
© Sioux 2014 | Confidential | 13
Model:
- Precise
- Complete
- Correct
Formal model
and verification
Source code:
- Java
- MISCRA C
- C++
- C#
Generate
formal model
Design
errors
Generate defect
free source code
from verified model
Guaranteed
equivalence
What is covered?
Interface model:
Specification of interfaces of components
“What”
Design model:
Implementation of interfaces
Internal implementation
“How”
Also the outside interfaces are specified
Interfaces to not-modelled components
© Sioux 2014 | Confidential | 14
Handwritten vs. generated
© Sioux 2014 | Confidential | 15
Interface
InterfaceInterface
Interface
Interface Interface
Handwritten vs. generated
© Sioux 2014 | Confidential | 16
Interface
Interface Interface
Handwritten code
Generated
code
Part of the model
Scope of model
Typically used to model behaviour described in:
State machines
Sequence diagrams
Algorithms
© Sioux 2014 | Confidential | 17
State machines
© Sioux 2014 | Confidential | 18
Large (encapsulated) state machines can be modelled
Sequence diagrams
© Sioux 2014 | Confidential | 19
Scope of model
Typically used to model behaviour described in:
State machines
Sequence diagrams
Algorithms
What is modelled?
All normal functionality (good weather)
All exceptional functionality (bad weather)
All illegal behaviour
© Sioux 2014 | Confidential | 20
MDD and Testing
© Sioux 2014 | Confidential | 21
Implementation
Component TestDesign
Integration TestArchitecture
System
RequirementsSystem Test
User
RequirementsAcceptance Test
Wish, need
Release
Formal
methods
MDD and Testing
Formal methods typically:
Applied at Design level
Only in Software
For part of the software
Not for legacy and 3rd party software
© Sioux 2014 | Confidential | 22
Implementation
Component TestDesign
Integration TestArchitecture
System
RequirementsSystem Test
User
RequirementsAcceptance Test
Wish, need
Release
Formal
methods
MDD and Testing
Part of the component tests are
not needed anymore
(Software Engineer)
Test activities of Test Engineer:
From integration level and up
Are still needed
But:
MDD has impact on quality of deliverables
© Sioux 2014 | Confidential | 23
Implementation
Component TestDesign
Integration TestArchitecture
System
RequirementsSystem Test
User
RequirementsAcceptance Test
Wish, need
Release
Formal
methods
Microscope Vacuum redesign
Vacuum software + firmware redesign
ASD has been used
Functionality tested manually (by experts)
Vacuum is critical part of system
verify reliability separately
AutoIT used for test automation
Freeware, mainly GUI tool
Only scripting used (no recording)
© Sioux 2014 | Confidential | 24
Test automation
First steps: quick and dirty
Fast feedback
Automation via
GUI
© Sioux 2014 | Confidential | 25
Test automation
© Sioux 2014 | Confidential | 26
UI-Controls and graphics used for
Test actions
Test verifications
Reliability tests
System tests on vacuum system
Executed for long time (days)
Not only software but whole system is tested
Test automation
© Sioux 2014 | Confidential | 27
Testing experiences
Testing is still necessary
Integration (with other SW, with HW)
System testing
Model incorrect Rubbish in, rubbish out
New insights, so model will change
Functionality must still be verified (and errors are found)
Programming errors or integration issues:
In handwritten code
On boundary between generated and handwritten
code (hard to analyse)
Not in generated code!
© Sioux 2014 | Confidential | 28
Testing experiences
Reliability of generated code: very high!
But crashes can still occur... (but not in generated code)
Reliability tests
HW failures/wear detected much earlier
Days instead of months
Integration and system test
not hampered by reliability issues
big improvement, less cycles
© Sioux 2014 | Confidential | 29
Testing experiences
Models are much easier to review, understand and adapt
than code
Design discussions are held early in project/sprint (as it
should be!) defects prevented!
Refactoring (relatively) simple and reliable
(less regression)
Fits well in agile approach
© Sioux 2014 | Confidential | 30
Product Risk Analysis
© Sioux 2014 | Confidential | 31
3
3
15
15
Impact
L
ike
liho
od
Comp1
Comp2 Comp3
9
9
Product Risk Analysis
© Sioux 2014 | Confidential | 32
3
3
15
15
Impact
L
ike
liho
od
Comp1
Comp2 Comp3
9
9
III I
IV II
Could test Must test
“Won´t test” Should test
Product Risk Analysis
© Sioux 2014 | Confidential | 33
3
3
15
15
Impact
L
ike
liho
od
Comp1
Comp2 Comp3
9
9
Product Risk Analysis
© Sioux 2014 | Confidential | 34
3
3
15
15
Impact
L
ike
liho
od
Comp1
Comp2 Comp3
9
9
MD
D
Conclusions
Component tests (white box) could be skipped
Internal working is correct
Although functionality can be incorrect
Integration and system testing a lot smoother
Less interface issues
Less reliability issues
Reliability testing reliability measurement
Integration and system test more efficient (less cycles)
Architecture and design takes more effort
More re-designs with ASD are planned
© Sioux 2014 | Confidential | 35
Questions
© Sioux 2014 | Confidential | 36
