20140925 fistb keynote
DESCRIPTION
These are the slides from my well-received keynote at the Finnish Software Testing Board's FiSTB 2014 conference, held in Helsinki on 25 September 2014. This was a great event with many fantastic talks — see more at http://testingassembly.fistb.fi.TRANSCRIPT
Beyond Agile Testing to Lean Development
James O. Coplien Gertrud & Cope
BEYOND AGILE TESTING TO LEAN DEVELOPMENT - Jim Coplien • Testing is a business function that should be focused on risk • Lean defines testing as waste — why do it? • Automated tests should reflect requirements, but testers shouldn't test requirements • Testers are the team's conscience 50 Minutes
1 20140925FiSTBKeynote.key - 27 Sep 2014
2-1 20140925FiSTBKeynote.key - 27 Sep 2014
2-2 20140925FiSTBKeynote.key - 27 Sep 2014
3-1 20140925FiSTBKeynote.key - 27 Sep 2014
3-2 20140925FiSTBKeynote.key - 27 Sep 2014
4-1 20140925FiSTBKeynote.key - 27 Sep 2014
4-2 20140925FiSTBKeynote.key - 27 Sep 2014
5-1 20140925FiSTBKeynote.key - 27 Sep 2014
5-2 20140925FiSTBKeynote.key - 27 Sep 2014
http://www.ambysoft.com/essays/agileTesting.html
6 20140925FiSTBKeynote.key - 27 Sep 2014
What we Wish Agile Testing Were
(Big thanks to Michael Bolton and James Bach)
7 20140925FiSTBKeynote.key - 27 Sep 2014
Agile Testing means “Test-first”
… and almost always at the unit level
Specification by example. “Context-driven testing is sometimes considered a ‘flavor' of agile software development.” — http://searchsoftwarequality.techtarget.com/definition/context-driven-testing
8-1 20140925FiSTBKeynote.key - 27 Sep 2014
Agile Testing means “Test-first”
… and almost always at the unit level
Specificatio
n by Examp
le Contextual testingAutomationJUn
it
TDD
"At the basis of the test automation pyramid is unit testing. Unit testing
should be the foundation of a solid test automation strategy and as such represents the largest part of the
pyramid.” — Mike Cohn
Specification by example. “Context-driven testing is sometimes considered a ‘flavor' of agile software development.” — http://searchsoftwarequality.techtarget.com/definition/context-driven-testing
8-2 20140925FiSTBKeynote.key - 27 Sep 2014
Agile Testing means “Test-first”
… and almost always at the unit level
Specificatio
n by Examp
le Contextual testingAutomationJUn
it
TDD
Processes and tools over
Individuals and Interactions
Specification by example. “Context-driven testing is sometimes considered a ‘flavor' of agile software development.” — http://searchsoftwarequality.techtarget.com/definition/context-driven-testing
8-3 20140925FiSTBKeynote.key - 27 Sep 2014
Part I: (Unit) Testing is Muda
9 20140925FiSTBKeynote.key - 27 Sep 2014
Unit Testing is One of the Least Efficient Ways we Know to Remove DefectsRemoval Step Efficiency
Formal Inspections 93%Static Analysis 55%
System Test 36%Functional Test 35%
Component Test 32%
Unit Test 32%
Desk Check 27%Prototyping 20%
Acceptance Tests 17%
Regression Tests 14%
(“Software Defect Origins and Removal Methods,” Draft 5, 28 December 2012, http://www.ifpug.org/Documents/Jones-SoftwareDefectOriginsAndRemovalMethodsDraft5.pdf
10 20140925FiSTBKeynote.key - 27 Sep 2014
Why doesn’t unit testing work?
We can’t test things of business value
Not knowing what to test, we test many things that don’t need testing
Automated crap is still crap
11 20140925FiSTBKeynote.key - 27 Sep 2014
Unit tests can’t test things of business value
In an OO world you can’t tell, from the code, whether a method will ever be executed — or in what sequence
There is rarely any explicit link from unit tests to anything of business value
What’s amazing is that we continue to test… Paths that will never be exercised Methods that are never used Ungrounded method invocation sequences
12 20140925FiSTBKeynote.key - 27 Sep 2014
Not knowing what to test, we test things that don’t
need testing
We test scenarios that will never be exercised in the real world
We test methods that will never be invoked
13 20140925FiSTBKeynote.key - 27 Sep 2014
Automated crap is still crap
Automation for its sake is waste
Design requires intelligence: intelligence cannot be automated
Autonomation instead of automation
You still need exploratory and experience-based testing
Get manual procedures working first and then automate the mundane parts
Design for Test like they do in hardware
14 20140925FiSTBKeynote.key - 27 Sep 2014
Automated testing is not testing
http://qz.com/196200/toyota-is-becoming-more-efficient-by-replacing-robots-with-humans/ Max Niesen, April 7, 2014 James Bach: “Testing Requires Inference.” “Exploratory inference may sound like a strange idea. It means one idea leading to another in ways you can’t predict in advance” [and, therefore, can’t script].
15 20140925FiSTBKeynote.key - 27 Sep 2014
The software is trained to pass the tests
"In software testing, it gets progressively harder and harder to find defects. If we run the same test or use the same technique over and over, the effort becomes less effective each time we do it, because we find only the easy-to-find bugs. The software is trained to pass the test. …” — Ross Smith, Microsoft
16 20140925FiSTBKeynote.key - 27 Sep 2014
That kind of test thinking, leads to a bottom-up
procedural architecture
Citation: Siniaalto and Abrahamsson, Does Test-Driven Development Improve the Program Code? Alarming results from a Comparative Case Study. Proceedings of Cee-Set 2007, 10 - 12 October, 2007, Poznan, Poland.
17 20140925FiSTBKeynote.key - 27 Sep 2014
18 20140925FiSTBKeynote.key - 27 Sep 2014
Part II: Why do we unit test?
Because we can
It doesn’t require teamwork
We don’t have to talk with the business
We get to interact with tools instead of people
Our decisions are emotional, not rational
Last point: Most tests are irrational: i.e., you can never presume software correctness from the tests. But we do feel better by having checked, so we test anyhow.
19 20140925FiSTBKeynote.key - 27 Sep 2014
Nokia Test Score 2: Testing
No dedicated QA - 0
Unit tested - 1
Feature tested - 5
Feature tested as soon as completed - 7
Software passes acceptance testing - 8
Software is deployed - 10
20 20140925FiSTBKeynote.key - 27 Sep 2014
Part III: You Will Have Bugs!
Scenarios we test in the lab
!
!
! !
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
21 20140925FiSTBKeynote.key - 27 Sep 2014
Testing waste
Scenarios we test in the lab
!
!
! !
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail
22-1 20140925FiSTBKeynote.key - 27 Sep 2014
Testing waste
Scenarios we test in the lab
!
!
! !
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
Type 1 Muda
Type 1: Defects
Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail
22-2 20140925FiSTBKeynote.key - 27 Sep 2014
Testing waste
Scenarios we test in the lab
!
!
! !
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
Type 2 Muda
Type 1 Muda
Type 1: Defects
Type 2: Stuff customers
don’t ask for
Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail
22-3 20140925FiSTBKeynote.key - 27 Sep 2014
Testing waste
Scenarios we test in the lab
!
!
! !
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
Type 2 Muda
Type 1 Muda
Type 4 Muda
Type 1: Defects
Type 2: Stuff customers
don’t ask for
Type 4: Overprocessing
Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail
22-4 20140925FiSTBKeynote.key - 27 Sep 2014
Testing waste
!
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
Type 1 Muda
Type 4 Muda
Type 1: Defects
Type 2: Stuff customers
don’t ask for
Type 4: OverprocessingType
1 Muda (field)
Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail
22-5 20140925FiSTBKeynote.key - 27 Sep 2014
Testing waste
!
!
!
Scenarios exhibiting
faults
!
Scenarios invoked in the field
Type 1 Muda
Type 4 Muda
Type 1: Defects
Type 2: Stuff customers
don’t ask for
Type 4: OverprocessingType
1 Muda (field)
Removing tests leaves field faults
undetected!
Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail
22-6 20140925FiSTBKeynote.key - 27 Sep 2014
Reduction of WasteType 1: Better testing (e.g., going beyond contextual testing to exploratory & experience-based testing) (and better yet, better analysis and design, where the real payoffs are)
Type 2: Just get rid of the tests
Type 4: Remove redundant and tautalogical tests
Type 1 in the field: Ship your tests!
23 20140925FiSTBKeynote.key - 27 Sep 2014
Don’t take the tests out when you ship!Remember, you will have bugs in the field!
Deployment is your largest test driver
Ship your tests as part of the code
IN case of a failure, it saves you from Type 9 muda (unsafe execution)
Be customer-focused
24 20140925FiSTBKeynote.key - 27 Sep 2014
Defensive Programming
… is largely the incorporation of tests in the code
Long known for high-reliability systems
The value of asserting the obvious
Design by Contract
25 20140925FiSTBKeynote.key - 27 Sep 2014
Assertions make code readable!
- (void) prepareForSegue: (UIStoryboardSegue*) segue sender: (id) sender { assert(segue != nil); if ([[segue identifier] isEqualToString: @"SegueAvatarPickerToPickOneLoginPlayer"]) { assert((UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPhone)); // The field parentViewController_ (instance variable) is // maybe null if we are in iPhone mode and doing a real // segue, instead of a popover as we do on the iPad. So // get the right object from the segue record. KnowsyPlayerPickerCommonViewController *parentViewController = [segue destinationViewController]; [parentViewController setKnowsyData: knowsyData_]; } else { // iPad - no special processing - should be no popover assert(parentViewController_ == nil); } }
26 20140925FiSTBKeynote.key - 27 Sep 2014
Agile testing isn’t about tools!
It’s about the test effort being customer-focused
It’s about partners instead of customers
It’s about “stopping the line” when discovering a defect — even in deployment
It’s about quality at every step
These are all Lean ideas
27 20140925FiSTBKeynote.key - 27 Sep 2014
Part IV: Beyond agile testing, Lean testingWhen a test finds a bug, it’s about fixing the process — not the bug
The long-term payoff is higher!
No bug-tracking systems: we fix problems NOW.
The tester should be the team’s conscience
Conscience… Curmudgeonly and like Emperor’s New Clothes — not in the sense of standing in the way. That’s business’s job.
28 20140925FiSTBKeynote.key - 27 Sep 2014
Other Lean-isms
Deming: Quality is inversely proportional to the number of testers
GM kept around its cars for 1 - 6 weeks for test and repair: Toyota drove them off the end of the assembly line onto the car carriers.
No unit - subsystem - system - integration - acceptance test leveling.
29 20140925FiSTBKeynote.key - 27 Sep 2014
Conclusion
Be humble about what tests can accomplish
It’s not about testing, but value
Design tests to assess properties of business value
Go beyond unit testing to defensive programming and design-by-contract
As agile testers, focus on process first and product second
30 20140925FiSTBKeynote.key - 27 Sep 2014
31-1 20140925FiSTBKeynote.key - 27 Sep 2014
31-2 20140925FiSTBKeynote.key - 27 Sep 2014
32 20140925FiSTBKeynote.key - 27 Sep 2014