paul gerrard - the redistribution of testing – where to innovate and what to renovate - eurostar...
DESCRIPTION
Presentation from the 20th annual EuroSTAR Software Testing Conference in Amsterdam, NetherlandsTRANSCRIPT
![Page 1: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/1.jpg)
Paul Gerrard, Gerrard Consulting
The Redistribution of Testing: Where to
Innovate and What to Renovate
www.eurostarconferences.com
@esconfs
#esconfs
![Page 2: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/2.jpg)
The Redistribution of Testing: Where to Innovate and What to Renovate
Paul Gerrard [email protected]
Twitter: @paul_gerrard
Web: gerrardconsulting.com
![Page 3: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/3.jpg)
Theme: If testing is
being redistributed,
what do we need to
re-invent?
![Page 4: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/4.jpg)
(Why) is a re-think
required?
![Page 5: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/5.jpg)
The Redistribution of Testing
• At Eurostar 2011 in Manchester some people were saying testing as we know it is dead or dying
• We think that testing isn’t dying… – It’s being “redistributed”
– There will be significant upheaval for many people
• There are pressures outside testing that will force these changes upon us
• If you aren’t ready to change, you might have to look for a different job: – Some doors are closing. (That might be a good thing)
– But other doors are beginning to open
Intelligent Testing, Improvement and Assurance Slide 5
![Page 6: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/6.jpg)
The new testing squeeze
• Developers adopting test-first, unit-testing, continuous integration
• Stable software, sooner? Good! • But are fewer system and
acceptance testers needed?
• Larger projects treat testing as a standard, factory process
• Certification promotes this
• Certified testers are a commodity • “Anyone can test” attitude prevails • Offshored testing puts unskilled
certified testers out of a job
• Some Agile projects squeeze testers out
• Testers as ‘eyes and ears’ of product owners?
• Is it testing? Can testers cope? • Is Agile testing test assurance in
the small? • A different set of skills are required
Intelligent Testing, Improvement and Assurance Slide 6
![Page 7: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/7.jpg)
A methodological shift…
• A methodological shift from staged to iterative and now ‘continuous’
• Just as we get used to Agile, we now need to consider continuous ‘Specification by Example’ (SbE) approaches (also see TDD, ATDD, BDD)
• SbE promotes a continual process of specification, exampling, test-first, and continuous integration
• Where does the tester fit?
1990s 2000s 2010s
Waterfall Agile Continuous
Intelligent Testing, Improvement and Assurance Slide 7
![Page 8: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/8.jpg)
Specification by example
• SbE promises much:
– Consistent requirements, stories/examples tests and code for the lifetime of a product
– Extremely rapid end-to-end turnaround – 24hours?
– Continuous Integration is the heartbeat, life-support and early warning system
• Of course, workflow, user experience and end-to-end testing are still required
• But these can be managed by business analysts, UX experts and the stakeholder community.
Intelligent Testing, Improvement and Assurance Slide 8
![Page 9: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/9.jpg)
The testing ‘crunch’
• Test teams are being pushed…
– From the ‘low-value’ norm of unskilled teams…
– To upstream, business-savvy, workflow-oriented,
UX-aware testing experts, with new tools
• There is a blurring and maybe merging of roles:
– BAs who can test… or testers who can BA?
• How many of the current generation of testers
and test managers will cope with this change?
Intelligent Testing, Improvement and Assurance Slide 9
![Page 10: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/10.jpg)
Predictions
For the next five years four
three Two?
For an article setting out my reasons and thinking:
http://gerrardconsulting.com/index.php?q=node/602
![Page 11: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/11.jpg)
Predictions 1
1. Half of the current onshore testing community will see their job disappear
2. Business test planning will become part of up-front analysis and design (where it should be)
3. Functional unit testing driven by stories (mostly); Structural testing defined by developers
4. Requirements and business models will become ‘executable’ through stories and examples
5. Some system tests and most acceptance tests increasingly business model-driven
Intelligent Testing, Improvement and Assurance Slide 11
![Page 12: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/12.jpg)
Predictions 2
6. Business models plus stories and examples will increasingly be ‘contractual’
7. System Test detailing – the creation of executable manual tests will be outsourced
8. Manual system test execution will be outsourced (and in the cloud)
9. Maybe half of all acceptance tests will be automated in CI…forever
10.New tools will emerge that manage requirements, stories, scenarios, workflows, prototyping and testing
Intelligent Testing, Improvement and Assurance Slide 12
![Page 13: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/13.jpg)
Where to Innovate and
What to Renovate
![Page 14: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/14.jpg)
A goal network (aka results chain or
logic model)
Slide 14
RISK GOAL
Every project has a network
of dependent interim and
ultimate goals threatened by
risks
Your strategy will identify
the test activities that will
measure goal achievement
and evidence these risks
The ultimate
business goal
![Page 15: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/15.jpg)
Test activities overlay the goal network
(not all goals in scope)
Slide 15
RISK GOAL
Test Phase/Activity
![Page 16: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/16.jpg)
Test strategy and requirements
validation are part of analysis • More emphasis on
business test analysis
• Test strategy is a key
part of business analysis
• Business model includes
stories, scenarios and
process workflows
• Continuous delivery
depends on trusted
requirements
• Advocate test as BOTH
goal and risk based
• We address risk AND
measures achievement
• Work closer with BAs
or even become one
• Requirements validation
is part of analysis
(DeFOSPAM)
Intelligent Testing, Improvement and Assurance Slide 16
![Page 17: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/17.jpg)
Let go of feature testing (checking)
• Features checking is
better done by
developers with tools
• Developers lose their
safety net, but take on
responsibility for their
share of finding bugs
• Scrap incident reporting
(bugs fixed the same
day that they are found)
• Example and validate
requirements with good
stories and scenarios
• System ‘feature’ tests
shift from checking to
exploratory
• System business tests
use business scenarios
plus workflows, e2e
Intelligent Testing, Improvement and Assurance Slide 17
![Page 18: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/18.jpg)
From traditional delivery…
… to Continuous Delivery
Intelligent Testing, Improvement and Assurance Slide 18
![Page 19: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/19.jpg)
The Deployment Pipeline
• Automated Unit
tests
• Automated
Acceptance Tests
• Manual User Tests
Intelligent Testing, Improvement and Assurance Slide 19
![Page 20: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/20.jpg)
Structured Agile Continuous
We are in transition • Developers are adopting new architectures and a more
streamlined approach to development
• Requirements half-life of weeks or months
• Agile evolving towards continuous delivery
• Kanban-Ichiwichi-Hokikoki-Kamikaze-Driven Development – Whatever!
• A hybrid Agile-Structured process:
– Structured Analysis, Agile Delivery?
– That would be SAAD
• Continuous delivery -»-» ‘executable specifications’
Intelligent Testing, Improvement and Assurance Slide 20
![Page 21: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/21.jpg)
Continuous delivery requires
continuous testing and new skills • Re-distributed
functional testing
• A focus on continuous
test and assurance
• Service integration
replaces system
integration
• (Many) acceptance tests
automated for all time
• Non-functional skills -
UX, security, privacy,
performance are in
demand (always was)
• Balancing competing
non-functional
requirements
• User tests focus on
changes, reconciliation;
less focus on e2e.
Intelligent Testing, Improvement and Assurance Slide 21
![Page 22: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/22.jpg)
Test management shifting to
supplier management • Unskilled, onshore test
teams are being
outsourced, offshored
• Trusted requirements
can be contractual
• Scripting tests from
trusted requirements is
largely mechanical – so
outsource it
• Scripted test execution,
automated or manual is
outsourced too
• Role of test manager is
shifting:
– Managing supplier
relationships, not teams
– Managing responsibilities
and information, not test
assets
Intelligent Testing, Improvement and Assurance Slide 22
![Page 23: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/23.jpg)
Test management shifting to
project intelligence management • Testing is not a phase,
it’s a continuous service
• Testing delivers data,
information, intelligence
• Testing provides data on
what the business cares
about:
– Achievement
– Risk
– Coverage (confidence)
• Testers in a technical
bubble need to look
towards the business
• Testing measures both
technical and business
achievement
• Test management shifts
to project intelligence
management.
Intelligent Testing, Improvement and Assurance Slide 23
![Page 24: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/24.jpg)
What about tools?
• Traditional toolsets support 1970s thinking:
– Staged/waterfall
– Testers and testing separated from the rest of project
– Mostly manual, scripted testing
– Tabular & test case-based (Excel works fine. Still!)
• Existing tools focus on test as a purely technical,
internal activity dominated by software testing as
defined in the certification schemes (70s thinking)
• It’s hard to see how these tools can evolve.
Intelligent Testing, Improvement and Assurance Slide 24
![Page 25: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/25.jpg)
Tools: closer to stakeholders
• Stakeholder management: identification, responsibility, commitment, expectations
• Business modelling: processes, collaboration, workflows
• Goal, results and benefits realisation management
• Risk identification, assessment, risk response
• Requirements elicitation, analysis and validation
• Dictionary-based impact analysis, traceability, test coverage and reporting
• Test support: from clerical record-keeping towards knowledge management, modelling and test design.
Intelligent Testing, Improvement and Assurance Slide 25
![Page 26: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/26.jpg)
Tools: project instrumentation and
reports against business models • The business model, stories and scenarios drive
automated checking (ATDD, BDD, SbE etc.)
• Automated checking will includes callbacks to report test status to provide continuous feedback
• Manual, scripted tests might run as cycles, but test status feedback is continuous, not monthly/quarterly
• Manual exploratory test logging adds “colour” to the stark, black and white status of automated checking
• Test status and coverage reporting by goal, by risk, by process, by stakeholder, by user role, business concept – on demand.
Intelligent Testing, Improvement and Assurance Slide 26
![Page 27: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/27.jpg)
Summary
![Page 28: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/28.jpg)
Redistribute testing; reinvent your
role – your choices as test lead 1. Managing/monitoring outsourced/offshore resources
who detail and execute tests
2. Move upstream with stakeholders and analysis
– Providing testing and assurance skills to business
3. Managing the Business Model knowledge base;
– Analysing business impact of change
4. Provide an assurance function to teams, projects, and business stakeholders; be the “TESTMASTER”
5. Manage the project information flows to/from development teams and the CI/test process.
Intelligent Testing, Improvement and Assurance Slide 28
![Page 29: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/29.jpg)
The phase after development is
REWORK, not Testing • Managers might never name this phase
correctly – it exposes them to the truth
• Testing must align with project activity and development, not follow it
• We have to move away from the siege mentality of ‘independent testing’
• Testers must let go of the test phase, embrace the change, not resist it
• Make that leap.
Intelligent Testing, Improvement and Assurance Slide 29
![Page 30: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/30.jpg)
Thank-You
check out businessstorymanager.com
to see how we support these ideas
with technology.
![Page 31: Paul Gerrard - The Redistribution of Testing – Where to Innovate and What to Renovate - EuroSTAR 2012](https://reader034.vdocument.in/reader034/viewer/2022051610/54973de6b47959604d8b5187/html5/thumbnails/31.jpg)
The Redistribution of Testing: Where to Innovate and What to Renovate
Paul Gerrard [email protected]
Twitter: @paul_gerrard
Web: gerrardconsulting.com