agile test automation strategy, v2
TRANSCRIPT
MJ Half-‐day Tutorials 10/3/16 13:00
Test Automation Strategies for the Agile World
Presented by:
Bob Galen
Velocity Partners
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Bob Galen Velocity Partners An agile methodologist, practitioner, and coach, Bob Galen ([email protected]) helps guide companies in their adoption of Scrum and other agile methodologies and practices. Bob is a principal agile evangelist at Velocity Partners; president of RGCG; and frequent speaker on software development, project management, software testing, and team leadership. He is a Certified Scrum Coach, Certified Scrum Product Owner, and an active member of the Agile and Scrum Alliances. Bob published Scrum Product Ownership–Balancing Value from the Inside Out.
1
Test Automation Strategies for the Agile World
Bob Galen President & Principal Consultant
RGCG, LLC [email protected]
Copyright © 2016 RGCG, LLC 2
Introduction Bob Galen n Independent Agile Coach (CEC) at RGCG, LLC n Director, Agile Practices at
n Somewhere ‘north’ of 30 years overall experience J n Wide variety of technical stacks and business domains n Developer first, then Project Management / Leadership, then
Testing n Senior/Executive software development leadership for 20+ years n Practicing formal agility since 2000 n XP, Lean, Scrum, and Kanban experience n From Cary, North Carolina
Bias Disclaimer:
Agile is THE BEST Methodology for Software Development…
However, NOT a Silver Bullet!
2
Outline
n Traditional Automation – Business Case & ROI n 3-Pillars n Agile Test Automation Pyramid n Agile Automation – Business Case & ROI n Implementation Strategy n Communication n Wrap-up
Copyright © 2016 RGCG, LLC 3 3
Let’s start with… Traditional Automation Strategy n What are your current strategies towards:
q Test Automation q Frameworks q Tooling q Maintenance q ROI q Team structure
n Get together in “pairs” and chat about this for 20 minutes. List your approaches and tactics.
n Then leverage SWOT or FFA analysis to capture your current position. Then we’ll gather your results…
Copyright © 2016 RGCG, LLC 4 4
3
Let’s start with… Traditional Automation Strategy n SWOT – Often used to analyze strategic direction or
positions. List the “position”, then identify aspects of: q Strengths q Weaknesses q Opportunities q Threats
Often in Quadrants
n Force Field Analysis – again, list a position q Forces (FOR) è q Forces (AGAINST) ç
Copyright © 2016 RGCG, LLC 5 5
Traditional Automation Business Case & ROI
Copyright © 2016 RGCG, LLC 6
4
It’s simple really…
n Manual testing is: q Slow q Labor intensive q Not intellectually stimulating q Error prone q Iterative q Did I say slow?
n And we’re testing software…right? q So we ought to be able to automate the testing.
Copyright © 2016 RGCG, LLC 7 7
It’s simple really…
n Automated tests are: q Fast q Low/no errors q Free after initial development q Run continuously q Did I say fast? q Did I say ‘free’?
n And we’re testing software…right? q So we ought to be able to automate ALL of the testing.
Copyright © 2016 RGCG, LLC 8 8
5
9 Copyright © 2016 RGCG, LLC 9
ü Capture size of the team ü Cost per tester – usually hourly ü # of Test cases to be run ü Time to run them ü Cyclical nature of the testing cycles (regression, integration,
system, etc.) ü Configurations ü Number of tests run per release ü Number of releases per year
n Do the math to come up with iterative testing run-time investments…and costs
Test Automation ROI Sample Points – Manual Testing
10 Copyright © 2016 RGCG, LLC 10
n Then contrast that against the cost for automation: ü Time / cost to establish automation infrastructure ü Time / cost to establish automated tests ü Time / cost for ongoing maintenance and results analysis ü Cost for automation tooling – depends on ‘type’, ex: Keyword
Driven ü Cost per automation developer vs. manual tester ü Automation replacement target for manual test cases
n Do the simple, discrete math to come up with cost & time
to replace manual testing run-time and ROI savings
Test Automation ROI Sample Points – Automation Development
6
11 Copyright © 2016 RGCG, LLC 11
n Run more tests, faster and with fewer people (lower costs)
n VALUE being driven by: Running tests! n Did I say with fewer people (testers)?
q We can hire more developers with the cost savings! And get MORE done…
q We can even reduce the size of the bathrooms J
Traditional Business Case
Copyright © 2016 RGCG, LLC 12
http://www.aspiresys.com/testautomationroi/index.php#ROI
7
Copyright © 2016 RGCG, LLC 13
http://www.aspiresys.com/testautomationroi/index.php#ROI
Sample of Test Automation ROI calculators n Aspire Systems seems to have pulled down their link,
here are other samples…
q http://sourceforge.net/projects/ta-roi/
q http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=58:roi-calculator&catid=37:calculators&Itemid=65
q http://www-01.ibm.com/software/rational/offerings/testing/roi/tool/ROI_Rational.html
Copyright © 2016 RGCG, LLC 14
8
But often there is underestimation of risks… So the ROI was always a ‘fuzzy’ estimate § Underestimating the nature of a software project
§ Architecture & design, complexity, risks, unknowns, ambiguity, etc.
§ Underestimating parallel software projects (product + automation) complexity § Integration, dependencies, coordination
§ Underestimating ongoing support & maintenance § Repairs & updates § Designing and adding new tests § Impact of new product technologies
§ Underestimating skill levels for automation development
Copyright © 2016 RGCG, LLC 15 15
It’s also very dangerous! Will Robinson… § It’s commoditizing your testing activity. Could you imagine –
§ Measuring developer value by LOC…and doing ROI? § Measuring architectural integrity by complexity analysis…and doing
ROI? § Measuring product owner value by number of backlog elements…and
doing ROI?
Based on some sort of automation introduction? Of course not. It’s insane.
§ Then why do we do it with tests…and testers?
Copyright © 2016 RGCG, LLC 16 16
9
17 Copyright © 2016 RGCG, LLC 17
n We’re all using them, so they must be good and the approaches must be ‘correct’…right?
n However, there are challenges— q Training q Size q UI-centric q COST q Can’t test everything (Back-end database code) q One-size fits all strategy q Limited developer / whole team usage
Traditional Test Automation Tools Just a quick stop…
Typical Deployment Level of effort
Copyright © 2016 RGCG, LLC 18
Initial design & Architecture
Initial Test Case Automation
Technology Disruption
Ongoing Maintenance
MORE
Typical Time & Budget Investment
LESS
é Level of Effort -
Investment ê
10
Now it may sound…
n Like I’m opposed to test automation. Truly, I’m not. I think automation rocks! Or like I’m opposed to gaining a ‘return’, absolutely not. q Automation is clearly a foundation
element for increased quality and going faster – releasing more often
n But, this traditional model or approach of focusing on ROI…is sort of insane
Copyright © 2016 RGCG, LLC 19
3-Pillars of Agile Quality & Testing
Copyright © 2016 RGCG, LLC 20
11
Copyright © 2016 RGCG, LLC 21 21
3-Pillars Genesis
n I’ve learned that “Balance” is important n A sad tale of:
q Thousands of ATDD testing; Gherkin run amok q All of them are working; continuously testing; increasing
“coverage’ and life is Good!
n BUT q These same teams couldn’t write a cohesive User Story to save
their life q So, where were the Acceptance Tests coming from?
3-Pillars of Agile Quality
Copyright © 2016 RGCG, LLC 22
Development & Test Automation
• Pyramid-based Strategy:
(Unit + Cucumber + Selenium)
• Continuous Integration
• Attack technical infrastructure in the Backlog
• Visual Feedback – Dashboards
• Actively practice ATDD and BDD
Software Testing
• Risk-based testing: Functional & Non-Functional
• Test planning @ Release & Sprint levels
• Exploratory Testing
• Standards – checklists, templates, repositories
• Balance across manual, exploratory & automation
Cross-Functional Team Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews & Standards
• Active Done-Ness
• Aggressive Refactoring of Technical Debt
• User Stories, “3 Amigo” based Conversations
• Whole Team Ownership of “Quality” • Knowing the Right Thing to Build; And Building it Right
• Healthy – Agile Centric Metrics • Steering via: Center of Excellence or Community of Practice
• Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement
12
Foundation of the 3-Pillars
Copyright © 2016 RGCG, LLC 23
• Whole Team Ownership of “Quality”
• Knowing the “Right” thing to Build AND Building it “Right”
• Healthy – Agile Centric Metrics
• Steering Required – CoE or CoP
• Strategic balance across 3 Pillars; Assessment,
Recalibration, and Continuous Improvement
• Whole team view includes building it right, everyone tests, everyone demo’s, etc.
• Focus on features/stories, confirmation, conversation, and getting them staged
properly OVER testing
• 4-tier metrics: Quality, Value, Prediction, Team
• Agile strategies need light-handed “steering”; establish a CoE (heavier weight) or a CoP
(lightweight)
• Consider finding an assessment framework and then tying it to your strategy
measurement, recalibration, and continuous improvement.
• Make the foundation visible thru information radiators and metrics
3-Pillars of Agile Quality
Copyright © 2016 RGCG, LLC 24
Development & Test Automation
• Pyramid-based
Strategy: (Unit + Cucumber + Selenium)
• Continuous Integration
• Attack technical infrastructure in the
Backlog
• Visual Feedback – Dashboards
• Actively practice ATDD and BDD
A central part of agile adoption is focusing on CI, 3-tiered Automation development, and Dashboards to
begin incrementally building coverage for faster feedback on changes.
100% automation is NOT the Goal!
In the interim, Hardening or Stabilization Sprints and
having a risk-based Release Train concept help
It’s important that Test or QA not ‘own’ the tooling or all of the automation efforts. The strategy can come from QA, but the tactical automation development is
best left to the team.
Mature teams invest in Automation, Tooling, and Technical Debt reduction as part of Done-ness and
continually add it to their backlogs
13
3-Pillars of Agile Quality
Copyright © 2016 RGCG, LLC 25
Software Testing
• Risk-based testing: Functional & Non-
Functional
• Test planning @ Release & Sprint levels
• Exploratory Testing
• Standards – checklists, templates, repositories
• Balance across manual, exploratory &
automation
Exploratory Testing (SBET with pairing) can be an incredibly effective way to establish a whole-team,
collaborative view towards quality and testing. It also emerges new tests.
Leverage ‘plans’ as a whole-team collaboration-conversation mechanism; at Sprint and Release
levels.
Do not measure testing or tester progress; instead, measure throughput, output, sprint outcomes, and
done-ness escapes at a team level.
You need a balanced test team; not everyone needs to be able to program. But everyone needs to be
passionately skilled testers with curiosity.
Agile testing is a Risk-Based play in every Sprint and across a release sequence.
3-Pillars of Agile Quality
Copyright © 2016 RGCG, LLC 26
Cross-Functional Team Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews & Standards
• Active Done-Ness
• Aggressive Refactoring of Technical Debt
• User Stories – 3 Amigo based Conversations
One of the hardest areas to get ‘right’ culturally. It needs leadership alignment from Quality/Testing to Product to Development and a consistent voice of
whole-team approaches.
This is where LEAN Thinking lives, where whole-team collaboration happens, where professionalism
and craftsmanship are held dear.
I like the view of testers becoming the VOC, champions of quality, and consistent questioners of
what is being build. Are we solving the right problems…as simply as possible. Notions of Minimal
Viable Product / Feature help with focus.
And yes Virginia, there ARE standards, templates, and a focus on x-team consistency!
14
Agile Test Automation Pyramid
Copyright © 2016 RGCG, LLC 27
Copyright © 2016 RGCG, LLC 28
Agile Test Automation Pyramid n Coined by Mike Cohn, ~ 2005 n Establishes the validity of turning Traditional Automation
– upside down
n Invests: q Most effort (%) in Unit Tests q Moderate effort (%) in Middle-tier tests (business logic, API,
component, feature) q Least effort (%) in UI-centric, top down tests
n Granularity of the tests is part of the strategy
15
Copyright © 2016 RGCG, LLC 29
Agile Test Automation Pyramid Often: n Tied to Definition-of-Done from a maintenance
perspective q You break it, you fix it
n Percentages vary; intent does not n Whole-team ownership of automation
q Although it skews at either end of the pyramid n Run as much as possible
q Check-in, Overnight, Cyclically q Prime Directive = Real-time Feedback
Agile Test Automation Pyramid
Copyright © 2016 RGCG, LLC 30
+80%, UI Centric Automation
0-10%, Service or Middle Tier Automation
0-10%, Unit level Automation
0-10%, UI Centric Automation
20-40%, Service or Middle Tier Automation
50-60%, Unit level Automation
Traditional Agile
16
Agile Test Automation Pyramid Mike Cohn; Lisa Crispin & Janet Gregory http://behaviordrivendevelopment.wikispaces.com/Testing
Copyright © 2016 RGCG, LLC 31
Definition of Done iContact example
n As part of story design, consider 3-levels of automation required for solid implementation: q Unit q Cucumber q Selenium
n If the team warrants the automation has value (ROI) then automate it
n Implement those automated test cases WITH the story n Demonstrate them in the Sprint Demo
Copyright © 2016 RGCG, LLC 32
17
Definition of Done iContact example
n Previous sprint automation needs to continue to work; if you break it…then Fix It!
n Previous CI/CD – automation wiring needs to continue to work; if you break it…then Fix It!
n All appropriate (RBT) Acceptance Tests should be regressed so that we haven’t “lost value”
n And in Readiness Criteria q Automation research spikes and architectural look-ahead
Copyright © 2016 RGCG, LLC 33
Brainstorm… Agile, Multi-tiered Automation n Get together in small groups of 4-6 to discuss
n Take a few minutes and think about your current automation approaches: q Tooling, approaches & strategies, strengths, weaknesses,
opportunities, maintenance challenges, future technology, etc.
n What sorts of adjustments would you need to make to take this approach?
n What would be the largest challenges in taking this approach? How might you overcome them?
n Do you “buy” the whole-team view to automation?
Copyright © 2016 RGCG, LLC 34
18
Agile Test Automation ROI
Copyright © 2016 RGCG, LLC 35
What agile has exposed to me… wrt/Automation & Testing n Testers vs. a Whole Team view
q Build teams in ratios – developers vs. testers q Instead invest in cross-functional teams composed of the skills to design,
construct, test, and deliver excellent products
q Only testers write test automation q The entire team writes test automation and, oh by the way, can test as
well
q Only testers test the application q The whole team participates in writing / running automation, functional
testing, exploratory testing, regression testing, etc.
Copyright © 2016 RGCG, LLC 36
19
What agile has exposed to me… wrt/Automation & Testing n Traditional vs. Open Source Tools
q Instead of leveraging singular, UI-centric tools q Leverage multiple tools at all-tiers of your application stack
q Higher cost q Lower cost and a sense of “Community”
q Developers won’t use them; costs q Everyone is leveraging the same tool-sets; AND integrating them in
Continuous Integration and Continuous Deployment
Copyright © 2016 RGCG, LLC 37
What agile has exposed to me… wrt/Automation & Testing n People are your primary value proposition
q Instead of looking at testers as a commodity q Consider good testers to be as valuable as any other team member
q Instead of thinking that testing is a by-rote exercise with little intellectual pursuit
q Testing is a profession. Testers are valuable craftspeople. Testing is an intellectual exercise.
q Instead of thinking the value is in the test cases q The value is in the testers (people) continuously reevaluating the right
things to test at the right time…for valuable feedback
Copyright © 2016 RGCG, LLC 38
20
Copyright © 2016 RGCG, LLC 39
http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html
The NEW Test Automation Business Case n So, don’t
q Count pennies q Count heads q Count cycles
n Work as part of a TEAM q Focus in on testing continuously – just the right things at just the
right time q Providing feedback, transparency, and reaction q Allowing your teams time to learn and improve q Solving your customers’ problems; delivering value
Copyright © 2016 RGCG, LLC 40
21
41 Copyright © 2016 RGCG, LLC 41
n Run more tests, faster, unattended, continuous feedback, learning & adjustments q Yes, saving time. Now what to do with it???
n Drive your new VALUE by: q Early feedback for adjustments—risk avoidance, learning, solving
real problems q Having more time to continuously refine your test cases
(automated and not) q Having more time to focus on building Quality into your products q Creatively testing the application (Exploratory Testing, Test Idea
Generation, Test Design)
The NEW Test Automation Business Case
So, with the additional time… INVEST in your testers & quality ü In building In Quality ü In attacking the customers REAL problems ü In better testing ü In more balanced automation ü In team training; in slack time ü In better collaboration ü In doing more than thought possible ü In creative solutions ü In adapting to new technologies ü In having FUN
Copyright © 2016 RGCG, LLC 42
22
How do we “Sell” this change? n You don’t sell it from a testing perspective. Nor from a bean-counting perspective.
n You allow the results to begin speaking for themselves as we begin focusing on and talking about— q How one builds truly innovative and creative software teams q That understand and solve customer problems q With simple solutions that fundamentally work q And do so iteratively & quickly, while nimbly adjusting to
customer feedback
Copyright © 2016 RGCG, LLC 43
But Bob…
n That doesn't work in the “Real World”!
n My boss would never go for it without ROI justification and hard measures
n Testing IS a commodity
n We’ve done automation perfectly – seeing none of the issues/risks you’ve pointed out
Copyright © 2016 RGCG, LLC 44
23
But Bob…
n Think about what I’m saying; taking the time and…
q Invest in people q Invest in better quality & testing q Invest in better product solutions to customer problems q Invest in improved workflow and delivering high customer value
WOW – how crazy is that? n Meet in small groups of 2-3 (pro / con) and chat for 10
minutes about the “sides” and the possibility for change? n Then we’ll debrief…
Copyright © 2016 RGCG, LLC 45
Agile Test Automation Implementation Strategy
Copyright © 2016 RGCG, LLC 46
Automation is one of the foundational investments of effective Agile Adoptions. It’s not a choice, it’s an imperative! It’s a cost of doing business.
24
Implementation Strategy
Typically a 5-Phased Strategy
1. Strategy – This session: 3 Pillars + Pyramid + Team
2. Team Formation 3. Tools Selection 4. Training & Infrastructure Development 5. Automation Development 6. Care & Feeding
Copyright © 2016 RGCG, LLC 47
Phase 2 Team Formation n GOAL…whole team ownership n Skill-set discussion; SDET vs. Test Professional? n May influence
q Development focus on Unit Testing q Automation team to “gain momentum” q Agile execution (Scrum, Kanban, transparency, demo, etc)
n Connect the dots architecturally n Bias: I like to lead automation with a Test Automation
Architect q Rather than with someone from the development team
Copyright © 2016 RGCG, LLC 48
25
Phase 3 Tools Selection n Open Source vs. Commercial n Singular, all-purpose vs. Tool-kit
q Cobble together or Integrate disparate tools
n Always perform a “Bake off” within your teams n If Open Source, consider support implications n Connect to development tooling
q Including CI and CD
n Additional Considerations: Test data, virtualization, DevOps
n Be prepared to iterate, learn, and possibly fail
Copyright © 2016 RGCG, LLC 49
Phase 4 Training & Infrastructure n Design, coding, and documentation standards n Templates n Community of Practice
q Example code, standards, forum, collaborative groups
n Models: q Keyword, Data-driven, Table-driven, DSL, Model-driven, others q Test Suite; Hierarchy, Naming
n Commercial: invest in training and/or consulting n Open Source: hire and/or acquire direct expertise
q Also can build the expertise
Copyright © 2016 RGCG, LLC 50
26
Phase 5 Automation Development n Move it to your teams ASAP n Make it a part of Definition-of-Done
q It should become a natural outcome and not a choice or question
n Complete it within each sprint q SAFe as a goal model for completing work in the same sprint! q Salesforce as well
n It needs to be a negotiated part of your Product Backlog n Have % targets; n For every story:
q The TEAM decides the requisite automated tests at each level of the pyramid?
Copyright © 2016 RGCG, LLC 51
Phase 5 Automation Development Attacking the Pyramid
n Usually I attack the Unit-level first q Development team engagement; sheer #’s q Baseline quality improvement q Integrated with CI/CD; fast feedback
n Then select either middle or UI tier; context based q Tools selection q Training q Framework development
Copyright © 2016 RGCG, LLC 52
27
Phase 5 Automation Development Attacking the Pyramid
n Complete one layer nearly completely before moving to the next layer
n Unit Test Strategy – Don’t dig the “hole” any deeper q Legacy vs. New q Encapsulate vs. Proper unit tests q Refactoring implications
Copyright © 2016 RGCG, LLC 53
Phase 6 Care & Feeding n Definition of Done & Product Backlogs n Automation evolution roadmap
q Merge with your architectural look-ahead q Merge with your Product Roadmap q Factor in ongoing automation investments
n Refactoring & Repairs as part of your Technical Debt handling
n Consider it at the same level of investment as your application code
n Stop-the-Line n Commitment
Copyright © 2016 RGCG, LLC 54
28
Communication
n Leadership Vision q High-level Plans, Strategic sharing with your teams q Leadership collaboration (Development + QA) q Intent to doggedly pursue automation
n Information Radiators n Roadmaps & Product Backlogs n Sprint & Release Reviews
q Even though it’s hard to show “working software” q Let the “process” transparently communicate your progress,
impediments, plans, challenges
Copyright © 2016 RGCG, LLC 55
Let’s end with… Agile Automation Strategy n Based on what you’ve heard, discussed, learned…
n Get together in “pairs” or small groups and chat about this for 20 minutes. List your strategic approaches. Then leverage FFA to capture your driving forces FOR and AGAINST moving towards Agile Automation
n Most importantly, focus on strategies to overcome your obstacles.
n Then we’ll gather your results…
Copyright © 2016 RGCG, LLC 56 56
29
Copyright © 2016 RGCG, LLC 57 57
Copyright © 2016 RGCG, LLC
Wrap-up THANK YOU!
§ I hope I’ve added to your previous assumptions and understanding § Inspired you to change your view towards Automation
ROI and investment § Introduced you to models for attacking Agile Automation Strategy and ROI
§ Final questions or discussion?
58 58
30
Contact Info Bob Galen Principal Consultant, RGalen Consulting Group, L.L.C.
Experience-driven agile focused training, coaching & consulting
Cell: (919) 272-0719 [email protected] www.rgalen.com
@bobgalen https://www.linkedin.com/in/bobgalen
Podcast on all things ‘agile’ -
http://www.meta-cast.com/
59 Copyright © 2016 RGCG, LLC 59
References
Copyright © 2016 RGCG, LLC 60
31
Links
n http://www.agileengineeringdesign.com/2012/01/7-deadly-sins-of-automated-software-testing/
n Link to PDF of Pillar-1 chapter n Links to “old” STP 3-part automation series n Link to Mary Thorn’s – Agile Test Manager article n + other materials, join the mailing list for download
password at - http://goo.gl/ORcxbE
Copyright © 2016 RGCG, LLC 61
Open Source Tools
n Unit Level q x-Unit q Junit, nunit, phpunit,
n ATDD q FitNesse, Robot Framework
n BDD q Cucumber, JBehave
n UI q Selenium, Watir, HP-UTF
Copyright © 2016 RGCG, LLC 62
32
…DD much from the XP space n TDD - Test Driven Development
q Test-first; Unit Tests as a design aid q Debate around creating the unit tests “first”
n ATDD - Acceptance Test Driven Development q User Story – Acceptance Tests q Executable Requirements
n BDD - Behavior Driven Development q jBehave, Cucumber; Gherkin – Given, When, Then q More useful phrasing
n Tests as functional documentation n Product Owners, Customer – writing tests
Copyright © 2016 RGCG, LLC 63
Agile Testing Quadrants Brian Marick; Lisa Crispin & Janet Gregory
Copyright © 2016 RGCG, LLC 64
Exploratory testing Scenarios
Usability testing UAT
Alpha / Beta
Unit tests Component tests
API tests
Functional tests Story tests Examples Prototypes Simulations
Performance testing Load testing
Security testing Non-functional requirements
Q1
Q2 Q3
Q4
Automated & Manual
Automated & Manual
Manual
Automation, Tools, and
Manual
Business Facing
Technology Facing
Sup
porti
ng th
e Te
am C
ritique the Product