stop the test automation roi-based justification insanity -...

Stop the Test Automation ROI-based Justification Insanity

Bob Galen President & Principal Consultant

RGCG, LLC [email protected]

Copyright © 2012 RGCG, LLC 3

Introduction Bob Galen n  Somewhere ‘north’ of 30 years experience J n  Various lifecycles – Waterfall variants, RUP, Agile, Chaos… n  Various domains – SaaS, Medical, Financial Services, Computer

& Storage Systems, eCommerce, and Telecommunications n  Developer first, then Project Management / Leadership, then

Testing n  Leveraged ‘pieces’ of Scrum in late 90’s; before ‘agile’ was ‘Agile’ n  Agility @ Lucent in 2000 – 2001 using Extreme Programming n  Formally using Scrum since 2000 n  Currently an independent Agile Coach (CSC – Certified Scrum

Coach, one of 50 world-wide) q  at RGCG, LLC and Director of Agile Solutions at Zenergy Technologies

n  From Cary, North Carolina n  Connect w/ me via LinkedIn and Twitter if you wish…

Bias Disclaimer: Agile is THE BEST Methodology for Software Development…

However, NOT a Silver Bullet!

Outline

n  Traditional Automation ROI n  Traditional Business Case n  Risks & Reality n  Agile as an Automation Disruptor n  New Business Case n  Investments & Selling n  The Big But n  Wrap-up

Copyright © 2012 RGCG, LLC 4 4

It’s simple really…

n  Manual testing is: q  Slow q  Labor intensive q  Not intellectually stimulating q  Error prone q  Iterative q  Did I say slow?

n  And we’re testing software…right? q  So we ought to be able to automate the testing.


It’s simple really…

n  Automated tests are: q  Fast q  Low/no errors q  Free after initial development q  Run continuously q  Did I say fast? q  Did I say ‘free’?

n  And we’re testing software…right? q  So we ought to be able to automate the testing.


7 Copyright © 2012 RGCG, LLC 7

ü  Capture size of the team ü  Cost per tester – usually hourly ü  # of Test cases to be run ü  Time to run them ü  Cyclical nature of the testing cycles (regression, integration,

system, etc.) ü  Configurations ü  Number of tests run per release ü  Number of releases per year

n  Do the math to come up with iterative testing run-time investments…and costs

Test Automation ROI Sample Points – Manual Testing


n  Then contrast that against the cost for automation: ü  Time / cost to establish automation infrastructure ü  Time / cost to establish automated tests ü  Time / cost for ongoing maintenance and results analysis ü  Cost for automation tooling – depends on ‘type’, ex: Keyword

Driven ü  Cost per automation developer vs. manual tester ü  Automation replacement target for manual test cases

n  Do the simple, discrete math to come up with cost & time

to replace manual testing run-time and ROI savings

Test Automation ROI Sample Points – Automation Development


n  Run more tests, faster and with fewer people (lower costs)

n  VALUE being driven by: Running tests! q  http://www.aspiresys.com/testautomationroi/index.php?

action=__reset

n  Did I say with fewer people (testers)? q  We can hire more developers with the cost savings! And get

MORE done… q  We can even reduce the size of the bathrooms J

Traditional Business Case


http://www.aspiresys.com/testautomationroi/index.php#ROI

But often there is underestimation of risks… So the ROI was always a ‘fuzzy’ estimate §  Underestimating the nature of a software project

§  Architecture & design, complexity, risks, unknowns, ambiguity, etc.

§  Underestimating parallel software projects (product + automation) §  Integration, dependencies, coordination

§  Underestimating ongoing support & maintenance §  Repairs & updates §  Designing and adding new tests §  Impact of new product technologies

§  Underestimating skill levels for automation development


It’s also very dangerous! Will Robinson… §  It’s commoditizing your testing activity. Could you imagine –

§  Measuring developer value by LOC…and doing ROI? §  Measuring architectural integrity by complexity analysis…and doing

ROI? §  Measuring product owner value by number of backlog elements…and

doing ROI?

Based on some sort of automation introduction? Of course not. It’s insane.

§  Then why do we do it with tests…and testers?



n  We’re all using them, so they must be good and the approaches must be ‘correct’…right?

n  However, there are challenges— q  Training q  Size q  UI-centric q  Cost q  Can’t test everything (Back-end database code) q  One-size q  Developer / whole team usage

Traditional Test Automation Tools Just a quick stop…

Typical Deployment Level of effort


Initial design & Architecture

Initial Test Case Automation

Technology Disruption

Ongoing Maintenance

Typical Time & Budget Investment

Now it may sound…

n  Like I’m opposed to test automation. Truly, I’m not. I think automation rocks! Or like I’m opposed to gaining a ‘return’ I’m not q  Automation is clearly a foundation

element for increased quality and going faster – releasing more often

n  But, this traditional model or approach of focusing on ROI…is sort of insane


What agile has exposed to me… wrt/Automation & Testing

n  Testers vs. a Whole Team view q  Build teams in ratios – developers vs. testers q  Instead invest in cross-functional teams composed of the skills to design,

construct, test, and deliver excellent products

q  Only testers write test automation q  The entire team writes test automation and, oh by the way, can test as

well

q  Only testers test the application q  The whole team participates in writing / running automation, functional

testing, exploratory testing, regression testing, etc.



n  Traditional vs. Open Source Tools

q  Instead of leveraging singular, UI-centric tools q  Leverage multiple tools at all-tiers of your application stack

q  Higher cost q  Lower cost and a sense of “Community”

q  Developers won’t use them; costs q  Everyone is leveraging the same tool-sets; AND integrating them in

Continuous Integration and Continuous Deployment



n  People are your primary value proposition

q  Instead of looking at testers as a commodity q  Consider good testers to be as valuable as any other team member

q  Instead of thinking that testing is a by-rote exercise with little intellectual pursuit

q  Testing is a profession. Testers are valuable craftspeople. Testing is an intellectual exercise.

q  Instead of thinking the value is in the test cases q  The value is in the testers (people) continuously reevaluating the right

things to test at the right time…for valuable feedback



http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html

The NEW Test Automation Business Case n  So, don’t

q  Count pennies q  Count heads q  Count cycles

n  Work as part of a TEAM q  Focus in on testing continuously – just the right things at just the

right time q  Providing feedback, transparency, and reaction q  Allowing your teams time to learn and improve q  Solving your customers’ problems; delivering value



n  Run more tests, faster, unattended, continuous feedback, learning & adjustments q  Yes, saving time. Now what to do with it???

n  Drive your new VALUE by: q  Early feedback for adjustments—risk avoidance, learning, solving

real problems q  Having more time to continuously refine your test cases

(automated and not) q  Having more time to focus on building Quality into your products q  Creatively testing the application (Exploratory Testing, Test Idea

Generation, Test Design)

The NEW Test Automation Business Case

So, with the additional time… INVEST in your testers & quality ü  In building In Quality ü  In attacking the customers REAL problems ü  In better testing ü  In more balanced automation ü  In team training; in slack time ü  In better collaboration ü  In doing more than thought possible ü  In creative solutions ü  In adapting to new technologies ü  In having FUN


How do we “Sell” this change? n  You don’t sell it from a testing perspective. Nor from a bean-counting perspective.

n  You allow the results to begin speaking for themselves as we begin focusing on and talking about— q  How one builds truly innovative and creative software teams q  That understand and solve customer problems q  With simple solutions that fundamentally work q  And do so iteratively & quickly, while nimbly adjusting to

customer feedback


But Bob…

n  That doesn't work in the “Real World”!

n  My boss would never go for it without ROI and hard justification

n  Testing IS a commodity

n  We’ve done automation perfectly – seeing none of the issues/risks you’ve pointed out


But Bob…

n  Think about what I’m saying; taking the time and…

q  Invest in people q  Invest in better quality & testing q  Invest in better product solutions to customer problems q  Invest in improved workflow and delivering high customer value

WOW – how crazy is that? n  Meet in small groups of 2-3 (pro / con) and chat for 10

minutes about the “sides” and the possibility for change? n  Then we’ll debrief…


Copyright © 2012 RGCG, LLC

Wrap-up

§  Hope we challenged your existing assumptions a bit §  Inspired you to change your view towards Automation

ROI and investment §  What did I miss?

§  Final questions or discussion?

Thank you!

27 27

Contact Info

Bob Galen Principal Consultant,

RGalen Consulting Group, L.L.C.

Director of Agile Solutions, Zenergy Technologies,

Experience-driven agile focused training, coaching & consulting

Contact: (919) 272-0719

[email protected] [email protected]

www.rgalen.com

Blogs Project Times -

http://www.projecttimes.com/robert-galen/ Business Analyst – BA Times -

http://www.batimes.com/robert-galen/ My Podcast on all things ‘agile’ -

http://www.meta-cast.com/

Scrum Product Ownership – Balancing Value From the Inside Out published by RGCG in

2009.


stop the test automation roi-based justification insanity -...

Documents