mark meninger-feb-2010

An Examination of Test Automation Use Cases(and the factors behind the examination)

January 19, 2010

Mark Meninger

Confidential McAfee Internal Use Only

An Examination of Test Automation Use Cases

Contents

• The Speaker

• Presentation Objectives

Objectives and Perspective

• Objectives of test automation

• GUI vs non-GUI

• A look at 2 products

Test Automation Philosophy

Test Automation Use Cases

How does this all fit in?

January 20, 2010

The Speaker (Mark_Meninger@McAfee.com)

• Current role @ McAfee: Driving test automation on Consumer side

• Functional GUI automation (support for L10n) across 3 sites

• Functional non-GUI automation (development focused)

• Automated Performance testing

• Previously @ RIM

• As test automation manager for end-user BlackBerry testing

• RIM Test Automation Conference Chair

• A student of test automation and testing

• Worked (and work) with some very capable individuals

• Made some good mistakes

• Learned from my mistakes

• Little experience with Agile

• Presentations @ StarEast, QAI, KWSQA

• My Blog (as of this Friday): http://automatedtestingblog.blogspot.com

January 20, 2010

Presentation Objectives

1. Talk about my learnings and philosophy of test automation

2. Hear your perspective on what I’m talking about

3. Evaluate test automation use cases/examples (group-

exercise)

a) Interface complexity

b) Speed of test automation

c) Speed of execution

d) Derived value

e) How this fits into the testing cycle

4. In the context of Agile, I would…

January 20, 2010

Mark’s Test Automation Philosophy

January 20, 2010

Manual Testing - A path in the trees

• Strategy/Plan (resourcing, infrastructure, approach)

• Need to learn the SUT:

• User-stories

• Use Cases

• Tools

• Technologies

• Testing types & cycles (regression, defect, smoke)

• Sprints

• Apply common techniques

• Scripted test cases (with different methodologies),

• Exploratory testing

• Then testing can begin

• I’ve seen most testing organizations do this well to varying degrees

January 20, 2010

Automated testing – Find a path in the forest

• Test automation is less of a cookie-cutter approach than manual testing

• Common tasks in test automation

• Interface, framework, integration

• Very specific considerations:

• The technology of the system under test (SUT)

• The technologies used to test the SUT

• Impact on the development/testing schedule

• These details impact the success of the

implementation, the derived value, the

time to delivery, availability etc

• This makes each test automation

implementation unique and challenging

• Not all orgs do test automation well

January 20, 2010

Evolution of the Approach to Test Automation

In the beginning:

Here is a test automation tool… now automate! (seen it)

January 20, 2010

And Then:

Evaluate tools, choose one and automate!

(done it)

January 20, 2010

After much learning (now do it!) :Examine my test automation requirements

a) Look at

• Objectives

• SUT technology, roadmap

• Available budget, resources

• Available tool technologies, capabilities and constraints

• SDLC methodology (Agile, Waterfall etc.)

• Development culture

• Testing culture

• Relationships

b) Put together my business case

c) Start with a careful plan

January 20, 2010

Objectives of test automation

1. Executed successfully

relatively early within the

development/testing cycle

2. Reduction in manual testing

�Test functionality sooner

rather than later in the cycle

�Interface used is available

and stable

�Automated execution as soon

as a build is released:

� Evenings, Weekends

�Automated execution run in

parallel

� Execution across platforms and

test suites

� Limited by availability of

infrastructure

January 20, 2010

Test Automation Objectives

3. Provides reliable results on

1st run

4. The solution is scalable &

maintainable

�Tests can be executed repeatedly

with minimal false negatives

�Regular testing of interface and

framework essential

�Changing interface not breaking

�We can increase coverage of

manual test cases without excessive

framework growth

�Grow test case numbers with

confidence:

� 10’s -> 100’s

� 100’s -> 1000’s

January 20, 2010

Test Automation Objectives

5. Agile Environment

�Out-of-the-box, light-weight test

automation supports planned sprints

�Test automation team structured,

organized to support agile testing

requirements

�Solution, approach is flexible and

expandable

January 20, 2010

GUI vs non-GUI Technology Considerations

GUINon-GUI(some API)

January 20, 2010

Test Automation – The Big Question

UI-Logic Interface

And Another

The Basement

Another Abstraction

Do we start here?

Or here?

Where?Where?Where?Where?

SoftwareAbstractions

First Business Logic Layer

January 20, 2010

Test Automation - GUI Fat Client (Generalities)

• Interface

• Typically complex - more points of failure

• Changes more frequent

• Increased maintenance

• Automation - dependent upon being stable

• Technology

• Vendor tools provide better support

• Scripting technology usually has limited support (i.e. VBScript)

• Performance

• Management for unresponsive GUI

• GUI adds performance hit each time accessed

• Framework can add considerable overhead

GUI - Fat Client• Skill-set

• Requires solid technical skill-set with good design and programming abilities

• Deeper test automation skill-set and experience required

• More expensive

• Localization

• Running identical functional tests across localized builds

• Language independence support is essential to support L10n automation

January 20, 2010

Test Automation – GUI Web (Generalities)

• Interface

• Typically simpler – fewer complexities to manage

• Changes frequently over time

• Automation – dependent upon being stable

• Technology

• Good open source (Selenium, Watir)

• Performance

• Fewer if no performance issues

• Good available open source frameworks

• Skill-set

• Better use of ‘scripters’ rather than developers

• Less investment required in more experienced test automation skill-sets

• Less expensive

• Localization

• Language independent testing can be supported

GUI - Web

January 20, 2010

Test Automation – non-GUI (Generalities)

• Interface

• Stable and reliable earlier in the dev/testing cycle

• Less changes over time

• Fewer points of failure

• High-level testing can be supported

• Usually needs to be modified for increased testability over time (fits Agile well IMO)

• Technology

• Choose scripting technology with robust library support

• Integrate into development cycle

Non-GUI (API) Automation

• Performance

• No GUI impact - much faster

• Choose integrated framework with little overhead

• Skill-set

• Requires knowledge of underlying business logic, application architecture and design

• More expensive

• Localization

• Issues around language independence (localized strings integrated into GUI abstractions)

January 20, 2010

GUI Automation

• Automation Implication

• Coverage

• Good top-to-bottom testing solution

• Broader

• # of automated test cases dependent upon GUI comlexity

• Development (GUI complexity)

• Higher maintenance costs

• More effort to write tests

• Longer development times

• Execution

• Dependent upon stable GUI

• Slower execution times

• More expensive (resources, equipment and license cost)

• Value-add later in dev/testing cycle for products with major GUI changes

January 20, 2010

Non-GUI Automation

• Automation Implication

• Coverage

• Not top-to-bottom

• Deeper testing

• Larger # of automated test cases

• Development

• Easier to write tests

• Shorter development times

• Lower maintenance costs

• Execution

• Ready when the API are ready

• Test much sooner in the dev/test

• Test results faster; test more often

• Gate before release to testing

• Cheaper (resources, no licenses)

• Add quality paradigm to development organization (code for testability)

• Value-add sooner in dev/testing cycle

Non-GUI Automation

January 20, 201022

RTQA Build Release Cycle

API Automation

Quality AssuranceStarts

Automated

API Testing

Manual

Testing

TestsPass?

Automated

API Testing

Automated GUI

Testing & Manual

Testing

TestsPass?

Manual TestingGUI Automation

Start End

YesYes

January 20, 2010

Automated Testing Coverage

Complex GUI - Cost for Coverage

• Complex GUI applications will incur an increasing cost to

automate larger #’s of test cases

• Cost: Effort, Infrastructure, Dollars, Time for Execution

Automation Ceiling

Costs become prohibitive

GUIGUIGUIGUI

January 20, 2010

Automated Testing Coverage

Automation Philosophy Review – Cost for Coverage

• Non-GUI Automation has lower cost for coverage

• Cost: Effort, Infrastructure, Dollars, Time for Execution

Automation Ceiling

Less expensive

GUIGUIGUIGUI

NonNonNonNon----GUIGUIGUIGUI

Consumer Test Automation Framework (CTAF)/QTP/MAGI

MAGI FrameworkServices

Core Functions

CTAF Extensions

HP Quick Test Pro

GUI HTMLControl

ID HTMLControl

CTAF External LibrariesGUI/Table

LanguageDependentAssert

VBScript Test Script

CTAF Internal Libraries

Unit Test

Helper

GlobalMPF MVS

Lots of librariesto build(and support)

Complex Framework(yet very helpful)

Our ownextensions

Python py.test/nose(?) Test Automation Framework

Python

Libraries

COM InterfaceImplement IDispatch

High-Level Class

Python Test Script

CTAF API Python

Libraries

Example API Test Automation – Using Python

Python hasnumerousrich and diverse libraries

Framework exists which nicely integrates with Python

January 20, 2010

A look at some interfaces

• Google Search

• McAfee Security Center (2010 release)

Disclaimer

• These thoughts are my own and are not necessary correct

• Part of the process would be for me to understand the

technology and determine the best approach

(Finding a path through the forest I find myself in)

Points to Evaluate

• What approach would you take?

• What are the risks of doing it this way?

• What are the costs/investment requirements of doing this? (be

specific)

• What are the gains of doing this?

• Why would you do this?

• How would you integrate this into a testing cycle?

• Why would you integrate this into the testing cycle this way?

• How could this approach be fit into an Agile/Lean

methodology?

• What type of buy-in would you need to achieve this?

• What relationships would you need to establish to be

successful?

January 20, 2010

What I will focus on

• Interface complexity

• Speed of test automation

• Speed of execution

• Derived value

• How this fits into the testing cycle

January 20, 2010

Test Automation Use Cases – Google Search

•Google Search

– From: http://en.wikipedia.org/wiki/Google_search

– PageRank Logic

– synonyms, weather forecasts, time zones, stock quotes,

maps, earthquake data, movie showtimes, airports, home

listings, and sports scores

– prices, temperatures, money/unit conversions ("10.5 cm in

inches"), calculations ( 3*4+sqrt(6)-pi/2 ), package tracking,

patents, area codes,[6] and rudimentary language

translation of displayed pages

– ‘I’m feeling lucky’

– Privatization logic (Switzerland)

– Ajax code

– ….

January 20, 2010

Our job to automate Search logic • Core functionality

• Page rank is an algorithm that evaluates an index using

supposedly over 200 factors

January 20, 2010

•What I’d do– Back-end (non-gui) automation

– Focus just on the engine and it’s data

– Evaluate the probability and weighting of each factor

– Generate a list of results and probably would fit into

some level of confidence of accuracy

– Rendering of data could be easily verified manually

January 20, 2010

•How• Work with developers to build a testing engine that could handle

‘plug-ins’ of new testing factors

• A common testing pattern would be for each factor

• Determine how each factor would return results on its own or in

interaction with another factor

• Integrate this all into the testing engine

• Establish a common mechanism for executing tests, gathering

expected results and comparing actual results

• Utilize the common testing pattern

• Utilize the same pattern for evaluation in the testing engine

• Drive a common testing interface across those adding new ranking

functionality to the google search engine

January 20, 2010

Interface complexity

• Interface would be simple – an API

• The combination of algorithms (factors) would make this

complex

Speed of test automation delivery

• Fast

– Could start writing tests for each factor

– Could start building in some relationships for each factor

• Slower

– Integration of everything together

– This would also include the test engine

January 20, 2010

Speed of execution

• Very fast

• Could run very often

Derived value

• Substantial

• Could fully automate the algorithm

• If the integration of factors together could be done

successfully, this would be substantial

How this fits into the testing cycle

• Immediately

• Write code, test code

• No throw-over to SV&V/QA

January 20, 2010

Cialis - erectile dysfunction drug

Radical Prostatectomy - is an operation to

remove the prostate gland and some of

the tissue around it Very fast

January 20, 2010

Test Automation Use Cases – McAfee Security

Center

•McAfee Consumer Product (2010)

– Security Center

– Firewall, AntiVirus, AntiSpam, Parental Controls…

– Focus on AntiVirus

– Verify GUI and Functional behaviour

January 20, 2010

• McAfee Consumer Product (2010)

– GUI – thick GUI

January 20, 2010

• How

• Vendor tool

January 20, 2010

Center

•What I’d do & How– Get a good GUI automation tool

– Find a manageable way to integrate with the GUI DOM

• Re-usability

• Maintainability

– First go: limit my automation to easiest test cases

– Find a good framework to integrate with my GUI

automation tool

• Automate the automation

– Aim would be reduce manual testing with an eye to

reduce maintenance

January 20, 2010

Center

• Interface is very complex

– Lots of objects to manage (lots of points of failure)

– How we work with the interface is complex under the

covers (constrained by tool)

• Slow!!

– Most off-the-shelf have limited library support

– Have common set of libraries across the products

– Need to build libraries for each product at the GUI & functional

– Integration into the framework could take even longer

January 20, 2010

Center

Speed of execution

• Slow!

• Fat client GUIs are usually slower

• Hooks from application driving the GUI adds overhead

• Integrating a framework that drives all this adds additional

Derived value

• Some!

• New GUI; some changes – later in the cycle

• Same GUI; little changes – earlier in the cycle

• Need more infrastructure to support

January 20, 2010

Center

• If no GUI changes - > Right Away

• GUI changes - > Later

• Testing cycles will take longer

• Slower to execute

January 20, 2010

Test Automation – The Big Question

Test Automation Use Cases – McAfee Security Center

UI-Logic Interface

And Another

The Basement

First Business Logic Layer

Another Abstraction

Do we start here?

Or here?

SoftwareAbstractions

Center

•What I’d do & How– Examine the layers below the GUI

– Can I use any of them to automate testing?

– Work to drive this idea within the organization

– Most likely some developer has thought the same thing

– That is my foothold and I build on that

January 20, 2010

Center

• Interface is less complex

– New technology learning curve

– Easier to call some API than manage GUI objects

– Less change; more stability

– Fast!!

– Available frameworks (don’t need to build your own) (i.e. py.test)

– Access to rich scripting environments & libraries (i.e. Python) – don’t

need to build your own

– Less points of failure to manage

January 20, 2010

Center

Speed of execution

• Fast!! No GUI overhead

• Integrated framework will also add little overhead (i.e.

TestNG, Py.test)

Derived value

• Higher value

• Almost always test earlier in the cycle

• Test more frequently

• Integrate into the development cycle rather than QA

– Quality moving upstream

January 20, 2010

• Have your test automation specialist begin to develop a methodology that

will fit your agile development cycle

• Tell her/him what your requirements are and ask for a solution that will

meet this

• Optimize your test automation execution for Agile!

• This will most likely be an out-of-the-box approach to test automation

• All pay-offs should be well understood:

• Light-weight, quick to execute, easy to develop, not-too-deep solution

• Heavier, complex, longer-to execute, harder to develop, deeper test

automation solution

January 20, 2010

• They should have a good understanding of the available technologies to

use and what the trade-offs are for each

• What solution will best meet the above requirements?

• Let that test automation specialist know what’s needed and why. This will

hopefully inspire them to build the solution that meets these needs

• Integrate automated testing into your testing cycles

• Fit the automated testing for a given sprint/release

• Each automated testing approach will have a given set of benefits and

• Choose the automated testing items that make most sense

Consumer Applications QATest Automation Strategy – Detailed Review

January 20, 2010

Thank-youMark_Meninger@McAfee.com

As of this Friday

http://automatedtestingblog.blogspot.com

mark meninger-feb-2010

Documents

betydningen af narrativer - holdspil.ku.dk · dette, er at...

hug hug auto-id data work team - update berlin, 1 st feb...

st mark missionary baptist church announcements - 2010 feb...

g020009-00-l e7 lessons learned mark coles feb. 4, 2002

mark a. evans, klein isd pdas in the classroom session id#:...

mark & michelle...

2017 bookmark web - amazon web services · 2017. 1. 4. ·...

here come the pharisees! mark 2 13 -- 3-6 - feb 3 2013

welcome to mrs. meninger & ms. wurm fourth grade meet the...

maxim sl secu mark u conversi on mark income college name k...

nasjonal kartlegging av unges utdannings- og …...6...

climate change (and you) uf, school of building construction...

vgrwebsites.blob.core.windows.netvgrwebsites.blob.core.windows.net/youngfoundations/...-feb...

procurement and g-cloud | mark pinheiro | feb 2014

mark sheet for iii-feb-2015.xls

ashburton district council solid waste activity management...

mark stafford smith, science director climate adaptation...

why your networking is not working kim meninger executive...

28 feb 2012 - asia opportunity, by mark ross [questex asia,...

and the beat goes on . . . mark 3 7-35 - feb 10 2013