Automation & Performance Testing

1. Automation Overview

2. Project assessment

3. Approach to building automated tests

4. Identifying tasks for automation

5. How should time be costed?

6. Building a tool kit

7. Testing environments

8. Training, Development, Resourcing

9. AXA Wealth Basingstoke & Bristol, identifying Automation needs

10.What Exists

11.Percentage of Coverage

12.Road Map

13.Suggested Priority List

14.Questions

Agenda

• What is test automation?

• Record and Playback, easy right?

• Who should build Automated tests?

• Full time or Part time.

• Time, expense & resource.

Automation Overview

• Test approach covers a section on automation and performance.

• Our technical testers are consulted and a decision is made on the benefits of automation.

• Documented proof we have considered these options – great for audit!

Project assessment for automation & Performance

• Project lead approach.

o Project output should be a set of automated testso Working more closely with development teams:o Buddying automated testers with developers during projects.

Big advantage with Visual Studio to deliver A/T’s as a project output and to smooth the C# learning curve.

• Building automated regression tests to replace manual

o Being held accountable! o Stakeholders should be identified. o Benchmark targets set.o Weekly demos to stakeholders.o Stakeholder knowledge is captured.o No surprises on delivery.

Approach to building automated tests

• Project lead approach.

• Building automated regression tests to replace manual• Focus on high resource drain areas relieving frequent resource effort

and high risk areas, as identified by the test managers.

Identifying tasks for automation.What should drive the priority?

• Complex Functionality

• Frees up experience test resource to analyse

20%

• High Resource• Repetitive Tasks• Common

functionality

80%

• Project Level

• BAU

How should time be costed? breaking down barriers

• Functional Testing • VS2010 Coded UI (Preferred option)

• HP Quick test Pro (Embassy, Siebel)

• Non Functional Testing• VS2010 Performance & Load tests (Preferred option)

• HP Performance Center (Embassy, Siebel)

• Other• Link Checker

• Data generator tools

Building a tool kit

• A known state of test data that can easily be redeployed before tests are re-run.

• Example: doing a switch, that policy would always be available and not used by another tester or disinvested in LIVE and therefore no longer available.

• Embassy data migration work may be reusable to snapshot database.

• Performance Environments

• You know and understand everything that happens in the enviroment.

• Can corrupt databases rapidly.• Cut down LIVE environment, ring-fenced LAN, Benchmarked times and

performance.

• Visual Studio 2010 Lab Management can spin up virtual environments

• Sponsor?

Testing Environments (Automation & Performance)

• Structured professional training in all tool sets.

• Roll out automated tests to other project members, testers, dev’s.

• Mentor and support other test analysts.

• Evaluate recruitment requirement permanent / contractor balance.

• Are the contractors allocated to the new OD available?

• Microsoft time to realise our goal of making VS2010 our preferred tool set.

Training, Development, Resourcing

• What tech test have we got on the Elevate Platform (Automation & Performance)

• What is the strategy for Bristol?

• AXA Wealth website, Performance on Elevate, more….

 AXA Wealth Basingstoke & Bristol. Identifying

Automation needs.

AreaTests Status

OLS Pension Quotes 380 WorkingOLS USP Quotes 30 WorkingOLS Bond Quotes 120 WorkingEmbassy Investment Bonds 18 WorkingEmbassy Direct / Debit (a/c sort validation)

33 Working

Total 581  

What exists (Basingstoke)

Area Tests StatusOCC Tests 13 Effort required due to data changeSiebel 30 Effort required due to data changeTotal 13

Area Tests StatusEmbassy Add Policy 141 Nearly workingEmbassy USP 23 Nearly workingEmbassy Disinvestments 25 Nearly workingTotal 189  

Area Tests StatusChameleon 171 Not Working (Winrunner)Embassy Segmented Bonds 22 Not Working (project was abandoned)Total 193  

Grand Total 976  

• Difficult to translate:

• Manual regression is written in business scenarios, automation deals with functionality areas.

• Automation covers more checks such as, drop down lists, images text on screen, although dilutes exploratory testing to a degree.

• Current Successes

Percentage of coverage

Project Name

Release Date

Test function Tools

Aon Hewitt's May Performance Test Visual Studio 2010

July Release July Automated Regression

Run QTP Regression Pack

September September Automated Regression

Run QTP Regression Pack

AWWIPS Nov AutomationPerformanceBrowser checkerLink Checker

VSCodedUI / QTP Visual Studio 2010BrowserCam / BrowseraLink Checker

Bond (Embassy Light)

Nov – Jan Automation QTP Automation / VS2010

Belfast December Functional Manual Fast Forward - VS2010

FNZ On-going Automation QTP Automation

Embassy Regression

On-going Automation QTP Automation

This years high-level road map

• AXA Wealth Website - 2 perm (training) - length of project, automation & Performance.

• FNZ Regression - 2 Contractors, 1 Perm (overseeing 25%) – 6 Months.

• Embassy Automation regression - 2 Contractors, 1 perm (overseeing 25%) - 6 months.

• Elevate Automation - Understand what's there

• Elevate Performance – We understand there are LIVE issues that need addressing

• Bond (Embassy Light) – Needs to be accessed

• Future projects (Automation & Performance) - To be accessed

Quick priority list

Questions?