automation from start to finish

Testing Automation From Start to Finish

Liang Gao

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

2

Testing Automation Attribute:

贵expensive

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

3

Testing Automation Attribute:

冰山iceberg

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

4

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Frequently functionality regression testing Sample based performance testing Repetitive work for system and solution

testing

5

Automation is useful for….

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Faster turnaround Reduce human errors Cost saving

6

Frequently functionality regression testing

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

7

Sample based performance testing

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Semi-automation Configure 1000 VPN tunnels Create 10000 routing table entries …..

8

system and solution testing

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

9

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

No matter manual or auto

10

The Goal

Accurate

Testing Report

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

11

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

12

Remote Control Your Products

Refrigerator

Scanner

Tablet

Telecommuter House with Router

Television

Television - Closed-circuit

Truck

Video Camera

Video Phone

Tunnel

Set Top Box (STB)

Printer

Personal Computer Web Browser

Weather Station

Web Server

Woman - End User

Woman - Sitting

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

2 ways of software manual testing (or Device Control) Mouse and Eye (expectation) Keyboard and Eye (expectation)

Automation is to repeat Mouse and Eye – GUI automation Keyboard and Eye – CLI automation

13

Remote Control Your Testing

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Windows based GUI automation AutoIT

Browser based GUI automation QTP WinRunner Silktest AutoIT Selenium

14

GUI Automation

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Expect TCL Expect Perl Expect Python Expect

15

CLI Automation

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Most network devices (network equipment) Management interface: Browser or console

Mobile devices Keyboard simulation Serial port, USB port or Bluetooth

16

Embedded devices State of the Union

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

17

A Sigma Approach (in progress)

Replay keyboard simulation, Screen capture and Image

comparison

Result Verification (manual, only once)

Keyboard simulation, predefined key sequence (XML), render engine and

Screen capture

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

18

Pick the Tool is just beginning万里长征第一步

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

19

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Lessons learned: API (Library) design is key for Productivity Modularity Maintainability Expansion

Don’t start with programming, start with API design

磨刀不误砍柴工 – It is true…

20

API Design

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Design pattern or other OO technique apply Need senior software architect blessing More design brain storming sessions, more

reviews, the better. Spend more time on the API design, it will

pay off in the long run

21

API Design

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

22

API Principle

Control Library

API

Scripts

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Script only contains testing actions, no software independent info

23

API Design - Principle

•Connect to Device

•Issue XXX command to send SMS messages

•Verify messages have been sent successfully

•Verify messages saved in the SMS sent folder

•Delete message

•Verify message has been deleted

Device_id = connect_to_device();

Status = Send_SMS(device_id, $string);

Assert($status);

Status = Get_SMS(sent, $string)

Assert($status)

Status = delete_SMS(sent,$string);

Assert($status)

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

API User Manual

Most important document for script development

API 名称

描述

输入参数 参数 1参数 2参数 3

返回值

使用举例

24

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

25

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

26

Script Development Process

×Ô¶¯»¯½Å±¾¿ª·¢Á÷³Ì²Î¿¼

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

All common development coding standard applies Version control Standard template Standard header 20% of the code is comments Well indented Use API and functions Enough error checking

27

Define Script Quality

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

28

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Testing script has it own quality too Exactly match the manual testing steps Just by reading logs, we know exactly what

scripts are trying to do Device and hardware/software environment

independent Enough debugging message when test case

failed Clean out environment before exit or before

execution 29

Define Script Quality

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

30

Script Quality Checklist

Others Script Quality ChecklistScript Quality

ChecklistSigma

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

31

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Always remember our goal You need at least an execution engine for

Error resuming batch run Report generation

32

Automation Framework ？

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Want to design and implement an Automation Framework?

Checkout

<<Automation Framework: Design and Implementation>>

33

Automation Framework

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

34

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

35

Execution Process

Always remember the goal

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Image upgrade process Parallel execution Total run time Debugging time Report collection and generation

36

Execution Process

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Where to get the image Can you automatically get the image and

upgrade to your product Call this a “clean” procedure

37

Image upgrade process

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

To save total run time – sometimes even automation is too long

This is where the script need to be environment independent

Be aware on the equipment cost issue How many parallel scripts can an engineer

run and manage (also equipments)

38

Parallel execution

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Total run time per engineer Total Debugging time per engineer Total reporting time per engineer

39

Run Time

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

If it is parallel, need a process to parallel report collection and generation

Testing Information Database? Web Interface?

40

Report collection and generation

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management

41

Agenda

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Who write the scripts? Who write the framework Who execute the scripts Are they happy

42

Team Management

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Whoever write the test plan, write the test scripts Knows test case the best Not professional coder Hard for project management

Professional scripter Need to understand the test case first High productivity Good Quality

43

Who Write Scripts

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Or any other testing related tools If a tester develop a great testing tool, who

maintains it? Who to coordinate the common share library

development Who to develop testing tools if there is a

request Most of the time developer does this. If have resources, think about a Tools Team

44

Who Write Framework

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Whoever write the scripts, execute the scripts Low productivity Bad script portability Unpredictable schedule (bad for scheduled release)

Dedicated resources (professional executer) Streamline resources High productivity Fixed total run time Most modern companies do it this way

45

Who Execute the Scripts

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Test cases writer and professional scripter Scripter and executer

46

Interaction process

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Repeatable test cases

47

Case Writer and Scripter

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Script integration process Newly develop scripts submit to runner Use “State” to manage

Script maintenance process New image change Use “State” to manage

48

Scripter and Runner

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Manual tester, scripter are doing creative things

Script runner may be board Role rotation Clear role for a clear career path

49

Team Building

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

50

Streamlining the team

脚本开发

脚本执行

脚本开发 脚本开发 脚本开发

脚本执行 脚本执行

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

No matter manual or auto

51

The Goal

Accurate

Testing Report