test automation principles, terminologies and implementations
Post on 20-Mar-2017
443 Views
Preview:
TRANSCRIPT
Test Automation
Principles, Terminologies & Implementations
Steven LiSteven004@gmail.com
2
Outline
•Test Automation Principles & Terminologies Why Test Automation How to do Test Automation Terminologies about Test Automation (Sycamore)
• Implement the Terminologies in Automation Scripts General Conventions An Example - PET
Why Test Automation
4
Top 10 Reasons for Automating Tests - What people said
1. Manual testing sucks.
2. Tool vendor said “Capture replay works.”
3. Fun to watch the dialogs popping on the screen.
4. “Afterwards we can fire all those pesky testers.”
5. Everybody else is doing it.
6. “No humans were harmed in the testing of this software.”
7. Big bucks already spent on the test tool.
8. Looks good on the resume.
9. No Testing for Dummies book ... yet.
10.Keep the intern busy.
5
Reasons for Automating – What people want • Speed up testing• Allow more frequent testing• Reduce manual labor costs• Improve test coverage• Ensure consistency
• Simplify testing• Define the testing process• Make testing more interesting and challenging• Develop programming skills• Justify cost of the tools• Of course we’ll have automation!
How to do Test Automation
7
Keep it Simple, Sustainable
•Uniform/Common infrastructure
•Appropriate tools
•Manage source code, test data, tools
•Agreed Conventions within the team
•High-quality Scripts
•Gradual Test Automation
8
Quality of Automation Scripts•Maintainability
Will the tests still run after product design changes? Will tests for 1.0 work with 2.0? And can be easily updated?
•Reviewability Can others review and understand what is being covered? Are the scripts documented?, can we make sure it matches the original design? How about the coverage? How easy another engineer use the scripts for the next release?
•Reusability To what degree the assets can be reused to create more, different tests? Can they be reused in different releases, in different environments?
•Dependability Integrity Reliability
•Repeatability•Independence•Performance•Simplicity
Terminologies for Test Automation
10
Architecture for AutomationTest Planning Test Preparation Test Perfecting Test Performing
Test Suite
Logical TestBed
Test Case
Test Case
…
Test Case
Test Suite
…
Physical TestBed
Test Case
Test Case
…
Test Case
Test Case
Logical TestBed
Test Case
Test Case
…
Test Case
Test Case
Test Plan
…
Test Step 1Test Step 2…Test Step n
Test Task
…
…
Test Report
Specify TestBed & TestCase ListFor a release Set a task to run a plan
ScheduleRuntime Options +
ScheduleRuntime Options +
Detailed Runtime InformationDetailed TestBed Information
Test Case Result
Test Case Result
…
Test Case Result
Detailed Runtime InformationDetailed TestBed Information
Test Case Result
Test Case Result
…
Test Case Result
Revised Test SuitesTest Plans
Logical TestBed
Test CaseTest Case…
Test Case
Physical TestBed
Test Case
Test Case
…
Test Case
Execute a task
11
Test Bed•Briefly, a test bed is the definition of the environment for a specified test. A test bed include the hardware/software and configuration/provisioning.
•There are two kinds of test beds: Logical TestBed. A logical test bed depicts all the elements and
configuration required for the test, but not indicate which exactly elements will be used (i.e. the index or addresses are not required.)
Physical TestBed. A physical test bed is an instantiation of a logical test bed. A physical test bed exactly indicates which elements are used in the test, and a test task can be really run on the test bed.
• Base State of a test bed The initial state of the test bed, ready for test case (group) start to
run with it. A self-cleaning test case (group) should get the test bed back to the
base state when the execution completed.
12
Test Case (group)•A test case is a sequence of operations against a system whose results can be used to determine if a requirement or use case upon an application of the system is partially or fully satisfied.
A test case contains a set of test steps, and based on a particular test bed.
A test step is an atomic statement of a test case, can have pass/fail result. Usually it contains one operation and the expected result.
A set of test cases can be grouped in a test case file.
•Test Case File (test group): A test case file defines one or multiple test cases, which is based on
a clean test bed for running, and do self-cleaning after performing all the cases one by one.
A test group are a set of test cases saved in one test case file.
13
Test Suite
•A suite of test cases, which share the same test bed to verify a specified feature in a specified test area. Usually, there are multiple test cases in one test suite.
•In automation scripts, a test suite includes: A test suite file: to describe the basic information of this test suite A logical test bed: to define the test bed which the test suite is based
on One or more test case files: each test case file is self-cleaning, so
that each file can run independently.
•Usually, test suite is feature oriented. It can be created once the feature is introduced, and keep being improved along the product life, or even product family life.
14
Test Plan•A test plan is an instantiation of a subset of a test suite, in which a set of test cases selected for a particular test for a feature in a program; and indicate the physical test bed for test performing.
E.g. we have a BLSR_4F_2Node test suite, then we can make a plan for SN16K R9.0 BLSR 4F 2Node Regression test, targeted run it in two days.
•In automation scripts, a test plan includes: Test suite path/name The physical test bed, which is an instantiation of the logical test bed A test case file list, selected from the test suite
•Usually, a test plan is feature and release oriented. For different releases, one can have different test plan for one feature. And, a test plan should be defined in the test preparation stage.
15
Test Task•Test Task is a runtime concept, it defines when and how to execute a test plan. Once a test task is executed, a test report should be reproduced
•In automation scripts, a test task could be a command in CLI, an item in TMS, or a file, which includes:
The test plan path/name A schedule for the test plan running Runtime options, e.g. Halt or skip if any failure? Wait or fail if the test
bed is not available? Recycle or one-time running?
•Multiple test tasks can be created for a test plan, that means, a test plan can be executed multiple times in a release.
•Usually, a test task is created in test performing stage by the tester for one-time running, and will not be reused.
16
Test Report
•A test report is a result record of a test task.
•Test reports can also be organized by test plans or test suites for broadened view to analyze requirements satisfaction.
•A test report should have the following information Runtime information: e.g. who/where/how had run this task with
what options? Detailed test bed information: e.g. build no. for each node, serial
numbers of elements, software versions Test results for each case and each step.
•A test report can be valuable just with a specified test bed on which the test was running.
17
Outline
•Test Automation Principles & Terminologies Why Test Automation How to do Test Automation Terminologies about Test Automation (Sycamore)
•Implement the Terminologies in Automation Scripts General Conventions An Example - PET
18
Classic Reasons for Automation Failure10. No clear objectives for automation. 9. Minimal or No attention of non GUI automation. 8. Lack of methodology to success of the project. 7. Automation is easy Just record and playback. – 6. Aim for 100% automation. 5. Testing for automation scripts. 4. No clarity of real cost involved in automation. 3. Automation is part time job of testers. 2. Wrong selection of Test cases. 1. Not treating automation as software development.
General Conventions
20
A flexible framework•General framework allows users define format and language
•User defined languages for: Test bed Test cases Test report Test task Scheduler ….
•General format XML Perl, Python Simply BDD language, Spreadsheet
21
Plug-in technology
•A platform should support a general framework
•All support simple plug in
•
22
An example: PET
•Pet: Peal Engine for Test It is not just for Perl testing, but Implement a framework to support different test format It is independent to run as an application Or, to be used as a library for your bigger framework
•Key engine: Test Recorder
•Download from: https://github.com/steven004/Pet
An example: PET
24
What is Pet?•Pet – Perl Engine for Test
Perl: Pet is based on Perl. It can run on almost any platforms Engine: It will speed up your work, and provide lots of benefit Test: It is designed for test, not just for test execution, but test
management as well
IQstream
WebUI
SN9K/3K/16K
…
Support different products
Console
HTML
TMS
…
TEA XML Perl …
Support different format/language
Support different outputs
Test Plans/Suites
Pet Engine
Library
Key-Words
Objects
…
Test Cases
Test Beds
Test Recorder
25
Why you need Pet? •TestRecorder Module embedded in Pet
You will get test results via different channels Console output, HTML files for Web, E-mail notification, TMS database, and more if you request All the channels can be configured, you can select any of them as you like/need when running your scripts.
A bunch of test recorder functions provided
•Multi-format supported for scripts writing Pure-perl TEA format compliant Or mix-programming
You can use Perl in your TEA case file; or use TEA format in Perl scripts
Spreadsheet cases
• Scripts running control A mechanism provided to run scripts case by case, or step by step Runtime control provided, by “Ctrl-c” Failure handling, for collect information in your scripts. Error handling, If there is some syntax problem in your scripts, Pet will
let you know.
26
How to run Pet – the 1st step
• Pet looks complicated, but we make it simple
• Let’s start① Log into an automation Server using your unix account
• 172.21.7.200 (Shanghai) or 172.31.115.6 (Chicago)• Run Pet
-bash-3.2$ pet/usr/bin/pet requires arguments. Try ‘pet –h’ for help.-bash-3.2$ pet -h
Usage: Pet [-cnhpv] [testplanfile] [switches] [arguments] -u UserName indicate the who is running the test scripts. Pet will get the user name from OS if no this option -h print Usage hints -l OutputDir indicate the output directory of log. \tmp\pet\username\ will be used for the default output directory -m the mail address for test result reports -v print version, subversion <includes VERY IMPORTANT Pet functionality info> -d [:debugger] run program under debugger (future) -b BuildNumber indicate the build number of the system/software under test -n new a test plan file in current directory based on the existing test bed and test case files
-p parse the test plan and print the case list to the standard output -r ReleaseName indicate the release name/number of the system/software under test -o [tms/console/html/xml/txt] indicate the environment of the Pet. Local run by default -c Pet will just check test bed availability, instead running all cases in the test case list -t TaskID indicate the test task ID for TMS. This option is just required by TMS to run the Pet -f [KEEP_RUNNING/STOP_ON_STEP_FAIL/STOP_ON_CASE_FAIL/PROMPT_ON_FAIL/PROMPT_BY_STEP/PROMPT_BY_CASE/
TEST_QUIT] continue the program or stop it when there is a case fail
27
Pet running examples
•Run a Perl file $ pet example.pl –r 1.0
•Run a Perl-format test plan $ pet example_pl.xml –r 1.2 –b B2.8
•Run a Tea-format test plan $ pet example_tea.xml –r 1.2 –b B2.7 –m steven004@gmail.com
•More parameters $ pet example_tea.xml –r 1.2 –b B2.7 –u xli –m
steven004@gmail.com –t 53432 –o CONSOLE – l /tmp/xli
•Parse TestPlan $pet –p exampletp.xml
28
Outline•Pet Basic
•Quick Start Understand test plan, test suite, test bed and test suite Create your own test task
Testbed modificationTestcase files selection
Where you get the results?Email, HTML, TMS?
• I want to do some change
•Advanced: How can I use Pet to write scripts
•Dig into Pet
29
Terms – Test Plan/Bed/Suite/Task/Report•See details in
Previous slides
•Pet is to run a test plan in which Test Bed, Test Suite, Test Case list are specified
•A test plan example<?xml version="1.0" encoding="UTF-8" ?> <!-- Test Engineer: Guo, Hugh --><!-- Program: Pet Example --><!-- Description: Tea-type example test plan --><TestPlan
SuiteFile="../exampleSuite.pl" PhysicalTestBedFile="../PhysicalTestBeds/example_tb.ttb"><TestCases>
<File>../TestCases/example_tc.ttc</File></TestCases>
</TestPlan>
! Use Relative paths in your test plan, instead of absolute ones!
30
Create your own test task
•Create/Modify TestBed file
•Create/Modify test plan file Indicate the test suite file Indicate the test bed file TestCase files selection Again, Use the relative paths, instead of absolute ones
•Demo …
31
Where is my test results?
•Important Environment Variables Get it by running ‘set’ in Linux-bash-3.2$ set…TESTMAILTO=steven004@gmail.com #results will be sent to this mail.TESTRECOPT='TESTSUMMARY|TESTDETAIL|TESTLOG|ERRORLOG|WARNINGLOG|INFOLOG|DEBUGLOG‘
# All logs will be recordedTESTRECPATH=/shared1/users/IQstream_Result/auto_pet/output/ #The place to save output resultsTESTRECTO='CONSOLE|HTML|EMAIL‘ #You can get results through Console, Email and HTML files.…
•Demo…
32
Outline•Pet Basic
•Quick Start
•I want to do some change Use default output options Set Pet Environment Variables
TESTRECOPT, TESTRECPATH, TESTRECTO, TESTMAILTO Set Pet Options in your scripts
•Advanced: How can I use Pet to write scripts
•Dig into Pet
33
All about TestRecorder
•TestRecorder Manual https://github.com/steven004/Pet/blob/master/Documents/
TestRecorder.htm •TestRecorder Training Slides
https://github.com/steven004/Pet/blob/master/Documents/Use%20PET%20step%20by%20step.ppt
•Default TestRecorder Options had been set in automation servers for all users
For TestRECTO, TESTRECPATH, TESTRECOPT
•You need to set TESTMAILTO
•You can also set your own Recorder Options in your profile
34
Use different options for a particular test
•Two ways to go 1) Specify your options as arguments when you run your test
pet myplan.xml –m notme@abc.com –r 1.2 –b 1 –l /tmp/xli/ This time, notme@abc.com will get the result, and all results
files will be save in /tmp/xli in the automation server
pet myplan.xml –u goldman –r 1.2 –b 3.8 –f STOP_ON_CASE_FAIL This time, pet will think the goldman is running the scripts, instead of
you, the one log into the server, and scripts will stop once there is a case fail.
2) Set in your scripts (not recommended) See more in TestRecorder Training Slides
35
Outline•Pet Basic
•Quick Start
• I want to do some change
•Advanced: How can I use Pet to write scripts Call Flow in Pet TestSuite file writing TestBed & TestCase file
Pure Perl programming TEA format programming Mix-mode programming
•Dig into Pet
36
Call Flow in Pet
•Typical call flowImport TestSuite(run testsuite file)
Import TestBed(run PhyTestBed file)
TestBed Setup
Test Start
Test End
TestBed Cleanup
Run CaseFiles #1 #2
… #n
Case, Step & TestLog related functions (defined in TestRecorder) can be used here, and all activities will be recorded by Pet.
37
TestSuite file•TestSuite file has to be a Perl file•It could be very simple•The simplest example: # A Test Suite Example
use IQsteam::Objects; # it means you will test a IQstream system
•The TestSuite.pl template: ## use the necessary packages here
use IQsteam::Objects; sub TestBedSetup {
#add your scripts here#
1;
}
sub TestBedClearup {
#add your scripts here#
1;
}
1;
38
TestSuite file is serving the test suite
•Anything related to the test suite should be defined in it
•It may contain The packages to be used in the test suite The attributes of the test suite
Test Bed base state definition (TestBed Setup) How to recover the test bed after test completes Any variables/data to be used in the test suite, but not related to a
particular physical test bed Any functions definition for test cases & Test bed initiation
E.g. the testsuite.pl defined for Lab Pcap running
39
TestBed & Test Case files•Test Bed design
Define variables/data related to particular test beds, i.e. Any parameters may change when you change your test bed to do the same test
•Test Case file design Test Case file is just to define test procedures Test Case files should not be modified when you run your test on
different test beds Case / Step / TestLog related functions defined in TestRecorder will be
used in TestCase Files
Test Cases defined in one TestCase file should be self-clean for better test plan making
40
TestBed & TestCase Files – Pure Perl•TestBed Example (from WebUI test suite)
our $BrowserType='*iexplore';our $BrowserURL='http://172.21.5.109';our $CNIPAddress='172.21.5.109';our $client_IP='172.21.160.154';our $PortNumber='4444';
All the definitions are about the particular test bed, and can be easily changed when you run it in another test bed.
•TestSuite ExampleCaseStart(“Just an example");{
InfoMsg(“All the following steps should pass!");Step(3*2==6, "3*2==6");Like(“Shanghai”, “Sh”, “Shanghai =~ Sh");Is(3*2, 6, "3*2 is 6?");
} CaseEnd();……CaseStart(“Other Cases….”); {…} CaseEnd();
More Functions, please see TestRecorder Manual
top related