business unit or product name © 2007 ibm corporation introduction of autotest qing lin
TRANSCRIPT
Business Unit or Product Name
© 2007 IBM Corporation
Introduction of Autotest
Qing Lin <[email protected]>
Business Unit or Product Name
© 2007 IBM Corporation2 Jun 19, 2012
Agenda
Overview of Autotest
Autotest Structure
Autotest Client Harness
Adding tests to Autotest Client
Analyze the test results
Integrate in Jenkins
Business Unit or Product Name
© 2007 IBM Corporation3 Jun 19, 2012
Overview of Autotest – What is Autotest?
Autotest is a framework for fully automated testing.
It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.
It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.
Business Unit or Product Name
© 2007 IBM Corporation4 Jun 19, 2012
Overview of Autotest – Goal
The aim is to produce a client which is open source, implicitly handles errors, produces consistent results, is easily installable, simple to maintain and runs either standalone or within any server harness.
Business Unit or Product Name
© 2007 IBM Corporation5 Jun 19, 2012
Overview of Autotest – General Features
Support for functional, performance, stress tests
Simple inclusion of tests
– Trivial to incorporate existing tests
– Easy to write new tests
– Possible to write very complex tests
Complete control over environment
Consistent, machine-parseable results output
Tests can be executed standalone or as part of a larger-scale system
– Easy to run a prepackaged collection of tests
Business Unit or Product Name
© 2007 IBM Corporation6 Jun 19, 2012
Autotest Structure
Users-->Autotest Client
Users-->Autoserver-->Autotest Client
Users-->Frontend-->Autoserver-->Autotest Client
Business Unit or Product Name
© 2007 IBM Corporation7 Jun 19, 2012
Autotest Client Harness
Harness running on the machine under test
Written in Python– APIs to easily compile and run C code
Each test has a simple wrapper class
Standardized results reporting for all tests
Comprehensive error handling
Control flow is managed by a control file
Business Unit or Product Name
© 2007 IBM Corporation8 Jun 19, 2012
Autotest Client Harness – Anatomy of a test
Business Unit or Product Name
© 2007 IBM Corporation9 Jun 19, 2012
Autotest Client Harness – Test Class initialize()
- This is run before everything, every time the test is run.
setup()
- This is run when you first use the test, and normally is used to compile the source code.
warmup()
-For performance tests that need to conduct any pre-test priming to make the results valid. This is called by job.run_test before running the test itself, but after all the setup.
run_once()
- This is called by job.run_test N times, where N is controlled by the iterations parameter to run_test (defaulting to one). It also gets called an additional time with profilers turned on, if you have any profilers enabled.
postprocess_iteration()
- This processes any results generated by the test iteration, and writes them out into a keyval. It's generally not called for the profiling iteration, as that may have different performance.
postprocess()
- [DEPRECATED] This is called once to do postprocessing of test iterations, after all iterations are complete. Please use postprocess_iteration instead.
cleanup()
-Used for any post-test cleanup. It always gets called, regardless of the success or failure of the test execution.
Business Unit or Product Name
© 2007 IBM Corporation10 Jun 19, 2012
Adding tests to Autotest Client – Contents Today
Tests / Testsuites
It only takes about 10 minutes to wrap a test
Business Unit or Product Name
© 2007 IBM Corporation11 Jun 19, 2012
Adding tests to Autotest Client
An example control file (eg tests/mytest/control)
A test wrapper (eg (tests/mytest/mytest.py)
Some source code for the test (if it's not all done in just the python script)
The name of the subdirectory (tests/mytest), the test wrapper (mytest.py) and the name of the class inside the test wrapper (mytest), all match .
Business Unit or Product Name
© 2007 IBM Corporation12 Jun 19, 2012
Adding tests to Autotest Client – control file
Path :tests/mytest/control
Business Unit or Product Name
© 2007 IBM Corporation13 Jun 19, 2012
Adding tests to Autotest Client – test wrapper
Path :tests/mytest/mytest.py
Business Unit or Product Name
© 2007 IBM Corporation14 Jun 19, 2012
Adding tests to Autotest Client – test call process
Business Unit or Product Name
© 2007 IBM Corporation15 Jun 19, 2012
Analyze the test result
View the rough result
– ./tools/scan_results.py results/default/status
Business Unit or Product Name
© 2007 IBM Corporation16 Jun 19, 2012
Analyze the test result
Result directory
– results/default
Detailed info for each case
Business Unit or Product Name
© 2007 IBM Corporation17 Jun 19, 2012
Analyze the test result
Result in HTML
Business Unit or Product Name
© 2007 IBM Corporation18 Jun 19, 2012
Integrate into Jenkins – Create a Job
Business Unit or Product Name
© 2007 IBM Corporation19 Jun 19, 2012
Integrate into Jenkins – Configuration I
Name
Workspace
Configuration Page
Business Unit or Product Name
© 2007 IBM Corporation20 Jun 19, 2012
Integrate into Jenkins – Configuration II Run test script
Business Unit or Product Name
© 2007 IBM Corporation21 Jun 19, 2012
Integrate into Jenkins – Configuration III Email Notification
Business Unit or Product Name
© 2007 IBM Corporation22 Jun 19, 2012
Integrate into Jenkins – Get the Result in Log