tempest: integrated openstack testing · tempest con guration i around 200 con g options currently...
TRANSCRIPT
Testing before Tempest
I Every project had its own unit tests
I Some projects had functional tests
I There was nothing to test the whole picture
What is Tempest?
I The official OpenStack integration test suite
I Only black-box testing, only interacts with OpenStack REST APIs
I Over 2300 tests covering every incubated and integrated project
I Used as the gating tests for every integrated project
Tempest GrowthI Started in late Diablo to solve the lack of integrated testingI Became a gating test suite in the Folsom cycleI Was one of the 5 most active projects in IcehouseI Testing with tempest is now a project graduation requirementI Being used in refstack as part of defcore validation
Growing Pains
I Huge continued increase in the number of projects and tests
I The common code to handle all the APIs becomes more complex
I Managing individual tests becomes very difficult
I Total run time continues to grow
I Tempest configuration becomes more involved to handle all the services andconfigurations
Design Principles
I Should be able to run against any OpenStack Cloud from devstack to a publiccloud
I Everything explicit
I Only use public interfaces (REST APIs)
I Should be self cleaning
I Self Testing
API Tests
I Directly test the OpenStack REST APIs
I Uses a unified tempest REST client
I Fall into 2 categories positive and negative
I Negative tests are starting to be automatically generated from api schemas
Scenario Tests
I Through path tests of functionality
I Designed to simulate a typical end user action workflow
I Use official python-*clients
I Ideally will involve integration points between projects
CLI Tests
I Test the official client CLIs
I Quick sanity checks of functionality in an OpenStack environment
I Only perform read-only operations
I Mostly a matter of convenience for gating
Stress Tests
I Use tempest to generate load on a cloud
I Dedicated scenarios for generating load
I Can specify any existing tempest test to also be a stress generator
I All scenario tests are automatically added as stress tests
I Run stress tests serially or in parallel looping for a set period of time
3rd Party and Unit Tests
I 3rd Party TestsI Tests for in-tree non-OpenStack APIsI Currently only tests Nova’s EC2 implementation
I Unit TestsI Tempest eventually reached a size where its operations were too complex to take for
grantedI Started late in Havana because of bugs in wrapper scripts ”super passing” the gateI Unit testing coverage was expanded significantly during Icehouse
Parallel Tempest
I Tempest is capable of running it’s tests in parallel
I More closely simulates real load on a cloudI Greatly improves speed of running tests:
I Serially gate runs took 1 hr with <1000 testsI Current gate runs are 40min to 1 hr with 2300 tests
Tenant Isolation and Parallel Execution
I Tenant isolation enables running each test class in isolation
I When enabled each test class creates an isolated user and tenant
I Execute tests in parallel at the class level
Branchless Tempest
I Starting with Icehouse tempest will no longer have stable branches
I For gating purposes we will use master tempest against stable branch proposedcommits
I All proposed tempest commits will run against master on the other projects andthe currently supported stable releases (Icehouse forward)
I This should help us ensure greater api consistency between releases
Tempest Configuration
I Around 200 config options currently
I Intent of config file is to tell tempest what can be run
I What should be run is the domain of the test runner
I Sample config file documents every option
I Intended to be able to cover any OpenStack configuration
I Should have sane defaults (ie match project defaults)I To setup a basic config file:
I Configuring only basic auth and connection info should be neededI Also specify flavor and image ids to use for creating serversI List which services are available and any required configuration for those services
Future Work
I Expansion of autogenerated negative tests
I More Unit test coverage
I Expansion of feature flags in config file
I Configuration file validation and generation tooling
I Possibly a new service for running tempest
Where to get more information
I Tempest Repository
I Tempest Documentation
I IRC #openstack-qa channel on Freenode