automation engineers way to continuous integration · •qa automation manager •conference...
TRANSCRIPT
Automation engineers way to Cont inuous I ntegrat ion
Va d z i m Z u b o v i c h
A p r i l 2 0 1 9
Vadzim ZubovichS o f t w a re t e s t i n g t e a m l e a d e r
S o f t w a r e t e s t i n g t e a m l e a d e r
R e s o u r c e m a n a g e r
Va d z i m Z u b o v i c h
• QA Automation manager
• Conference speaker
• COMAQA activist
• Automation trainer and lecturer
Talk Structure
• Briefly about CI
• Typical steps
• Potential difficulties
• Extra tips
• Summary
Briefly about CIW h a t i s i t ?
Practice of integrating new code all the time
Briefly about CIW h a t i s i t r e a l l y ?
Integrating code means testing it
Briefly about CIC I t o d ay
Evolved from Unit testing to full-scale integration and E2E testing
Briefly about CIW h y d o w e n e e d i t ?
“Continuous Integration doesn't get rid of bugs, but it does make them dramatically easier to find and
remove”
– Martin Fowler
Briefly about CIW h y d o A E n e e d i t ?
Main issues of lacking CI for automation engineers
Briefly about CIW h y d o A E n e e d i t ?
Issue 1: Reliance on someone to run tests
Briefly about CIW h y d o A E n e e d i t ?
Issue 2: Lack of automated test data preparation
Briefly about CIW h y d o A E n e e d i t ?
Issue 3: Issues with tests dependency / parallelism are not spotted until the full tests set is ran
Briefly about CIW h y d o A E n e e d i t ?
Issue 4: No general tests health status check
Let ’s go!
Typical way
Let ’s go!F i r s t s t e p
Need to automate our tests build and run
CI server selection
Food for thoughtW h a t t o c o n s i d e r
Developers CI server
Food for thoughtW h a t t o c o n s i d e r
Licenses
Food for thoughtW h a t t o c o n s i d e r
VCS support
Let ’s go!
Tool selected
Moving on!S e c o n d s t e p
Creating an independent job that runs tests on schedule
What then?T h i rd s t e p
We need test data!
What then?T h i rd s t e p
Adding data within tests (Before fixtures)
Adding data generationD ra w b a c ks
• Longer testing time
• Data cleanup can be an issue
• “Garbage” data accumulated during test runs
What then?T h i rd s t e p
Adding data generation through scripting
Adding data generationD ra w b a c ks
• Need to perform rollback
• Large scripts / high error probability
• “Garbage” data accumulated during test runs
• Direct manipulations of the DB may harm existing data
Alternative?F o u r t h s t e p
Adding “mock” database
Adding “mock” databaseD ra w b a c ks
• Need extra coordination with dev team
• Takes more time to initially create
• If DB is subject to change full replacement will be required
What next?F i f t h s t e p
Running in parallel
Running in parallelI s s u e s / c o u n t e r m e a s u r e s
• Need to make sure the framework is thread-safe
Difficulties: Resolution:
• Group tests smart
• Critical components initialization shouldn’t be static
• Need to maintain tests independence
Extra t ipH e l p i n g t o o l s
• Selenium Grid
Consider using:
• Cloud platforms
• Selenoid
What next?A n d t h e n
Test reporting
Food for thoughtW h a t t o c o n s i d e r
Readability
VS
Food for thoughtW h a t t o c o n s i d e r
Customization
Food for thoughtW h a t t o c o n s i d e r
Team access
Food for thoughtW h a t t o c o n s i d e r
Dynamics tracing
Reporting
Got it right
And then?F i n a l s t a g e
Pipeline – a set of consequent builds
And then?F i n a l s t a g e
Two-stage deployment pipeline
Three-stage pipelineUnit API E2E
Unit API
And then?F i n a l s t a g e
If an E2E stage finds a bug it is fixed with high priority and within the next build the run of this one test is included into CI
All setF i n a l s t a g e
Maintain the order
Extra t ipsF ra m e w o r k d e s i g n
Framework should support maximumparametrization
Extra t ipsF ra m e w o r k d e s i g n
Cross-platform capability
Extra t ipsVa r i e t y o f e n v i ro n m e n t s
How many environments should we have?
DevUnit QAAPI UATUI
Summary
• DO get involved in CI process
• Divide tests in layers
• Think about parametrization
• Pay attention to test reporting
• Consider platform-independence
• Use environments wisely
Thanks for your attention
Skype: csi.vadimzubovich