automation engineers way to continuous integration · •qa automation manager •conference...

Automation engineers way to Cont inuous I ntegrat ion

Va d z i m Z u b o v i c h

A p r i l 2 0 1 9

Vadzim ZubovichS o f t w a re t e s t i n g t e a m l e a d e r

S o f t w a r e t e s t i n g t e a m l e a d e r

R e s o u r c e m a n a g e r

Va d z i m Z u b o v i c h

• QA Automation manager

• Conference speaker

• COMAQA activist

• Automation trainer and lecturer

Talk Structure

• Briefly about CI

• Typical steps

• Potential difficulties

• Extra tips

• Summary

Briefly about CIW h a t i s i t ?

Practice of integrating new code all the time

Briefly about CIW h a t i s i t r e a l l y ?

Integrating code means testing it

Briefly about CIC I t o d ay

Evolved from Unit testing to full-scale integration and E2E testing

Briefly about CIW h y d o w e n e e d i t ?

“Continuous Integration doesn't get rid of bugs, but it does make them dramatically easier to find and

remove”

– Martin Fowler

Briefly about CIW h y d o A E n e e d i t ?

Main issues of lacking CI for automation engineers

Briefly about CIW h y d o A E n e e d i t ?

Issue 1: Reliance on someone to run tests

Briefly about CIW h y d o A E n e e d i t ?

Issue 2: Lack of automated test data preparation

Briefly about CIW h y d o A E n e e d i t ?

Issue 3: Issues with tests dependency / parallelism are not spotted until the full tests set is ran

Briefly about CIW h y d o A E n e e d i t ?

Issue 4: No general tests health status check

Let ’s go!

Typical way

Let ’s go!F i r s t s t e p

Need to automate our tests build and run

CI server selection

Food for thoughtW h a t t o c o n s i d e r

Developers CI server

Food for thoughtW h a t t o c o n s i d e r

Licenses

Food for thoughtW h a t t o c o n s i d e r

VCS support

Let ’s go!

Tool selected

Moving on!S e c o n d s t e p

Creating an independent job that runs tests on schedule

What then?T h i rd s t e p

We need test data!

What then?T h i rd s t e p

Adding data within tests (Before fixtures)

Adding data generationD ra w b a c ks

• Longer testing time

• Data cleanup can be an issue

• “Garbage” data accumulated during test runs

What then?T h i rd s t e p

Adding data generation through scripting

Adding data generationD ra w b a c ks

• Need to perform rollback

• Large scripts / high error probability

• “Garbage” data accumulated during test runs

• Direct manipulations of the DB may harm existing data

Alternative?F o u r t h s t e p

Adding “mock” database

Adding “mock” databaseD ra w b a c ks

• Need extra coordination with dev team

• Takes more time to initially create

• If DB is subject to change full replacement will be required

What next?F i f t h s t e p

Running in parallel

Running in parallelI s s u e s / c o u n t e r m e a s u r e s

• Need to make sure the framework is thread-safe

Difficulties: Resolution:

• Group tests smart

• Critical components initialization shouldn’t be static

• Need to maintain tests independence

Extra t ipH e l p i n g t o o l s

• Selenium Grid

Consider using:

• Cloud platforms

• Selenoid

What next?A n d t h e n

Test reporting

Food for thoughtW h a t t o c o n s i d e r

Readability

VS

Food for thoughtW h a t t o c o n s i d e r

Customization

Food for thoughtW h a t t o c o n s i d e r

Team access

Food for thoughtW h a t t o c o n s i d e r

Dynamics tracing

Reporting

Got it right

And then?F i n a l s t a g e

Pipeline – a set of consequent builds

And then?F i n a l s t a g e

Two-stage deployment pipeline

Three-stage pipelineUnit API E2E

Unit API

And then?F i n a l s t a g e

If an E2E stage finds a bug it is fixed with high priority and within the next build the run of this one test is included into CI

All setF i n a l s t a g e

Maintain the order

Extra t ipsF ra m e w o r k d e s i g n

Framework should support maximumparametrization

Extra t ipsF ra m e w o r k d e s i g n

Cross-platform capability

Extra t ipsVa r i e t y o f e n v i ro n m e n t s

How many environments should we have?

DevUnit QAAPI UATUI

Summary

• DO get involved in CI process

• Divide tests in layers

• Think about parametrization

• Pay attention to test reporting

• Consider platform-independence

• Use environments wisely

Thanks for your attention

Skype: csi.vadimzubovich