agile test management joins DevOpsMartin Koch | 2019

It’s 2019 and test management needs to move faster than ever before.

How to combine classic manual testing, test automation, DevOps and smart test management for better results with reduced effort and headaches.

We are showing in an hands on approach, how modern QA and tooling can make your life easier.

Мы привезли подарок ...

Для того, чтобы принять участие врозыгрыше – пришлите текст

“Testcon2019”

На номер:

+7 929 646 8235

Победитель получит приз на стендеaqua в Зале№1 😃

2

who am i?

Martin Koch

Head of Product Management aqua

m.koch@andagon.com

aqua cloud GmbH (andagon)

Germany

andagon.com/en/aqua

3

aqua

• Test management software made in Germany

• Integrates very tightly with Jira

• Manages and simplifies test automation

• Powerful & flexible reporting

4

The challenge in QA

Businesses need to react fast to stay competitive

Do more with less.

5

The need for speed

Complexity of software systems: rising 📈

6

Systems

Complexity

time

The need for speed

Complexity of software systems: rising 📈

Number of Platforms: rising fast📈• OS x Web browsers

• mobile & smart devices

• # of IoT Devices and Platforms

• microservices architecture with multiple services

7

# of Platforms

Systems

Complexity

time

The need for speed

Complexity of software systems: rising 📈

Number of Platforms: rising fast📈• OS x Webbrowsers

• mobile & smart devices

• # of IoT Devices and Plattforms

• microservices architecture with multiple services

Number of testers ️♀️: rising at the same rate?

8

# of Platforms

Systems

Complexity

time

What do the Developers do?

• Shorter Release Cycles10 years ago: months – years

today: shorter and shorter,

multiple times a day

• Continuous Delivery & Continuous Integration

• DevOps

• Agile

9

How to we keep up this speed in QA?

How do we integrate withDevOps processes?

10

Integrate with your Development Pipeline

11

DevOps unwrapped

12

plan code build test release deploy operate monitor

Agile Development

Continuous Integration

Continuous Testing

Continuous Delivery

DevOps

We want Continuous Testing

Faster feedback for QA & devs

Integrate with DevOps pipeline

Needs automated tests• Unit Tests (Developer Tests)• Service Tests• UI Tests

We want

• environment-triggered tests

• traditional developer/tester-triggered tests

13

Diagram: environment-triggered test

Automated Testing ≠ Continuous Testing

14

Test Pyramid

Mike Cohn - Succeeding withAgile

15

End2End: Brittle, break easily unless build smart

Simulate a real user

slower

faster

more integration

more isolation

Are we building the right code?

Are we building the code right?

Test Pyramid in detail

16

Manual Tests

Automated Component Tests

Automated Unit Tests

Automated Integration Tests

Automated API Tests

Automated E2E Tests

Tools

17

Manual Tests

Automated Component Tests

Automated Unit Tests

Automated Integration Tests

Automated API Tests

Automated E2E Tests

Test management suite:

e.g. aqua

UI Tests: Selenium, Appium, Ranorex,

Managing E2E: aqua

xUnit, TestNG,

Mockito, Jasmine

SoapUI, FitNesse, Cucumber,

REST-assured, Postman

Granularity of tests

1. Write tests with different granularity

2. The more high-level you get, the fewer tests you should have

A failure in a high level test means

a) a bug in your functional code

b) a missing / incorrect unit test

Tip: If this bug gets fixed, write the unit test first to make sure it stays fixed.

18

How to build End-2-End tests

• Build End-2-End tests from smaller tests, stack them, combine them

• Reference test scripts, never copy them

For UI tests either:

a) Build simple UI tests that you can record very quickly and re-record them when they break

b)Build smart UI tests• Dedicated Object Repositories• Invest extra time in solid object recognition paths, IDs, etc.• Write code

and of course use a good test automation tool

19

Key take aways

For agile & DevOps speed: Automation!

Make good use of Unit Tests• Cheaper• Focused • Easier to maintain• Bug is fixed? -> Write another unit test

Good End-2-End & UI tests can be expensive (good software* can help enormously)

Build up End-2-End Tests from existing tests.

Combine multiple test levels

Invest in good test data

20

* aqua!

How can you manage all this?

aqua provides a way from manual to automated testing

aqua manages • Manual Tests• Unit Tests (Developer Tests)• Component Tests• Integration Tests• API Tests• End-2-End Test

aqua orchestrates your test automation

21

From manual to automated tests

Good practice:

1. Write a manual test case first

2. Automate this test case

3. Ideal: You manage manual and automated test & scripts in the same tool

4. Ensure traceability

22

user storymanual test

caseautomation

script

Picture: Manual test case in aqua

From manual to automated tests

23

Connect with your Continuous Integration (CI) Server

Moving to DevOps, Developers set up their CI Server and Build Pipelines

This is where the Unit Tests reside and get triggered.

Next up:

We want

• all our test results in one place

• environment-triggered tests

• traditional developer/tester-triggered tests

• Unit, Service, UI & End-2-End tests

Connect your test management solution with your CI server and DevOps pipeline!

24

Jenkins CI

Continuous Integration

Get Unit Test results in the your Test Management solution

25

aqua Jenkins Plugin Test Management

Unit Test results

Case 1 • Save unit test results to your test management suite

• Holistic Reporting • unit test results• manual test results• any automation test results

• Full traceability from user stories to all test results

Continuous Integration

Get Unit Test results in the your Test Management solution

26

aqua Jenkins Plugin Test Management

trigger test automation

Case 2• Start automation scripts managed by your

test managemet suite

• This can be complex End-2-End test scenarios using multiple technologies.

Example:

1. DB script inserts new products into DB2. Selenium creates new Customer3. Selenium buys 2 products4. SoapUI checks results over REST API

Continuous Integration

Get Unit Test results in the your Test Management solution

27

Jenkins CI Test Management

Trigger

Build Jobs / Unit Tests

Case 3 • Start build jobs

• Deploy new software to QA environment

• Run Unit Tests

Trigger an Test Scenario from your CI Server

Example:

• aqua Jenkins Plugin

• Test scenario contains multiple testcases

• All test cases can contain multiple automated test scripts and test data

28

DevOps example test caseThis aqua test case does:

1. Trigger Job in Jenkins

2. Checkout GIT

3. Build of application (SUT)

4. Execution of Unit-Tests

5. automatic Deployment on Tomcat-Server

6. Trigger Functional Test in aqua (TS000265 - Functional Test PetClinic)

7. Results are written to Test Szenario TS000265 -Functional Test PetClinic

8. Detailed results of the single test case can be found here: TC000264 - Petclinic Smoketest

9. Results from Jenkins are written to: TC000266 -Unit test result PetClinic

29

Agents – they make your life easier

• aqua agents manage test tools for you

• They

1. Wait for a task2. They fetch everything they need: test scripts, test data,

files3. Execute a test for you4. Monitor the execution5. Send live updates to the server and you6. They make your life easier

30

31

aqua | Integrations

aqua | Jira

32

Continuous Testing and all is good?

Not so fast...

... flaky tests

33

Tension between QA and development

• Cooperation with developers can be difficult

• Nobody likes to be told they're doing something wrong.

• Classic problem in many companies:

•"Works on my machine"

•"This can't happen."

•"Your test is broken."

34

More Tension!

Example:

• Developer checks in new source code

• Automated tests run on every check-in/build

• Test results show 1 to N failures

• Developer checks his code for errors

Failures from automated tests that are not defects.

Developers spend time searching and debugging just to notice that the test was wrong…

35

The false alarm

Loss of confidence in test results

Loss of confidence in QA

It's difficult to regain lost trust.

36

Flaky Tests!

What are flaky tests? Unreliable tests.

Flaky tests run sometimes green, sometimes red, then sometimes greenagain

… without changes to the SUT or the test scripts

Randomly occurring problems

Flaky tests are poison.

"Unreliable tests are worse than missing tests.”

Analyse them. If they aren’t real bugs: Fix your test automation scripts fast!

37

Test Data

• Don‘t skimp on good test data

• Invest time upfront in a good test data sets & pipeline• Saves a lot of time later!• Script: Write test data to environment • Run tests• Script: Tear down / Clean up used data or environment

How to get good test data?

a) Grab it from production• Secure sensitive parts & anonymize where needed

b) Generate it with dedicated tools• Make your test data smart• A few dynamic parts

• Don‘t underestimate the importance of good, available test data

38

Use agile in your day to day tasks!

• Adopt agile processes in QA

• Organize your QA tasks• Writing new test cases• Reviews• Automating• Updating & maintaining • Managing test environments• Preparing test data• Research new technologies and tools

• Match test cases to user stories & development tasks

39

Picture: aqua scrum board

International Projects?

Screenshots: aqua machine translation 40

41