end-to-end test automation for both horizontal and vertical scale

End to End Test Automation for Both Horizontal and Vertical Scale

Erdem YILDIRIM

2000 – Software Developer

2006 – Discovered Test Automation &

Decided to be Test Engineer

2010 – Test Lead at Defence Technologies and Engineering (STM)

2013 – Test Lead at İnnova(Telecommunication)

About Me

Linkedin: https://tr.linkedin.com/in/erdem-yildirim-0b5ab466

E2E Test Automation

(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales)

Test Automation Strategy

(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains)

E2E Test Automation Experiences

(ROI, The Problems Faced In the Process)

Need for Both Manual and Automated Testing

(Is Manual Tests Still Needed? Cross Tests)

Agenda

Agenda

E2E Test Automation

(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales)

Test Automation Strategy

(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains)

E2E Test Automation Experiences

(ROI, The Problems Faced In the Process)

Need for Both Manual and Automated Testing

(Is Manual Tests Still Needed? Cross Tests)

Machines are great for runningrepetetive / straightforward scenariosthat don’t require any intelligence orexperience.

Test Automation

End to end testing is verifying that all units of anapplication and all subsystems of a system interact asexpected with each other that the system as a wholeworks as intended.

E2E Test Automation

Ideal Software Test Automation Pyramid

UI

Tests

Integration

Tests

Unit Tests

Manual

(Scripted)

Test

It is OK that automating whole Unit, Inregration, UI levels is great.

Can we improve it?

More Ideal One

CI –

Co

ntin

uo

us In

teg

ratio

n

Exploratory

Test

An Extra Level: Test Data Preparation

How to prepare test dataset and bringdatabase to known state for test cases before

every test run?

Is Teardown Method OK?

Test Data Preparation Automation

Teardown Method with Automation

Setup

Exercise

Verify

Teardown

SUT

Install

Create

Return

ValuesIndirect

Inputs

Record

Mock

Object

Advantages of AutomatingTest Data Preparation Phase

Faster coding,

Faster test executions,

Clean test environment,

Testing with real and more complex test data,

Loose coupling between data and code layers

Facilitates testing for multiple customers (each with a DB)

So, we were able to automate theentire vertical E2E testing:

Unit, Inregration, UI +Test Data Preparation levels

Can we improve it further?

E2E Test AutomationHorizontal & Vertical Scale

UI

Tests

Integration

Tests

Unit Tests

Test Data Preparation

CI –

Co

ntin

uo

us In

teg

ratio

nHorizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay

Vertical E2E

Implementing

tests on all level

of the pyramid

E2E Test AutomationHorizontal & Vertical Scale

UI

Tests

Integration

Tests

Unit Tests

Test Data Preparation

CI –

Co

ntin

uo

us In

teg

ratio

nHorizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay

Vertical E2E

Implementing

tests on all level

of the pyramid

Prerequisites of Horizontal E2E Tests

All system and subsystems’ test environments shouldbe ready.

If they are not ready, mocking work should be performed.

Prerequisites of Vertical E2E Tests

Having a supporting test/development strategy likeTDD, BDD, Continuous Testing

It should be driven by all stakeholders (dev, tester, pm, tl) in Project.

Advantages of Vertical andHorizontal E2E Tests

Advantages of Horizontal E2E Tests:

High business logic coverage,

Testing from the user perspective,

Prevents production incidents,

Gives the confidence that everything is OK

Advantages of Vertical E2E Tests:

High code coverage,

Fast execution,

More determined and focussed tests,

Useful for especially safety critical softwares

UI

Tests

Integration

Tests

Unit Tests

Test

Type

Low

Medium

High

Business Logic

Coverage

Code

Coverage

Low

Medium

High

Execution

Time/Costs

Short/

Low

Medium

Long / High

E2E Test Automation

(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales

Test Automation Strategy

(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)

E2E Test Automation Experiences

(ROI, The Problems Faced In the Process)

Need for Both Manual and Automated Testing

(Is Manual Tests Still Needed? Cross Tests)

Agenda

E2E Test Strategy

Which projects’ tests strategy should be designed as E2E?

Can all tests be written as E2E ?

Test Strategy

I Put That Sh t on Everthing*

Alternate Test Automation Approach

Continuous Testing

To develop software within shorter delivery cycles; Agile, DevOpsand Continuous Delivery approaches are on the forefront.

Continuous Delivery supports Continuous Testing which is a testing strategy that consists of a large number of automated unit and acceptance tests but a small number of automated end-to-

end tests.

Continuous Testing claims that E2E tests are inefficient and theyshould be limited.

It is a means of using Vertical E2E Test Automation effectively but avoiding Horizontal E2E Test Automation

Which Test Automation Strategyis the Best?

There isn’t one absolute strategy that is valid forevery SUT

Criteria / PrinciplesTo Form Test Strategy

Test Types / Methodologies that your staff is strong,

Coverage Strategy: Business Logic Coverage or CodeCoverage

Execution Time Priority

Cost

These Criteria should be considered with the dynamics and needsof Project, Staff, Company, Customer.

It is not an obligation to select & implement a strategystrictly.

You can mix the approaches and form your ownstrategy based on your needs.

It’s about finding the right balance.

Criteria / PrinciplesTo Form Test Strategy

Key Factors To Form & Implement a Successful Test Automation Strategy

Manage test releases effectively

Establish test measurement strategy & metrics

Establish test management strategy

Choose your test delivery model

Determine & implement test tools strategy

Assess your test & process maturity

Feasibility Analysis

Evaluation of Tools / Methodologies

Forming Automation Strategy (Plan, Design)

Implementation

Test & Maintain

Key Factors To Form & Implement a Successful Test Automation Strategy

Based on Delivery Strategy

Automated test executions should be planned andclear CI roadmap should be outlined based on

needs.

An ExampleAutomated Test Execution/CI Strategy

Based on Below Criteria

Delivery Velocity; should be fast so

smoke Unit test first, smoke UI test second,

full test sets last-after deploy in order.

Test Risk Level: Medium; smoke tests for all test types runsbefore deployment. But it is risky to run full sets after

deployment, since bug could stay undetected.

Dev TeamVersionControl

Build & UnitTests

UI Test Automation

(Smoke)Deploy (Prod)

UI Test Automation

(Smoke)

UI Test Automation

(Full Set)Monitoring

Dev CI Pre-Production Production

Check inTrigger

Trigger

Approval

Approval

Approval

Approval

Feedback

Feedback

Feedback

Feedback

FeedbackAutomated Test RunsAfter Nightly Build [1]• Unit Test

Automation• Web Service Test

Automation• UI Test Automation

(Full Set)• Cross Browser &

Platform Tests

Automated Test RunsAfter Weekly Build [2]• Automated Non

Functional Tests(Performance & Load Tests)

• Static Code QualityAnalysis

Feedback

Feedback

Feedback

Communication

Collaboration

Automation

Living Documentation

Drives whole project team

Healthy test basis / requirements

Replaces scripted test with automation scripts

To Strengthen the Automation Process:BDD – Behaviour Driven Development

Benefitsof E2E Test Automation

Velocity

Stability

Efficiency

Money

E2E Test Automation

(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales

Test Automation Strategy

(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)

E2E Test Automation Experiences

(ROI, The Problems Faced In the Process)

Need for Both Manual and Automated Testing

(Is Manual Tests Still Needed? Cross Tests)

Agenda

E2E Test AutomationExperiences

7 active projects’ tests that are being automated

as E2E

ROI

Project Name # of E2E Test

Scenarios

Manual Test

Execution Time

Automation

Execution Time

Ratio of Saving

Legalite 235 6 Days (48 hrs) 1,5 Hour 32 times

HOPE 85 2 Days (16 hrs) 1 Hour 16 times

PROTTON 1200 5 Days (40 hrs) 2 Hours 20 times

TTSis 2000 8 Days (64 hrs) 3 Hours 21 times

Planor 110 3 Days (24 hrs) 2 Hours 12 times

TTS 120 2 Days (16 hrs) 1 Hour 16 times

ATS 30 1 Day (8 hrs) 1 Hour 8 times

Challenges / Problems FacedIn the Process

TechnicalStaff Training and MotivationChoice of Tool and Methodology

PlanningDoing Feasibility Analysis Unproperly.Wrongly designed test automation strategy.

ManagerialDoing incrementally and showing the work & results.

Test Team (Agile) Organizational Structure

PM

PM

PM

PM

Team

Team

Team

Team

T L

Test Engineers TL: Test Lead

PM: Project Manager

Me!

E2E Test Automation

(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales

Test Automation Strategy

(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)

E2E Test Automation Experiences

(ROI, The Problems Faced In the Process)

Need for Both Manual and Automated Testing

(Is Manual Tests Still Needed? Cross Tests)

Agenda

Testing cost: testing accounts for roughly 26% of IT budgets

*According to a 2014 industrial survey of 1543 executives from 25

countries

Lack of testing is even more costly; the global cost of locating

and removing bugs from software has risen to $312 billion

annually and it makes up half of the development time of the

average project.

* A 2013 study by the Cambridge University

Testing Cost vs Lack of Testing Cost

Trying to find defects on System UnderTest using human intelligence andexperience.

Manual testing

Is Manual Test Still Needed?

Mostly: Yes!

Avoid Scripted Testing;

Automate it instead

Concantrate on Exploratory Testing;

Machines are great for running straigtforward

tests

Humans are great finding new defects

Support

your team

by moving testers

across teams

and a little

bit of fun

At the End of Bug Fest

Team Facility;Playing Tabu, Surprise Gifts,

Coffee, Free Time

Thank You

Erdem YILDIRIM

erdemcp@hotmail.com

Linkedin

• V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? A multi-vocal literature review," Information and Software Technology, vol. 76, pp. 92-117, 2016.

• Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl

• V. Garousi and D. Pfahl, "When to Automate Software Testing? A Decision-Support Approach Based on Process Simulation," Wiley Journal of Software: Evolution and Process, vol. 28, pp. 272-285, 2016.

Effective Software Testing: 50 Ways to Improve Your Software Testing, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA ©2002

End-to-End Integration Testing Design, Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International

Distributed End-to-End Testing Management, Enterprise Distributed Object Computing Conference, 2001

HP Capgemini. Sogetti, World quality report 2014-2015. www.sogeti.com/solutions/testing/wqr/, Last accessed: Sept. 2015.[2] T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen, “Reversibledebugging software,” University of Cambridge, Judge Business School, TehnicalReport, 2013

How Google Tests Software, James A. Whittaker

Behavior Driven Development Webapps mit Groovy Spock und Geb

http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/

http://techarcis.com/services/end-to-end-test-strategy/

References