lights-out testing for lights-on business

1

Lights-Out Testing for Lights-On Business

Ty Henson

September 20, 2017

2

Digital RiskIt’s not just about security, hackers, and data theft

3

Agenda

• Effective automation testing

• Creating on-demand automation

• The right tool for the job

Presented by Ty HensonWorksoft, Senior Solution Architect

4

Late Defect Discovery is Expensive and Disruptive!

0

9599

22781331

$18 ($0)

$180 ($1.7 M)

$360 ($820K)

$1,800 ($2.3 M)

$0

$200

$400

$600

$800

$1,000

$1,200

$1,400

$1,600

$1,800

$2,000

0

2,000

4,000

6,000

8,000

10,000

12,000

Discovered inRequirements/Specs

Discovered in TestExecution

Discovered in UAT Discovered inProduction

Defect Correction Cost (5 Project Sample)

Defects Cost to correct per defect

5

Mainframe Client Server Web 1.0 Web 2.0 / Mobile HTML 5 VR & Biometrics

1980 2000

1990

2020

2010 2030

Modern Day UI EvolutionAs the way applications are written changes. The way we test must also.

6

Business Analysts/QA

“Does my Business Work?”

Developers/Test Engineers

“Does my Software Work?”

UI Tests – The Bulk of Business Application Testing

6

End to End (UI)

Integration ( API)

Unit (Code)

UI Testing Remains the Most Challenging to Automate.

Last 10% takes 90% of the time!

7

Digital Business ProcessesComplex

8

1 Discover

Business Analysts/Super Users

2 Automate

Quality Assurance/Automation Professionals

3 Run

IT Operations/Project Management

Capture, visualize, document & analyze end-to-end business processes

Worksoft Analyze®

Build and maintain a reusable portfolio of business process test automation

Worksoft Analyze®, Worksoft Certify®

Run automation at industrial scale for projects & ongoing operations

Worksoft Certify®,Impact, Execution Manager, Business Process Procedure

The Worksoft Solution

9

Demonstration

10

Agenda

Effective automation testing

Cross-Platform End to End

• Creating on-demand automation


12

• End to End Process

• Enter Order

• Outbound Delivery

• Post Goods Issue

• Invoicing

Test Data Concepts

13


+ Add Inventory

• Enter Order



• Invoicing

Test Data Concepts

14


+ Find or Generate Materials

+ Add Inventory

• Enter Order



• Invoicing

+ Cleanup for Repeatability

Test Data Concepts

15

Agenda



Creating on-demand automation

Self-sufficient and repeatable


16

• Hundreds of tests under management

across multiple applications

• Tests must often be scheduled in

sequence with complex dependencies

• Difficult to manage distributed and

diverse testing resources

• Central monitoring can be a challenge

• Testing time is strictly limited and

closely scrutinized

The Trouble with Automation

17

UI Automation Challenges

Access Orchestration Scale

18

1 Discover

Business Analysts/Super Users

2 Automate

Quality Assurance/Automation Professionals

3 Run

IT Operations/Project Management

Capture, visualize, document & analyze end-to-end business processes

Worksoft Analyze®

Build and maintain a reusable portfolio of business process test automation

Worksoft Analyze®, Worksoft Certify®

Run automation at industrial scale for projects & ongoing operations

Worksoft Certify®,Impact, Execution Manager, Business Process Procedure

The Worksoft Solution

19

Industrial Scale Intelligent Scheduling

• Distributable agents to [geographically distributed] physical or virtual workstations

• Mass parallel execution ability

• Automatic load balanced execution

• Dependencies to control intelligent execution

Lights Out Testing and Continuous TestingIndustrial Scale Intelligent Scheduling and Interoperable Execution

Interoperable Execution

Providing applications like those above the ability to trigger automation execution

• Rest API calls

• Command line interface (CLI)

And more…

20

Running Certify From CI - Jenkins

Jenkins builds and creates build number

Build Number

Call Execution Manager

Execution Manager

REST

Run Certify Tests

Get Status

Remote Test Lab 1CI/CD Server

EMAgent

EMAgent

EMAgent

On premise Test Lab 2

21

Demonstration

22

Lights Out Testing (LOT) provides the ability to

execute a battery of integrated tests at pre-

determined times. These tests will be re-runable

and will be based on exception reporting,

requiring manual intervention only if the test fails.

Tests can be scheduled at predetermined times

and leverage:

• Off-hour bandwidth

• Use of infrastructure and personnel for other

tasks

• The execution of large, data centric integrated

tests during off hours

• Elimination of manual validation and audit

What is Lights Out Testing (LOT)?“Because we’re covering all

of the business processes

in an automated fashion,

we’re finding defects that

also give us a snapshot of

the data that’s causing a

script to fail. Because we’re

now on a 24-hour testing

cycle, we can isolate the

cause more efficiently,

which also results in cost

savings.”

— Sada Krishnan, IT Project

Manager, Reliant

23

Cross Platform End to End Automation

Bidirectional Communication with Other Applications and capabilities

And more…

SOA Rest APIMock

Services

File

SystemRelationa

l Queries

Microsoft

OfficeExecutio

n

Execution Manager

Interoperability

24

Quality and Validation for DevOps and Packaged Application Testing SDLC

Requirements

• Test Cases Documented

Architect Develop

• Build Unit tests

CI &

Unit Test

• Run Unit Tests

CD to Staging Integration

• API Functional

• API Load Testing

• Application Functional

UAT

• End to End

• Requirement validation

• Exploratory Testing

CD to Prod Smoke

• Production Validation

Prepare


Discover

• Business process changes

• Integration

Realize

• End to End



Go Live Run


• Developer based testing specific to

development of custom applications

• Application functional testing common to

both Testing SDLC – UI and end user

focus

Ide

a

Ma

jor

Developer

Testing

25

Supporting Continuous Integration and Continuous Delivery

Develop

Code Check in

BacklogGroom Stories

Sprint starts

CI Continuous Integration CD Continuous Delivery

Development

Sprints

Requirements


Architect Develop

• Build Unit tests

CI &

Unit Test

• Run Unit Tests

CD to Staging Integration

• API Functional

• API Load Testing

• Application Functional

UAT

• End to End



CD to Prod Smoke


Worksoft

Execution

Manager

Application is deployed and connected

End-to-end is available

Development testing when

code is not connected

26

Digital Business ProcessesComplex

27

Agenda



Creating on-demand automation

Self-sufficient and repeatable

The right tool for the job

Presented by Ty HensonWorksoft, Senior Solution Architect

28

Case Study:

US Federal Government Program

10 million+ participants; 400 core processes & 500,000

business process steps tested nightly

Virtual

Machines (150+)

29

Case Study:

Global Luxury Goods Manufacturer

Enterprise wide automation effort for end-to-end business

process validation with Worksoft Certify.

Significant reduction in production defects for SAP, Web

applications and more.

600 hours of automation

running daily on 30+ virtual machines

55,000 hours

of manual

labor annually

replaced

80% business

coverage of 500

core processes.

SCALE SAVINGS COVERAGE