Data, Data Everywhere

Orasi Software Inc.

david guimbellot

• You can’t test quality into a system by hiring a million testers

• It’s not the quantitative testing but the qualitative

Test Data Management

2

The Myth of Production Datax-axis is the customer’s age y-axis is the total the order

3

Minimum Data – Maximum Coverage

4

SAP-1

Data Attribute Mining Data Test Conditions

Test Matching

DB-2

XML

Matched Test

Automated Scripts

Automated Testing – Data Matching

5

Production Attributes

Scenario Subset

Covered SubsetProduction & Enhanced Attributes

Generated Coverage

Covered Data in Development

6

Discover your data

• Use mathematical algorithms to sample data and understands the gaps in your coverage

• Discover the exact data you need to satisfy your testing• Multi-dimensional cubed views allow you to understand how the

data is related• Compare production and development data to indicate progress in

creating data

Profiling

7

Quality Pipeline

Orasi Software Inc.

Acceptance Test Start Create Test VM Validate OS, FS,

Users Setup Testing toolsValidate

services & licenses

Fetch Scripts from SVN Validate scripts

Test Data ProcessValidate data readiness

Execute AcceptanceTest

Publish Test Results

Production Data for Testing – The Gold Copy

Test2

Prod Copy

AuditSeed DataCross Ref

MaskSubset Dev1Test1

Prod

Gold Copy

Test + Development

Dev2

Production

Gold Copy

Orasi Software Inc.

Test Integration

Orasi Software Inc.

First major iteration

Create Test DB1 from Gold Master

Connect test servers to Test DB1

Create checkpoint on Test DB1

RepeatUntil test

pass is Done

Deploy latest changes into test

environmentExecute tests

Reset to Test DB1 checkpoint

A B

Augmenting Data for Coverage

Orasi Software Inc.

First major iteration

Create Test DB1 from Gold Master

Connect test servers to Test DB1

Create checkpoint on Test DB1

RepeatUntil test

pass is Done

Deploy latest changes into test

environmentExecute tests

Reset to Test DB1 checkpoint

Examine data coverage using

Dev DB1

Develop data gen scripts to improve

coverage using Dev DB1

Run data gen scripts at point A

A B

Re-analyze data coverage at

point B on Test DB1

Testing OK ANDCoverage OK

False Run data generation scripts to

improve Gold Master

True

Create Dev DB1 from

Gold Master

Orasi Software Inc.

Build server pipeline

Coverage pipeline

Test Start

Testing setup Validate test readiness

Execute AcceptanceTest

Publish Test Results

Capture DB baseline

Publish DB coverage results

Examine DB after test

Setup instrumented build for C#

Publish coverage to Sonarqube

Generate coverage report

Attach Jacoco for Java

Publish coverage to Sonarqube

Generate coverage report

Centralized BuildStart

Compile & Publish Errors/Warnings Static Analysis &

Publish== 0Meets TargetsUnit Test &

PublishCoverage OK?

No ErrorsLaunch Acceptance

Test

> 0

Build FailureNotify recent

editorsFailed Unit Testing Criteria Failed Static Analysis Criteria

TDM Team – Specialization

TDM Role

Manage test data environment

Analyze TDMproblem space

Create data pools

Validate data pools

Deliver data

Improve test data mgmt

Optimizetest execution

<<include>>

<<include>>

<<include>>

Get access rightsto required DBMS

Keep tools up99.9%

<<include>>

<<include>>

Examine data model

Review productionincidents

<<include>>

<<include>>

Optimize delivery latency

<<include>>

Analyze datacoverage

<<include>>

Assign specific data to testers

<<extend>>

Orchestrate datageneration

<<extend>>

Design mask & subset scripts

Create scripts to increase coverage

<<extend>>

<<extend>>

Test data matches changes & biz rules

<<include>>

TDM Role: Data Design

Manage Test DataDuring Design

Visualize andModel Data

Request Updates toTest Data

Request New TestData

Validate DesignAlgorithms with Test Data

Build Test Data Set

Deliver Schema toPartner

Estimate Data DesignCosts and Resoruces

Unit Test withMock DB

Component Test withMinimum DBMS Solution

TDM Role in Support

Data RelatedProduction Support

Generate DBSnapshots for Test

RCA productionincidents

Resolve DataAnomalies

Errors Traced fromCustomer or Trusted source Errors traced from

tools or reconciliation

Analyze data changeimpacts of design

Estimate DBA/SAtools and script costs

Estimate trainingcosts

Estimate storagecosts

Estimate IO costs

TDM Role: Building Test Data

Build Test Data set

Run ComplianceChecklist

Evaluate datasensitivity

Scrub DataPerform

multi-level authz

Review DB SchemaCreate and tagdata varieties

Scope existingdata and optimize Build error

condition data

Build boundarycondition data

TDM Role: Execution

Manage data duringtest phases

Estimate test teamcosts and resources

Evalaluate testartifacts for TDM

Examine test casesfor coverage

Examine automationscripts for data coverage

Track & prioritizedata demand list

Measure impact ofrequests

Optimize mechanismthat generate data

Build Test Data set

Participate inperodic audits

Manage storage &space requirements

Evaluate datacoverage

TDM Role: Audits

Prove Safe DataPractices during Audit

Gather user listwith read access

Gather OwnershipList

Review practice ofdata from production

Review data scrubprocess

Review Stage toTest/Final process

Configuration, Data, Infrastructure

Tuning, A

lgorithms, Security, etc.

Sour

ce

Code

Build

A

rtifa

cts

Dep

loy

Scrip

ts

Sys Admin TasksVagrant, Chef -test kitchen

xlDeployDatabases

Delphix, CA TDM

Smart MonitorsAppDynamics, Splunk, Arcsight

Unit TestingJunit, Nunit Continuous Integration

Maven, Jenkins, BambooStatic Analysis

SonarQube, Fortify, PMD

PerformanceLoadRunner, Jmeter

PenetrationWebInspect

Device & UIMobileLabs,

MobileCenter, HP UFT

Com

plet

e St

ack

Filter View of Defects

Orasi Software Inc.