Migration and consolidation at GE:All You ever wanted to know about Real

Application Testing

Földi Tamás – Starschema Kft.

GE Lighting EMEA

Data warehouse

Oracle 9i, HPUX

13 000 tables

~1400 ETL jobs

~8000 ETL mapping

Reporting

BusinessObjects 6.5

180 standard, 860 ad-hoc report

1200 uses (average 60-80 concurent)

www.starschema.net

BI Upgrade: project goals

www.starschema.net

Support

• Replace non-supported applications

New functions for Business

• Modern dashboards and reports

Performance

• Apply industry specific best practices

Cost reduction

• Standard, consolidated solutions

BI Upgrade: project objectives

www.starschema.net

Oracle 11g

Merge instances

Automatic Storage

Management

SQL Plan Management

SQL Tuning, Parallel Query

BusinessObject XI 3.1

Upgrade to version XI 13.0

Migrate reports to WebI format

LiveOffice

Metadata Manager

PL/SAP Connector

Efficient SAP interface

New staging system

Advanced Analysis (OLAP)

Supply Chain

Commercial Finance

Sales

Oracle action plan

www.starschema.net

Consolidation

Merge Stage and DWH instances

Less license, more performance

Upgrade from 9i to 11g

SQL tuning (orainit, PX, SPM)

Compression

Storage

Reorganize table spaces (ASSM)

Automatic Storage Management (ASM)

Remove unused objects (based on audit)

Oracle upgade risks

Software errors

Bugs, incompatibilities

Data mismatch

Cannot reach the goals

The performance gain is less than expected

www.starschema.net

www.starschema.net

www.starschema.net

www.starschema.net

Real workload

• Executed SQLs

• Timings

• Network load

Evaluation

• Reports

• SQL tuning sets

Real Application Testing

www.starschema.net

Testing process

www.starschema.net

www.starschema.net

RAT preparation

Make a plan

Merge instances

Rearrange table spaces

Tune parameters

Apply patches

Oracle 9.2.0.8 + „one-off patch” 9373986

Oracle 11.2.0.2 + „one-off patch” 11691834

Estimate storage

Do test recordings

Disable API loads (direct path)

www.starschema.net

Timing

www.starschema.net

Production

Install patches

Full backup

Restart database

Start recording

RAT system

Build new environment

Full backup

Preprocess recordings

Playback

Restore

Tuning sets

Reports

Recording

Rec files

Could be several gigabytes

Performance impact

ARCHIVE mode

Disable „direct path”

Etc. 10% performance degradation

Select interesting period

Record 24 hour workload

Started in the same time on all instances

www.starschema.net

www.starschema.net

Playback preparation

Consistent state

All data are the same on prod and RAT

Reset system clock to same time as production

Backup

RMAN backup or Snapshot Standby

Processing

Process one time, playback many times

Filtering

Non-used schemas (EM, AWR)

www.starschema.net

Consolidation

www.starschema.net

RAT officially does not support multiple records playback on single instance.

But there is a workaround.

Playing records from multiple instances on single node: Sources must be captured in same time

All rec files must be copied into single folder, but with one header and footer

Only datetime based playback is supported, SCN based sync is not working

Playback

Iteration

Scenarios with different prameters or environments

Workload playback

Go back to the starting state

Make AWR baselines

Replay clients

Client Calibration Advisor

Synchronization

When replaying from multiple instances, only timestamp based is working

www.starschema.net

Reports

www.starschema.net

DB Replay

• Data divergence

• Execution errors

• Performance impact (ADDM,

ASH, AWR)

SQL Performance Analyzer

• Compare performance (CPU,

IO, etc.)

• Tune regressed SQL using

SQL Tuning Advisor

• Analysis results can be

used to seed SQL Plan

Management repository

Results

Consolidation

Two, merged database instances can ran on

the same hardware and instance

Reduced license and maintenance fees

Data integrity

There was no data divergence

Optimization

Regressed SQL statements were isolated

10-15% performance gain

www.starschema.net

RAT vs. Without RAT

www.starschema.net

DB Replay

Wo DB Replay

0

20

40

Development/

RAT setup

Install

simulation environment

Generating

Workload

Run Test Evaluation

Reports, tuning sets

Re

qu

ire

d r

es

ou

rce

s (

da

ys

)

10

DB Replay: 2 weeks

Without DB Replay: 10 weeks

Required time for testing

30

4 5 0

1020

20

5

Summary

Easy-to-use

Real workload instead of synthetic

Make backups before all steps

Must have for multiple instance consolidation

Efficient with AWR

Non-supported loads (direct path)

Non-used schemas must be filtered out

Use PL/SQL interface instead of EM

www.starschema.net

www.starschema.net

Thank You

Used pictures

http://www.flickr.com/photos/govert1970/87971108/sizes/l/

http://www.flickr.com/photos/museeinformatique/3783304683/sizes/l/

http://www.flickr.com/photos/littledebbie11/3208631777/sizes/l/

http://theinspirationroom.com/daily/2008/mercedes-crash-test-for-breast-

cancer/

http://www.flightsafety.com/html/pdf/0981_sim_bro_final.pdf

http://tonguc.files.wordpress.com/2009/07/11g_real_application_testing.ppt

www.starschema.net