Active Data Warehousing

Can you have your cake and eat it too ?

Jeff MonicoGeneral Manager Information Systems

Agenda

INTRODUCING STAR TRACK EXPRESS

THE ‘REAL TIME ENTERPRISE’

THE TRIGGER FOR CHANGE

ACTIVE DATA WAREHOUSING

THE RESULTS

OPERATIONAL INTELLIGENCE

QUESTIONS

Company Overview

Australian owned and operated Express Freight Company.

Founded in 1974 by Greg Poche. Greg’s influence and core guiding principles are very much part of the business ethos today.

December 2003 saw Australia Post and Qantas form a joint venture to acquire Star Track Express from its founder.

On the 18th May 2011 the retail division of Australian Air Express was merged into Star Track Express to create Australia’s largest door-to-door express service provider.

4,500+ Employees, 3,750+ Vehicles (Vans, Trucks, Prime Movers, Trailers, Forklifts).

Each day over 400,000 items of freight are picked up and delivered, with an industry leading ~99% success rate.

Core Services

How we collect and use data

Business activity has a very short life cycle, with data being collected and used at many points during that lifecycle……

TIME EVENT DESCRIPTION16:45 Freight picked up18:30 Freight unloaded at Sortation Depot18:40 Freight sorted at Sortation Depot into Lane 2119:00 Freight loaded onto interstate trailer M20005:00 Freight unloaded at Sortation Depot05:35 Freight sorted at Sortation Depot onto Centre Wishbone07:00 Freight loaded onto delivery truck ABC12309:30 Freight successfully delivered09:30 Consignment Note signed by Bob.

16:45

05:00

18:30

05:35

18:4019:00

09:30

07:00

DATA COLLECTION (SCANS)

DATA USAGE

Problems we face

Large Volumes of Data

Producing millions or records per day which are needed for operational and analytical queries.

Fast Response Times

Users demand sub second response times on operational queries.

Near Real Time Loads

Real time business needs real time data from a variety of sources. Latency of more then a minute or two is an issue.

Large Data Sets

Extending the data being collected from financial, to analytical, to transactional created exponentially larger data sets……..

Low Latency

Delivery Event Wireless IP OLTP SQL Server

Sortation Event

File Transfer

Mainframe (RMS)

EDW

Data comes from multiple sources, from inside and outside of local networks and needs to be available with very low latency………

?

?

Varied Query Response Needs<1s <1s

<1m<10s

Business event

Data captured

Intelligence delivered

• 1 connote, 5 cartons and 1 pallet arrive at destination depot, pallet goes to bulk warehouse

Missed Opportunity• Two Trucks are dispatched to same location for

delivery• Customer is curious and a bit frustrated as to

why 2 trucks are needed…

TDWI The Business Case for Real-Time BIBased on concept developed by Richard Hackathorn, Bolder Technology

Situation Gained Opportunity• Cartons and bulk are loaded onto one truck for

delivery• Customer is pleased to receive the consolidated

delivery

Action taken

Accelerating Decisions

Time

Low latency, fast response times and effective BI together helps accelerate decisions and create business value…..

Valu

e

The trigger for change

By 2010 we had identified two needs in the business…….

ODSEDW

ADW

New Enterprise Data Warehouse to meet data growth as we extended the subject areas captured.

New Operational Data Store to act as central data repository for SOA based architecture.

Single platform to provide both an Enterprise Data Warehouse and an Operational Data Store – an Active Data Warehouse

What is an Active Data Warehouse ?

Active LoadNeed to be able to stream data into the

ADW near real-time.

Active AccessNeed to be able to access data in the ADW

in near real time.

Active Availability

Need to have extremely high availability.

Active Workload Management

Need to be able to handle mixed workloads and ensure SLA’s are met.

Activating the Data Warehouse means more then just a very powerful fast database server……

MPP Data Warehouse Server

Large volumes, active load and active access needs a different type of database technology – Massive Parallel Processing……

EXAMPLE: 100 random cards, return all 7 of Hearts.

1 ‘CPU’100 cards.

100 ‘CPU’ Cycles

100 Seconds

2 ‘CPU’s’50 cards each.

100 ‘CPU’ Cycles (50 cycles each)

50 Seconds

2x Improvement in Time

NO Improvement in CPU

4 ‘CPU’s’25 cards each

Each ‘CPU’ holds a suit.

25 ‘CPU’ Cycles from 1 CPU(3 ‘CPU’s’ Idle)

25 Seconds

4 x Improvement in Time4 x Improvement in CPU

Active Workload Management

Active management of workload is needed to ensure information is delivered at the ‘right time’ and at the lowest total cost of ownership……

Streaming Data

Traditional batch processing will not deliver the near real-time data loads an Active Data Warehouse demands……

EDW

ENTER

PRISE SERVIC

E BU

S

Change Data

CaptureOLTP

RMS

OLTP

Information Delivery

Traditional Business Intelligence tools (reporting) will not deliver information in a manner that enables fast decision making……

¨ Well designed Dashboards deliver information which can be consumed very rapidly using good visual design;

¨ Visual based ad-hoc analysis tools (Explorer) provide users capability to rapidly get to the information they need from vast data sets;

¨ Mobile delivery provides information to people when and where they need it.

Architecture

ENTER

PRISE SERVIC

E BU

S

Change Data

Capture

ADW

Data Services

WebIDashboards

The results¨ August 2010 – ‘Lift and Drop’ of existing SQL Data

Warehouse onto Teradata ADW. Approximately 500GB migrated, and approximately 1,000 complex BI queries per day are run against this.

¨ November 2010 –ODS and new Web Site goes Live. Supporting hundreds of thousands of tactical queries per day on the ADW platform.

¨ Now ADW platform is supporting this mix of BI and ODS loads and queries, and has maintained 100% up-time and 100% of queries meeting SLA’s.

¨ 1 DBA + 1 Platform = Low Total Cost of Ownership.

ODS

EDW

Operational Intelligence

Continuous update and time-sensitive

queries become important

OPERATIONALIZINGWHAT IS happening?

Event-based triggering takes

hold

ACTIVATING MAKE it happen!

Primarily batch and some ad hoc

reports

Increase in ad hoc analysis

ANALYZINGWHY

did it happen?

REPORTINGWHAT

happened?Analytical modeling

grows

PREDICTINGWHAT WILL

happen?

Value

Complexity

Continuous Update/Short Queries

Event-Based Triggering

Batch

Ad Hoc

Analytics

Questions, Comments and Observations