An Agile Approach to Building & An Agile Approach to Building & Managing Data WarehousesManaging Data Warehouses

A Briefing by WhereScape

Mary Edie Meredith, Sr. Technical AnalystMary Edie Meredith, Sr. Technical Analyst- maryedie@wherescape.com- maryedie@wherescape.com

2

Why do Data Warehouse Projects struggle ?

Gartner notes that over 50% of data warehouse projects fail or go wildly over budget

1. Inaccurate business requirements - #1 problem IDC

2. Poor development productivity

3. Slow development cycles

4. High cost of resources

5. High TCO

6. Poor documentation – usually the last thing that is considered &

never up to date.

7. Poor data quality

8. HIGH RISK

3

Where did they go wrong? – one real problem is the “Big Bang” project approach

“Incremental Incremental Data Warehouse Development –

The Only Way to Fly” Bill Inmon, Jan 8, 2009, (BeyeNetwork)– “There are many reasons the ‘Big Bang’ approach doesn’t work … “but at the heart is inability of the

development analyst to gather requirements in the manner prescribed by the SDLC”– “End users of analytical systems need to know what the possibilities are before they can articulate the

requirements.”

The goal is NOT to build a Data Warehouse, but rather…– Deliver real valueDeliver real value– Create a solution that is adaptable becauseCreate a solution that is adaptable because responding quickly to change brings competitive advantage

– Create a process Create a process to develop and maintain the solution that is trustworthy and sustainable

4

How would agile proponents approach the problem? From the agile manifesto: //agile

• Early, frequent, and continuous test and delivery of valuablevaluable working working software (every 2 wks-2mos).

• Welcome changing requirementsWelcome changing requirements, even late in development. • Business people, developers work together daily work together daily throughout the project.• Build projects around motivated individualsmotivated individuals. Give them the environment and

support they need, and trust them to get the job done. • The most efficient, effective method of conveying information to and within a

development team is face-to-faceface-to-face conversation. • Continuous attention to technical excellence technical excellence and good design good design enhances

agility. • Simplicity--the art of maximizing the amount of work not donemaximizing the amount of work not done--is essential. • At regular intervals, the team reflects the team reflects on how to become more effective, then

tunes and adjusts its behavior accordingly.

5

What is uncomfortable about this approach?

• The further out in time, the less a project team can say about what will be accomplished.

• An agile approach can break the rules.– Agile implementers sometimes wrongly assume you can break ANY rule.– Shortcuts do not equal Quality Pragmatism

• Classic trade-offs for project managers - Schedule/ Scope/ Resources/ Quality – agile leaves little wiggle room.

• Does not lend itself to outsourcing, distributed teams.• Having a close working relationship with business users does not solve

the difficulty determining requirements.

And ….

6

If I could deliver something meaningful in weeks

DON’T YOU THINK I WOULD HAVE, ALREADY.

7

Agile Approach Versus Traditional Approach

Docs?

8

What really works using agile “The WhereScape Way”

• A Governance structure – Strategy, Architecture, Roadmap, Standards– Goals, sponsors, infrastructure, data governance ….

• New Development Paradigm for delivering data - RED– ETL tools are great for moving data, but RED can do DW part better.– Integrated Development using one metadata driven tool.– Do the data delivery in the database.– Incorporate Business Rules into data delivery process

• Iterative workshops with business users– Use REAL DATA for flushing out requirements (RED enables this)– Track all issues discovered, especially data quality

9

Agile in Operation

• Integrate analysis, design, creation, data delivery, deployment, iteration

• Useful even if you just need to provide the presentation layer

• Feedback from business users on live data part of the development process

Live Data Workshop

Business UserSessions

10

Speeding up the development by leveraging metadata, embedding best practice methods

dim_customer_key

dss_update_time

11

Data Warehouse Scenario – Build a Sales Fact

12

Star schema creation scenario – start with load table

SourceWarehouse

Oracle, SQL/Server, Teradata, DB2Oracle, SQL/Server, Teradata, DB2Native RDBMS, ODBC accessible, FilesNative RDBMS, ODBC accessible, Files

13

RED Browser Mode

Metadata

Results

ActionsDrag and Drop Target Area

BrowsingConnections

Choose connection and filtering

14

For the Teradata shop -

15

Star schema creation scenario – start with load table

SourceWarehouse

Oracle, SQL/Server, Teradata, DB2Oracle, SQL/Server, Teradata, DB2Native RDBMS, ODBC accessible, FilesNative RDBMS, ODBC accessible, Files

16

Drag and Drop Example: load source data

17

Drag and Drop Example: load table properties

18

Drag and Drop Example: load table storage mapping

19

Drag and Drop Example: load table “create and load”

metadata

20

Drag and Drop Example: load table results

create

generated load script execution

21

Drag and Drop Example: load table results

create

generated load script execution

Display Data

22

Stage table creation scenario – the stage table

SourceWarehouse

Foreign dimension Keys, lookups

Source table join

23

Stage table: start with load_order_header (Drag and Drop)

24

Add columns from load_order_line (Drag and Drop)

Load_order_headerColumn metadata

25

Add columns from load_order_line (Drag and Drop)prevents duplicate column names

Load_order_headerColumn metadata

26

Add FK cols to Stage Table – Drag and Drop dim_*

Drag and drop Dimension table keys

27

Column Metadata easily altered

28

Column Transformations – Business Rules, Computed Fields, String Manipulation, Type Conversion, Null handling,…

29

Create the Stage Table (right click object)

30

Create the update procedure (object Properties)

31

…then select Procedure Type

32

… then specify the Join statement

add appropriate clauses

Numerous joins supported

33

…indicate the business key to identify SK in DimensionPrompts if column names match

34

…indicate the join column if names are different

35

Procedure is created, compiled. Execute Procedure.

36

Display Data

37

Fact table creation scenario – Sales Fact table

SourceWarehouse

38

Create the Fact Table from the Stage table

39

Metadata leveraged to create the code

Dimension tables are created with

“zero” row for unknowns

Join metadata

Transformation for quantity column

40

Auto generated stored procedure code …

• Keeps all the data movement in the database• Provides consistent variable naming, coding best practices• Utilizes custom parameters you can embed in metadata• Includes error checking and rollbacks• Preserves the metadata for easy modification• Can augment with custom procedures• Includes features best practices for various object types

o Can handle slowly changing dimensions (all three types)o Procedure provided to populate and update time dimensiono Handles code for surrogate keys, update and life-span dateso Creates Unknown Row for each dimension tableo Accounts for missing dimension key matches in source data

Let’s advance developers can skip the mundane

Allows less experienced developers to be productive

41

Generated Procedures with version compares

42

Next Step – Business User review

Easy vehicles to show this to Business users:

Output table data to Excel

Stress test with SSAS cube

43

Create a SSAS Cube for Business User EvalDrag and Drop Fact to OLAP Cube target

Creates OLAP dimensions

Creates OLAP measure group

44

Create a SSAS Cube for Business User EvalSlice and Dice in Analysis Services

45

Capturing Metadata - Lineage information

46

Leveraging Metadata: Reports

47

Ready to Deploy

48

Scheduler to manage objects and data flow

Run in parallel

49

Scheduler to manage objects and data flow

Run in parallel

50

Diagrammatical View Example: Update Job

51

Application Files to promote to QA and Production

52

Leveraging Metadata: Auto Producing Documentation

53

User Documentation

Where RED fits

55

"WhereScape promised a lot and the product has delivered.  We are very happy with the amount of time it is saving us in development, as well as the documentation it is producing and the built-in scheduler.  I am very happy with the purchase.“

"We estimate the development lifecycle is 20-25% of what it was previously when we were hand-codingWe estimate the development lifecycle is 20-25% of what it was previously when we were hand-coding."."

Dan Mosher, Director of Enterprise Data Warehousing

                                                                                                          

“WhereScape RED offers IPC a sophisticated Lifecycle Methodology sophisticated Lifecycle Methodology that guides us through the process of building our data warehouse. RED

creates integrated database objects such as tables, indexes, procedures, etc; produces standard yet customizable T-SQL code and

auto-generated user and technical documentation.”

Maylee Sanchez, Sr. Database Administrator

Some WhereScape Customers

57

58

ConclusionConclusion

• Build Your Data Warehouse Solution

– Way Faster– Way Cheaper– Ready for Change

• Get Full Documentation– For Users– For Techies

• And DO IT THE AGILE WAY

60

Tools and Reports

61

Additional CUBE Features

• Can add MDX calculations to the cube metadata for calculated members

– Specify font, foreground/background colors, boldness, display format, non-empty behavior, order number, client visibility

• Canned MDX calculations – Month/Year to date, Moving Qtr/Year, same month previous year, previous year to date.

• Can specify Post Create or Post Update XML/A Scripts– Allows features built outside of RED to be added to the Schedule cube processing (e.g. security roles added,

perspectives, translations )

• Cube properties include – Processing modes for Cubes (Regular, Lazy Aggregation) and priority– OLAP dimension processing (together or separately)– Cube visibility to client applications– Default Measure and estimated rows

• Can optionally drop Dimensions, Measure Groups, Cubes, and Cube databases from within RED.

• Can manage KPIs, partitioning, and processing for measure groups