rapid bi with ldms.ppt

8/14/2019 Rapid BI with LDMs.ppt

1/28

Using Pre-Packaged Data Models

to Support Rapid BI Development

David Schoeff, Teradata Corp.

Jeff Hoffer, University of Dayton

1 Jeffrey A. Hoffer


2/28

Overall Agenda

Overview of iLDMs

Learnings from case studies of iLDMapplication

Workshop on using iLDMs in your organization

Jeffrey A. Hoffer 2


3/28

What Weve Learned From

Contrasting Case Studies

Jeffrey Hoffer

University of Dayton

3 Jeffrey A. Hoffer


4/28


5/28

Learning Resources

On www.teradata.com Search on Hoberman or logical data models, especially see

Leveraging the Industry Logical Data Model as Your Enterprise Data Model

Search on agile business intelligence

On www.beyenetwork.com

See Dan Linstedt blog, and The 2-Month Data Model by Bill Inmon

Search on logical data models or industry data model

On www.tdwi.org

In White Papers, search on agile business intelligence or industry datamodel

Hay, D.C. 1996, Data Model Patterns: Conventions of Thought,and 2006,Data Model Patterns: A Metadata Map

Silverston, L. various dates, several volumes of The Data Model ResourceBookand various articles from 2002 in DM Review

Moss, Larissa, President, Method Focussee articles, seminars on agile BI

And, of course, there is Modern Database Management.

5 Jeffrey A. Hoffer
http://www.teradata.com/http://www.beyenetwork.com/http://www.tdwi.org/http://www.tdwi.org/http://www.tdwi.org/http://www.beyenetwork.com/http://www.beyenetwork.com/http://www.teradata.com/


6/28

Traditional (Invented Here) Database

Development Process

Conceptual Data

Modeling: detailed

metadata

Conceptual/EnterpriseData Modeling: scope,

ISA, EDM

Logical Data

Modeling:

integrate,

normalize, integrity,

security

Database

Definition: schema,

documentation,

installation, training

Tuning: integrate

new requirements,

improve, fix (mini

cycles of Analysis,

Design,

Implementation)

Physical /technical

database design:

technology design,

performance

6 Jeffrey A. Hoffer


7/28

Database Development with Prototyping

(A Learning Together Approach)

Identify

Need

DevelopInitial

Prototype

Revise &

Enhance

Prototype

Implement &

Use

Prototype

Convert to

Operational

Form

Conceptual Data

Modeling:

preliminary CDM

Initialrequirements Logical Database

Design: detailed

requirements

Physical Database

Design: new

database contents ,structures, programs

Database

Implementation:

coding, integrate

contents

Database

Maintenance:

evaluate and

enhance

Database

Maintenance:

tune, improve for

performanceNew

requirements

Working

prototype

Deficiencies

Next version

If prototype is

inefficient

7 Jeffrey A. Hoffer


8/28

Two Case Studies:

LDMs Work in a Variety of Situations

Case Study A On-line retailer

Young

Highly competitive, rapidly

changing Information-driven

Dynamic, immersedleadership team

Turbulent period, neededsolution quickly

Business analystsembedded in units

LDM as golden model

Case Study B Technology provider

Mature

Innovative, detail-oriented,

comprehensive Highly analytical

Decentralized leadershipteam

Constant pressure andenvironmental changes

Diversified structure forbusiness analysts

Internal systems as goldenmodel

8 Jeffrey A. Hoffer


9/28

Data Modeling Process Changes for

Rapid BI: Case Study A

Background: Hoffer, Watson, and Wixom

Large, on-line retailer >300 hourly/daily reports

>400 Business Object IDs

also SAS, on a Teradata EDW platform

Critical need to get a BI environment up before thenext Christmas buying season (core needs of

marketing, merchandising, and auction parts ofbusiness met in 9 months)

Limited internal resources due, in great measure, tosimultaneous implementation of a new ERPoperational system.

9 Jeffrey A. Hoffer


10/28

Overview of Results:

Case Study A

LDM was about 80% right before customization (used severalLDMs for different industries represented by companys offerings)

Cost of an LDM is about one DBA for one year

Saved time, improved quality, less re-work

LDM did not allow them to develop new environment piecemeal

needed quick start with a solid foundation for future of rapidlychanging businessenterprise perspective from beginning

Collaboration of external consultants

3 for one month, 2 for another 5 months, 1 for another 6 months andinternal data analysts

Key for short- and long-term success was to involve internal dataanalysts, who do evolution of data modeling

Acquisition of the LDMs was one of the key strategic things (we)did to gain quick results and long-term success with datawarehousing and BI. DW Director.

10 Jeffrey A. Hoffer


11/28

Overview of Results:

Case Study B

Why did they use LDMs? Use data consistently throughout BI applications

Adhere to government regulations

Understand data across organization using common names

Supplier = Vendor, Commodity = Material Comprehend transformations (part of LDMs)

Can combine / use for analytics data we didnt know could beanalyzed together

Allows for normalized data structures to be traversed from

any where to any where without introducing reportinganomalies

Allows for quicker building of dimensional star schemas(dependant data marts) because of ease to negotiate datastructures.

Jeffrey A. Hoffer 11


12/28

Database Development with LDMs

Identify

Need

Evaluate

Alternative

Packages

Customize

LDM

Evolve for

New Needs

6 months from need to firstapplication

2 weeks for data model

90 days for first application

9 months from need to all

phase I applications

Initial

Infrastructure

Applications &

Infrastructure

Evolution

2-week

release

packages

Evolve for

New NeedsEvolve for

New NeedsApplication

package

developmentoverlaps



13/28

Observations About

Customization Identify entities, attributes,

relationships in the LDMthose youneed for the future Concentrate on details for those you

need first

Create a phased roadmap (can useentity clustering to show thisfunctional decomposition for data)

Rename data to local terms

Refine LDM to local business rules

Map LDM data to current databases(e.g., to design migration plans andload processes)


Customize

LDM


14/28

What Is Mapping?

The process of relating each LDM data element with asource Do we need it? (now, later?) from either current systems or

LDM

Where do we get it?

When do we get it?

How do we define it and what do we name it?

Does it need to be transformed? Or do we need more atomicsource?

Does source system need to be improved? It is NOT about resolving conflicts between source systems

or fixing source systems

It is NOT about designing/writing the ETL.



15/28

Key Points About Mapping

Some elements will be missing in LDM andcurrent databasesthese become obviousbecause of LDM

Are mismatches really needed?

Avoid temptation to always accept current databasesas tie-breaker

Encourage thinking of the possibilities fromelements in LDM not in current databases

Current databases are often poorly documented,which makes process difficult

Watch for duplicate, inconsistent entries of the samedata in different databases.



16/28

The LDM is comprehensive in business rules (e.g.,cardinalities and generalization) and can becomplex; thus it is flexible to change Do you really need all this complexity? Do we need

something more restrictive? Does comprehensiveness suggest opportunities?

Smartly tailor LDM to organization

LDM updates can react to changing standards and

regulations Current environment likely has different standards

and regulations for different sources.




17/28

Engage users and managers early because youhave a validated prototype data model from thestartthe LDM provides a visual, comprehensive

checklist of possible questions Would we ever have a customer order with more

than one customer?

Might an employee also be a customer?

Give special attention to elements of LDMs that SMEsdid not mention in interviewsWill we ever go inthat direction? a basis for impertinence its allabout the questions you ask!




18/28

Mapping is criticalcant afford to do a bad

job

Mapping projects are great student projects in

a capstone courserequires integration of

data and systems knowledge and skills, with

understanding of differences across platforms,

ETL, timing, etc.




19/28

More on Customization

Even with good mapping, do data profiling toidentify overloading, obsolescence, emptycolumns, hidden (undocumented) requirements,outliersthe proof is in the data Understand reasons for inconsistencies

Poorly designed databases

Accuracy of current data, which you do not want to migrateto new database for analyticsa time for data cleansing

Investigate reasons for missing data for mappedattributes Application software errors, human data entry errors,

optional data (subtypes).



20/28

Data Profiling a Must

Profiling = statistical analysis to uncover hidden patternsand flaws

Look for outliers

Sorting by date can reveal overloading and patterns for

empty values, or when data moved columns over time, orshifts in data

Can match shifts in data to major system changes

Empty columns can imply entity subtypes

Wide tables can imply denormalization, which canencourage erroneous data

Can be used to identify flaws in current systems, need forcleanup efforts, and need to improve database design.



21/28

A Chance to Verify

Business Rules

Verify each business rule (in the LDM) for your

organization

Review metadata (names, definitions, data types,

formats, lengths, cardinality, etc.) with the bestSMEs

Business rules dictate transformations of

operational data into analytical database Different operational systems may = different

business rules.



22/28

Observations About Evolve

As new business needs arise, conduct minicustomization projects to extend currentimplementation from LDM with a different focus (theLDM implementation easily scales as an architectural

foundation for agile development) Dynamic businesses will yield extensions to LDMs, so

vendors like feedback

LDMs provide the flexibility and speed to react to (toanticipate) new needs

BI systems are not complex (although theinfrastructure is), which is why LDMs are valuable andagile development works.



23/28

PMI View of Agile Project Management

Source: Sliger, M.

A Project

Managers Guide

to Going Agile,

Rally Software

Development

Corp., 2006



24/28

Typical Evolve Scenario



25/28

An Environment Conducive

to Rapid BI: Case Study A

Organizational Climate Compelled to do rapid development of infrastructure

and applications Business moves quicklydot.com or swarming mentality

when leadership turns their focus to it Attitude of weve defined it, lets get it done, then move

on perfection not critical

Leaders see firm as an information company An interaction of technology and retail

Using technology and information well is a competitiveadvantage

Needed a drastic change to jump start the transformationthe LDMs

LDM also overcomes the hazards of swarminglack ofarchitecture/plan.



26/28

LDM and Organizational Fit LDMs essentially modify the agile approach initially by

making the business define core requirements upfrontinfrastructurebut still supports iterative

evolution A balance to swarming

Leadership team sets priorities and is willing to evolvein phases (normal agile chunk approach)

Synergistic initiative gets greatest attention LDM supports iteration, which builds trust

Incremental changes (2-week chunks of work) showscontinuing commitment (rather than one time, big bangchange), which also builds trust.





27/28

Need tech and business savvy people

Business analysts embedded in each business area

(removes bureaucracy), and report to both VP of

business area and head of BI applications, which

creates deep knowledge about business and

facilitates rapid development

Business managers with strong technical aptitudeand skillsa hiring priority..





28/28

Workshop Questions

To start, do you have any questions about the iLDM?

How does your ERD match up with iLDM?

What difficulties do you have merging the iLDM with yourERD?

In your environment, which model trumps the other andwhy?

Is the iLDM more than you need? Why? How deal withthat?

Are there things missing in iLDM that you need in your

environment? What kinds of resistance would you get for using an iLDM?

How would you make use of an iLDM in your environment?


rapid bi with ldms.ppt

Documents