claudia imhoff intelligent solutions, inc. october 16,...

Claudia Imhoff

Intelligent Solutions, Inc.

October 16, 2012

Getting Data In: Answering the Challenge

of Growing Sources of BI Data

Sponsors

3

Speakers

Claudia Imhoff President and Founder,

Intelligent Solutions, Inc.

Chris Hagans VP of Operations,

WCI Consulting

Copyright © 2012, Intelligent Solutions, Inc., All Rights Reserved 4

Agenda

Why We Need a Stable and Robust Architecture

for Data Integration

Today’s BI Architecture

Getting Data In Best Practices

Getting Started

Copyright © 2012, Intelligent Solutions, Inc., All Rights Reserved

Questions You Need to

Answer

Why are you implementing DW/BI at all?

Mission, vision, guiding principles

What types of decisions does your company

need to make?

Strategic, tactical and/or operational BI

What types of data are needed?

Real time, low latency, historical

Have you got the right skills, methodology,

requirements gathering process?

GDI v GIO, prototype creation, iterative processes

5


Questions You Need to

Answer

What is the state of your data quality?

DQ varies depending on the analytic and application

Good enough data may be fine

Do you have the appropriate data integration and

data quality technologies?

Data integration

Data quality

How secure is your environment?

Security / privacy procedures in place

Alerts to security breaches

6


Information Consumer

Task-oriented business users who

consume BI to support day-to-day

operations

Gather info to increase personal

knowledge, make decisions but

don’t have time, experience or

inclination to create inputs

Info may be delivered by:

Operational applications, e.g.,

available services or products

Analytical applications, e.g., report

or dashboard

Contextual applications

Users: general public, customers,

partners, suppliers, employees

7


Information Producer

Improves business operations,

make tactical and strategic

decisions

Explores, analyzes, and

produces actionable BI

analytics

Builds customizable BI

components and publishes BI-

related info for use by

information consumers

Participates in BI governance

committee

Users: data scientists, power

business users and business

analysts

8


BI/DW Builder

Traditionally responsible for

building DW and/or BI solutions

Due to budget, resource, or priority

issues are often bottlenecks in

deploying BI

Are responsible for:

Providing access to source data,

ideally via common business views

Developing customizable BI

components for other IWs

Participating in BI governance

committee

Monitoring use of BI solutions by

other IWs

Users: central/business unit IT,

information producers

9


Agenda





Getting Started


Why Do You Need An

Architecture?

Reusability of components – reduces cost,

complexity, improves productivity

Permits optimization of each component – data

warehouse, data mart, experimental sandboxes

Increases flexibility in terms of selection

Tools

Technologies

Techniques


Architectural Objectives

Understand how functional, procedural and

technical aspects of business and IT

environments pertain to BI effort

Gauge company’s state of preparation to develop

and support a BI environment

Look for specific risk mitigation recommendations

to improve potential for success

Extended to embrace new BI capabilities (e.g.,

Big Data, advanced analytics, self-service BI)

Assessment


Integration

Availability

Quality

Capability

Usability

Security

Business Intelligence

Getting

Data In

Getting

Information

Out


Getting Data In

Integration

Availability

Quality

Capability

Usability

Security

Getting

Data In

Getting

Information

Out


Architectural Objectives

Part One – Getting Information Out:

Make BI results easy to consume and

enhance

Make BI tools easy to use

Part Two – Getting Data In:

Make BI/DW solutions fast to deploy & easy to

manage

Make access easy to all sources of data – not

just data warehouse data

15


Make DW Solutions Fast to

Deploy & Easy to Manage

Improves the productivity of both BI/DW builders and


Provides faster time to value

Enables applications to be built that were not possible

with earlier technologies

Enables a business unit to deploy its own solutions

BI requirements:

Support lower-cost BI/DW solutions that are easy to administer

Provide good performance and scalability for complex analytical

workloads and high data volumes

Support easy administration

16


Make it Easy to Access Data

Improves productivity of both BI/DW builders and


Enables self-service BI by enabling motivated information

consumers to become information producers

Extends reach of BI giving access to many new sources

of data – mitigating business workaround or end runs

BI Requirements:

Make dispersed data easy to access without IT assistance

Support new types of business data and business content

Support easy administration

17


Agenda





Getting Started


Focus – Appropriate Data

Integration Technology

Reduce and standardize data integration

technologies

Determine need for different integration

techniques

Data consolidation (ETL/CDC)

Data propagation (EAI)

Data virtualization (EII)

19


Focus – Data Integration

Architecture

Extract, transform and load (ETL) – assimilates data from

sources into integrated, consistent data repositories,

preparing it for BI processes; target is a database

Change data capture (CDC) – accesses critical data in

real-time during day to support operational BI; combines

with ETL

Enterprise application integration (EAI) - a strategy or

framework for centralizing & optimizing application

integration; target is an application

Enterprise information integration (EII) – framework for

real-time access of disparate data types from multiple

sources; target is an end user

20


Focus – Technical Metadata

Where did the data come from?

What happened to it as it was processed?

Where did it end up?

What are the integration processes?

What are the data quality processes?

What are the data quality metrics?

Found in data integration technologies, data

modeling technology, etc.

21


Focus – Self-Service BI

Easy access established to multiple sources of data (DW,

external data, individual databases, etc.)

Clear data lineage and glossary – easily accessible

Creation of library of starter components (report

templates, common algorithms, stored models, etc.)

Appropriate and timely education (and training) for

information workers

Balance between IT-generated and IW-generated

decision objects

22


Focus: Governance

May be difficult to set up in such a distributed environment

Include ability to rate, annotate value of each component

Identify whether data used is “governed” (e.g., in a data

warehouse or MDM environment) or “ungoverned” (e.g.,

individual spreadsheets, external source)

Encourage reuse of prebuilt components (report

templates, customizable dashboards, widgets)

Control what data populates different repositories

(warehouse, mart, etc.) through standard ETL processes

Determine which user-defined analytics, reports, etc.,

should be brought back into the governed environment for

use by others

23


Focus: IT Organizational

Considerations

BI/DW builder must have ability to administer and manage

infrastructure

Must be able to monitor the environment

Have insight into information workers’ activities, determine

performance of environment, ensure access via unified semantic

layers, determine if source should be governed

Must have oversight into the environment

Can determine if source is governed or not, what sources are

used, who else is using them, determine if a popular analytic or

report is mission-critical

Note: LOB IT or information producers may have to take

on some of previously traditional central IT roles

Security of data, adherence to privacy policies

24


Focus: Integration

Infrastructure*

homogeneous/heterogeneous centralized/distributed federated/dispersed

Source data

organization

Data integration technology

Data integration

technique & mode

real-time near-real-time point-in-time

read-only/read-write

structured semi-structured

unstructured packaged application

EAI/Web service metadata

Source data type

Data scale

number of data sources data store size

data store volatility

Target data currency & access

consolidation virtualization propagation

changed data capture synchronous/asynchronous on-demand pull/event push

ETL (event/demand) EAI, EDR

EII, ECM, CDC

restructuring reconciliation

cleansing aggregation

Source data transformation

25 * Slide compliments of Colin White, BI Research


Integration Infrastructure

Variables: Example* Variable Source 1 Source 2 Target

Data source type SAP ERP package Oracle RDBMS IBM DB2 RDBMS

Data source

organization

distributed BTx store centralized BTx store centralized ODS

Data scale

- data size

- data volatility

500GB

1,000 records/hr

250GB

250 records /hr

1,000 GB

600 records/hr

Data quality good average -

Data rules restructuring restructuring

content cleansing

-

Target data currency - - 1 hour

Target data access - - read-only

Data integration technique

asynchronous propagation to

staging

asynchronous CDC to staging

consolidation from staging

Data integration technology

EAI message Q CDC ETL

* Slide compliments of Colin White, BI Research 26


Agenda





Getting Started


Best Practices for Getting

Data In

1. IT needs to monitor BI environment

There needs to be a layer of administration and

manageability

Give IT insight and oversight when information workers

deploy, share, and collaborate using BI capabilities

IT can see usage of any BI that an information worker

publishes

Who published it, data sources used, and who else is using it

Which queries are too costly, long-running, or bog down

performance of other queries

IT not only needs to manage BI, but also needs to

secure, validate and audit it

28



Data In

2. Allow data warehouse system to be used with

other sources of data

Extend BI environment to include “other” sources of

data – e.g., consumer’s personal spreadsheets,

unstructured data, external data, etc.

Create experimental or sandboxes for exploration or

ad hoc queries – may be persistent or temporary

Monitor usage of these sources and the BI objects that

use them

29



Data In

3. Your integration infrastructure must handle all

sources and targets of data

Fill out a spreadsheet with variables for data

integration

Communicate what is available to Getting Information

Out Team(s)

Determine mechanisms for handling emergency

situations (workarounds and end runs)

Communicate any and all quality problems to

operations and GIO folks

Much hinges on data integration capabilities

30



Data In

4. Ensure you have best technological support for

High performance – data integration, loading, querying

High reliability – integration processes, data

consistency, response times

High flexibility – data usable for any and all forms of

analytics

High scalability – seamless capacity

Low maintenance – easy to change and enhance

Determine your standard set of GDI technologies

31



Data In

5. Watch your costs!

A major product differentiator

If you already have a BI vendor’s platform in place, you

can often add a self-service capability with minimal

effort and cost

Many vendors offer entry level products geared toward

companies with limited budgets

Some companies use open source solutions but there

may be additional “deployment” costs

Consider software-as-a-service offerings to cut capital

and IT staff costs

32


Summary

To accommodate all analytic needs, you will need:

Historical data – data warehouse and data marts

Near current data – operational data store

Current data – operational systems

Structured and unstructured data

All need to be tightly integrated and coordinated

Seamless interfaces

Reusable and consistent

Timely and reliable

Seamless scalability

BI must be extended beyond traditional architectural

borders!

33


Best Practices Combined

Focus on Getting Information Out to stimulate BI adoption

– Part 1 (October 04, 2012)

1. Don’t assume that simply installing easy-to-use BI tools will make

your environment easy to consume and use

2. Don’t give information consumers too much responsibility

3. Support collaborative business intelligence

4. Understand requirements of information consumers, provide

appropriate tools/reports/dashboards

5. Create a starter set of standard BI components

6. Create appealing, personalized dashboards, distribution

mechanisms

34


Best Practices Combined

Focus on Getting Data In – Part 2 (this webcast)

1. IT needs to monitor BI environment

2. Allow data warehouse system to be used with other

sources of data

3. Your integration infrastructure must handle all

sources and targets of data

4. Ensure you have best technological support

5. Watch your costs!

35

Chris Hagans, VP of Operations

WCI Overview

Operations team are FTE’s; no contractors

BI experts: Since 1998

Range of expertise: BI, dashboards, EIM, SaaS, Mobile, Data Cleansing, Data Warehousing

Creator of real time BI consulting offering www.instantaccessbi.com

www.wciconsulting.com

SAP PartnerEdge Global

Fastest Growing

Reselling Partner of the

Year

The BI/DW Organization Perspective

- Meeting and addressing the business goals and objectives

-Integrating and providing valuable data

- ETL/quality/propagation platforms

-New systems, new acquisitions, new platforms, changing requirements

- Addressing growth and/or change

- Flexible, nimble, reusable

-Awareness of what is the business going to get excited about

- In memory

- Unstructured

- Mobile

- Emerging technologies

-Right Technologies/platforms for the future

The BI/DW Builder Perspective

- Understand

- Where we are and are going

- What the business wants

- Support of the business

- Is the business going around me

- Are they engaged with me

- Engage the business in the POC/design process, leverage their work

- Embrace and engage the data “producers”

-Promoting the BI/DW landscape

- Educate, advertise, brand, and communicate

- Where “should” the DW be used

- Tooled correctly

- People/Team, future thinkers, data “scientists”

- Monitoring

- We are business “enablers”

40

Questions??

Contacting Speakers

• If you have further questions or comments:

Claudia Imhoff

[email protected]

Chris Hagans

[email protected]

claudia imhoff intelligent solutions, inc. october 16,...

Documents