sap data quality
TRANSCRIPT
-
8/8/2019 SAP Data Quality
1/58
Developing a Data Quality
and Integration Strategy
Jonathan G. GeigerIntelligent Solutions, Inc.
April 28, 2010
-
8/8/2019 SAP Data Quality
2/58
Sponsor
-
8/8/2019 SAP Data Quality
3/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Topics
Complexities
Expectations Setting
Assessment
Improvement
Strategy
3
-
8/8/2019 SAP Data Quality
4/58Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Topics
Complexities
Typical Complexities
Why They Exist
Expectations Setting
Assessment
Improvement Strategy
4
-
8/8/2019 SAP Data Quality
5/58Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved5
Data Integration
DataWarehouse
OperationalData Store
Operational Data
ExternalData
InternalData
. . . and a miraclehappens here . . .
Data integration is very complex!
-
8/8/2019 SAP Data Quality
6/58
-
8/8/2019 SAP Data Quality
7/58Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Capture Complexities
Data requirements
Best source of the data
Business rules for capturing the data
Data meaning
External data requirements
History requirements
Currency requirements
Privacy and security requirements
Audit and control requirements
Metadata 7
-
8/8/2019 SAP Data Quality
8/58Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Cleansing, Transformation &Integration Complexities
Data quality
Data integration
Data transformation
Data enrichment
Error handling
Privacy and security requirements
Audit and control requirements
Metadata
8
-
8/8/2019 SAP Data Quality
9/58Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Load Complexities
Currency
Privacy and security requirements
Audit and control requirements
Metadata
9
-
8/8/2019 SAP Data Quality
10/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved10
Why Complexities Exist
Problem RecognitionData deficiencies are often
not recognized
Responsibility Overlaps and gaps
DisciplineData is called an asset but
not managed as such
Benefit RecognitionPeople are getting work
done`
-
8/8/2019 SAP Data Quality
11/58
-
8/8/2019 SAP Data Quality
12/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved12
De
fectRate
Time
Target
Quality - Definition
Quality is conformance to requirements
Quality is not
.... (necessarily) zero defects
-
8/8/2019 SAP Data Quality
13/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved13
Quality - Definition
Quality is conformance to requirements
Conformance to what?
Whose requirements?
How are requirements set?
What degree of conformance?
-
8/8/2019 SAP Data Quality
14/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved14
Stewardship
A steward is one who is called upon to exerciseresponsible care over possessions entrusted tohim or her
(adapted from Websters dictionary)
The steward does not own the possessions
The steward has a responsibility affecting theprocesses that impact the possessions
-
8/8/2019 SAP Data Quality
15/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved15
Data Steward
Exercise responsible care over the dataresources of the enterprise
The steward does not own the data
The steward impacts processes that affect the dataand its use
Acquisition
Management, maintenance and storage
Dissemination
Disposal
-
8/8/2019 SAP Data Quality
16/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved16
Data Planning Roles
Stewardship responsibilities
Provide input to the subject area model
Provide input to the business data model (business
rules, definitions, etc.) Establish metadata management strategy
Custodianship responsibilities
Develop the subject are model
Develop and maintain the business data model
Establish metadata management strategy
-
8/8/2019 SAP Data Quality
17/58
-
8/8/2019 SAP Data Quality
18/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved18
Data Management Roles
Stewardship responsibilities Establish and monitor data demographic expectations
Establish archival and disaster recovery rules
Provide metadata content Custodianship responsibilities
Transform the business data model into system andtechnology models
Establish (technical) data naming standards
Manage metadata
Manage data storage (design, reliability, security,recoverability, archival and restoration, etc.)
-
8/8/2019 SAP Data Quality
19/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved19
Data Dissemination Roles
Stewardship responsibilities Establish privacy and security policies
Define standard query and reporting requirements
Establish capability requirements
Establish quality expectations
Establish policies and guidelines for information use
Provide metadata content
Custodianship responsibilities Ensure adherence to privacy and security policies
Providing input to the quality expectations
Manage and provide metadata
-
8/8/2019 SAP Data Quality
20/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved20
Data Disposal Roles
Stewardship responsibilities
Establish retention rules
Establish erasure rules
Custodianship responsibilities Provide input to retention rules
Provide input to erasure rules
-
8/8/2019 SAP Data Quality
21/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved21
Making It Real
Two ways to approach stewardship
Data stewards are assigned to a specific data subjectarea customer, product, order, etc.
Data stewards are assigned to a particular functionsales, marketing, finance, etc.
There are benefits and drawbacks to eachapproach
In either case, good communication ismandatory
-
8/8/2019 SAP Data Quality
22/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved22
Executive Oversight
Cross-functional committee
Provides authority to the data stewards
Provide resources for data stewardship andinformation management
Resolve conflicts
-
8/8/2019 SAP Data Quality
23/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved23
Prioritization
Too many data elements to do at once
Need to categorize data
Criticality
Visibility
Usage
Sanctioned Projects
-
8/8/2019 SAP Data Quality
24/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Topics
Complexities
Expectations Setting
Assessment Continuous Improvement
Data Profiling
Symptoms vs. Root Causes
Improvement
Strategy24
-
8/8/2019 SAP Data Quality
25/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Continuous ImprovementProcess
25
PLAN
DOCHECK
ACT
Data profiling typicallystarts here
Reactive actionstypically start here
Proactive programsstart here
Some companies starthere, following existing
processes
Data profiling typicallystarts here
-
8/8/2019 SAP Data Quality
26/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved26
Data Profiling Framework
Data Cleanup and Business Process AdjustmentData Cleanup and Business Process Adjustment
PLAN
ACT
CHECK
DO
Framework courtesy of SAP
-
8/8/2019 SAP Data Quality
27/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Data Profiling in Context
Diagnostic step to understand data meaningand quality
Priorities dictate scope
Business data model provides business rules Quality expectations provide perspective
Data profiling reveals conditions
Analysis determines actions Expectations adjustment
Corrective actions
Preventive actions27
Root Cause Analysis is
-
8/8/2019 SAP Data Quality
28/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved 28
Root Cause Analysis isPerformed
Major Cause Major Cause
Major Cause Major Cause
Characteristic
AB CD OTHERS
%#
-
8/8/2019 SAP Data Quality
29/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Topics
Complexities
Expectations Setting
Assessment
Improvement
Data warehouse implications
Upstream implications
Strategy
29
-
8/8/2019 SAP Data Quality
30/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Data Warehouse Implications
Data handling options
Accept
Reject
Fix
Adopt default value
Error handling options
Suspend data awaiting correction
Transmit correction to source
Transmit need for corrections
30
Continuous Improvement
-
8/8/2019 SAP Data Quality
31/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Continuous ImprovementProcess
31
PLAN
DOCHECK
ACT
Data profiling typicallystarts here
Reactive actionstypically start here
Proactive programsstart here
Some companies starthere, following existing
processes
Data profiling typicallystarts here
-
8/8/2019 SAP Data Quality
32/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Topics
Complexities
Expectations Setting
Assessment
Improvement
Strategy
Major components
32
-
8/8/2019 SAP Data Quality
33/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Framework & Business Drivers
Relate to Enterprise Quality ManagementApproach
Formal or informal
Goals
Understand business drivers and needs
Business intelligence / operational systems
Strategic / tactical / operational
Declare strategy
Mission statement
Guiding principles
33
-
8/8/2019 SAP Data Quality
34/58
-
8/8/2019 SAP Data Quality
35/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved 35
Subject Area Model
Business Data Model
OperationalSystem Model
Data WarehouseSystem Model
Technology Models
Data Models
Continuous Improvement
-
8/8/2019 SAP Data Quality
36/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Continuous ImprovementProcess
36
PLAN
DOCHECK
ACT
Data profiling typicallystarts here
Reactive actionstypically start here
Proactive programsstart here
Some companies starthere, following existing
processes
Data profiling typicallystarts here
-
8/8/2019 SAP Data Quality
37/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved 37
Data Profiling Framework
Data Cleanup and Business Process AdjustmentData Cleanup and Business Process Adjustment
Framework courtesy of SAP
-
8/8/2019 SAP Data Quality
38/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved 38
Roles and Responsibilities
Executive Oversight
Data Stewardship
Data Custodianship
Data Providers
Data Users
T l d T h l
-
8/8/2019 SAP Data Quality
39/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved 39
Tools and Technology
Database management system
Data modeling
Data profiling
Metadata management
D Q li M i
-
8/8/2019 SAP Data Quality
40/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Data Quality Metrics
Data usage
Data quality improvement
Benefits attained
40
-
8/8/2019 SAP Data Quality
41/58
T i
-
8/8/2019 SAP Data Quality
42/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved
Topics
Complexities
Expectations Setting
Assessment
Improvement Strategy
42
Ab t I t lli t S l ti
-
8/8/2019 SAP Data Quality
43/58
Copyright 2010 Intelligent Solutions, Inc., All Rights Reserved 43
About Intelligent Solutions
Founded in 1992 by Claudia Imhoff Received outstanding recognition for client satisfaction
(Dun and Bradstreet survey of our clients) Internationally recognized industry expertise Full line of Corporate Information Factory and CRM
courses BI and CRM Consulting services
Mentoring Assessment and Planning Management
Design and Implementation
International client base in all industry verticals
SAP Solution OverviewEnterprise Information Management
-
8/8/2019 SAP Data Quality
44/58
Enterprise Information Management
Kristin McMahonDirector, Enterprise Information ManagementSAPApril 28, 2010
Poorly Managed InformationLeads to Inefficiency and Risk
-
8/8/2019 SAP Data Quality
45/58
Leads to Inefficiency and Risk
Over 51% of organizations estimate datarelated issues cost their company over
Forbes Insight
90% of all businesses still do not havesufficient policies in place to meet data
governance regulations.IT Policy Compliance Group
$5 million.
-
8/8/2019 SAP Data Quality
46/58
-
8/8/2019 SAP Data Quality
47/58
Build an Information Driven Organization
-
8/8/2019 SAP Data Quality
48/58
Provide all users with data thatis complete, accurate andaccessible
Improve business insightand decision making
Provide high quality data to allbusiness processes
Increase operationalefficiency and reduce costs
Enhance informationgovernance via policy-baseddata management
Meet compliance andregulatory requirements
SAP Provides Best-In-Class EIM SolutionsDeliver Information That Is Complete, Accurate, and Accessible
-
8/8/2019 SAP Data Quality
49/58
p
Data Integration & Quality Management:
SAP BusinessObjects Data Services
SAP BusinessObjects Data Federator
SAP BusinessObjects Text Analysis
SAP BusinessObjects Data Insight
SAP Data Migration services
Master Data Management:
SAP NetWeaver Master Data Management
SAP Master Data Governance for Financials
SAP Data Maintenance by Vistex
Enterprise Data Warehousing:
SAP NetWeaver Business Warehouse
SAP NetWeaver Business Warehouse Accelerator
SAP BusinessObjects Rapid Marts
SAP BusinessObjects Metadata Management
Content & Information Lifecycle Management:
SAP NetWeaver Information Lifecycle Management
SAP Extended ECM by Open Text
SAP Document Access by Open Text
SAP Archiving by Open Text
SAP 2007 / Page 49
What Are the Sources of Bad Data Problems?
-
8/8/2019 SAP Data Quality
50/58
SAP AG 2010 / 50
EnterpriseInformation
EmployeeData Entry
CustomerSelf-Service
DataMigrationProjects
ITApplication
Updates
Purchasedor RentedExternal
Data
The Data Quality Framework
-
8/8/2019 SAP Data Quality
51/58
SAP 2009 / Page 51
CONTINUOUSMONITORINGMEASURE
ANALYZE
PARSE
STANDARDIZE
CORRECT
ENHANCE
MATCH
CONSOLIDATE
YOUR DATA
Data Assessment
Enhance Data Cleansing
Match &
Consolidate
Continuous Monitoring
Data Quality ApproachThe Three Rules of Data Quality
-
8/8/2019 SAP Data Quality
52/58
SAP AG 2010 / 52
Rule 1: Analyze your data
Profile, query, extract and in every other waybecome intimately familiar with data content at adetail level. If you take a high-level approach todata quality, you will waste time discussing whatthe data might look like.
Rule 2: Define your scope
All data quality projects uncover hidden issues.Be very clear about what is, and is not, relevantto your current effort.
Rule 3: Cleanse your data and track yourresults
Data quality is not a one-time process. It is anongoing process of monitoring and correctingyour data. You should know that: 1) new qualityneeds are being met and 2) new businessprocesses are being monitored.
What is the definition of clean data?
Who defines clean?
Who owns it over time? Which entities have the most issues? Where are the issues originating from?
Which business processes are affected?
What business benefit can be achieved? How clean does it need to be? People, process, and tools?
Define stakeholders to analyze and clean Define processes to clean, monitor and
maintain cleanliness Acquire necessary tools to assist
Business and IT collaboration through
visualizing information governance metrics
-
8/8/2019 SAP Data Quality
53/58
SAP 2009 / Page 53
Business users can easily see howtheir information measures upagainst information governancerules and standards
IT can easily share data qualitymetrics to business users andinvolve them in owning the dataproblem
Building a Roadmap for Enterprise Information
Management is Key for Success
-
8/8/2019 SAP Data Quality
54/58
1. DataREADINESS
4. DataGOVERNANCE
Understand what data
assets you have andhow they are beingused
Deliver trustedinformation repeatableand reliably at the rightform, to the right place atthe right time
2. DataINTEGRATION &CLEANSING
3. DataCONSOLIDATION
Understand
Govern
Consolidate
Understand
Consolidate
Understand Understand
Consolidate diversemaster data landscapesand increase trust andreliability in information
Technology enablingpeople to implement arepeatable process tomanage the use, qualityand lifecycle of
information
People & Process Maturity
Value
Cleanse Cleanse Cleanse
Data Quality Provides Value ThroughoutPortfolio maps to the People and Process Maturity
-
8/8/2019 SAP Data Quality
55/58
END-TO-END DataManagementFull EnterpriseCOVERAGE
1. DataREADINESS
4. DataGOVERNANCE
2. DataINTEGRATION &CLEANSING
3. DataCONSOLIDATION
Dash Quality
MDG
MDM
Dash Quality
MDM
Dash Quality Data Quality
People & Process Maturity
Value
Dash Integrator Dash Integrator Dash Integrator
SAP 2009 / Page 55
Time to Value: Fast and cost-effective integration with existing
Why SAP?The Best Choice for EIM
-
8/8/2019 SAP Data Quality
56/58
SAP and non-SAP systems
Proven Customer Value: Matureoffering and large install base ofcustomers supporting criticalbusiness scenarios
Market Leadership: Analystrecognition and customerimplementation success
Comprehensive Solutions for EIMStrategyOne-stop for end-to-end informationgovernance and management
SAP 2007 / Page 56
Q ti ??
-
8/8/2019 SAP Data Quality
57/58
Questions??
-
8/8/2019 SAP Data Quality
58/58