strategy for data quality
DESCRIPTION
Presentation from an Ark Conference on establishing a Strategy for Data Quality.TRANSCRIPT
![Page 1: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/1.jpg)
1
Establishing a Strategy for Enterprise Data QualityBarry Williams Principal ConsultantDatabase Answers Ltd.Ark Conference 1st April 2008
![Page 2: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/2.jpg)
2
Establishing a Strategy for Enterprise Data Quality
Overview
• Identifying the Infrastructure (data arch)
• Setting a Quality Control Initiative (tools)
• Developing Plans to enrich Quality (data platfm)
• Getting Started
![Page 3: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/3.jpg)
3
Establishing a Strategy for Enterprise Data Quality
What is Data Quality ?
TDWI says …
Wikipedia says …• Many things• Good enough (!!)
Barry says …• “Fit for Purpose”
![Page 4: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/4.jpg)
4
Establishing a Strategy for Enterprise Data Quality
1. Identify the Infrastructure
• The Framework
• As-Is and To-Be
• Roles for Everybody
![Page 5: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/5.jpg)
5
Establishing a Strategy for Enterprise Data Quality
Fifteen Years Experience
• Barclays (1993) • Barclays (1998)• Centrica (2001)• Cisco (2003)• Ealing (2005-2008)
![Page 6: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/6.jpg)
6
Establishing a Strategy for Enterprise Data QualityStarting out at Barclays Bank (1993)
![Page 7: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/7.jpg)
7
Establishing a Strategy for Enterprise Data Quality
From Experience to Infrastructure
Framework• Data Governance• Data Quality Architecture• Data Quality Metrics• Tools
![Page 8: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/8.jpg)
8
Establishing a Strategy for Enterprise Data Quality
Basic Data Quality Architecture
• An Entry-Level System• Rules in SQL
![Page 9: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/9.jpg)
9
Establishing a Strategy for Enterprise Data Quality
Intermediate DQ Architecture
• Add Library of Scripts• Produce Reports
![Page 10: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/10.jpg)
10
Establishing a Strategy for Enterprise Data Quality
Advanced DQ Architecture
• Within Governance Framework
![Page 11: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/11.jpg)
11
Establishing a Strategy for Enterprise Data Quality
Tomorrow’s DQ Architecture
• Web Services-based
![Page 12: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/12.jpg)
12
Establishing a Strategy for Enterprise Data Quality
DQ Real-Time System• Validate in Batch• Validate Data on Entry
![Page 13: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/13.jpg)
13
Establishing a Strategy for Enterprise Data Quality
A Data Quality Dashboard
![Page 14: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/14.jpg)
14
Establishing a Strategy for Enterprise Data Quality
Data Quality Metrics
What Makes a Good Metric ?• Clear and Agreed Definition • Easy to Measure • Relevant to the Business
![Page 15: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/15.jpg)
15
Establishing a Strategy for Enterprise Data Quality
2. Setting a quality control initiative
• Establish the Objectives
• Define the Data Quality Architecture
• Top-Down and/or Bottom-Up
• Choose Tools or DIY …
![Page 16: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/16.jpg)
16
Establishing a Strategy for Enterprise Data Quality
Tool Vendors – DIY
Suitable where :-• Limited Scope
• Simple DQ Rules
• Templates are usable
![Page 17: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/17.jpg)
17
Establishing a Strategy for Enterprise Data Quality
Tool Vendors – Niche Players
• Ab-Initio (Data Profiling)
• InfoShare (Customer Matching)
• InSource (Data Warehousing)
![Page 18: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/18.jpg)
18
Establishing a Strategy for Enterprise Data Quality
Tool Vendors - Gartner
• Gartner’s Leaders Quadrant– DataFlux– Data Foundations (‘Cool Vendor’)– IBM– Trillium
![Page 19: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/19.jpg)
19
Establishing a Strategy for Enterprise Data Quality
Tool Vendors DQ-as-a-Service
• Boomi
• SalesForce and Business Objects SalesForce and Informatica
Talend
![Page 20: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/20.jpg)
20
Establishing a Strategy for Enterprise Data Quality
Tool Vendors – Open Source
Talend – Chinese Office Data-Integration-on-Demand
SQL Power - Canadian geared to Data Warehousing
![Page 21: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/21.jpg)
21
Establishing a Strategy for Enterprise Data Quality
Tool Vendors – SQL Power Data Profiling
![Page 22: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/22.jpg)
22
Establishing a Strategy for Enterprise Data Quality 3. Developing plans to enrich the quality
Data Quality is an Enterprise Issue• Top-level Support• Data Governance • Master Data Management• Customer Data Integration
![Page 23: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/23.jpg)
23
Establishing a Strategy for Enterprise Data Quality
The Plans
• Determine Your Data Platform• Establish the Roadmap• Agree Business View of Data• QA is a stethoscope
![Page 24: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/24.jpg)
24
Establishing a Strategy for Enterprise Data Quality
The Data Platform
• Each Stage builds on the previous one5) BI Data Mart
1) Properties - Gazetteer
2) Services - Directorate- Service Name
3) Customer Master Index
4) Customer Services
![Page 25: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/25.jpg)
25
Establishing a Strategy for Enterprise Data Quality
Single View of the Customer
Customer- Date- Standard Debt Type- Amount
Housing Benefits Overpayments
Council Tax
Parking Fines
Business Rates
Rent Arrears
• Requires Quality to Consolidate Data
• Needs Customer Data Integration Software
eg InfoShare, DataFlux (MDM/CDI)
![Page 26: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/26.jpg)
26
Establishing a Strategy for Enterprise Data Quality
Framework for Performance Management Participants
• Directors, Managers, Business Partners,etc.
Performance Reporting• Traffic Lights• Key Performance Indicators• BVPIs • Drill-Down• Reports, etc.
Data Quality Standardisation Layer• Enterprise Data Model• Single View of the Customer• LGSL, Master Data Management, etc.
![Page 27: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/27.jpg)
27
Establishing a Strategy for Enterprise Data Quality Enterprise Data Model
• Comprehensive, Generic and Unique
• A Standard way to integrate Customer Data
• Over 200 Entities in 14 Functional Areas
• Defines Data Standardisation Layer in SOA
![Page 28: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/28.jpg)
28
Establishing a Strategy for Enterprise Data Quality
Enterprise Data Model
![Page 29: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/29.jpg)
29
Customer Area
Property Area
Service_Request
Customer - Organisation - Person
Geographic_Address(Std = Gazetteer LLPG)
Service Catalogue(Std=LGSL/IPSV)
Service Delivery Area
Establishing a Strategy for Enterprise Data Quality
EDM Diagram Extract
Customer_Address_Occupancy
![Page 30: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/30.jpg)
30
Establishing a Strategy for Enterprise Data Quality Data Standardisation Layer
DATA QUALITY LAYER
- Mapping from Vendor-specific to Ealing Standards,(LGSL, e-GIF, Ethnic Origins, etc.) - Customer Master Index, Enterprise Data Model
BI Data Marts- Social Services- Street Environment- BVPIs, KPIs
Services - ERDMS File Plan- LGSL / IPSV (Govt Standard)
Customers - Matches
Customer Histories - Links to LOBs
Lines of Business (LOBs)
Data Quality Audit- Data Profiling - Gazetteer Validation
CRM- Customer Profiles- Good/Bad Customers
Reference Data - Ethnic Origins - Vehicle Makes and Models
Self-Service Portal- Enquiries
![Page 31: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/31.jpg)
31
Establishing a Strategy for Enterprise Data Quality
Determine the Standards• Easy where defined
• LGSL /IPSV, BVPIs
• Aim for Buy-In
• Create Glossary for Mapping
• Look for obvious Data Leaders• eg Social Services for Ethnic Origins
![Page 32: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/32.jpg)
32
Establishing a Strategy for Enterprise Data Quality
4. Steps in Getting Started
• Identify Business Drivers
• Decide Roles and Responsibilities
• Agree Overall Timetables
• Consider Data Quality Audit
![Page 33: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/33.jpg)
33
Establishing a Strategy for Enterprise Data Quality
Identify Business Drivers
• Over 200 Legacy Systems
• 300,000+ customers – Ethnic Origin Breakdown ?– Customers receiving multiple Services ?
• Need Single View of the Customer
• Standards are essential for BI
![Page 34: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/34.jpg)
34
Establishing a Strategy for Enterprise Data Quality
Roles and Responsibilities
• Senior Management
• Line-of-Business Managers
• Data Stewards
• DQ Professionals
![Page 35: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/35.jpg)
35
Establishing a Strategy for Enterprise Data Quality
Identify Business Champions
• With Vision • Evangelists
• High-Profile Service
• Successful Track-Record
![Page 36: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/36.jpg)
36
Establishing a Strategy for Enterprise Data Quality
Agree an Overall Timetable
• One Year Targets
• Three months Targets
• Quick Wins
• Road Map
![Page 37: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/37.jpg)
37
Establishing a Strategy for Enterprise Data Quality
Decide the Approach
• Top-Down and/or Bottom-Up
• POC or ‘Feasibility Study’ • Management Involvement
• Success Criteria
![Page 38: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/38.jpg)
38
Establishing a Strategy for Enterprise Data Quality
Consider a Data Quality Audit • Sell the Importance
• Can use SQL
• Data Profiles suggest Standards
• Obtain Buy-In from Data Owners
• Slice down the Organisation
![Page 39: Strategy For Data Quality](https://reader036.vdocument.in/reader036/viewer/2022062613/5447b8acafaf9f25708b463e/html5/thumbnails/39.jpg)
39
Establishing a Strategy for Enterprise Data Quality
Contact Details
• Barry Williams– [email protected]
• Database Answers Web Site– www.databaseanswers.org/data_cleansing.htm
• Community of DB Professionals– Databaseanswers.ning.com