lucius mcinnis, systems engineer – client services group kam wong, solutions architect – iway...
TRANSCRIPT
![Page 1: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/1.jpg)
Lucius McInnis, Systems Engineer – Client Services GroupKam Wong, Solutions Architect – iWay Software
March 22, 2012
Getting Data Ready for WebFOCUS
1
![Page 2: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/2.jpg)
Data Quality/Business Intelligence Lexicon
2
GIGI
GOGO
GIGO Garbage-In-Garbage-Out
1960’s Dance Craze (Image: target.com)
1958 Romantic Musical (Image: imdb.com)
![Page 3: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/3.jpg)
Get Rid Of The Garbage…
3
• Access
• Cleanse
• Standardize
• Monitor
• Manage
• Accurate data promotes accurate information and decisions…
![Page 4: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/4.jpg)
4
• ERRORS
• CONFUSION
• DUPLICATION
When Business Data Is Not Managed
![Page 5: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/5.jpg)
AGENDA
5
Fraud, Waste, and Abuse
Operations and Financial Mgmt.Information
Risk, Compliance, and Governance
Revenue Generation
Quality of Care/Service.
• The Path from Data to Information• Access to Data• Data Quality• Master Data Management/Data Synchronization
• Demonstration
![Page 6: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/6.jpg)
Path from Data to Information
6
Infrastructu
re
•Allow for access to data
•Real-Time and Batch Information Movement
•Reusability
DataQualit
y
•Allow for Real-Time Data Quality
•Correct Data Quality issues before they propagate
Master
DataManageme
nt
•Centralize the management of information
•Control the information throughout to organization
![Page 7: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/7.jpg)
Path from Data to Information
7
Infrastructu
re
•Allow for access to data
•Real-Time and Batch Information Movement
•Reusability
#1
![Page 8: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/8.jpg)
Integration Approach – Start with an Integrated Infrastructure
8
![Page 9: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/9.jpg)
Pre-packaged Integration Components
9
SFA/CRM
Amdocs/Clarify BMC/Remedy MSDynamics Oracle/Siebel Salesforce.com SAP
Data Warehouse
DB2 ETL Oracle/Essbase MS SSAS/OLAP Netezza SAP BW Teradata
B2B
Internet EDI Legacy EDI MFT Online B2B XML
ERP/Financials
Ariba I2 JD Edwards Lawson Manugistics Microsoft Oracle SAP
Industry
ACORD CIDX HL7 RNIF SWIFT 1Sync
Legacy Systems
CICS IMS VSAM .NET Java TUXEDO MUMPS
![Page 10: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/10.jpg)
Enterprise Data Integration Scenario
10
…
Data Sources
Data IntegrationData Quality
ReportsDashboards
![Page 11: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/11.jpg)
Path from Data to Business Intelligence
11
DataQualit
y
•Allow for Real-Time Data Quality
•Correct Data Quality issues before they propagate
#2
![Page 12: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/12.jpg)
The Business Value of Data Quality
12
• Improves customer-facing processes: Promotes accurate client address and household information
• Enables advanced analysis: Facilitates the use of data-mining, market predictions, fraud detection, and future client value
• Credit and behavioral scoring:Helps financial institutions improve risk management - Basel Capital Accord III (2010)
• Assists healthcare organizations:Develop an Enterprise Master Patient Index (EMPI) leveraging connectivity to legacy systems and databases
![Page 13: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/13.jpg)
Data Quality Center – Profiling
13
Profiling – Technical (Pre-built)• Basic Analysis
• Minimums• Maximums• Averages• Counts• Etc.
• Patterns / Masking• Extremes• Quantities• Frequency Analysis• Foreign Key Analysis
• Profiling – All• Charting• Grouping / Aggregate• Drilldown / Interactive Displays
![Page 14: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/14.jpg)
Data Quality – Cleansing
14
•Parsing•data parsed into components (pattern based)
•Standardization•transformation into standard format (Jim Smith -> James Smith)•standard and nonstandard abbreviations (Str. -> Street)•language-specific replacements
•Data quality validation•validation against rules •validation against reference tables
•Large number of domain oriented algorithms
•Address•Party•Vehicle•Name•Identification number•Credit Card number•Bank account number
•Extension by custom validation steps
•using complex function and rules including
•Levensthein distance•SoundEx•internal (java-based) functions
![Page 15: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/15.jpg)
Data Quality – Match & Merge
15
•Unification•identification of the candidate groups
•company•address•person•product•…etc.
•Deduplication•best representation of the identified subject•golden record creation
•Identification•new data entries – to identify subject (person, address, etc.) to which the new record is connected (matched)
•Fuzzy logic and scoring•Same name + same address•Same name + similar address•Similar name + same address•Similar name + similar address
•Complex business rules•using sophisticated algorithms and functions including
•Levensthein distance•Hamming distance•Edit distance•Data quality scores values•Data stamps of last modification•Source system originating data
![Page 16: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/16.jpg)
16
Data Quality:Issue Management
![Page 17: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/17.jpg)
Data Quality Issue Management
17
![Page 18: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/18.jpg)
Issue Tracker Portal – Workflow Management
18
![Page 19: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/19.jpg)
Issue Tracker Portal – Issue Resolution (1)
19
![Page 20: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/20.jpg)
Issue Tracker Portal – Issue Resolution (2)
20
![Page 21: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/21.jpg)
Path from Data to Business Intelligence
21
Master
DataManageme
nt
•Centralize the management of information
•Control the information throughout to organization
#3
![Page 22: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/22.jpg)
Moving Towards MDM from Data Quality
22
1. Matching: Identification, linking related entries within or across sets of data.
2. Merging: Creation of the golden data based on one or several reference source and rules.
3. Propagating: Update other systems with Golden Data if required.
4. Monitoring: Deployment of controls to ensure ongoing conformance of data to business rules that define data quality for the organization.
![Page 23: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/23.jpg)
MDM Architectures
23
Master is Single Version of Truth Data Quality at Master Updates occur at Sources Updates propagated to Master
Master
Source Source
Source Source
Consolidated
Registry Style
Master
Source Source
Source Source
• Other Styles Supported
• Multiple Versions of Truth
• Data Quality is Ongoing
• Updates occur at Sources
• Keys and Metadata in Registry
• Updates propagated to other Sources
![Page 24: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/24.jpg)
Project Successes – Pathway to Maturity
24
1. Start with Data Profiling• Understand the data you have• Identify inconsistencies in the data• Disseminate the information about the data quality
2. Continue with Data Quality• Validate, standardize and cleanse for purpose
• Automate the process
• De-duplication (Match & Merge)
3. End with Master Data• Synchronize with closed loop feedback integration
• Provide a single view for all stake holders
Getting to MDM – “Golden Data”
4. Implement Data Governance – Issue Tracking
![Page 25: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/25.jpg)
25
Demonstration
![Page 26: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/26.jpg)
26
Data Management Life-Cycle
![Page 27: Lucius McInnis, Systems Engineer – Client Services Group Kam Wong, Solutions Architect – iWay Software March 22, 2012 Getting Data Ready for WebFOCUS 1](https://reader035.vdocument.in/reader035/viewer/2022062421/56649e2e5503460f94b1e774/html5/thumbnails/27.jpg)
Thank You! - Questions?
27
iWay SoftwareBecause Everything Should Work Together.
WebFOCUS Because Everyone Makes Decisions.