Тестирование данных с помощью data quality services (ms sql 12)
DESCRIPTION
Презентация доклада Дмитрия Романова на конференции SQADays-14, Львов 8-9 ноября 2013TRANSCRIPT
![Page 1: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/1.jpg)
Quality Assurance for Data with Data Quality Services (MS SQL 12)
Dmitriy RomanovItera Consulting, Kiev
![Page 2: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/2.jpg)
Dmitriy Romanov
Areas of expertise:
Test Automation for various projects in:Business IntelligenceRIABilling systems
![Page 3: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/3.jpg)
Agenda
• Intro– Data Quality – what it is about ?– Data Quality in Business Intelligence projects– Tools selection
• Data Quality Services– Structure– Project component– Data Quality routine
• Conclusions
![Page 4: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/4.jpg)
Typical information flow
![Page 5: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/5.jpg)
Data Quality Components
DATA QUALITY
Validity
Accuracy
Consistency
Integrity
Timeliness
Completeness
![Page 6: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/6.jpg)
Data Quality IssuesBefore QA :
After QA :
![Page 7: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/7.jpg)
Data Quality: What is it?
Business intelligence (BI) is a set of methodologies, processes, and technologies that
transform raw data into meaningful and useful information for business purposes.
Data Quality – represents the degree to which Data is suitable for business usages
![Page 8: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/8.jpg)
Data Quality: Tools selection
Custom Tools• Variety of technologies• Flexibility• Accuracy
PROS
• Higher Competence level in business area / tech. stack
• Lots of development efforts
CONS
3rd-party software• Established methods, standards,
algorithms• Open / Expandable / Reusable• Lower entry level for newcomers
PROS
• Scalability / performance issues• Limitations
CONS
![Page 9: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/9.jpg)
Gartner Magic Quadrant for BI platformsCHALLENGERS LEADERS
NICHE PLAYERS VISIONARIES
COMPLETENESS OF VISION
ABILITY TO EXECUTE
![Page 10: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/10.jpg)
Data Quality: tasksData Quality Services (DQS) is a Knowledge-Driven data
quality solution enabling data stewards to easily improve the quality of their data
Cleansing Matching
Profiling Monitoring
![Page 11: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/11.jpg)
DQS: Knowledgebase creation process
Build
Use
DQ Projects
KnowledgeManagement
Match & De-dupe Correct & sta
ndardize
Manage Knowledge
Connect
EnterpriseData
ReferenceData
Cloud Services
KnowledgeBase
Discover /
Explore Data
Notifications
Progress
Status
![Page 12: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/12.jpg)
MatchingReference
Data
DQ Clients
DQ Server
DQ Projects Store
Common Knowledge Store
Knowledge Base Store
DQ Engine
3rd Party / Internal
SSIS DQ Component
DQ Active Projects
Published KBs
Knowledge Discovery
Data Profiling & Exploration
Cleansing
Azure Market Place
Reference Data API(Browse, Get,
Update…)
RD Services API
(Browse, Set, Validate…)
Data Domains
DQS User Interface
DQS Structure
![Page 13: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/13.jpg)
DQS Usage
Knowledge Base
Reference Data Definition
Values/Rules
New
Suggestions
Correct & Corrected
Invalid
Source DQS CleansingComponent
SSIS Package
Destination
Reference Data Services
DQS Server
Design Run
Monitor Review & Manage
Activity Monitoring
Interactive Cleansing Project
![Page 14: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/14.jpg)
Real Examples
![Page 15: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/15.jpg)
Business Case – Source Data Quality Assurance
Source Data
Oracle
DB2
csv
Screen
DQS
Load
KDVH
ConfirmStatus
“Ready to load”
DQ Reports
Data steward - requesting source data fixing
ETL
Data steward - managing data KB- monitoring DQ process
![Page 16: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/16.jpg)
How DQS could help QA Engineer ?
• In general it allows to bring closer things Data Analytics usually deal with
• Helps to understand underlaying data better • Introduce measurement and manageability to DQ
matters• Increase re-use/decrease re-work• Open and extendable proposal of new standard to
store and treat Knowledge Bases on iterative basis
![Page 17: Тестирование данных с помощью Data Quality Services (MS SQL 12)](https://reader034.vdocument.in/reader034/viewer/2022050904/545bee1ab1af9f280a8b45a0/html5/thumbnails/17.jpg)
Thank you