qa challanges in data integration
TRANSCRIPT
QA Challenges in Data Integration Projects &
iCEDQ SoftwareData Warehouse | ETL | Data Migration | Data Lake | MDM
ETL Testing & Monitoring Platform
The QA Challenge
Lack of Automation
Lack of Agility
Clutter
No Test Driven Development
Lack of Regression Testing
Data Warehouse | ETL | Data Migration | Data Lake | MDM
www.icedq.com
vs. iCEDQ has in-Memory Rules Engine
Tests ETL processes by Validating the output data
Validation Rule
Reconciliation Rule
Tests ETL Transformations by Reconciling Source Data Vs. Target Data
ETL
Too many tools – Excel, SQL, Scripting
Not repeatable and consistent Can’t test Millions of rows Forced to do sampling Time wasted in coding rather
than testing Disorganized Time & resource consuming
A
iCEDQ Shift
www.icedq.com
CClutter vs. Organized & Documented
START
Business Requirements
Test Results
Workflow
QA Dashboard
Audit (Testing) Requirements
Audit Rules & Scripts
Test Cases
iCEDQRepository
iCEDQ Shift
www.icedq.com
Conventional Waterfall SDLC vs.
Testing is too late in the project. 8th Step! QA team sitting idle for developers to finish Last minute rush
Step 1. Get Requirements
Step 2. Design the
TablesStep 3 Create Data Mapping
Step 4. Develop ETL
Code
Step 5. Get Test
RequirementsStep 6. Write
Test CasesStep 7. Write Physical Test Step 8. Test Step 9.
Deploy
Step 1. Get Requirements
Step 2. Design the
Tables
Step 3 Create Data
Mapping
Step 4. Develop ETL
CodeStep 5.
TestStep 6. Deploy 7 8 9
RefactorReduce
timeline by 33%
Step 5. Get Audit Requirements
Step 6. Write Test Cases
Step 7. Create
Physical Rules
iCEDQ enables Test First Approach Test Driven Development - TDD Lift n’ Shift the QA to the beginning of project
iCEDQ -TDD
DiCEDQ Shift
www.icedq.com
Qualit
y ?
Start
QA QC
Regression Testing & Release Management
Conventional data integration testing lacks… Tools for regression
testing Sign-Off & release
management Continues Build
Management Quality Control
iCEDQiCEDQ Shift
Business Requirements
Audit Requirements
Test Cases
Regression Testing Deployment
Release Management
Quality Control & Production Monitoring
Source Sys.
Biz. Users Management
E
www.icedq.com
Why iCEDQ?
Automation Test high volume of data Regression Testing Need Agile and/or Test Driven Development Lower Project Risks & Cost Improve Productivity Compliance Security Organized & Collaborative Reporting, Dashboard & Notification Transparency & Feedback Shrink Project Timeline
Data Warehouse | ETL | Data Migration | Data Lake | MDM
www.icedq.com
Development & QA Testing
Customer Finds It Post Release
Leads to Litigation, Fines, Reputation Loss
1x
100x
10,000xSave $$$$$$
Early is Better!
Cos
t of D
elay
$
Time www.icedq.com
iCEDQ SERVER
iCEDQ Architecture
Audit Rules Engine
iALM iGOVERNiQAiAUDIT Scheduler
RepositorySQL Server |Oracle
Configuration Files
Parameter Files Log Files
Data Exceptions
Files
GUI Web Service CLI/Shell RPC Report
JDBC
Text File
Cobol*
Other Files
Mainframe*
Data Adapters
Oracle | SQL Server | Redshift | DB2 | My SQL |
Netezza |…
Hadoop HIVE | HP Vertica
XML | EDI | JSON
CSV | Fixed Width | Excel
Cobol Files
ISAM | KISM | KSDS | QSAM |VSAM
BIG DATA
Redshift, HADOOP, Hive, S3, HP VERTICA, Cassandra, MONGODB, HDFS…www.icedq.com
Who uses iCEDQ?
Stock Exchange Banks Insurance Manufacturing Healthcare E-Commerce Manufacturing …
www.iCEDQ.com
game changer:nouna newly introduced element or factor that changes an existing situation or activity in a significant way
ETL Testing & Monitoring Platform
- Merriam Webster
www.iCEDQ.com