data: migrating, distributing and audit tracking michelle ayers, advisory solution consultant email:...
TRANSCRIPT
1visionsolutions.com
Data: Migrating, Distributing and Audit Tracking
Michelle Ayers, Advisory Solution ConsultantEmail: [email protected]
Mark Scanlon, UKI Regional Director
October 2013
visionsolutions.com 3
Data Migration
Traditional Approach – ‘Big Bang’AllUsersOffSystem
UsersBackOnSystem
Monday AM
EODFriday SundaySaturday
DOWNTIME
Backups Copy Data Transform Data Testing/verify
Issues:• Hope you can complete all steps in available time window…
• Data volumes becoming ever larger – takes longer to copy and convert data
• Business becoming more 24 x 7 - reduced downtime window opportunities
• Final testing/verification limited to what remaining time you have• Many stages are dependant on completion of previous stage
visionsolutions.com 6
Data Migration
More Complex:
SourceData
TargetData
DT ShareCaptures Changes
DT ShareCaptures Changes
DT ShareApplies trans to Staging Database
DT ShareApplies trans to Staging Database
IBMi Oracle
Administration & Monitoring(Director, MonCon & Console)
LAN / WAN
Data path
CustomPrograms
CustomPrograms
CustomPrograms
CustomPrograms
Staging Data
Staging Data
DT ShareCaptures Changes
DT ShareCaptures Changes
DT ShareApplies trans to Staging Database
DT ShareApplies trans to Staging Database
LAN / WAN
Data path
visionsolutions.com 7
Data Sharing
Typical Approaches:
Application Database
Apps
Website
ERP Application
Business Intelligence
User Queries
• Lots of demands on the central database• All users/applications using same data therefore all should be consistentHowever:• Performance issues – too much access of same data• Users running inefficient SQL statements• Increased security risks from many different communication jobsCan lead to productivity issues for core business processes
visionsolutions.com 8
Offloaded data warehouses have been around some yearsHowever:• Data often loaded by batch overnight• May even need to reload whole tables each nightIssues:• Data is out of date quickly• Every day the database grows a little bit bigger, therefore each night the
load job takes a little bit longer…
Data Sharing
Typical Approach Using Offloaded Data Warehouse:
Application Database
ERP Application
Business Intelligence
User Queries
Data Warehouse
visionsolutions.com 9
Data Sharing
Alternative Approach – ‘CDC’ Change Data CaptureCustomer Story – Red Wing Shoe Company - USA
OFFLOAD QUERY & REPORTING PROJECTOFFLOAD QUERY & REPORTING PROJECT
Manufacturing
MS SQL Server
Query reports
Data Warehouse load
Double-Take ShareReal time CDC replicationwith transformation
Reduce CPU and I/O overheadon production systemimprove user response times
Many cost effective tools available on MS SQL serverplatform for query reports
Data is already partially‘scrubbed’ and availablefor loading data warehousesand data marts withoutperformance impact on production system
Production System Offload Query System
IBM System i DB2Lawson M3 (Movex)
visionsolutions.com 10
Data Sharing
Alternative Approach – ‘CDC’ Change Data CaptureCustomer Example – Application Interface
Bi-directional replication
IBM System i DB2Manufacturing ERP
Introduced automated manufacturing inspection tracking and also faster Data Warehouse Load.
MS SQL Server
Double-Take ShareReal time CDC replicationwith transformation
Production line information is transformed and sent to ERP
Production SystemAPPLICATION INTEGRATION PROJECTAPPLICATION INTEGRATION PROJECT
Query reports
MS SQL Server
Double-Take Share Replaced batch reload of data warehouse with DT Share creating incremental extracts of last changed row, and deleted transactions.
Third Party Inspection Application
visionsolutions.com 11
Data Auditing
Traditional Approach to Field Audit Tracking
Application Database Audit FilesBefore and after changes captured via embedded program codeOr trigger programs
• Older applications may be using embedded code to capture changes – high overhead to maintain and ensure nothing missed
• Trigger based capture can have database overhead though should ensure call changes captured
• ERP provided solution can be limited:• May only support pre-selected tables/columns• Restricted data filtering capabilities
visionsolutions.com 12
Data Auditing
Alternative Approach – ‘CDC’ Change Data Capture
FIELD AUDIT TRACKING PROJECTFIELD AUDIT TRACKING PROJECT
Application Database Audit FilesDouble-Take ShareReal time CDC replication captures before and after valuesIBMi uses journal receiver entries
• Capture changes made within or outside the ERP application• Optionally add row/column filtering• Choose columns from any database table, including local modification• Target audit files are custom designed• Add target columns to hold trans type, timestamp, user id• Access to audit data via custom queries/reports• IBMi M3 java database has journaling switched on by installation – DT
Share can use these journal receivers.
Target auditfiles can be heldin same databaseor different, sayMS SQL Server