brochure - chain-sys platform suite for data migration ...data clean & lean data hadoop...
TRANSCRIPT
Data Migration Platform
appTM
MIGRATE
DATA RECONCILIATION
DATA PROFILING
DATA COEXISTENCE
DATA MAINTENANCE
DATA QUALITY
MANAGEMENT
CUTOVER PLANNING
AND EXECUTION
DATA MIGRATION
Data migration done in the traditional way, writing your own conversion programs, is a painful and expensive proposition. Conversion programs don’t get ready before approximately six months into the project, pushing the exercise of examining data for correctness and standardization too close to the Go-Live date. Data quality issues do not get the needed attention.
“Data Conversion was a big part of enabling our successful milestone, ‘System Integration Testing’. Chain-Sys overcame many challenges, shortened the conversion cycle and completed conversions on time.”
-CIO of a major iconic EnergyCompany
?
?
Problem
Solution
appMIGRATETM
Field proven appMIGRATE provides instant relief for data migration challenges. Being a product with ready to use configurable templates with no need for programming, appMIGRATE allows Business Users to see production data in the target instance as early as the first test cycle. It gives them ample time to correct, clean, consolidate, standardize and harmonize their data. The tool plays well with agile methodology. Data risks are identified early in the project and solutions made ready and applied.
The tool reduces production cutover time and the overall project timeline. appMIGRATE product and processes are battle tested. It helped several fortune 1000 companies including GE, Cummins, Caterpillar, Iron Mountain, Canon, CH2MHill, Lifeway Christian Resources and others to migrate their complex data in a scalable and repeatable manner.
The appMIGRATE Solution with 1000+ ready-made templates provides accelerated data conversion and ensures data quality. The appMIGRATE Solution has been recognized by the industry and is being utilized by Fortune 1000 companies for meeting their challenging data conversion requirements.
appMIGRATE tool provides end to end data migration processes right from Requirements Definition to successful Cutover. All these processes and activities come built inside the tool, allowing data migration SDLC processes to become repeatable, scalable and predictable.
“Never before we had all data converted before the first test cycle. With this tool and the methodology, we are free to concentrate on other business and process issues.”
-ERP Practice head of a majorSystems Integrator
Product Architecture:Template Adapters Interface Adapters Extract Adapters appLOAD Adapters Log & Notification
Handler
appMIGRATE
Schedule Engine appEXTRACT Service FTP Service appLOAD Service Log & Notification Handler
appINTERFACE
Template Processor Loader Processor Pre-validation Engine Error Correction Cleansing Handler
appLOAD
Instance Handler Data Processor Validation Engine Log & Notification Handler
Template Adapters
BASE COMPONENTS
Query Parser Extract Engine Error Handler
appEXTRACT
Accelerated ETL:
EBS LEGACY SAPSALES FORCE
DATA MART
TARGET*
TARGET*
DQM/MDM LOAD/DELTA LOAD
EXTRACTION
CLEANSE/TRANSFORM
Automated documentation for Audit & Reporting
Compare with target
Predefined Load
Predefined Extract
EXTRACTION
Ready-made extraction templates will provide earlier view of the source data in the Data Mart.
Predefined extraction templates available for Master and Transactional data (both open and historical)
All data fields are extracted from Legacy Systems
Raw data is profiled and map reduced using Hadoop
Ready-made templates accelerate target instance mapping
CLEANSE/TRANSFORM
Data is pre-validated prior to loading into target Oracle EBS/SAP environment
Automated sequencing of Data migration objects
Preserves the data linkages for all dependent data elements
Raw data is profiled and map reduced using Hadoop
Repeatable and flexible Change Data Capture(CDC) Cutover
HISTORICAL CONSIDERATIONS
Post processing builds reference to historical source
Pre-built historical sequencing of master/transaction data
Front-end screen to build historical mapping
Integrity for historical drill-down to the transaction level
Dimensional reconciliation engine for source and target
Includes standard fieldsIncludes user defined fields
* Oracle Cloud Applications or Oracle E-Business Suite or SAP ECC 6.0 or SAP S/4HANA or Other Enterprise Systems
appMIGRATE Architecture:
“For a large document management company we reduced Item Master data size to about 5% of the original size.”
Data Profiling approach :
DATA MART
ANYSOURCE
DATAPROFILING AT SOURCE INSTANCE
DATA MART
DATAPROFILING AT TARGET
Raw Data
Corrected Data
Clean & Lean Data
Hadoop
appMIGRATETM
Data Profiling
“appMIGRATE performs data profiling on the source data to reduce the data volume and explore data quality issues. Data profiling is done with the extracted data against the target Applications Reference data and Setup data dependencies with predefined rules. Business process verification is another highlight. Data profiling engine is also used for on-going ERP wellness activities”
Profiling source data improves data quality and reduces data quantity. Profiling the extracted data against the target reference data and business logic is built inside the appMIGRATE tool. Each Template comes packed with hundreds of pre-validations.
appMIGRATE helps investigate if existing data can be used for other purposes.
Improves the ability to search the data by tagging it with keywords, descriptions or assigning categories.
Allows assessment of risk involved in integrating data for new applications, including the challenges of joins.
Helps understanding data challenges early in any data intensive project, so that late project surprises are avoided. Finding data problems late in the project can lead to delays and cost overruns.
Provides enterprise view of all data, for uses such as Master Data Management where key data is needed, or Data Governance for improving data quality.
TARGET*
* Oracle Cloud Applications or Oracle E-Business Suite or SAP ECC 6.0 or SAP S/4HANA or Other Enterprise Systems
“DQM is a prime mover of any successful Data Migration project. It is a mandatory process and cannot be wished away.”
appMIGRATE provides extensive features and functions for Data Quality needs of Data Migration projects. The major features offered include:
Data Quality Management
appMIGRATE Data Migration process specifies Data
Data De-Duplication
Data Cleansing
Data Harmonization
Data Consolidation
Data Enrichment
Data Standardization
ANYSOURCE
Raw Data
Corrected Data
DATAMART
DATAPROFILING
Hadoop
INSTANCE
Data Cleansing Data Consolidation
Data Enrichment Data Standardization
Data Construction
+
Quality Management activities as “Mandatory” for all Data Migration projects. The DQM activity is generally carried out in the early phase of the project and it is a precursor to Data Migration activities. The appMIGRATE tool has powerful features to collect high volume of master data into its data mart for performing DQM activities. It uses consolidation algorithms along with Hadoop technology to identify the matches and no matches for the collected data. Provides a good user experience in classifying the matches into “false positives” or “false negatives”. When the user merges the matching records, individual values can be copied from one to another. Using “Merge” activity, data gets harmonized using data cross references.
Data Coexistence
Enterprises do not adopt simultaneous “big bang” Go Live into a single ERP system from all their geographies and divisions. In a phased approach, existing data interfaces cannot be interrupted for the legacy systems, when feeder systems have already been migrated over. appMIGRATE Module fits the bill for quick interface building between systems, both batch and real time.
During ERP Implementations, appMIGRATE is used for building the integrations which are needed for the coexistence of the old and the new applications. The appMIGRATE Extract and Load Adapters are reused for the coexistence tasks.
Clean & Lean Data
TARGET*
* Oracle Cloud Applications or Oracle E-Business Suite or SAP ECC 6.0 or SAP S/4HANA or Other Enterprise Systems
Data Reconciliation
“Chain-Sys Reconciliation engine is used for validation of the Data Migration results. Also it is used for on-going ERP wellness activities”
Technical and Functional reconciliation is a mandatory step in Chain-Sys Data migration process. It ensures smooth transition to the target application. It also helps with the ongoing Data Maintenance efforts. appMIGRATE consistently delivers 99.999999% data reconciliation success in every project.
appMIGRATE tool provides out of the box data reconciliation processes and functions for both Technical and Functional reconciliations. The tool provides complete drill down capable dashboards for identifying the discrepancy between source and target systems. During cutover, appMIGRATE helps you with the reconciliation needs between Sub Ledger and Main Ledger (Finance and Cost). Within Sub Ledger accounting transactions and activities modules are tracked and reported using this engine.
“The reconciliation on USD 500 million came to USD 7. The CFO pulled out USD 7 from his pocket and presented it to the CEO during go live. The technical team worked on it later and brought reconciliation to zero.”
LOADINGLoading into Target, Build Cross Reference
VALIDATE AND TRANSORM
Data Mart, Simulation,Bulk Transform
SOURCE RECONCILIATION
Extracted Vs Transformed
EXTRACT SOURCE
Database, File Systems,Queues, Webservices
TARGETRECONCILIATION
Extracted vs Loaded,Fieldwise, Dollarwise, Quantitywise etc.
General Ledger Journals Reconciliation Results with Drill Down features (Periodwise, Journal Categorywise, Journal Sourcewise, View overall data, View Mismatch data):
appMIGRATETM
“appMIGRATE achieved a record 3 hours cutover time at a very Transaction Intensive major Travel site.”
Cutover Planning
“appMIGRATE Cutover has always been smooth – with no data issues after Go Live.”
appMIGRATE provides features and functions for a smooth cutover. Smooth cutover is achieved by a combination of features including: Change Data Capture, Data Scalability functions and Configuration Management.
Change Data Capture
The appMIGRATE product offers powerful Change Data Capture functionality. Using this feature, the source application tables redo log files are monitored at a scheduled time frequency and the changes are moved into the Change Data Capture tables. From the Change Data Capture tables, the data is extracted, transformed and loaded into Target Application. appMIGRATE offers Templates for “Update” of the data entities along with “Create” option, which will be used for updating the target Application. Templates for timestamp based CDC are available.
Data Scalability Functions
For Data Migration situations of large data volumes, complex validations and intricate transformations, appMIGRATE offers the following features:
“appMIGRATE successfully processed over 520 Million records of installed base for a major appliances brand in USA”
Clustering multiple application servers to load balance
Create multiple JVMs under the application server to process more threads
Process Extractions in parallel and Process Loads in parallel
Execute multiple threads simultaneous based on number of cores and processors
Split the loads into smaller batches of data and process in parallel
Reduce the IO against the database drastically
CLUSTER
MULTIPLE JVM
PARALLEL PROCESSING
MULTIPLE THREADS
SPLIT PROCESSING
MERGE OPTION
FEATURE DETAILS
s appLOAD
TM
appMIGRATETM
appSILHOUETTETM
appBITM
Street
Zip
StreetZip
Zip Extn
appAUGMENTERTM
appMDMTM
app CONNECTTM TM
www.chainsys.com1-855-appLOAD Option 1
Copyright © 2016 Chain-Sys Corporation
CHAIN SYS
Define Data Object, Attribute, Validation, Transformation and Cleansing Rules and Requirements
appMIGRATE Adapters are Configured
appMIGRATE Loader is Configured
appMIGRATE Server
Initial Purging, Archiving, Cleansing Transformation and Pre-Validation is Performed
Target Update Data File
Upload Programs areExecuted
Target System
appMIGRATE DataStaging Table
Data is ManuallyCleansed andTransformed
Purge and ArchiveData Files
Error and Exception Reports
Errors are Corrected
Source Systems
appMIGRATE process diagram:
appMIGRATE DataStaging Table
Pre-Validation is Performed
Target System
There are multiple challenges with traditional Master Data Management approaches. The products and processes are cumbersome to implement. They need many other add on tools and technologies for making the core MDM product to work. The MDM products are compartmentalized by domains such as: products, customers, suppliers etc. Most MDM solutions are tightly coupled with the vendor’s own ERP products. It takes a long time for implementation mainly due to activities such as Analysis and Discovery for the Hub Data Model, Configuration and Development of the Hub, Creation of the workflow governance for Activation, and Approvals and Confirmation for create/update of the master data. Traditional MDM Products do not offer industry specific solutions for the master data attributes and inter dependencies of the attributes, which leads to a long time taken for the clients to create attribute mappings. Due to these shortcomings, many enterprises are apprehensive about implementing traditional MDM products for their businesses, even though they are fully aware of the benefits of MDM.
? Problem
Chain-Sys has critically analyzed the above problems and offers a complete MDM Product, appMDM™. appMDM™ focusses on and satisfies all the MDM processes. It is highly configurable and enables master data on-boarding and maintenance. appMDM™ provides industry focused templates for all verticals. The unique and powerful templated approach for Hub modeling and configuration helps you to create a complex Hub within hours rather than months. The Cloud enabled integration engine enables you to connect the MDM Hubs with Mobile, On Premise and Cloud Applications with no programming for both in bound and out bound transactions. The inbuilt features support high volume scalability and complex business validations for your data management needs. appMDM™ customers are getting full comfort with their master data quality and data governance policies, with the unique “Get Clean Stay Clean” preventive control approach. appMDM™ satisfies the end to end Master data management process and system needs.
? Solution
Cloud Enabled.
Single Product solving all MDM challenges.
Accelerated implementation using templated approach.
Minimized data risks and Improved Security.
Improved data quality and data ownership.
Approval Workflows for assured data success.
Integrates with Oracle E-Business Suite, SAP, MS Dynamics, SFDC, Chain-Sys ERP, PeopleSoft, JD Edwards, Oracle Fusion Applications, and Oracle Cloud Applications.
The value propositions:
A Single Source System is designated as the “System of Record” for Master Data.
MASTER DATA MANAGEMENT
Chain-Sys provides a configurable Platform for users to create their Data Model, User Interfaces, and Business Validations for the MDM Hubs. The Platform automates the code creation for the MVC architecture. The Data Architects can create any complex Data Hubs using this platform rapidly.
CHAIN-SYS PLATFORM
Chain-Sys Platform:
Chain-SysPlatform
Multiple DomainHubs
No Programming
Data QualityManagement
appMDM provides “Central Data Hub” to process Master Data Maintenance Transactions (Add, Change, Archive, Purge).
A “Master Data Governance” group controls the Master Data Management process and safeguards the data.
appMDM consolidates master data from multiple sources into a Central Data Hub (Data Mart) using Data Quality Management features. The Data Hub allows consolidated reporting and query of master data, inventory, cost, and account balances etc.
Work Flow functionality supports data governance and approval processes and procedures.
Control of privileges at organization/object/attribute level ensures data integrity.
DM with
appMDM provides complete flexibility on the Domains. It provides the following out of the box domains for Products (Products, BOM Structures, Formulas, Routing, and Recipe), Contacts (Customers, Customer Sites, Customer Contacts, Suppliers, Supplier Sites, Supplier Contacts, Employee and Employee Contacts) and Operations (Fixed Assets, and GL Code Combination). Designing a custom domain is easy. Users can create domain hubs for their specific needs with no programming efforts, by simple configurations. Chain-Sys strongly believes in the template based approach for creating Hubs. appMDM provides templates for all their standard domains against major applications such as: SAP, Oracle EBS, Oracle Fusion, JD Edwards, PeopleSoft, Microsoft and Chain-Sys ERP.
MULTIPLE DOMAIN HUBS
Screen Design Workbench:
Build MDM Hub easily using Chain-Sys Platform:
Product Hub rapidly created using Chain-Sys Platform:
Chain-Sys provides a state-of-the-art Data Management tool for configuring Migration and Integration objects. Chain-Sys offers the Industry Gold Standards: appMIGRATE and appINTERFACE tools for performing the Data Migration and Integration for MDM efforts.
appMDM DATA MANAGEMENT – NO PROGRAMMING, SIMPLE CONFIGURATIONS
Outbound Interface to Consumer Application:
The appMDM’s Data Management solution satisfies complex data management needs such as: Data Profiling, Data Assesments, Data Synchronization, Change Data Capture, Data Harmonization, Data Visualization and Data Classification.
Design and maintain the enterprise data model that supports desired business application landscape and capabilities
Ensure that Data Management efforts (Business or IT) are accelerated and consistent with enterprise core process performance
Accountable for data quality; resolve information and cross process issues
Enforce global data rules and standards monitoring master data quality metrics and initiating corrective actionCorrect Pre-validation Errors
Maintain an enterprise wide consolidated “data mart” for reporting , query and analysis
Establish a centralized Data Governance Organization to:
appMDM’s Data Quality Management comprises Data Consolidation, Data Standardization, and Data Harmonization activities. Consolidating the data into the Hubs is a critical activity. The system collects data from multiple source systems and sends it to target MDM Hubs to perform Consolidation. Hadoop technology is extensively used. appMDM provides the “Matches”, and the Data Stewards or the Data Owners can classifythe matches as “False Positive” or “False Negative” and perform the merge, drop or migrate actions.
DATA QUALITY MANAGEMENT
Source System
appMDM
appMDM Governance
System ofRecord
appMDMData Steward
ConsumerSystem
Workflow
Extract newMDMTransactions
Corrected andApprovedTransactions
Extract andWorkflowNotify
Updated withLocal Attributes
“Work Flow functionality supports the data governance and approval processes and procedures”
Chain-Sys provides multiple roles to the MDM users such as: Operations, Data owner, Data Steward, Data Guardian, Data Architect etc. All these roles are used in configuring the workflow for any new Master Data introduction into the hub and also for processing the inbound master data that is interfaced from other Consumer Systems.
DATA GOVERNANCE
Data consolidation engine is key for identifying duplicate data and consolidating them. Data harmonization is achieved using data cross referencing of merged data against parent data. The data consolidation engine uses multiple string based mathematical algorithms to identify the groups for de-duplicates. It also identifies matching percentage for the match records. The matching percentage should be equal or greater than the threshold percentage assigned to the consolidation project task. If the data comes up with less than the threshold percentage, the system marks them for No-Matches.Data owners get an opportunity to review the apMDM dashboard for the results of the consolidation project execution and can drill into the Matches and No-Matches data. They classify data for merge, migrate, drop and on hold. Figures below illustrate data consolidation. Main screen shows process summary results, and the Consolidation Results screen shows the results for sample Customer, “Alcoa”. The results are used for merging, dropping and migrating. Data hierarchy for the Customer and Supplier data are also established.
Data Consolidation - Process Summary and Match/No Match Dashboard
Consolidation Main Page:
Consolidation Result for Companies related to “ALCOA”:
Shows the results of the data consolidation for “ALCOA”: