accelerated etl generation for microsoft ssis
DESCRIPTION
Acceler8 - Move from bespoke, hand-coded ETL packages to model-driven ETL generationTRANSCRIPT
-
Batch
Module
Batch
Instance
Module
Instance
Module Type
Event Log
Event Type
StatusArea
Layer
Area Type
Frequency
ParameterModule /
Parameter
Environment
Module
Instance
Audit
Acceler8
Accelerator for Microsoft SSISData Integration Development
-
Challenges
Organisations face common challenges managing and leveraging corporate data assets
Lengthy time to deployment
High data complexity
Extensive data dependencies
Inability to respond to changing business requirements or a changing environment due to inflexibility
Lack of qualified development resources
Poor maintainability
Acceler8 was developed to address these
challenges!
1Visit Analytics 8 1800 463 656 [email protected]
-
Typical Data Warehouse project
ETL Development takes 60 80 % of the time
of a typical data warehouse project
2
Analysis & design
Testing
ETL DevelopmentData Modelling
Visit Analytics 8 1800 463 656 [email protected]
-
Acceler8
Move from
bespoke, hand-coded ETL packages
to
model-driven ETL generation
3Visit Analytics 8 1800 463 656 [email protected]
-
Some success metrics
- 12 new source tables from source to staging to integration layer in
2 hours
- 160 source tables added from source to staging and integration
layer in less than 3 days
- New customer feed added to existing marketing analytics
environment in 3 days
- Reduced regression testing after change by 74%
4Visit Analytics 8 1800 463 656 [email protected]
-
Acceler8 What is it and what does it do?
Acceler8 auto-generates SSIS packages, reducing development time and improving development
consistency
Structured approach to data integration, consistently maintaining high quality standards
Documentation, templates, built-in error handling, auditability and process control
Designed to facilitate a more flexible and affordable ETL development cycle
5Visit Analytics 8 1800 463 656 [email protected]
-
Acceler8 foundations
Reduce complexities
Decouple warehouse logic and business logic
Minimise dependencies between ETL modules
Re-run any module without compromising data quality
Automate recovery from failure
Design for change
Streamlined addition of new data sources
New data and reporting requirements met without having to go back to square one
Generate ETL processes
Traceability and auditability
Parallelise loading
Top down, pattern-based
Support Agile development
6Visit Analytics 8 1800 463 656 [email protected]
-
Where we are today from an SSIS perspective?
Manual development
Manual development supported by a conceptual framework
ETL Generation based on conceptual framework
7Visit Analytics 8 1800 463 656 [email protected]
-
Benefits
Development time for integration is greatly reduced
Increased quality and consistency
Reducted development errors
Delivers a documented, sound foundation for the Data Warehouse
Improved maintainability and support
Increased Agility of ETL development
Reduction of cost and time-to-value
8Visit Analytics 8 1800 463 656 [email protected]
-
Customers that have already benefited (SSIS)
9Visit Analytics 8 1800 463 656 [email protected]
-
Out of the box No need to worry about
10
Indexes
Keys and constraints
Hierarchies
Time variance
Key distribution
Caching
Compression
Initial Loads
CDC
Process control
(Data) Recovery
DDL/DML definitions and conventions
Late arriving information
File paths
Package Configurations
SCD1/2/3/6 implementation
Fill Factors
Maintenance Plans
Data Type conversions
Scripting
Sort order
Lookups
Data Access modes
Validation
Compliance and Lineage
Date calculations
Visit Analytics 8 1800 463 656 [email protected]
-
Structure
11
Presentation Layer
Integration Layer
Staging Layer
Op
era
tio
na
l M
eta
Da
ta
Ex
cep
tio
n H
an
dli
ng
Design Patterns
Implementation
Patterns
ETL Generation
Technology independent;
What do we need to do?
Technology Specific;
How do we implement
a design pattern in
SSIS?
Outline Architecture
Documentation
Structure
Visit Analytics 8 1800 463 656 [email protected]
-
About us
Founded in 2002 in Australia
Offices in Sydney, Melbourne, Brisbane, Chicago, Raleigh and Dallas
85+ Consultants
Cross industry
Technology agnostic, 100% services organisation
Consulting, Training, Support, Software Procurement
Data and Analytics:
Better understand your market, customers and business
12
Leverage your data to hit your targets
www.analytics8.com
Visit Analytics 8 1800 463 656 [email protected]
-
What We Do
13
Strategic Services
Data & Analytics Strategy & Roadmaps
DW, BI & ETL Architecture
Data / Business Modeling
Project Management & Governance
Competency Centers
DW, BI & ETL Assessments
Tool / Vendor Selection & procurement
assistance
Training Support / Help Desk
Implementation Services
Business Intelligence and Analytics
Data Warehouse and Data Integration
Visit Analytics 8 1800 463 656 [email protected]