applying software engineering practices to etl

8/12/2019 Applying Software Engineering Practices to ETL

1/34

Applying Software

Engineering Practices to ETL

December 2010


2/34

3

Disaster or Development

Disaster Development

Process adds the structure

MaintenancePreparation Development

Testing

Design

Migration

Maintenance


3/34

4

Agenda

We will discuss process best practices in the

following areas

Preparation

Design

Build

Testing

Migration

Maintenance Wrap Up


4/34

5

Preparation Best Practices 1/7

Requirements

Environment

Standards

Conceptual Design

Initiation Meeting

Start

Check List


5/34

6


Required roles and responsibilities

Project manager/team lead

Design lead/ETL architect

Designers

Developers

Testers

Migration administrator

Solution architect DBAs

Business analyst


6/34

7


Prerequisites (do we know what we are doing?)

Requirements

Documented, reviewed and understood

Source(s) identified Data format, availability, location

Target(s) identified

Data model

Physical development environment created


7/348


Prerequisites (do we know how we are doing it?)

Standards

Solution specific

Naming conventions Error handling

Document templates

Tool/Technique specific

Use SQL for x

Dont use feature y

MyData repository

Media and format


8/349


Environment (do w e know where we are doing i t?)

Development

10% of production size

Developer sandbox accounts

Tool, database, etc.

Test

30% of production size

QA

As much as possible

Production


9/3410


Conceptual / logical design

Data flows

Duplicate data sources?

Modularization locations Data hops

Timing dependencies

Accounts updated weekly, customers monthly?

Plan it!

How do you distribute the work Similar complexity

Associated tasks (unit of work)

Subject area


10/3411


Prerequisites are OK, is the team?

Team review of requirements

Allows everyone to understand big picture

Team review of standards How well do this

Review overall time line / milestones

When well do this

Assign roles / responsibilities

Who will do what


11/3412

Design Best Practices 1/5


12/3413


Use standard design templates

Functional specifications

Technical designs

Traceability matrix

Keep GUI tool designs tool-independent

Address the CTQs of ETL in design documents

Performance

Error handling

Meta data

Reuse

Latency


13/3414


Achieve goal of design reviewsvalidate it

Expand design reviews outside of IT

Distribute designs prior to the review

Project it line by line

Get signoff

Formalize pre-production change requests

Design

Approve and sign

Maintain document of change requests


14/3415


Design for the tools strengths and weaknesses

Know your tool

Know your requirements

Design for both

Address the CTQs of ETL

Performance

Error handling

Meta data

Reuse

Latency ETL Tool

Strengths WeaknessesDesign


15/3416


Utilize prototypes to drive out detailed

requirements

Produce sample target data early

Start your unit test document now Build functional test cases during design

Developer will enhance with technical cases


16/34

17

Build Best Practices 1/4


17/34

18

Build Best Practices 2/4 Conduct pre-code reviews

Tool specific approach

One common reviewer to produce standard results


18/34

19


Leverage an object-oriented approach

Source and target definitions

Common calculations

Multiple instances of the same source system Initial load vs. ongoing load

Source Transformations Targets


19/34

20


Conduct code reviews

Match design

Conformance to standards

Utilization of common objects Meta data


20/34

21

Testing Best Practices 1/6


21/34

22


Test completelyaddress all types

of testing

Unitin sandbox development environment

Unitin development environment Stringin test environment

Systemin test environment

User acceptancein QA environment

Regressionall environments (possible exception of QA)


22/34

23


Leverage document templates (or test scripts)

with common test scenarios defined

Identify common tests to be performed during design phase

Create templates with the common tests Customize templates to apply to each program

Create and use a testing checklist

Are we testing in the right place

Are we testing with the right data set(s)

Are we in sync with any pre/post dependencies


23/34

24


Reuse test data

Create library

Create at least 2 records to test each scenario

Use unique set for each program, do not share Use production data, if possible

Each test should be able to be recreated at any time

Update and maintain library

Define processes around introducing new test case data into library

Build procedure to reset the test database

Document library

Understand what each record is testing


24/34

25


Roles and responsibilities

Designer

Creates test scenarios

Programmer Executes cursory test of program

Creates needed test data set

Tester

Executes test scripts

Validates results Lead

Reviews test results


25/34

26


Test execution

Execute tests completely, do not stop after first error

Programmer makes necessary program fixes

Re-test every scenario again, regardless of original pass/failstatus

Each scenario must pass in a single execution of test


26/34

27

Migration Best Practices 1/4


27/34

28


Plan to migrate Identify roles within the migrations

Develop an architecture that supports migration

Develop processes and procedures to support migration Create migration checklist

Code has been reviewed and approved

Migration request form has been filled out and approved

System test conditions are ready

Move units (defined by product)

Identify tools that will be used

Test migration strategy before development ends

Roleslimit privileges for migrating code


28/34

29


ETL migration flow

Development unit test

Coordinate move to Test

Testsystem test Coordinate move to Production

Production validation

PD T


29/34

30


ETL migration process

Code movement approval

User acceptance of unit / test

Migration checklist completed

Versioning

Regression testing

ETL backup and recovery

Utilize migration completion checklist

PD T


30/34

31

Maintenance Best Practices 1/3


31/34

32


Repeat the process!!!

Code change control

Requirements

Design Construct

Test

Regression testing

Migrate

Documentation

Upgrade Subject Area

Contingency Code


32/34

33


Emergency maintenance

Requirements

Quick gather: many times already known (bug)

Design Construct

Test

Usually cant be scheduled into system test

Can we test in QA?

Test in dev with system test data Migrate


33/34

34

Best Practices Wrap Up

OK, what did we just see?

A collection of simple activities grouped together into a process

Basic and sometimes obvious yet many times ignored

Prepare and plan

What, why, who, and when

How

Control it

Roles/responsibilities

Checklists


34/34

THANK YOU

applying software engineering practices to etl

Documents