Agile Database Techniques

Tim DawsonInternational Decision Systems

© 2006 International Decision Systems, Inc.

Speaker Background Chief Architect, International Decision Systems

object oriented development with relational databases for 15 years custom software development & consulting – 9 years independent software vendors – 6 years

IDS is an enterprise software vendor, not a tools vendor I’m not here to sell you anything! These items are tools and tricks I have developed over the years

Agenda Why do we need Agile Database Techniques? Common Best Practices Example Problem Approaches & Tools That Can Help

Why Do We Need Agile? “Shouldn’t we invest the analysis and design time to create the

proper data model upfront and get it right the first time?” Do your best, but don’t expect to get it perfect from the start – even

if you do, business needs change. It’s inevitable, so embrace it.

“But change is hard – I can live with that old table/column name.” You can for now, but just like bad code, eventually this stuff

catches up with you in increased maintenance costs. It used to be hard with Java too… Its not as hard as it looks - even on multi-application databases.

Typical Problems Developers need to make data model changes without

impacting other team members. Race Condition at Check-in Time

Need to be able to make changes to databases involving multiple applications. Scott Ambler’s book has some great ideas about this

DBAs have to maintain both “Upgrade” and “Create” scripts Does ERWin help? Which versions are you upgrading from and to?

Test Data is a major challenge Differing test data between Dev & QA, sometimes even between

QA, and definitely between versions of the application.

Common “Best Practices” Development Sandboxes

separate DB Schemas for each Developer (maybe even QA?) allows independent execution of data-altering tests allows one developer to work against a modified schema while

developing code that works with the new structure – preventing impact to the entire team

(not sufficient in itself - increases DBA workload unless automated; also have to worry about race conditions)

Common “test data” Allows Developers & QA to easily reproduce defects (can be difficult to keep track of across multiple versions)

Is that the best we can do? Any others?

Suggested Practices Put control where it needs to be

DBAs define structure Developers control timing

Create a flexible system of upgrade scripts sensitive to multiple release maintenance and branch development

Revision control your test databases ideally import / export to CSV revision alongside source code along with install scripts

Example Problem Movie Database

Start Simple – Implement Basic Functionality

Minor Requirement Changes

Major Requirement Change

Using ADDAM for Test Data, Lifecycle, & Multiple Releases

Movie Database (Initial Design)

Simple movie database, needs some work though, as requirements change.

Let’s start by creating the schema...

create table Movie ( movieID integer primary key, movieName varchar2(255), yearMade number(4,0));

create table Actor ( actorID integer primary key, firstName varchar2(255), lastName varchar2(255));

create table MovieActor( movieID integer references Movie, actorID integer references Actor);

Movie Database – Test Data Aliens,

starring Sigourney Weaver, Bill Paxton, Paul Reiser

Director James Cameron

Click starring Adam Sandler,

Christopher Walken, David Hasselhoff, Henry Winkler

Directed by Frank Coraci

Small Change (during Release 1.0)

NOT NULLs missinginsert into Actor (4,null,null);

Need to apply a change... How do we roll out to development, QA, production? What about branching?

alter table Actor modify (firstname not null);alter table Actor modify (lastname not null);

alter table Movie modify (moviename not null);

Bug Fix (scheduled for Release 1.1)

Some actors don’t have a last name, e.g. Madonna. Simple Change…

alter table Actor modify (lastname null);

Medium Change (Release 1.1)

The Director wants credit too. Easy enough to add…

create table Director ( directorID integer primary key, firstName varchar2(255) not null, lastName varchar2(255) not null);

create table MovieDirector( movieID integer references Movie, directorID integer references Director);

Larger Change What about other roles, such as Key Grip and Best Boy?

Larger Change

Merge MovieActor and MovieDirector into Credit Merge Actor and Director into Person Create Role to identify usage

Role is a “coded” table

Larger Change

Create Actor and Director views until reports are upgraded in next release.

Release Procedures Upgrade Scripts ready Need to generate updated Install Scripts (and with it Test Data)

This is particularly important for Product Development Also as a periodic practice per iteration in your project – keeps

“upgrade” times to a minimum when refreshing

How Does ADDAM Work? Install and Upgrade are very similar

install.drv, upgrade.drv specify directories to traverse master.drv inside directories specify scripts to execute VERSION and SCRIPT_LOG table

DDL Generator Generic or Vendor-Specific DDL Generation Generic uses JDBC DatabaseMetaData Oracle implementation uses dbms_metadata.get_ddl()

Test Data Import / Export Simple CSV codec; first line is column names “delete *” or “truncate” followed by batch insert statements

Current Best Practices Fall Short On medium-to-large teams, managing individual developer and

QA schemas is a huge time sink for development DBAs. Development DBAs are distracted from other key activities

Reviewing data model proposals Enforcing standards Performance tuning

More importantly, developers have to wait for their changes Saving binary snapshots of test data is also time consuming and

space-wasting compared to revision control of individual CSV files.

Lifecycle Automation Solves Problems By automating upgrades, control is in the right hands

DBAs control upgrade scripts Developers control timing of when scripts are applied No “race condition” to apply to all developer schemas after a

checkin that requires a DB change Simply uncomment needed scripts, “ant upgrade”, and check-in

when ready Huge benefits to QA – easy to build QA environments with test

data from any build label Package via ServletContextListener for auto-upgrade

Can also be used in production rollout Easy to package as a command line, or auto-upgrade

Other Useful References Scott Ambler’s guide to database refactoring

http://www.ambysoft.com/books/refactoringDatabases.html