KYSUC

Keep Your Schema Under Control

04.FEV.2016

Nuno Alexandre Alves

Speaker

• Nuno Alves

• Born at Coimbra / Lives in Leiria

• Degree in Informatics Engineering (ESTG-IPLeiria)

• Some experience with databases

• Nowadays working in Critical Software, SA

nuno.alexandre.alves@gmail.com

nuno.alexandre.alves

Disclaimer

This presentation is:• From my personal experience• My own suggestions

This presentation is NOT to:• Pointing fingers• Enumerate errors• This is better than that• Do not use this

Just another day in “paradise”

• How do you develop your scripts ?

• Do you have a ER model of your Database(s) ?

• Do you have your Database documented ?

• How do you control your scripts versions ?

• Do you have your test data under scripts ?

• How do you test your scripts ?

• How do you deploy your application and database to a specific version ?

Agenda

1. Goals

2. Diagrams (ER models)

3. Scripts

4. Deploy

5. Documentation

Goals for Today

• Updated ER model

• Scripts versioned

• Easy Deploy

• Easy Documentation

• Schema Under Control !

Agenda

1. Goals

2. Diagrams (ER models)

3. Scripts

4. Deploy

5. Documentation

ER Model (1/4)

Start New Project (from requirements)• Build your Conceptual Model• First Physical Model• First Scripts (version 1.0.0)• Print (paper not PDF) your Physical Model and put it

in a Wall next to the team (old-school but it works)

Assumptions:• High Level Architecture designed• RBDMS almost decided (for the first scripts)

ER Model (2/4)

Start New Project (already in PROD) / Maintenance• Ask for actual documentation (get lucky )• DDL of actual production database• Import DDL in your development servers• Reverse Engineering• Build Conceptual Model (from Physical) if

necessary• Script Physical Model with actual application

version (ex: v3.5.2 – it’s your baseline)

ER Model (3/4)

Development / Maintenance

• Keep your database changes updated in your ER Model

• Use Reverse Engineering to do checkpoints (compare)

ER Model (4/4)

Tools

• Use a Case Tool

• PowerDesigner

• Enterprise Architect

• …

Agenda

1. Goals

2. Diagrams (ER models)

3. Scripts

4. Deploy

5. Documentation

Scripts (1/10)

Scripts Development (projects reality)

• Manually

• I don’t have scripts. My ORM tool does it for me !?!

• Database is something to store some data on it. I don’t care.

Scripts (2/10)

Scripts Development (projects reality)

• WTF ?!

• Your medical records relies on a database system

• Your bank account relies on a database system

• Your salary relies on a database system

• …

• Your life relies on too many database systems

• Take your own conclusions

Scripts (3/10)

Then someday• How do I execute the database scripts ? Order By ?• Which development database is updated ?• How do you develop scripts ?

• One script vs multiple scripts ?• Full Scripts vs Partial Scripts ?• It’s your ORM your Database developer ?• Naming Conventions for objects (where is the guide?)

• I have a app log with this error “constraint SYS_098374 violated”. What is that constraint ?

• The column CGTRTHY stores what ?

Scripts (4/10)

Database Scripts Under Control• Why ?

• Is your source code under control ?• Does your source code have versions ?

• Database scripts = source code ?• I think Yes !

• Table definition• Table initial data• Function / Stored Procedure / Package

Scripts (5/10)

Decide your Database Storage Strategy• Check production for rolling projects• Decide according to hosting supplier or client needs

• Tablespaces• Data files• File groups

My Suggestion (minimum)• One Tablespace for Data• One Tablespace for Indexes• One Tablespace for LOBs

Scripts (6/10)

Decide your Database / Schema(s) architecture• One schema, multiple users with diff grants• Multiple schemas, one/multiple users

My Suggestion• Keep it simple• One schema/database, multiple users (grants/views)

• FKs between multiple schemas not supported in all RDBMS• Distributed Transactions (do you really want ?!)• Multiple queries (joins in app server)

• Are you creating your own RDBMS ?!

Scripts (7/10)

Decide your Database Naming Conventions• Easy identify a database object• Name every object in database

• Don’t let default names under RDBMS

Scripts (8/10)

Script like source code• Use SQL formatting (for human readable)• Comment your SQL• Define objects language: PT; EN; FR … (don’t mix)

Script for documentation• Comment every object (mandatory: wait and see)• Keep documentation in database dictionary• Useful for Reverse Engineering• Useful for Automatic Documentation

• COMMENT ON TABLE• COMMENT ON COLUMN TABLE_NAME.COLUMN_NAME

Scripts (9/10)

Tools• Manually (Notepad, VI, … is your friend!) • Liquibase• mybatis• MIGRATEdb• migrate4j• Flyway • …

Scripts (10/10)

Scripts Under Control ?

Flyway (1/6)

Overview• Scripts: SQL, Java• RDBMS Agnostic (JDBC driver)• Inside your source code• Placeholders (variables)• Multiple Execution Modes

• Java API• Maven / Ant• Command line

• Ex: V2_0_0_1_0001__script_name.sql

Flyway (2/6)

Decide your Script Naming Convention• V<JIRA_FIX_VERSION>_1_<4_digits>__<JIRA_KEY>.sql

• <4_digitos>• Execution order

Flyway (3/6)

Put some code in your Script• Add an header to your SQL file

• Add JIRA_KEY or JIRA URL as a comment

• Commit

Flyway (4/6)

Decide your Execution Mode• Command line (java -jar …)• Maven/ANT

• configure inside your IDE or in CLI

• Use Maven profiles is necessary (useful in multiple databases/environments)

Flyway (5/6)

Flyway in action• flyway:info

• Check schema version already installed

• You need flyway:baseline for the first time

Flyway (6/6)

Flyway in action• flyway:migration• Script execution

• With/without target release

Flyway

In Action

Agenda

1. Goals

2. Diagrams (ER models)

3. Scripts

4. Deploy

5. Documentation

Deploy (1/7)

Deploy Database Scripts• Do you test your scripts ? (how many times)• Do you re-create your schema from scratch ? • Do you have automatic deployment for your app ?

• And your database ?• Do you have night builds ?

• They include database too ? and automated tests ? • Do you simulate your client releases in development

environments ?• Do you have confidence in your own scripts ?

Deploy (2/7)

Deploy Database (full / upgrade)• Use your Continuous Integration

• Jenkins

• Team City

• Bamboo

• Other …

Deploy (3/7)

Jenkins + Flyway = easy database deploy• Why Jenkins ?

• Easy to configure• Parametrized builds• Many plugins (CVS, SVN, Maven, JIRA, Email, …)• Scheduler• Live Build Logs / Console output directly in browser• Why not ?!

Deploy (4/7)

Jenkins + Flyway = easy database deploy• Configure a Jenkins Job

• Checkout script from VCS• Put some parameters

• if necessary for customs build• for multiple versions• for multiple databases

• Execute Flyway• via Maven • via custom Shell Script / Windows batch

Jenkins + Flyway

In Action

Deploy (5/7)

Continuous Monitor your database• Oracle Enterprise Manager (Oracle)• pgwatch (PostgreSQL)• db2top (DB2)• SQL Server Management Studio (SQLServer)• MySQL Enterprise Monitor (MySQL)• …• Check administrator views (available in every RDBMS)

Deploy (6/7)

Continuous Monitor your database• Check some random SQL Statements

• CPU• I/O

• Get query plan for these statements• Help development team write better SQL• Create indexes according to business needs

• Monitor sessions• Check Connection Pooling parameters

• Min / Max Pool sizes (warn: multiple Connections Pools)

Deploy (7/7)

Help Development Team to help yourself• Write SQL Statements to log (debug mode)

• Ask for log4jdbc / P6Spy / … (with parameters)• Write query times to log• Analyze top time consuming queries• Analyze top running queries

• Ask for application times vs database times (in log)• Most of times the application consumes more time (yes it’s true)• Try to analyze how many queries are made for a business operation• Check the number of columns retrieved in every query (it’s really necessary 80

columns ?)

Agenda

1. Goals

2. Diagrams (ER models)

3. Scripts

4. Deploy

5. Documentation

Documentation (1/3)

How do you produce your database documentation ?• Manually• I don’t have database documentation• “My client doesn’t need this documentation”

• And your team ?• How do you maintain a project without documentation?• Most (~All) of your production tickets are resolved by SQL

statements !• Client business data is stored in “your” database (really?!)

Documentation (2/3)

Automatic Documentation• Use RDBMS data dictionary

• information_schema.tables, …• user_tables, …• syscat.tables, …• Remember COMMENT ON … from scripts ?

• “Export” to something useful• Word Document/PDF ? • Who reads a 800 pages document ?

Documentation (3/3)

Automatic Documentation• Perl script (reads data dictionary, outputs One HTML file)• Send to your client as a deliverable• Publish in your wiki• Open locally in your browser

• Produce custom CSS: 30m - 1h• Update Documentation: 1 minute

• Remember COMMENT ON

Summary (from initial goals) (1/2)

ER Model• Use a Case Tool• Keep your model updated

Scripts versioned• Use Flyway / Liquibase / Other• Script everything (data tests included)• Review your scripts

• Naming Conventions• Comments

Summary (from initial goals) (2/2)

Easy Deploy• Use Jenkins / Team City / Bamboo / Other• Keep your database under Continuous Integration• Schedule your database builds• Parametrize your database deploy• Monitor your database close• Release Matrix isn’t nice to have (is mandatory)

Easy Documentation• Comment your objects in data dictionary

Main Goal

Updated ER Model + Scripts versioned + Easy Deploy + Easy Documentation

= KYSUC

Extra Credits

Define / ask for backup/restore plan• Test your backups (restore is your friend)• Check backup/restore times• Check database growth

Compare data• Compare your production data with development data:

reference data• Try to equal PKs IDs across environments (for referencial data)

That’s It!

Thank you for your time.