transmart community meeting 5-7 nov 13 - session 3: transmart-data

transmart-dataManagement of tranSMART’s Environment

Gustavo Lopes

The Hyve B.V.

November 6, 2013

Gustavo Lopes (The Hyve B.V.) transmart-data November 6, 2013 1 / 22

Outline

1 ProblemsReproductibilityVersioning ControlAutomationWhy?!tranSMART Foundation’sVersion

2 Solution: transmart-dataGeneral DescriptionConfigurationDatabase Schema ManagementSeed DataETLRModules Analyses’RserveSolrtransmartApp Configuration

3 Limitations


Typical Branch Distribution

Grails Code

transmartApp (without fullrepo history, always withwrong ancestry information⇒ merging quite difficult)

RModules (if you’re lucky),but analyses definitions inDB not provided

Database

SQL scripts on top of GPL1.0 dump or later. Probablyinsufficent/won’t apply

Stored procedures for ETL.Overlapping definitions withyours, but no history ⇒merging quite difficult

Manual fixups alwaysrequired (even if justpermissions/synonyms)


Typical Branch Distribution (II)

ETL

High variablity in strategies

Instructions/sample datararely provided

Kettle scripts areproblematic

Solr/Rserve/Configuration

Solrschemas/dataimport.xmlperpetually forgotten

Idem for information on Rpackages

Sample configuration rarelyprovided


Versioning Control

Version control used ONLY for Grails Code. . .But often squashed and with wrong ancestor information.Forget about database, Solr, most of ETL.

Result

Merges are very difficult.

Changes cannot easily be tracked

Changes’ wherefores are unknown

Regressions are introduced (no conflicts)

Collaboration is based on e-mail attachments


Automation

Even with all the pieces. . .

Setting up a new branch takes days;weeks for non-basic functionality

No reproductibility in the process!

Result

Devs driven away from fully localenvironment (too much work)

Robust environment for CI passed over(too much work)

Bugs cannot be reliably reproduced (seealso: no consistent usage of VCS)

Time wasted with deployment specificmistakes/inconsistencies


Why?!

Guillaume Duchenne (public domain)

The “source code” for a work meansthe preferred form of the work formaking modifications to it.

— GPL v3, section 1

Is everyone holding back “source code”?More likely explanation:

No appropriate tooling being used


Situation for tranSMART 1.1

The situation is much better!Some problems remain, though.

The Good ,Create/populate DBis easy

Most stuff isversioned

CI for builds

Image available

Public issue tracking

The Bad /No Oracle supportChanges to DB scripts/seed data aread hoc (lax structure)No mechanism to support/compareschemas with other branchesR analyses are json blobs in TSVsNo VCS for Solr or Rserve/images’ setupSet up Sol/Rserve is time-consumingPopulation of DB with sample data is stilltime-consumingConfig changes required for dev


Description of transmart-data

We developed transmart-data to address most of these problems:

transmart-data is a set of

scripts for managing tranSMART’s environment and

certain application data (e.g. Solr schemas, DDL, seed data), whichis used by scripts and sometimes generated by them.

It has a makefile based interface.


transmart-data: Purposes

Purposes of transmart-data:

1 Allow setting up a complete dev environment quickly (< 30 min)

2 Bring versioning to the database schema and Solr files

3 Setup Solr runtime

4 Invoke ETL pipelines

5 Setup Rserve

Target audience: Programmers


transmart-data: Non-purposes

Non-purposes of transmart-data:

1 Setup a production environment(some components can be used)

2 New users evaluating tranSMART(use an pre-built image)

3 Building transmartApp or its plugin dependencies(build them yourself or use artifacts from Bamboo/Nexus)


Configuration

Environment variable based configuration

cp v a r s . sample v a r svim v a r s #e d i t f i l esource v a r s

PGHOST=/tmpPGPORT=5432PGDATABASE=t r a n s m a r tPGUSER=\$USERPGPASSWORD=TABLESPACES=\$HOME/pg/ t a b l e s p a c e s /PGSQL BIN=\$HOME/pg/ b i n /ORAHOST=l o c a l h o s tORAPORT=1521ORASID=o r c lORAUSER=” s y s as s y s d b a ”ORAPASSWORD=mypasswordORACLE MANAGE TABLESPACES=0#c o n t i n u e s . . .


Database Schema Management

Support for Oracle and Postgres

Postgres

Uses pg dump(all)

Parses the dump files

#Dumpmake −C p o s t g r e s / d d l dumpmake −C p o s t g r e s / d d l /

GLOBAL e x t e n s i o n s . s q lr o l e s . s q l

#Loadmake −C p o s t g r e s / d d l l o a d

Oracle

Queries dba * tables

Dumps DDL w/DBMS METADATA

#Dumpmake −C o r a c l e / d d l dump

#Loadmake o r a c l e


Seed Data

Only Postgres for now

#Dump#T a b l e s to dump i n p o s t g r e s / data/<schema> l s tmake −C p o s t g r e s / data dumpmake −C p o s t g r e s /common m i n i m i z e d i f f s

#Loadmake −C p o s t g r e s / data l o a d

#Load DDL and datamake p o s t g r e s

Only for basic stuff with no ETL!

Pretty fast (DDL+data loaded in 10s)


ETL (I)

Unified interface for ETL

Prepare dataset

1 Prepare ETL-specific sourcefiles

2 Prepare file with ETLspecific params

3 Upload dataset to CDN(optional)

For each new ETL pipeline,support must be added

Load dataset

make −C s a m p le s /{ o r a c l e ,p o s t g r e s } l o a d <type><s t u d y id>

#Example :make −C s a m p le s / p o s t g r e s

l o a d c l i n i c a l G S E 8 5 8 1

Everything is automated!


RModules Analyses’(tsApp-DB)

Situation in transmartApp-DB:

update searchapp.plugin_module

s e t params='{"id":" survivalAnalysis ","converter ":{"R":[" source(''|| PLUGINSCRIPTDIRECTORY|| Common/dataBuilders.R'')","source(''|| PLUGINSCRIPTDIRECTORY || Common/ExtractConcepts.R'')","source(''|| PLUGINSCRIPTDIRECTORY || Common/collapsingData.R'')","source(''|| PLUGINSCRIPTDIRECTORY || Common/BinData.R'')","source(''||PLUGINSCRIPTDIRECTORY || Survival/BuildSurvivalData.R'')","\ tSurvivalData.build(\n\tinput.dataFile = ''|| TEMPFOLDERDIRECTORY || Clinical/clinical.i2b2trans '',\n\tconcept.time=''||TIME||'',\n\tconcept.category=''|| CATEGORY ||'',\n\tconcept.eventYes=''|| EVENTYES ||'',\n\tbinning.enabled=''|| BINNING ||'',\n\tbinning.bins=''||NUMBERBINS ||'',\n\tbinning.type=''|| BINNINGTYPE ||'',\n\tbinning.manual=''||BINNINGMANUAL ||'',\n\tbinning.binrangestring=''|| BINNINGRANGESTRING ||'',\n\tbinning.variabletype=''|| BINNINGVARIABLETYPE ||'',\n\tinput.gexFile = ''||TEMPFOLDERDIRECTORY ||mRNA/Processed_Data/mRNA.trans '',\n\tinput.snpFile = ''||TEMPFOLDERDIRECTORY ||SNP/snp.trans'',\n\tconcept.category.type = ''|| TYPEDEP ||'',\n\tgenes.category = ''|| GENESDEP ||'',\n\tgenes.category.aggregate = ''|| AGGREGATEDEP||'',\n\tsample.category = ''|| SAMPLEDEP ||'',\n\ttime.category = ''|| TIMEPOINTSDEP||'',\n\tsnptype.category = ''|| SNPTYPEDEP ||'')\n\t"]}," name ":" Survival Analysis","dataFileInputMapping ":{" CLINICAL.TXT":" TRUE","SNP.TXT ":" snpData"," MRNA_DETAILED.TXT

":" mrnaData "}," dataTypes ":{" subset1 ":[" CLINICAL.TXT"]}," pivotData ":false ,"view ":"

SurvivalAnalysis "," processor ":{"R":[" source(''|| PLUGINSCRIPTDIRECTORY || Survival/CoxRegressionLoader.r'')"," CoxRegression.loader(input.filename=''outputfile '')","source(''|| PLUGINSCRIPTDIRECTORY || Survival/SurvivalCurveLoader.r'')"," SurvivalCurve.loader(input.filename=''outputfile '',concept.time=''||TIME||'')"]}," renderer ":{"GSP ":"/ survivalAnalysis/survivalAnalysisOutput "} ,... (goes on)'

where module_name = 'pgsurvivalAnalysis ';

Not very nice...


RModules Analyses’ (transmart-data)

In transmart-data:

One file per analysis

Files can be generated from DB data

Sanely formatted

But we really want to remove this from the DB!array (

'id' => 'heatmap',

'name' => 'Heatmap',

'dataTypes' =>

array (

'subset1' =>

array (

0 => 'CLINICAL.TXT',

),

),

'dataFileInputMapping' =>

array (

'CLINICAL.TXT' => 'FALSE',

'SNP.TXT' => 'snpData',

'MRNA_DETAILED.TXT' => 'TRUE',

),

'pivotData' => false,

...


Rserve

Targets for Rserve:

Download/build R

Install R packages

Start Rserve

Install System V initscript for Rserve

Idem for systemd

cd R

make -j8 bin/root/R

#some packages don 't support

concurrent builds

make install_packages

make start_Rserve

make start_Rserve.dbg

TRANSMART_USER=tomcat7 sudo -

E make install_rserve_init

TRANSMART_USER=tomcat7 sudo -

E make install_rserve_unit


Solr

Solr (4.5.0) automaticallydownloaded and configured

Solr cores automatically created

User only needs to create a schemafile and dataconfig.xml

#setup & solr (psql)

make start

#just configure

make solr_home

make <core >_full_import

make <core >_delta_import

make clean_cores

ORACLE =1 make start


transmartApp Configuration

Out-of-tree config management:

Targets for installing files

Zero configuration fordev!

Customization allowedwithout touching the targetfiles

Only supports ours branches

But a lot of configurationshould be in-tree instead!

#install everything

#previous files are backed

up

make install

#just one file:

make install_Config.groovy

make install_BuildConfig.

groovy

make install_DataSource.

groovy

#costumizations in:

#Config -extra.php

#BuildConfig.groovy (

limited)


Current Limitations

© Joost J. Bakker, CC BY 2.0

DB upgrades not handled

Only a few ETL pipelinessupported

Oracle support is behindPostgreSQL

Tooling shares repositorywith application data


transmart community meeting 5-7 nov 13 - session 3: transmart-data

Technology

s s y s d b

hyve b

e s t o d

r s vim v

c p o s t g r e s d

description of transmart

purposespurposes of

e source v