model driven software development - data model evolution

DataModel Evolution

Sander VermolenEelco Visser

This research is supported by NWO/JACQUARD project638.001.610, MoDSE: ModelDriven Software Evolution.

Data Models

User 1name bob real name Bob Johnsonemail [email protected]

Page 1title "The first page"isRedirect falsetext "Hello world"

Count page views

Version history

User 1name bob real name Bob Johnsonemail [email protected]

Page 1title "The first page"isRedirect falsetext "Hello world"

No page countNo revisions

Coupled Data Evolution

...

$dbh->bz_add_column('attachments', 'submitter_id', {TYPE => 'INT3', NOTNULL => 1}, 0);$dbh->bz_rename_column('bugs_activity', 'when', 'bug_when');_add_bug_vote_cache(); _update_product_name_definition(); _add_bug_keyword_cache();$dbh->bz_add_column('profiles', 'disabledtext', {TYPE => 'MEDIUMTEXT', NOTNULL => 1}, '');_populate_longdescs(); _update_bugs_activity_field_to_fieldid();

if (!$dbh->bz_column_info('bugs', 'lastdiffed')) {$dbh->bz_add_column('bugs', 'lastdiffed', {TYPE =>'DATETIME'});$dbh->do('UPDATE bugs SET lastdiffed = NOW()');

}

_add_unique_login_name_index_to_profiles(); $dbh->bz_add_column('profiles', 'mybugslink', {TYPE => 'BOOLEAN', NOTNULL => 1, DEFAULT => 'TRUE'});_update_component_user_fields_to_ids();$dbh->bz_add_column('bugs', 'everconfirmed', {TYPE => 'BOOLEAN', NOTNULL => 1}, 1);$dbh->bz_add_column('products', 'maxvotesperbug', {TYPE => 'INT2', NOTNULL => 1, DEFAULT => '10000'});$dbh->bz_add_column('products', 'votestoconfirm', {TYPE => 'INT2', NOTNULL => 1}, 0);_populate_milestones_table();$dbh->bz_alter_column('bugs', 'target_milestone', {TYPE => 'varchar(20)', NOTNULL => 1, DEFAULT => "'---'"});$dbh->bz_alter_column('milestones', 'value', {TYPE => 'varchar(20)', NOTNULL => 1});_add_products_defaultmilestone();

if (!$dbh->bz_index_info('cc', 'cc_bug_id_idx') || !$dbh->bz_index_info('cc', 'cc_bug_id_idx')->{TYPE}) {$dbh->bz_drop_index('cc', 'cc_bug_id_idx');$dbh->bz_add_index('cc', 'cc_bug_id_idx', {TYPE => 'UNIQUE', FIELDS => [qw(bug_id who)]});

}if (!$dbh->bz_index_info('keywords', 'keywords_bug_id_idx') || !$dbh->bz_index_info('keywords', 'keywords_bug_id_idx')->{TYPE}) {

$dbh->bz_drop_index('keywords', 'keywords_bug_id_idx');$dbh->bz_add_index('keywords', 'keywords_bug_id_idx', {TYPE => 'UNIQUE', FIELDS => [qw(bug_id

keywordid)]});}

...

Costly

High risk

Holds back the development process

Large infrequent development steps

Userid : integername : varcharrealName : varcharemail : tinytext

Pageid : integertitle : varcharauthor - User *isRedirect : booleancontent : text

set of

Userid : integer Uniquename : varcharrealName : varchar ?email : tinytext

Page : Mediumauthor - User min(1) max(8)content : textrefs : url * Indexed

abstract Mediumid : integer Uniquetitle : ANY_NAME

EvolvingData Models

SpecifyingData Model Evolution

Userid :: integername :: varcharrealName :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor UserisRedirect :: booleancontent :: text

Userid :: integername :: varcharrealName :: varcharemail :: tinytext

Pageid :: integertitle :: varcharcounter :: bigintegerisRedirect :: booleanrevisions set of Revision

Revisionid :: integerpage Pagecomment :: tinyblobtimestamp :: timerevisionText :: textauthor User

Added type revisions

Added attribute revisions

Moved content to revision text

Added attribute counter

Revisionid :: integerpage Pagecomment :: tinyblobtimestamp :: timeauthor User

Pagerevisions set of Revision

RevisionrevisionText :: text

Pagecounter :: biginteger

What happened?

add or remove entity

change name of entity

change type of set

8 Basic Transformations

add or remove property

change name of property

change type of property

move property

1 Advanced Transformation

addRevision

id : integerpage - Pagecomment : tinyblobtimestamp : timeauthor - User

addcounter : biginteger

At / Entity Page / Property Titleadd counter : biginteger

At / Entity[Id=''Page''] / Property[Id=''Title'']add counter : biginteger

at // Property [Id = ../Id]add counter :: biginteger

RevisionrevisionText :: text

At / Entity Revision / Property timeStampmove page.content to revisionText :: text

at Entity Page / Property Titleadd counter :: biginteger

at Entity Revision / Property timeStampadd revisionText :: textfrom page.content

;

EvolvingData Models

8 basic transformations

1 advanced transformation

Language to specify transformations

Positioning sub language

Specify data model evolutions

DataMigration

Technical Domain

WebDSL

Stratego/XT

Stratego/XTGeneric Aterm

SQL Databases (MySQL)

Because

we really like program transformations

generally richer than regular data acessing languages (SQL)

data migration = model transformation

Program transformation for data migration

Userid :: integername :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor User

User(id(1),name(“John”),email(“[email protected]”)

)

Page(id(2),title(“Hello World”),[author(1)]

)


)


)

<user><id>1</id><name>John</name><email>[email protected]</email>

</user>

<page><id>2</id><title>Hello World</title><authors>

<author>1</author></authors>

</page>



User(id(1),name(“John”),.....

)


)

Remove Attribute (1)

Signature:User := Id * Name * Email




)


)

Remove Attribute (2)



User( 0,[

id(1),name(“John”),email(“[email protected]”)

])

Page( 1,[

id(2),title(“Hello World”),author(0)

])

Generic Aterm (GTerm)

User( 0,[


])

Page( 1,[


])

XMI

<user id='0'><id>1</id><name>John</name><email>[email protected]</email>

</user>

<page id='1'><id>2</id><title>Hello World</title><author>0</author>

</page>

GTerm Characteristics

Explicit references

No nesting

Implicit sets/lists

Storage...

GTerm Transformation

Gterm libraryObject creationModifying attributes

(add, remove, change, rename, ...)Object equivalenceObject traversals(Object graph traversals)

Data model libraryType examinationSuper/Sub type handlingAbstract type handling

GTerm Storage

Large quantities of data...

Storage engine:In memory list based ~10KIn memory hash table based ~500KIn database ~25M ...

User( 0,[


])

Page( 1,[


])

GTerm Storage – In database (1)

0 User id 10 User name John0 User email [email protected] Page id 21 Page title Hello World1 Page author 0

0 User1 Page

GTerm Storage – In database

CREATE TABLE Attributes (id varchar(16),type varchar(30),name varchar(30),value text,INDEX USING HASH (id (5)),INDEX USING BTREE (v(10))

)

CREATE TABLE Objects (id varchar(16),type varchar(30)

)

GTerm Storage – In database (2)

GTerm Performance

Database indexes

Stratego memory usage

Parallel execution

GTerm Storage – Regular database

0 User id 10 User name John0 User email [email protected] Page id 21 Page title Hello World1 Page author 0

User:1 John [email protected]

Page:2 Hello World

PageUser:2 1

Old Database Generic Database

SQL Script

Data model

Migration (Stratego)

SQL Script

GTerm 2 SQL

New Database

ResearchrDBLP

<dblp><incollection mdate="20020103" key="books/acm/kim95/AnnevelinkACFHK95">

<author>Jurgen Annevelink</author><author>Rafiul Ahad</author><author>Amelia Carlson</author><author>Daniel H. Fishman</author><author>Michael L. Heytens</author><author>William Kent</author><title>Object SQL A Language for the Design and

Implementation of Object Databases.</title><pages>4268</pages><year>1995</year><booktitle>Modern Database Systems</booktitle><url>db/books/collections/kim95.html#AnnevelinkACFHK95</url><crossref>books/crc/KIM95</crossref>

</incollection>

....</dblp>

article {key : stringtitle : string *author : string *editor : string *booktitle : string *pages : string *address : string *journal : string *volume : string *number : string *publisher : string *crossref : string *series : string *school : string *chapter : string *month : string *year : string *url : string *note : string *mdate : string *cite : string *ee : string *cdrom : string *isbn : string *

}

book {key : stringtitle : string *author : string *editor : string *booktitle : string *pages : string *address : string *journal : string *volume : string *number : string *publisher : string *crossref : string *series : string *school : string *chapter : string *month : string *year : string *url : string *note : string *mdate : string *cite : string *ee : string *cdrom : string *isbn : string *

}

....

~ 800,000 Authors

~ 1,200,000 Publications

~ 14,000,000 Lines of XML

~ 16,000,000 Database records

DBLP Data

Publication {key : string Uniquetitle : stringauthors - Author + Indmonth : stringyear : stringdblpUrl : string ?doi : stringlinks - Link *abstract : stringnote : stringannote : stringmodified : datemodifiers - User *cites - Publication *ee : string ?isbn : stringissn : stringconflicts : bool

}

abstract PrintPublication - Publication {

pages : stringpublisher : string ?firstpage : intlastpage : int ?

}

Article - PrintPublication {journalname : stringvolumenumber: stringissuenumber : string

}

Alias {name : string Unique

}

AbstractAuthor {alias - Alias

}

Load objects into database

9 Stages of migration

Generate SQL

Migration Approach

BridgingMeta levels

article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

renameType(|oldType, newName)

?Transformation(path, Substitution(newName), _)

UPDATE At SET t=newName WHERE t = ...;

Proceedings - Collection {booktitle : string ?

}

Proceedings - Collection {conference : string ?

}

renameAtt(|oldName, newName, type, dmodel)

?Transformation(path, Substitution(newName), _)

UPDATE At SET n = newName WHERE n = oldNameAND t = <getSubTypeQuery(|dmodel)> type;

abstract Thesis - Publication {school : string

}

abstract Thesis - Publication {school : stringtype : string ?

}

...

?Transformation(path, Addition(Att(Name(attName), PrimType(_), annotations)),_);<getAttAnn(|"MinCard")> annotations; ?0

...

abstract Thesis - Publication {school : string

}

abstract Thesis - Publication {school : stringtype : string

}

<addDefaultAttributesToType(|type, <makeint> low, attName, attType, mmodel

)> model;

?Transformation(path, Addition(Att(Name(attName), PrimType(_), annotations)),_

);<getAttAnn(|"MinCard")> annotations; ?low

onType(addDefaultAttributes(|type, nr, attName, attType)

| type, mmodel)

1. find objects of type

2. divide into chunks

3. per chunk in parallel

Load objects in chunkper object

ssave object if changed

onType

Publication {key : string Uniquetitle : string ?authors : string + Indexedyear : string...

}

?Transformation(path, Substitution(DeclType(Name(newTypeName))), _)...Author alias mandatoryAuthor alias not unique

Publication {key : string Uniquetitle : string ?authors - Author + Indexedyear : string...

}

Author {alias : string

}

onType(for each author

create author objectset author attribute

)

Author {alias : string

}

?Transformation(path, Substitution(DeclType(Name(newTypeName))), _)...Alias name mandatoryAlias name unique

Author {alias : Alias

}

Alias {name : string Unique

onType(if alias exists then

set alias attribute to existing idelse

create alias objectset alias attribute to new id

)

IdentityPrimitive attribute addition (3)Complex attribute additionAttribute removalAttribute name changeAttribute move (2)Primitive type changeImplicit reference resolutionAttribute wrappingType additionType removalType name changeAbstract type handlingInverse annotation handling (2)Cardinality changes (2)

Supported transformations

Easy to define Hard to define

Easy to optimize No need to optimize

Expressive Limited expressiveness

Easy to abstract Abstraction near impossible

Performance OK Performance great

In memory vs. Database transformations

DetectingEvolution


}


}

diff?


}


}

Entity(Name(“article”),Name(“PrintPublication”),[

Att(Name(“reviewid”),PrimType(

Name(“string”)),[

MinCard(0),MaxCard(1)

]),...

])

Entity(Name(“Article”),Name(“PrintPublication”),[




]),...

])


}


}

Entity(Name(“article”),Name(“PrintPublication”),[




]),...

])

Entity(Name(“Article”),Name(“PrintPublication”),[




]),...

])

diff?


}


}

Removed type article; Added type Article

Added type article; Removed type Article

Substituted article name with Article

What happened?

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string

}

Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?

}

renamed rating; renamed number;changed rating cardinality; changed number cardinality

What happened?


}


}


swapped rating and number; renamed rating; renamed number

What happened?


}


}


swapped rating and number; renamed rating; renamed number

deleted Article; added Article

What happened?

Addition: 0.8 * relativeSize4

Removal: 0.5 * relativeSize4

Substitution 0.4 * relativeSize6

Weighing transformations

Custom weights:Type removalType substitution (0)Attribute substitution (0)

On both versions at the same time

Bound on weightIncreasing bound

Weight computation caching

Try them all!

HeterogeneousCoupled Evolution

Horizontal Generalization

Username :: varcharrealName :: varcharemail :: tinytext

Pagetitle :: varcharauthor UserisRedirect :: boolean

Entities

Properties

Types

Meta model / Grammar

Lists

More types

Inverse associations

Abstract types

Vertical Generalization

Heterogeneous Coupled Evolution

ofSoftware Languages

The ingredients

Software Language Definition Formalism

EvolvingSoftware Language

Software

DiverseEvolution

What did we generalize?

Why did we generalize?

A GenericArchitecture

Input

Coupled evolution scenario

Mapping from Mi to Mi+1

Output

Domain Specific Transformation Language

(DSTL)

Transformation Interpreter

Software Migration

Stratego

SDF SDF

Entity* > DataM ModelId "{" Prop* "}" > Entity EntityId "::" Type > Prop Prop"int" > Type Int"bool" > Type BoolId > Type "set of" Type > Type SetNAME > Id Id

Entity* > DataM Model

Lists

"add" Entity > LocalTransformation"remove" > LocalTransformation

[...]

NAME > Id Id

Lexicals

"substitute" NAME > LocalTransformation

''...''

"int" > Type Int"bool" > Type BoolId > Type "set of" Type > Type Set

Multiple productions

"substitute" Type > LocalTransformation

>*

"at" APath LocalTransformation > Transformation

Type checking

Generation of local transformation domainsAPath type derivation

Type checking

.../...

Transform

Constant

Larger grammars

Interpreter generation

Transformations library

Generic DSTL constructs

APath evaluation

A GenericArchitecture

Software Language Evolution

This research was supported by NWO/JACQUARD project638.001.610, MoDSE: ModelDriven Software Evolution.

model driven software development - data model evolution

Technology

user id

page id

indexedabstractmedium

id keywordid

addedtyperevisions revision

dbhdoupdate bugs

varcharauthor user

text author user