delta* - hvl

DELTA*A tool for database refactoring

Patrick Stünkel

Western Norway University of Applied Sciences*Supported by the "Modern Refactoring"bilateral SIU/CAPES project

November 21, 2018

IntroductionTheory

ApplicationRelated Work

Conclusion and Outlook

About MeMotivation

About Me

I B.Sc. and M.Sc. from FHDW Hannover in GermanyI There DELTA was developed.

I 3 years in industry as a Software EngineerI Since September 2017: PhD research fellow at Western Norway

University of Applied SciencesI Topic: Interoperability in Model Driven Software Engineering (MDSE)I Areas: MDSE, Bidirectional Transformations (BX), Co-Evolution

2 / 63

IntroductionTheory



About MeMotivation

Motivation

I Software refactoring is widely adopted and is performedautomatically...

I ... whereas database refactoring is not.I Ambler/Sadalage „Database Refactoring“, 2006 [1]

Definition 1 (Ambler): Database refactoring

A simple change to a database schema that improves its design whileretaining both its behavioural and informational semantics.

3 / 63

IntroductionTheory



About MeMotivation

Motivation


I ... whereas database refactoring is not.

I Ambler/Sadalage „Database Refactoring“, 2006 [1]



4 / 63

IntroductionTheory



About MeMotivation

Motivation





5 / 63

IntroductionTheory



About MeMotivation

Motivation




A simple change to a database schema that improves its design whileretaining both its behavioural and informational semantics. 6 / 63

IntroductionTheory



About MeMotivation

Distinction: code refactoring - database refactoring

What makes Database refactoring different ?

I deals with behavioural and informational aspects,

I affects actual stored data,

I requires manual effort,

I in general a database schema is shared between many differentapplications.

Requirement 1: Transition periods

The database has to be accessible through the new and the old schemaafter a refactoring for certain transition period because the differentdependent applications need time to adopt the changed schema.

7 / 63

IntroductionTheory



About MeMotivation









8 / 63

IntroductionTheory



About MeMotivation









9 / 63

IntroductionTheory



About MeMotivation









10 / 63

IntroductionTheory



About MeMotivation









11 / 63

IntroductionTheory



About MeMotivation









12 / 63

IntroductionTheory



About MeMotivation

Further requirements

Requirement 2: Generated Migrations

As the manual development of migration procedures is error-prone andthere are recurring patterns of migrations the migration code itself shallbe generated.

Requirement 3: Revertible Migrations

As a refactoring must not change the behaviour of the data or causeinformation loss the refactoring can be reverted. Therefore we require anundo-feature for our migrations.

Requirement 4: Avoid downtimes

As long downtimes have negative impact and productivity and sale theyshall be avoided whilst applying refactorings.

13 / 63

IntroductionTheory



About MeMotivation








14 / 63

IntroductionTheory



About MeMotivation








15 / 63

IntroductionTheory



About MeMotivation








16 / 63

IntroductionTheory



Refactoring ProcessRefactoring CatalogueMigration Strategies

Maintaining compatibility

Trigger Old and new schema running in parallel, trigger replicatechanges back and forth.

Views Views, representing the old schema, provide backwardcompatibility.

Batch Jobs Old and new schema running in parallel, Batch jobsreplicate changes back and forth on a regular basis.

17 / 63

IntroductionTheory





I Ambler and Sadalage suggest Trigger.

I We chose Views with Instead-Of-Trigger.

18 / 63

IntroductionTheory







19 / 63

IntroductionTheory







20 / 63

IntroductionTheory




Assumptions

I Database schema represents an object-oriented-model, i. e. set ofrelated entities.

I Object-Relational-MappingI EntityType → Table with surrogate IDI Attribute → ColumnI *:1-Association → Foreign-Key-Constraint

I ⇒ Schema satisfies 2NF.

21 / 63

IntroductionTheory




Assumptions




22 / 63

IntroductionTheory




Assumptions




23 / 63

IntroductionTheory




Catalogue

I Rename Table

I Rename Column

I Introduce Calculated Column

I Merge and Split Columns

I Spin-Off Empty-Table

I Move Column

I Merge and Split Table

I Transcode Foreign-Key

I Compose Foreign-Keys

24 / 63

IntroductionTheory




Introduce Calculated Column

25 / 63

IntroductionTheory




Merge and Split Columns

26 / 63

IntroductionTheory




Spin-Off Empty Table

27 / 63

IntroductionTheory




Move Column

28 / 63

IntroductionTheory




Merge and Split Table

29 / 63

IntroductionTheory




Transcode Foreing Key

30 / 63

IntroductionTheory




Compose Foreign Keys

31 / 63

IntroductionTheory




Offline Batch-Migration

I Shut down database.

I Apply schema changes and migrate data.

I Restart database.

Advantages Disadvantages

Easy to implement May cause long downtimes on large data sets

32 / 63

IntroductionTheory




Stepwise, Transactional Migration

I Create new schema (empty).

I Copy data to the new schema on-access or per batch.

I View layer on the new schema aggregates data from old and newschema.


Long downtimes avoided Higher complexity due to aggregationTemporary trigger required

33 / 63

IntroductionTheory




Two-Phase Migration

I Copy database.

I Apply schema changes to one copy (incl. data migration).

I Original schema receives triggers, which log every datamanipulation.

I The logged actions are applied to the new schema copy bit by bit.

I Eventually all data is migrated to the new schema.


Long downtimes avoided Higher complexity due to mergeConcurrent migration Temporary trigger for logging required

34 / 63

IntroductionTheory



Case study

Design Goals

I Independent, encapsulated refactorings

I A refactorings is composed of Deltas

I Virtual preview of schema modification

I Goal in the long run: Merge of refactorings!

35 / 63

IntroductionTheory



Case study

Design Goals





36 / 63

IntroductionTheory



Case study

Design Goals





37 / 63

IntroductionTheory



Case study

Design Goals





38 / 63

IntroductionTheory



Case study

Design Goals





39 / 63

IntroductionTheory



Case study

Design Goals





DELTA:

I Tool for database migration (currently Oracle)

I Java 8

I JavaFX-GUI

I Three-Layer-Architecture

I 50 Deltas (9 "real"refactorings)

40 / 63

IntroductionTheory



Case study

Starting point

I Address is part of Person

41 / 63

IntroductionTheory



Case study

Starting point

I Address is part of Person

42 / 63

IntroductionTheory



Case study

Step 1: Spin-Off the Address

I transition phaseI no distinction between legal and natural persons

43 / 63

IntroductionTheory



Case study


I transition phase

I no distinction between legal and natural persons

44 / 63

IntroductionTheory



Case study


I transition phaseI no distinction between legal and natural persons

45 / 63

IntroductionTheory



Case study

Step 2: Split in natural and legal persons

I no support for multiple tenants

46 / 63

IntroductionTheory



Case study

Step 2: Split in natural and legal persons

I no support for multiple tenants47 / 63

IntroductionTheory



Case study

Step 3: Implement Multi-Tenancy

I transition phase

48 / 63

IntroductionTheory



Case study

Step 3: Implement Multi-Tenancy

I transition phase

49 / 63

IntroductionTheory



Case study

Company Merger

I bad design: insuranceId and liabilityInsuranceId aredisjoint

50 / 63

IntroductionTheory



Case study

Company Merger

I bad design: insuranceId and liabilityInsuranceId aredisjoint

51 / 63

IntroductionTheory



Case study

Schritt 4: Concrete-Table-Inheritance →Class-Table-Inheritance

52 / 63

IntroductionTheory



Case study

Case Study: Wrap-Up

I Extract entities: transformation of associations 1:1 → 1:* → *:*

I Split entities: Single-Table-Inheritance → Class-Table-Inheritance

I Extend models (multi-tenancy) with calculated columns

I Transformation of models (Merge, „Transcode“etc.):Concrete-Table-Inheritance → Class-Table-Inheritance

53 / 63

IntroductionTheory



Case study

Case Study: Wrap-Up





54 / 63

IntroductionTheory



Case study

Case Study: Wrap-Up





55 / 63

IntroductionTheory



Case study

Case Study: Wrap-Up





56 / 63

IntroductionTheory



Existing tools

SQL Prompt VisualStudio-PlugIn, supports renaming and split-table

ApexSQL Only MS-SQL-Server, supports renaming, split-table andintroduce association-table.

Flyway Version control and management of migration scripts, noimplemented refactorings

Liquibase Same as Flyway but with SQL-Abstraction and somebuilt-in-refactorings (rename, split-table)

57 / 63

IntroductionTheory



Related research

I I. Skoulis et. al. Growing up with stability: How open-sourcerelational databases evolve [4]: study of the VCS-history to analysepatterns in the development of the database model.

I C. Curino et. al. Graceful database schema evolution: the prismworkbench [5]. Similar approach to DELTA, based on SQL rewriting.

I M. Pereira et al. Evolution of databases using petri nets [6]. Moregeneral approach on how to check dependencies betweenrefactorings.

58 / 63

IntroductionTheory



Conclusion

I Catalogue of database refactorings

I Backward-compatibility with Views and Instead-Of-Trigger

I Presentation of the different migration scenarios

I Open-Source-Tool: DELTA

59 / 63

IntroductionTheory



Outlook

I Evaluation of the prototype

I Support of the different migration scenarios

I Composition of refactorings

I Reordering of refactoring

THANK YOU FOR YOUR ATTENTION!

60 / 63

IntroductionTheory



Outlook

I Evaluation of the prototype

I Support of the different migration scenarios

I Composition of refactorings

I Reordering of refactoring

THANK YOU FOR YOUR ATTENTION!

61 / 63

IntroductionTheory



[1] Scott W Ambler and Pramod J SadalageRefactoring databases: Evolutionary database design. PearsonEducation, 2006.

[2] Martin FowlerRefactoring : improving the design of existing code. Addison-Wesley,1999.

[3] Michael LöweRefactoring information systems - association folding and unfolding.FHDW Hannover, 2013.

[4] Ioannis Skoulis, Panos Vassiliadis, Apostolos V. ZarrasGrowing up with stability: How open-source relational databasesevolve. Information Systems,53:363-385, 2015

[5] Carlo A. Curino, Hyun J. Moon, Carlo ZanioloGraceful database schema evolution: the prism workbench.Proceeding of the VLDB Endowment, 1:761-772, 2008

62 / 63

IntroductionTheory



[6] Marcia Beatriz Carvalho Pereira, Jorge Rady de Almeida Junior,Jose Reinaldo SilvaEvolution of databases using petri nets. Anais de XIX CongressoBrasileiro de Automatica, CBA 2012

63 / 63

delta* - hvl

Documents