merging and migrating: data portability from the trenches

Standard horizontal view logo.

Purposes: Website, Blogs, External PRs, External websites, Event websites.

Smaller lettering.

Purpose: Business cards.

Logo centralised.

Purpose: Events, stall headers, handouts.

Logo centralised.

Purpose: Events, stall headers, handouts.

AgendaTools & Support

Scenarios

Atlassian

Q&A

I have a cunning plan

BALDRICK, BLACKADDER http://www.fansshare.com/celebrities/blackadder-goes-forth/

APPROACHES CHECKLISTS

✅ Task1

✅ Task2

✅ Task3

...

Business as Usual

Infrastructure Change

Atlassian Cloud

Data Shaping

Scenarios

Business as Usual

Scenarios

Copy & Clone Driven by Operational Needs

Backup & Restore Driven by Business Requirements

Scenarios

Business as Usual

Managing Data

Business as Usual

XML Backups

Business as Usual > Managing Data

Backup

Restoration

Gotchas

Contents A zipped archive containing most of the data from your Jira system.

Location Found in the Jira home directory, within the exports folder.

Default Enabled by default, running nightly.

XML Backups

XML Backups

Backup

Restoration

Gotchas

Jira Admin UI Restoration is started through the Jira user interface.

Resource Intensive Intensive of use of Java memory, network & disk I/O.

Index Rebuilt After the restoration, a full reindex will be triggered.

XML Backups

Backup

Restoration

Gotchas

Missing in Action • Attachments • Bespoke Plugin Data • Plugin Installation Files

Never quite the same • Database record IDs are regenerated. • Application keys are unchanged.

Not Production Safe If Jira is updated during an XML backup, the resulting backup can be inconsistent or even corrupt.

File System & Database Snapshots

Business as Usual > Managing Data

Backup

Restoration

Gotchas

Contents Two separate snapshots taking the file system and database state, at a synchronised point in time.

Knowledge & Tooling Infrastructure and DBA knowledge required. Process & tooling is technology specific.

No Maintenance Mode Execute during an outage, or when minimal activity is running through Jira.



Backup

Restoration

Gotchas

Technology Specific Restore the entire snapshots of the file system & database directly.

Outage Required Jira must be offline for the restoration.

No Index Rebuilding The system will now be identical to that backed up.


Backup

Restoration

Gotchas

Identical in every way • Base URL • Application Links • Inbound Email

Work together Synchronising the snapshots can get tough.

✅ Method • XML • Snapshot

✅ Storage • Frequency & Retention • Constrains your RPO

✅ Security (Access) • Sensitive data • Availability

✅ Security (Crypto) • In Transit / At Rest / In Use • Key Management

✅ Data Sovereignty • Where is your data? • DR / Backups / Support

✅ Compliance • ISO 27001 / HIPAA • Privacy Shield / DPA / GDPR

Business as Usual

Checklist

Business as Usual


Atlassian Cloud

Data Shaping

Scenarios


Scenarios

File System Changing system architecture

Database Moving between vendors / systems

Scenarios


Transform Filter, modify and translate

the data as needed.

ETL

Load Hydrate the data into the

target system

Export Extract the data from the

source system


utf8mb4

JRASERVER-36135MySQL 4-byte character support

Transform Filter, modify and translate

the data as needed.

ETL

Load Hydrate the data into the

target system

Export Extract the data from the

source system


✅ Method • XML for DB changes • Snapshot for Lift & Shift

✅ Revisit Approach • Backups & Resilience Processes • Snapshot Consistency

✅ Ownership • Access & Control • DBA? Service Desks?

✅ Technology • Drivers, Firewalls, Tooling • Direct Integrations with FS & DB

✅ Conversion • DB schema changes • Indexes & Character Sets

✅ Dry Runs • Multiple iterations required • Outage sizing


Checklist

KEEP CALM

& DRY RUN

✅ Method • XML for DB changes • Snapshot for Lift & Shift

✅ Revisit Approach • Backups & Resilience Processes • Snapshot Consistency

✅ Ownership • Access & Control • DBA? Service Desks?

✅ Technology • Drivers, Firewalls, Tooling • Direct Integrations with FS & DB

✅ Conversion • DB schema changes • Indexes & Character Sets

✅ Dry Runs • Multiple iterations required • Outage sizing

✅ Immutability • Prefer new & migrate • How do we back out?

✅ Monitoring • Benchmark latency & responsiveness • Validate in all environments


Business as Usual


Atlassian Cloud

Data Shaping

Scenarios

Atlassian Cloud

Scenarios

Scenarios

Atlassian Cloud

XML

http://www.pngmart.com/image/tag/terminator http://www.mcgamer.net/threads/gameover-us-clan.95893/

http://www.pngmart.com/image/tag/terminator

✅ Method • XML only • No Infrastructural Access

✅ XML Backing Up • Are more resources needed? • Is an outage required?

✅ Duration • How long does it take? • Backup, Transfer, Restore

✅ Versions • What versions are needed? • What change is expected?

✅ Complexity • How simple is your case? • Is transformation needed?

✅ Atlassian ID • What is your username schema? • Does any mapping need to happen?

Atlassian Cloud

✅ Functionality • Map the differences • Include add-ons

✅ Licensing • What are the commercial changes? • Include add-ons and environments

Checklist

Business as Usual


Atlassian Cloud

Data Shaping

Scenarios

Data Shaping

Scenarios

MergingSplitting

Data Shaping

Scenarios

Splitting

Data Shaping

Snapshot Backup & Restore

Splitting

Snapshot Backup & Restore Duplicate the system from snapshot backups.

Splitting

Snapshot Backup & Restore Duplicate the system from snapshot backups. Remove unwanted data from either system.

Splitting


Constraints

Splitting


Constraints Target system must not exist.

Splitting


Constraints Target system must not exist. Same technologies in both systems.

Splitting



Use Cases

Splitting



Use Cases Where the source system cannot be impacted.

Splitting



Use Cases Where the source system cannot be impacted. Where cloning the system is straight forward.

Splitting

Tooling Based Copy

Splitting

Tooling Based Copy Setup a new system.

Splitting

Tooling Based Copy Setup a new system. Copy global configuration.

Splitting

Tooling Based Copy Setup a new system. Copy global configuration. Use tooling to copy across project config & data.

Splitting

Tooling Based Copy Setup a new system. Copy global configuration. Use tooling to copy across project config & data. Remove copied data from source system.

Splitting


Constraints

Splitting


Constraints Target system global configuration must match.

Splitting


Constraints Target system global configuration must match. Appropriate tooling must be purchased.

Splitting


Constraints Target system global configuration must match. Appropriate tooling must be purchased. Plugins data may be missed.

Splitting



Use Cases

Splitting



Use Cases Where the source system must remain live.

Splitting



Use Cases Where the source system must remain live. Where a relatively small amount of data needs to be split off.

Splitting

XML Backup & Restore

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup.

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup. Install plugins.

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup. Install plugins. Copy over the attachments.

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup. Install plugins. Copy over the attachments. Remove unwanted data from either system.

Splitting


Constraints

Splitting


Constraints Source system must have an outage for backup.

Splitting


Constraints Source system must have an outage for backup. Typically consumes the most resources and takes the longest.

Splitting



Use Cases

Splitting



Use Cases Where transformation of the backup is needed.

Splitting



Use Cases Where transformation of the backup is needed. When the new system architecture is different.

Splitting



Use Cases Where transformation of the backup is needed. When the new system architecture is different. Can restore specific projects. (Server/DC only)

Splitting

Merging

Data Shaping

Merging

Snapshot Backup & Restore Not possible as target system already exists.

Merging

Tooling Based Copy

Merging

Tooling Based Copy Align Jira & add-on versions in both systems.

Merging

Tooling Based Copy Align Jira & add-on versions in both systems. Sync target system global & project configuration.

Merging


Conflict Resolution Resolve any configuration differences:

Merging



Globals Users & Groups, Resolutions, Statuses

Merging




Project keys Including historic

Merging





Dashboards & Agile Rapid Boards & Filters Compound key: item name + owning user

Merging

Strategies





Merging

Strategies Technical





Merging

Strategies Technical Process





Merging

XML Backup & Restore

Merging

XML Backup & Restore Transform the XML backup in-flight.

Merging


Constraints

Merging


Constraints Sensitive to change.

Merging


Constraints Sensitive to change. Edge cases require testing.

Merging



Use Cases

Merging



Use Cases Where complex migrations are needed.

Merging



Use Cases Where complex migrations are needed. Extreme changes in data structure and architecture.

Merging

✅ Versions • See: Atlassian Cloud • Add-ons too

✅ Add-ons • Where is the data? • How is it managed?

✅ Scope • Whole or partial? • New or merge in existing?

✅ Conflict Resolution • Full assessment upfront • Tooling assistance helpful

✅ Transformation • What needs to change? • Users, Keys, Globals

✅ Validation • Missing Data (Script Fields) • Smoke Test (config item count)

Data Shaping

Checklist

Data Shaping

Tools & Support

Scenarios

Atlassian

Q&A

http://wheelhouseworkshop.com/2017/04/06/its-dangerous-to-go-alone-take-this/

Botron Software Configuration Manager

for JiraProject Configurator

for Jira

Atlassian Recommends

Tools & Support

AdaptavistAwnaba

Atlassian Support Standard or Premium.

Consider a TAM.

You are not alone!

Solution Partners Professionals who have done

this many times before.

Atlassian Community Documentation, Q&A,

support from your peers.

Tools & Support

Start off Simple

Tools & Support

Tools & Support

Scenarios

Atlassian

Q&A

• Merging and migrating are really tough problems

• We're incredibly thankful for our solution partners

• We recognize your pain and are finally ready to help

Announcing a dedicated migration team!

• Migration (not just export/import)

• Multiple products (not just Jira)

• Rome wasn't built in a day

• Server->Cloud to start

Cloud migration, worthy of Atlassian

Inviting Make migration feel like the natural next step in the customer

journey

Scalable Migrate everything from single projects to massive instances

with confidence

Reliable Tell customers what is happening, before, during, and after (and

make sure it does)

Flexible Support multiple

migration modes and approaches based on

customer needs

Tools & Support

Scenarios

Atlassian

Q&A

Tools & Support

Scenarios

Atlassian

Q&A?