merging and migrating: data portability from the trenches

Data PortabilityTales from the Trenches

DAN HARDIKER | CTO | ADAPTAVIST | @DHARDIKER

BRITTANY WISPELL | CONSULTANT | ADAPTAVIST | @REDHOTJIRAPEPPS

Standard horizontal view logo.

Purposes: Website, Blogs, External PRs, External websites, Event websites.

Smaller lettering.

Purpose: Business cards.

Logo centralised.

Purpose: Events, stall headers, handouts.

Logo centralised.

Purpose: Events, stall headers, handouts.

AgendaTools & Support

Scenarios

Atlassian

I have a cunning plan

BALDRICK, BLACKADDER http://www.fansshare.com/celebrities/blackadder-goes-forth/

APPROACHES CHECKLISTS

✅ Task1

✅ Task2

✅ Task3

Business as Usual

Infrastructure Change

Atlassian Cloud

Data Shaping

Scenarios

Business as Usual

Scenarios

Copy & Clone Driven by Operational Needs

Backup & Restore Driven by Business Requirements

Scenarios

Business as Usual

Managing Data

Business as Usual

XML Backups

Business as Usual > Managing Data

Backup

Restoration

Gotchas

Contents A zipped archive containing most of the data from your Jira system.

Location Found in the Jira home directory, within the exports folder.

Default Enabled by default, running nightly.

XML Backups

Backup

Restoration

Gotchas

Jira Admin UI Restoration is started through the Jira user interface.

Resource Intensive Intensive of use of Java memory, network & disk I/O.

Index Rebuilt After the restoration, a full reindex will be triggered.

XML Backups

Backup

Restoration

Gotchas

Missing in Action • Attachments • Bespoke Plugin Data • Plugin Installation Files

Never quite the same • Database record IDs are regenerated. • Application keys are unchanged.

Not Production Safe If Jira is updated during an XML backup, the resulting backup can be inconsistent or even corrupt.

File System & Database Snapshots

Business as Usual > Managing Data

Backup

Restoration

Gotchas

Contents Two separate snapshots taking the file system and database state, at a synchronised point in time.

Knowledge & Tooling Infrastructure and DBA knowledge required. Process & tooling is technology specific.

No Maintenance Mode Execute during an outage, or when minimal activity is running through Jira.

Backup

Restoration

Gotchas

Technology Specific Restore the entire snapshots of the file system & database directly.

Outage Required Jira must be offline for the restoration.

No Index Rebuilding The system will now be identical to that backed up.

Backup

Restoration

Gotchas

Identical in every way • Base URL • Application Links • Inbound Email

Work together Synchronising the snapshots can get tough.

✅ Method • XML • Snapshot

✅ Storage • Frequency & Retention • Constrains your RPO

✅ Security (Access) • Sensitive data • Availability

✅ Security (Crypto) • In Transit / At Rest / In Use • Key Management

✅ Data Sovereignty • Where is your data? • DR / Backups / Support

✅ Compliance • ISO 27001 / HIPAA • Privacy Shield / DPA / GDPR

Business as Usual

Checklist

Business as Usual

Atlassian Cloud

Data Shaping

Scenarios

File System Changing system architecture

Database Moving between vendors / systems

Scenarios

Transform Filter, modify and translate

the data as needed.

Load Hydrate the data into the

target system

Export Extract the data from the

source system

utf8mb4

JRASERVER-36135MySQL 4-byte character support

Transform Filter, modify and translate

the data as needed.

Load Hydrate the data into the

target system

Export Extract the data from the

source system

✅ Method • XML for DB changes • Snapshot for Lift & Shift

✅ Revisit Approach • Backups & Resilience Processes • Snapshot Consistency

✅ Ownership • Access & Control • DBA? Service Desks?

✅ Technology • Drivers, Firewalls, Tooling • Direct Integrations with FS & DB

✅ Conversion • DB schema changes • Indexes & Character Sets

✅ Dry Runs • Multiple iterations required • Outage sizing

Checklist

KEEP CALM

& DRY RUN

✅ Method • XML for DB changes • Snapshot for Lift & Shift

✅ Revisit Approach • Backups & Resilience Processes • Snapshot Consistency

✅ Ownership • Access & Control • DBA? Service Desks?

✅ Technology • Drivers, Firewalls, Tooling • Direct Integrations with FS & DB

✅ Conversion • DB schema changes • Indexes & Character Sets

✅ Dry Runs • Multiple iterations required • Outage sizing

✅ Immutability • Prefer new & migrate • How do we back out?

✅ Monitoring • Benchmark latency & responsiveness • Validate in all environments

Business as Usual

Atlassian Cloud

Data Shaping

Scenarios

Atlassian Cloud

Scenarios

Atlassian Cloud

http://www.pngmart.com/image/tag/terminator http://www.mcgamer.net/threads/gameover-us-clan.95893/

✅ Method • XML only • No Infrastructural Access

✅ XML Backing Up • Are more resources needed? • Is an outage required?

✅ Duration • How long does it take? • Backup, Transfer, Restore

✅ Versions • What versions are needed? • What change is expected?

✅ Complexity • How simple is your case? • Is transformation needed?

✅ Atlassian ID • What is your username schema? • Does any mapping need to happen?

Atlassian Cloud

✅ Functionality • Map the differences • Include add-ons

✅ Licensing • What are the commercial changes? • Include add-ons and environments

Checklist

Business as Usual

Atlassian Cloud

Data Shaping

Scenarios

Data Shaping

Scenarios

MergingSplitting

Data Shaping

Scenarios

Splitting

Data Shaping

Snapshot Backup & Restore

Splitting

Snapshot Backup & Restore Duplicate the system from snapshot backups.

Splitting

Snapshot Backup & Restore Duplicate the system from snapshot backups. Remove unwanted data from either system.

Splitting

Constraints

Splitting

Constraints Target system must not exist.

Splitting

Constraints Target system must not exist. Same technologies in both systems.

Splitting

Use Cases

Splitting

Use Cases Where the source system cannot be impacted.

Splitting

Use Cases Where the source system cannot be impacted. Where cloning the system is straight forward.

Splitting

Tooling Based Copy

Splitting

Tooling Based Copy Setup a new system.

Splitting

Tooling Based Copy Setup a new system. Copy global configuration.

Splitting

Tooling Based Copy Setup a new system. Copy global configuration. Use tooling to copy across project config & data.

Splitting

Tooling Based Copy Setup a new system. Copy global configuration. Use tooling to copy across project config & data. Remove copied data from source system.

Splitting

Constraints

Splitting

Constraints Target system global configuration must match.

Splitting

Constraints Target system global configuration must match. Appropriate tooling must be purchased.

Splitting

Constraints Target system global configuration must match. Appropriate tooling must be purchased. Plugins data may be missed.

Splitting

Use Cases

Splitting

Use Cases Where the source system must remain live.

Splitting

Use Cases Where the source system must remain live. Where a relatively small amount of data needs to be split off.

Splitting

XML Backup & Restore

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup.

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup. Install plugins.

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup. Install plugins. Copy over the attachments.

Splitting

XML Backup & Restore Setup a new system, restoring the XML backup. Install plugins. Copy over the attachments. Remove unwanted data from either system.

Splitting

Constraints

Splitting

Constraints Source system must have an outage for backup.

Splitting

Constraints Source system must have an outage for backup. Typically consumes the most resources and takes the longest.

Splitting

Use Cases

Splitting

Use Cases Where transformation of the backup is needed.

Splitting

Use Cases Where transformation of the backup is needed. When the new system architecture is different.

Splitting

Use Cases Where transformation of the backup is needed. When the new system architecture is different. Can restore specific projects. (Server/DC only)

Splitting

Merging

Data Shaping

Merging

Snapshot Backup & Restore Not possible as target system already exists.

Merging

Tooling Based Copy

Merging

Tooling Based Copy Align Jira & add-on versions in both systems.

Merging

Tooling Based Copy Align Jira & add-on versions in both systems. Sync target system global & project configuration.

Merging

Conflict Resolution Resolve any configuration differences:

Merging

Globals Users & Groups, Resolutions, Statuses

Merging

Project keys Including historic

Merging

Dashboards & Agile Rapid Boards & Filters Compound key: item name + owning user

Merging

Strategies

Merging

Strategies Technical

Merging

Strategies Technical Process

Merging

XML Backup & Restore

Merging

XML Backup & Restore Transform the XML backup in-flight.

Merging

Constraints

Merging

Constraints Sensitive to change.

Merging

Constraints Sensitive to change. Edge cases require testing.

Merging

Use Cases

Merging

Use Cases Where complex migrations are needed.

Merging

Use Cases Where complex migrations are needed. Extreme changes in data structure and architecture.

Merging

✅ Versions • See: Atlassian Cloud • Add-ons too

✅ Add-ons • Where is the data? • How is it managed?

✅ Scope • Whole or partial? • New or merge in existing?

✅ Conflict Resolution • Full assessment upfront • Tooling assistance helpful

✅ Transformation • What needs to change? • Users, Keys, Globals

✅ Validation • Missing Data (Script Fields) • Smoke Test (config item count)

Data Shaping

Checklist

Data Shaping

Tools & Support

Scenarios

Atlassian

Tools & Support

Scenarios

Atlassian

http://wheelhouseworkshop.com/2017/04/06/its-dangerous-to-go-alone-take-this/

Botron Software Configuration Manager

for JiraProject Configurator

for Jira

Atlassian Recommends

Tools & Support

AdaptavistAwnaba

Atlassian Support Standard or Premium.

Consider a TAM.

You are not alone!

Solution Partners Professionals who have done

this many times before.

Atlassian Community Documentation, Q&A,

support from your peers.

Tools & Support

Start off Simple

Tools & Support

Scenarios

Atlassian

Tools & Support

Scenarios

Atlassian

• Merging and migrating are really tough problems

• We're incredibly thankful for our solution partners

• We recognize your pain and are finally ready to help

Announcing a dedicated migration team!

• Migration (not just export/import)

• Multiple products (not just Jira)

• Rome wasn't built in a day

• Server->Cloud to start

Cloud migration, worthy of Atlassian

Inviting Make migration feel like the natural next step in the customer

journey

Scalable Migrate everything from single projects to massive instances

with confidence

Reliable Tell customers what is happening, before, during, and after (and

make sure it does)

Flexible Support multiple

migration modes and approaches based on

customer needs

Tools & Support

Scenarios

Atlassian

Tools & Support

Scenarios

Atlassian

Thank you!

DAN HARDIKER | CTO | ADAPTAVIST | @DHARDIKER

BRITTANY WISPELL | CONSULTANT | ADAPTAVIST | @REDHOTJIRAPEPPS

merging and migrating: data portability from the trenches

Software

lesson ppt trenches

trenches project

deep ocean trenches

merging funnels

health insurance portability and accountability act ... ·...

image merging

portability issues

java portability

trenches group activity

calculating estate tax portability exemption...

merging identities

in the trenches

v from trenches

the trenches

powerpoint trenches

portability final

portability law

the trenches - azbocug

by sharath.r. mnp 1. service operator portability 2....

merging & blending - topaz...