simplify your database migration to aws | aws public sector summit 2016

PowerPoint Presentation

Atul Ganatra, AWS Solutions Architecture

June 20, 2016Simplify Your Database Migration to AWS

2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Session outlineThe need for a database migration toolAWS Database Migration Service (AWS DMS) overviewAWS Schema Conversion Tool overviewDemoQ&A

We will be looking at some of the scenarios in which customers realize need to migrate the databases.

We will have high level overview of what the database migration service has to offer and how it works.

We will look at Schema Conversion Tool and see when you need to use it and how. This is also part of Database Migration Service, but works slightly differently.

And at the end we will see a demo of actual migration that uses both DMS and SCT.

We will leave few minutes at the end for questions. 2

Database migration challengesHow will my on-premises data be migrated to the cloud?How can I make it transparent to my users?How do I migrate schema objects and database code?Can I get help moving off of commercial databases?Can I avoid complex setup and migration procedures?Can I do it cost effectively?

As we all know the databases and the applications that use relational databases have been around for decades. As such, some decisions to use the databases have been dated and may not hold to be the best choice for the current situations. The landscape of the database world keeps changing and with that your needs and desire to use a particular platform are likely to change. E.g. now you have an option to use a database in AWS with RDS that takes away lot of burden of doing that you had no choice but to do it on your own.

So may customers see value in offering such as RDS or for that matter continue to run it like they have always run but want to run it in AWS using EC2 instances instead of managing those instances on their own. AWS offers very attractive value proposition to run the IT infrastructure which certainly includes your database. Often times customers are comfortable with the challenge it involves to move that database to AWS.

Database migration regardless of from where you are moving to where you are moving often requires careful planning and even with that involves experiencing decent amount of outage and impact to your customers. You may not be happy about that impact to your customers.

When you move the how your source database and target databases are going to communicate. What type of networking and security requirements you are going to have. Depending on the type of the data you may be concerned about that. Add to that the complexity of setting up that connection and tools to do the migration.

And last but not the least many customers feel that they need to move off of the commercial databases in order to stay competitive or continue to grow the usage of databases without incurring cost of expensive licenses.3

AWS Database Migration Service

Welcome to the AWS Database Migration Service. AKA DMS.

The service was introduced earlier this year and has received great response from our customers. We have thousands of customers already using DMS.4

Database Migration Service (DMS) goals

Minimize disruption to the applications

Make it easier to migrate databases to AWS

Provide choice and flexibility

As you seen with many AWS services that our goal is to relieve you from that undifferentiated heavy lifting so you can focus on what you are good at doing. You can focus on what your core competency is.

We want to do it so that has min impact to your business.

And another primary reason is that we want our customers to have choice of using various database options on top of AWS. You can migrate to different databases on RDS or run them on EC2. Its your choice. You can use DMS to move your data in or move it out. The requirement is that one end of the migration has to be in the AWS. So for whatever reason if your migration doesnt work out, you can use the same mechanism to get it out of AWS and take back to your on-prem environment or move it from RDS to database that is running on EC2. So we want to keep the migration options like some of the options that you may be familiar with that it can be done both ways.

5

Start your first migration in minutesKeep your apps running during the migrationMigrate to/from Amazon EC2 or Amazon RDSMove data to the same or different database engine Learn more at aws.amazon.com/dmsAWSDatabase Migration Service

* Like all AWS services, it is easy and straightforward to get started. You can get started with your first migration task few minutes.You simply connect it to your source and target databases, and it copies the data over, and begins replicating changes from source to target.

*That means that you can keep your apps running during the migration, then switch over at a time that is convenient for your business.

* DMS can be used to replicate within, to or from AWS EC2 or RDS databases. Basically, from anywhere to anywhere as long as one of them is an AWS resource.

*Again- it is heterogeneous ~. With DMS, you can move data between engines. Supports Oracle, Microsoft SQL Server, MySQL, PostgreSQL, MariaDB, Amazon Aurora, Amazon Redshift

* If you would like to learn more, go to

6

How does DMS work?

7

Create a replication instance

Replication instance

You start out your DMS migration effort by replication instance. A replication instance is your main compute unit that does the job of doing your migration.

We have several different types of replication instance you can choose depending on how large your migration project is. Mainly replication instance comes in two families. T2 and C4. We recommend using one of the T2 instance while you are checking your database migration out to see if things are migrating. Or when you are running small jobs on regular basis you can use T2 family. For large migration in production we recommend using one of the C4 instances. As you know the T2 instances comes with the baseline cpu performance which can burst depending the CPU credits that you accumulate. Certainly okay to use them for your dev/test scenarios.

The repl instance is created inside your VPC. You can put it in default VPC or on the VPC you create. Its going to need access to Internet or your on-prem resource depending where your source target are located. We will look at that little bit later. But you should keep it mind as you create your replication instance as to its going to need connectivity with your source and target.

Instance can be public or private depending on whether your source and target databases are available through the private connectivity from the VPC. I.e. VPN connection or Direct Connect or have the db in the same VPC, or peered VPC.

Similar concepts that apply to network configuration of an instance inside of a VPC applies also applies to replication instance.

Just like an EC2 instance, the replication instance is also protected by Security Group which needs to allow traffic between source and target endpoints.

8

SourceTargetCreate source and target endpoints


Next you want to create your source and target endpoints. The source endpoints can be any of the supported databases. You dont have to worry about managing odbc or jdbc driver or anything like that. When you create a source and target endpoint mainly you are required to provide is the host where your database is running, the port, the username and the password. There are additional things you can specify by custom parameters to influence the behavior the source or the target.

For example, Oracle requires the addition of supplemental logging and MySQL requires row-level bin logging. When using Amazon RDS as a source, we recommend ensuring that backups are enabled and that the source database is configured to retain change logs for a sufficient time (24 hours is usually enough).

The documentation describes all endpoint requirements for each source and target database engine.

The connection information you provide is encrypted by the service using a key in the KMS. By default the service uses an account specific default. You have an option to use a different key.

9

SourceTargetCreate a task to migrate the data

TaskReplication instance

now you are ready to create a task. Task is unit of your migration activity.

Task has mainly three key parameters:- source and target- type of task- tables to migrate

There are three types of tasks you can define:Full load: transfers the data from source to target and stopsFull load and on-going replication: starts with the full load and once finishes continues with on-going replication using Change Data Capture technique.Replication only: Captures only new transaction from the source and applied them to target

While creating the task you can also specify which schema and tables you want to load as part of this task. We will look at it little bit later as to how you can create multiple tasks for migration of the same database.

You can also specify certain transformation at the metadata level. Changing column names, or changing case of the table, adding prefix, etc. You can also drop a column during migration.

There are several other parameters you can specify such as how many tables you want loaded in parallel. The default number is 8, which you can change depending on how you want to perform the task.

You can put multiple tasks on the same replication instance. You should test your configuration out.

10

SourceTargetLoad startscaptures changes from source


Once you have the tasks defined, you basically start your task. Database Migration Service starts with what we call full load. While its doing the full load the new transaction that occur on the source database are captured using Change Data Capture on the source and staged on the migration instance.

Task caches the data mostly in the memory of the replication instance but depending how fast the target is able to take in the data and also the size of the data being transferred it may need to write to the disk of the migration instance. You get 50 GB or 100 GB depending which family you used for the replication instance11

SourceTargetLoad completeapply captured changes


Updates

Lets say after some time depending on the size of the database the full load is complete. The source and target reaches steady state or equilibrium. At this point the service is going start applying captured changes since it started the full load.

Depending on the source, you are going to have to enable transaction logs. E.g. in case of Oracle you have to enable archive logs and supplemental logging at the database level and also for each table. If you don't have that enabled, you are going to see some type of error in the logs and the task will fail.

12

SourceTargetChanges reach steady state


Updates

Once it catches up the source and the target will reach a steady state. What this means is that only the delta that is applied to the source is being moved to the target. There could be some replication lag but not much. You also have a choice of how frequently you want DMS to check for new changes. It can be as small as 1 sec. The default is 5 sec. Its one of the parameter you can specify while creating the task.

So you can see you still have the updates or the write going on with your source database.13

SourceTargetStop writes on source


Updates

You can stop the update on the source and make sure all the changes are flushed through the replication tasks and applied to the target14

SourceTargetAllow in-flight transactions to finish


You can stop the update on the source and make sure all the changes are flushed through the replication tasks and applied to the target15

SourceTargetPoint application to the target


Updates

At this point you are ready to move16

Full load is table by tableCreates tables on target databaseSets up metadata required on targetPopulates data from sourceEach process loads one entire tableCan use multiple processesTask can be paused Replication instanceSource

Target

8 processes at a time by default. Parameter that can be controlled by customer- Mention the phases in the project e.g.) Setup an empty shell, migrate, apply foreign keys, secondary indexes, etc.

Task loads data table by table during the full-load.You need to make sure foreign keys are disabled on the target database. E.g. there is a parameter for MySQL target to ignores the foreign key checks. You can also drop the foreign keys and recreate them when your full load is done.

If the target doesnt have the necessary tables to transfer the data it will create them including the primary keys.

Once the necessary table is created the task will transfer the data. Each process will load the entire table.

You can configure multiple processes to load tables in parallel. The default is 8, but you can adjust this number to either speed up the task or adjust the load on your source and target database. You should be mindful of how many tasks you run. The table copy requires a full table scan on the source and you want to make sure it doesnt adversely impact your source application.

You can also pause the task. If for any reason if you want to pause the task to check on something or if you have issue with connectivity that disrupts the tasks. It can be restarted. When the task restarts it will start from where it left. By default it will reload the table which it was in the middle of loading. You can configure it so that it doesnt reload the table.

17

Changes are transactional

SourceTarget

Updates

t1

t2

t1

t2

TargetReplication instance

As we saw earlier, mainly there are two phases of migration tasks. The full-load and then synchronize the transaction for ongoing replication.

The replication starts once the full-load is complete and its going to take little bit of time for it to be in sync with source because it would have built up some backlog of transactions that will apply in the order they were received.

The changes applied during this phase are transactional. So you want to make sure that if you are going to have on-going replication in your task, you include tables that can provide consistent transactions. In other words, you dont want to not include some tables that are in transactions and leave behind the others. This could create an error while applying this transactions on target if the tables dont exist and you may lose those transactions.18


Source

TargetMultiple targets

Target

Target

19

Migration Server

Source

TargetMultiple sources

Source

Source

20

Homogenous or heterogeneous


Microsoft SQL ServerMySQL


OracleOracle


OracleAmazon Aurora

The migration performed with DMS can be homogenous or heterogeneous. Meaning it can be same engine or different.21

Key featuresHomogenous or heterogeneous migrationsFull load migration, ongoing replication, or bothSupports variety of source and target databasesAllows migrating portion of databaseEncryption of the data on replication instanceNo agent on source or target database server

Here is the summary of key features that we reviewed.

You can use DMS for both homogenous and heterogeneous migrations

There three type of migrations it supports.

Variety of options for both source and target.

Allows you to migrate portion of your db by just including certain tables and columns.

Encryption of data on replication instance and endpoints using KMS

Its easy to configure because you dont need to install any software or agent on either the source or the target database.

22

SecurityUses AWS Identity and Access Management (IAM) to control accessSecure resources within Amazon Virtual Private Cloud (Amazon VPC)Uses AWS Key Management Service (AWS KMS) to encrypt data at rest on replication instance

Just like any other service in AWS you need proper IAM permission assigned to your user or role to be able to access DMS.

You can carry out a completely private migration project by keeping your resources private. The replication instance is launched inside of a VPC and you have an option to keep it private. You can create private connectivity from the replication instance to the your source and target and keep everything private.

DMS uses encryption using AWS Key Management Service to encrypt the data that is stored on the instance and also encrypt the endpoint information like user and password.

23

Migration phasesApply/create the empty database schema minus foreign key constraints and secondary indexesUse DMS to move the dataApply the remaining schema objectsStop writes on source databaseLet the changes synchronize to targetChange the app to point to target

24

Best practicesLoad multiple tables in parallelRemove bottlenecks on the targetUse multiple tasks for parallel loadManage load on the source databaseEnable logging for the migration tasksPlan schema conversion

Supported databasesSourceOracle, Microsoft SQL Server, MySQL, MariaDB, PostgreSQL

TargetOracle, Microsoft SQL Server, MySQL, MariaDB, PostgreSQL, Amazon Aurora, Amazon Redshift

26

AWS Database Migration Service pricing

T2 pricing starts at $0.018 per hourC4 pricing starts at $0.154 per hour(Pricing as of June 2016. See website for latest)

50 GB GP2 storage included with T2 instances100 GB GP2 storage included with C4 instances

Data transfer inbound and within AZ is freeData transfer across AZs starts at $0.01 per GBData transfer between AWS Regions and leavingAWS costs the same as when using EC2

27

Downloadable client installConverts your database schema including tables, stored procedures, views, and DMLAWSSchema Conversion Tool

JW- The AWS Schema Conversion Tool helps you convert your database schema from an Oracle or Microsoft SQL Server database, to a MySQL DB instance, an Amazon Aurora DB cluster, or a PG instance. The target can be either RDS or EC2 based

Desktop application available for Windows, Mac, and Linux

28

Key featuresPoint-and-click migration of schemaMigration Assessment ReportScript to generate schemaApply converted schema to target databaseConverts tables, stored procedures, views, and other objectsProvides action items for manual changes

Get help with converting tables, views, and code

SchemasTablesIndexesViewsPackagesStored proceduresFunctionsTriggersSequencesUser defined typesSynonyms

30

Supported conversionsSource DatabaseTarget Database on Amazon RDSMicrosoft SQL ServerAmazon Aurora, MySQL, PostgreSQLMySQLPostgreSQLOracleAmazon Aurora, MySQL, PostgreSQLPostgreSQLAmazon Aurora, MySQL

Demo

Expedia: On-line travel marketplaceMigrating some databases to Amazon AuroraKuldeep Chowhan, Principal Engineer, Expedia, Inc.:

The ease by which we can do this using the AWS Database Migration Service has simplified this process for us and enabled us to accelerate our migration efforts. The ability to closely monitor the process, the detailed logging feature, and the support we received from AWS have given us a great deal of confidence in a successful migration.

Worlds leading online travel company, with a portfolio that includes 150+ travel sites in 70 countries.

33

Thomas Publishing: Digital-friendly businessNeeded to grow database footprint but using Oracle would require significant upfront investment in both infrastructure and license expense.Wanted to migrate to Amazon AuroraDatabase Migration Service automated most of the work and dramatically reduced the manual effort involved in the code migration.Hans Wald, Chief Technology Officer, Thomas Publishing:"The AWS Database Migration Service will be a key enabler for our plans to migrate more databases to Amazon Aurora in 2016.

Connecting buyers and suppliers across all industrial sectors, evolving from an industrial trade print publisher into industrys most respected group of digital-friendly businesses.

34

Database migration partners

http://aws.amazon.com/dms/testimonials/#partners

35

Thank You!