sharepoint online migration api and improvements

SharePoint Online Migration API and Improvements

Simon BourdagesMicrosoft PM

© SPintersection. All rights reserved.http://www.SPintersection.com

Introduction

Overview Moving content into O365

Current tech New Capabilities Things to come

The Migration API and PowerShell Using the PowerShell Building Code for the API Tips and Lessons Learned

A dedicated API created by Microsoft in order to provide the 3rd party vendors a dedicated way to bring large volumes of contents into SharePoint Online and OneDrive for Business.

The Migration API is live since May 2015

The Migration API


Moving Content in O365Current Tech

The Migration API was built to bring large Amount of content into the O365 Cloud

Requires the Use of Azure Timer Job based Best way to avoid CSOM throttling Scalable and in continuous improvement



Source Package SharePoint/ODBFinal Destination

Azure Temporary Storage

File Share,SharePoint On-Prem and potentially any other data Source

Create package for the API to be able to accept it

Use the Power of ingestion of Azure to bring the content faster in Microsoft network

Timer Job based import in a scalable way that will not hurt the service using back end resources



File and Folders Basic Metadata Permissions Versions


Moving Content in O365New Capability


Encryption at Rest Alerts Taxonomy


Moving Content in O365Things to come


Encryption at Rest Alerts Taxonomy

Removing the need of a customer Azure Account TAP program available for Developers

More Based on Usage and Requests


PowerShell


The Migration API and PowerShellWhat is What

The PowerShell Commands PowerShell based Client side tool Performs the end to end content migration

The Migration API CSOM based Call Requires Transformation of data Server side Service Requires the content to already be in Azure container


The Migration API and PowerShellTooling responsibility

Source Package SharePoint/ODBFinal Destination


File Share,SharePoint On-Prem and potentially any other data Source

Create Package for the API to be able to accept it

Use the Power of ingestion of Azure to bring the content faster in Microsoft network

Timer Job Based import in a scalable way that will not hurt the service using back end resources

Client Tool Azure API SPO Migration API


PowerShellThe Commands

New-SPOMigrationPackage

ConvertTo-SPOMigrationTargetedPackage

Set-SPOMigrationPackageAzureSource

Submit-SPOMigrationJob

New

Convert

Set

Submit


PowerShellcommands Flow

Convert Set Submit

NewExported Content

Package SharePoint/ODBFinal Destination


File Share Server

SharePoint On Premise Export-SPWeb

From On-Premise


PowerShellSupported scenarios


File Share SP 2010, 2013, 2016


The first command will read the list of content targeted by the source path and then will generate the appropriate XML to perform migration. Here are the required parameters to enter (others are optional)

SourcefilesPath : pointing to your content you want to migrate OutputPackagePath : pointing to your Temporary folder

Ex:New-SPOMigrationPackage -SourcePath C:\Users\Pictures\ -OutputPackagePath C:\Users\test\tempPackage

PowerShellNew-SPOMigrationPackage


The Second command will simply convert the xml generated in your temporary folder into a format that the destination will understand. This will be the final package. Here are the required parameters to enter (others are optional)

TargetwebURL : point to the Web of the destination SourceFilesPath : pointing to your content you want to migrate SourcePackagePath : pointing to your Temporary package folder OutputPackagePath : pointing to your final package folder targetDocumentLibraryPath : This is the path to your destination library Credentials : SPO credential that has admin rights to the destination site

Ex:ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/ -SourcePackagePath C:\Users\test\tempPackage -OutputPackagePath C:\Users\test\SPOpackage -targetDocumentLibraryPath Documents -sourcefilespath C:\Users\Pictures\ -Credentials (Get-Credential [email protected])

PowerShellConvertTo-SPOMigrationTargetedPackage

https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/


The third command will simply convert the XML generated into a format that the destination will understand. Here are the required parameters to enter (others are optional)

SourceFilePath : Pointing to the content you want to migrate SourcePackagePath : Pointing to the final package folder AccountName : The name of the Azure storage account Accountkey : The key to the Azure storage account AzureQueueName : The name of the azure queue blob(optional but recommended)

We also recommend to take the output of this command and store it in a variable for the next command.$temporaryVariable = Set-SPOMigrationPackageAzureSource….

Ex:$azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath C:\Users\Pictures\ -SourcePackagePath C:\Users\test\SPOpackage -AccountName mystoragename -AccountKey 000000000000000000000000000000000000000000000000000000000000000000000000000000== -AzureQueueName migrationqueue

PowerShellSet-SPOMigrationPackageAzureSource


Ship drive requirements

Max 6TB drives SATA II or III Format the drive ( not a requirement but still makes things easier) Prepare your container Install the Azure tool :

http://go.microsoft.com/fwlink/?linkid=301900&clcid=0x409

http://go.microsoft.com/fwlink/?linkid=301900&clcid=0x409

The last command will simply convert the XML generated into a format that the destination will understand. Here are the required parameters to enter (others are optional)

TargetwebURL : Point to the Web of the destination MigrationPackageAzureLocations : The temporary variable from the previous

command Credentials : SPO credential that has admin rights to the destination site

Ex:Submit-SPOMigrationJob -TargetWebUrl https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/ -MigrationPackageAzureLocations $azureSource -Credentials (Get-Credential [email protected])

PowerShellSubmit-SPOMigrationJob

https://datamigration-my.sharepoint.com/personal/admin_datamigration_onmicrosoft_com/


PowerShellRemarks for PowerShell

Export-SPWeb Target a List Uncompressed

New-SPOMigrationPackage Be careful of SPO limitations

ConverTo-SPOMigrationTargetedPackage Using different output folder is recommended

Set-SPOMigrationPackageAzureSource The Azure Containers will be required for the Submit-SPOMigrationJob

Submit-SPOMigrationJob The destination list needs to exist


Migration API


Building code for the Migration API

Uses PRIME package formatting Recommended to stay with what the PowerShell commands supports SPO Migration TAP program

[email protected] The Migration API is agnostic of the source

mailto:[email protected]


The Migration API calls[ClientNS.ClientCallableMethod] public Guid CreateMigrationJobEncrypted( Guid gWebId, string azureContainerSourceUri, string azureContainerManifestUri, string azureQueueReportUri, MigrationOption AES256CBCKey)

[ClientNS.ClientCallableMethod] public Guid CreateMigrationJob ( Guid gWebId, string azureContainerSourceUri, string azureContainerManifestUri, string azureQueueReportUri,)


Migration Job Status

[ClientNS.ClientCallableMethod] public Guid SPMigrationJobState (Guid MigrationJobId)


The encryption method should follow the AES CBC 256 Standard The IV should be different for every file including the manifests in the

package. should be stored as a metadata on each files.

Name = [IV] Value = [Base64Encoded byte array of the IV]

Extra Requirement for Encryption


Migration API EncryptedQueue

Here is the JSON content in the queue message{"Label", "Encrypted"},

{"JobId", "[JobId value]"},{"IV", ”[IV value in base64format]”}, {"Content", “[encrypted message in base64string]”}

Once decrypted the messages will be the same as the API without encryption.


Migration API Logs

[Date]Message[Date] [Action] message[Date] [item type] [SPObject name] [SPObject path] [SPObject ID] [Action] message

Note that there may be other structures, and in some cases message will overlap into multiple lines, such as


Optimizations

Parallel jobs Find the perfect package size

Number of items Weight of metadata Weight in GB over the wire

Single Report queue for every jobs Keeping every threads running No over provisioning


Please use Event Board to fill out a session evaluation.

Questions?

Thank you!


Sample Code

$userName = "[email protected]"$adminSite = "https://contoso-admin.sharepoint.com" $sourceFilePath = "D:\data\documents\"$packagePath = "d:\data\documentPackage"$targetWebUrl = "https://contoso.sharepoint.com/sites/finance" $spoPackagePath = "d:\data\documentPackageForSPO"$targetLibrary = "Shared Documents" $azureAccountName = "acctName"$azureAccountKey = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000=="$azureQueueName = "contosoMigrationQueue" $cred = Get-Credential $userNameNew-SPOMigrationPackage -SourceFilesPath $sourceFilePath -OutputPackagePath $packagePath ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl $targetWebUrl -SourceFilesPath $sourceFilePath -SourcePackagePath $packagePath -OutputPackagePath $spoPackagePath -TargetDocumentLibraryPath $targetLibrary -Credentials $cred$azureSource = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFilePath -SourcePackagePath $spoPackagePath -AccountName $azureAccountName -AccountKey $azureAccountKey -AzureQueueName $azureQueueNameSubmit-SPOMigrationJob -TargetWebUrl $targetWebUrl -MigrationPackageAzureLocations $azureSource -Credentials $cred

mailto:[email protected]

https://contoso-admin.sharepoint.com/

https://contoso.sharepoint.com/sites/finance