Strategies for a Successful Migration to SharePointRuss HoubergSenior Technical ArchitectKnowledgeLake

About Me• SharePoint Microsoft Certified Master since 2009• Migration Engineer for 7 years• Co-Developed KnowledgeLake Migration Framework in 2007• Almost 9 years at KnowledgeLake• Currently in the middle of a 70 million document migration.

Over 30 million documents migrated since July 1, 2013. (About ½ million per day)

Strategies for Successful MigrationWhy Migrate to SharePoint?Migration Concepts

• Setting Expectations• Migration vs Upgrade• Difficult (Expensive) Migration Tasks

Assessment• Know Your Data• Know Your Environment

Building a Migration Plan• Migration Order• Content Mapping• Primary/Delta Migration Strategies

Execution• Migration Management• Maintaining the Integrity of the

Migration• Exception Mitigation

Finalizing the Migration• Back Up Everything!• Validation• Reconciliation Report• SharePoint Crawls

SummaryQuestions?

Why Migrate to SharePoint?• SharePoint is Flexible and Scalable

• Can be adapted to service a wide variety of requirement.• Can scale to 100’s of millions of documents

• SharePoint has a Healthy Ecosystem• Virtually every .NET or JSCRIPT developer can adapt to SharePoint development• Someone will be there when you need to upgrade in 3, 5 or 10 years from now

• SharePoint has full disposition and records management capabilities• Record Centers and Document Centers for archiving content• File plans and retention rules can be implemented

Why Migrate to SharePoint?• SharePoint has a robust client side and server

side API• Multiple ingestion technologies to support rich

integration• Client side API supports cloud migrations

• SharePoint has been refined over 15 years• Roots go back to the 90’s• FrontPage Server Extensions and Microsoft Site Server• Evolved into a rich portal platform SP2007• Significant scalability improvements in the Service

Architecture with SP2010, SP1013 and Office 365

• You probably already own it!• Included in most Enterprise License Agreements (ELAs)

Migration Concepts – Setting Expectations

Migration is unpredictable in many ways…• Duration is virtually impossible to predict without live metrics• Migrations tend to run in the order that documents were created

• Older documents may be larger or smaller than newer documents

• Source system, migration farm, and destination system must always be in harmony during a migration• Overloading the source system• Overloading the destination system• Maintaining storage space, database data files,

and database log files

• Expectations must be set CAREFULLY!

Migration Concepts – Migration vs UpgradeMigration• Importing content and metadata into SharePoint from any legacy

document repository… even if that legacy document repository is SharePoint.

Upgrade• The process of updating an existing SharePoint installation to a newer major version

(SP2003 to SP2007, SP2007 to SP2010, SP2010 to SP2013, or SP2013 to vNext).

• In-Place Upgrade: Executing the next version SharePoint installer on an existing SharePoint farm, upgrading all SharePoint components in place. Pre-requisite requirements are very important

• Database Upgrade: Installing a greenfield, next version SharePoint farm, then attaching previous version content databases and/or service application databases. Content database / site collection preparation is very important.

Migration Concepts – Migration vs Upgrade

SharePoint Upgrade is recommended when:• Existing farm is SP2007 or higher (Content Types not available in

SP2003)

• Existing taxonomy is well formed

• Site collections and content databases are properly architected

• Content is properly distributed (Folder structures that are not too shallow)

• Site templates used include Team Site, Document Center, or Records Center

• Migrating from a given version of SharePoint to the next major version of SharePoint

Migration Concepts – Migration vs Upgrade

Migration is recommended when:• Source document repository is not SharePoint (SP2007 or higher)

• Moving up more than one major version (i.e. SP2007 to SP2013)

• Taxonomy / Information Architecture needs to be redesigned

• Content DBs need to be split to improve content database performance or to meet RTO (Recovery Time Objective) or RPO (Recovery Point Objective)

• Content is improperly distributed (Folder structures that are too shallow)

• Custom site template was used for document repository site collection, making upgrade overly-complicated

Polling Question #1

What do you consider to be a “large” document repository?

a) 100,000 documentsb) 1 million documentsc) 10 million documentsd) 100 million documentse) 1 billion documents

Migration Concepts – Difficult Tasks• Full Text PDF Reprocessing –

Custom code to manage conversion of image-only PDF to full text PDF• Very math (CPU) intensive process• On average, it takes 6 seconds per PAGE to

perform Advanced PDF Rendition on an image-only PDF (extremely time consuming)

• Recommend migrate PDF first then reprocess documents AFTER they’re already in SharePoint

Migration Concepts – Difficult Tasks• Version Migration

• Some source systems support branching (Hummingbird), SharePoint does not• Different migration code depending on Major / Minor version support. Custom exporter is always

required.• SharePoint storage projection becomes more complicated when versions are in play, particularly

with RBS enabled• Usually requires “Legacy Mode” migration as opposed to the much faster “Express Mode”• Migration performance is nearly impossible to predict, even with live metrics• Legacy API is often COM Interop based which can lead to memory leaks and exporter crashes.

Additional migration management time is often required

Migration Concepts – Difficult Tasks• Annotation Migration

• Most legacy systems store annotations in a proprietary format, often in a binary file that we can’t decode without significant proprietary knowledge or, in some cases, an API

• Most legacy systems layer annotations over the graphic using their viewer. The annotations aren’t embedded in the TIF / PDF

• In some systems, the only solution is to redact annotations permanently into the graphic during export

• In virtually all cases, a custom exporter must be coded to work with the source system API in order to properly migrate the annotations. No “Express Mode” migration. Slower.

• Legacy API is often COM Interop based which can lead to memory leaks and exporter crashes. Additional migration management timeis often required

• Significant analysis time is highly recommended when annotations are in absolutely required

• Note that often times annotations are only needed during initial document processing, can be dropped during migration saving time and money.

Assessment – Know Your Data

• Interrogate your data, understand your corpus• Document Classes / Metadata Columns• Counts by document class and by year (extrapolate load rate)• Document Types (PDF, TIF, Office Docs)• Collaborative documents (DOCX, XLSX)• Transactional documents (TIF, PDF)

• Versions? Annotations?• Identify Content for Migration

• Should you exclude certain content for compliance or other purposes?

Polling Question #2Does your organization typically generate more Collaborative documents (DOCX, XLSX)

or more Transactional documents (high volume TIF or PDF)

Assessment – Know Your Environment

• Legacy System• Database version• API or direct DB/File access (Express Mode)• Current Storage Requirements

• Destination SharePoint Environment• Database Server Resources (For Migration

and for SharePoint)• Storage Resources and Expandability• RBS vs SQL Storage• Access to run tools directly on SharePoint

Servers or to configure migration servers as SharePoint Servers (release locally)

Migration Plan – Migration Order

Department by Department• Smallest First, Medium First, Largest First

• Quick wins / ROI vs Proving the entire solution• Delta migration during cutover “weekend”

SharePoint Day Forward• New content released to SharePoint

• Single migration for legacy content• Often migrate newest legacy content first• Dual system interaction required until migration is complete

•AP•HR• IT

Small

•Development•Professional Services

•MarktingMedium

•SalesLarge

Migration Plan – Content MappingThis is a BA / TA extension of the data interrogation:• Identify all source fields and their data types

• Identify any data that needs to be cleaned up or converted into different (sometimes multiple) fields

• Design Information Architecture (Content Types & Site Columns) to contain destination content

• Determine destination web / site collection structure• Consider unique web application for migrated content in a

previously live system (Crawl Benefits)• Always migrated content to “sub” site collections in a managed path and

not a root site collection• Evaluate the impacts of RBS on the number of documents stored in a given site collection

• Use data interrogation information to extrapolate necessary SharePoint content database and migration database storage requirements. Comprehensive storage architecture recommended.

Polling Question #3Do You Know Your Data?

A. Yep, we have someone monitoring and maintaining our document management system(s) regularly

B. No. We really need to do a better job of understanding our data and/or our DMS environment.

C. We don’t have a document management system and we really need to get organized

Migration Plan – Migration StrategiesSample Migration• Small comprehensive sample that provides business users with a set of

documents that can be manually compared and validated

Primary Migration• Initial metadata dump and migration of documents that are part of the

captured metadata at that point in time

• If new Document Management System is live then there may only be a Primary Migration

Delta Migration• Any content that was added or modified since the time of the initial

metadata dump.

• Usually there is just (1) delta. Usually it’s a “just in time” migration over a weekend.

• Sometimes there are (2) deltas if the primary migration is extremely large and takes months to complete.

Execution – Migration ManagementTest, Test, Test! Before launching any migration “live”, ensure that a sample migration has been executed. Verify that all documents and metadata match EXACTLY between the source and destination system. Business Users / Power Users are the best people to sign off on the testing. Nobody knows the data better than they do!

SharePoint Environment Configuration• The migration process from start to finish should be audit logged for

EVERY document!

• Should not be crawling migrated content until after migration is complete

• Consider turning off “Full” database recovery on destination content DBs. Still perform nightly full backups, but “Full” mode will fill up your SQL logs quickly.

• Ensure that SharePoint WFE resources can handle end users and migration processes simultaneously

• Monitor SharePoint and SQL resources (CPU, RAM, Storage) consistently throughout the migration to ensure that resources aren’t being depleted unexpectedly

Execution – Migration ManagementSharePoint Environment Monitoring• Keep an eye on those content database and log files! Ensure

you’re not running them out of space!

• Monitor destination libraries carefully. You should NEVER migrate all of your content into the root. Are you seeing the document counts you expect?

• Does it seem like you’re migrating but documents are not showing up in the libraries? You may have a required column not being populated leaving the document checked out.

• Keep an eye on SQL I/O on a regular basis to make sure you’re not overtaxing your SQL resources and impacting end users.

Execution – Maintaining Migration IntegrityBe EXTREMELY Careful with…• Update SQL statements (usually executed during data scrubbing)

• Delete SQL statements (rarely a good reason to use them)

• Multiple document versions that should be treated as one document for the purposes of document count reconciliation. When executing SQL against multiple documents, versioned documents must be considered.

• Link Level Consistency – If a content database goes offline or RBS storage fails, the entire corpus must be scanned and re-validated and then compared with the records in the migration database. The migration database and the information that SharePoint reports MUST be in sync!

Execution – Exception MitigationWhen a document fails to upload or validate…• Don’t panic! It happens. Almost every migration has a subset of documents that will not

migrate for various reasons (more on this later).

• Wait until the migration is complete before retrying broken documents.

• Validate that the document is NOT in SharePoint (can be tricky). If it is in there then check the metadata. Migrate or mark as migrated as necessary.

• Investigate root cause and mitigate as necessary• Unexpected document types may not migrate into

SharePoint correctly (EXEs, DLLs, etc)• SharePoint content DB or log is out of space. Can’t upload anything.• SharePoint timeout (sometimes we just need to retry).• Data Type transformation issues will often cause upload errors

(Usually related to improperly formatted dates)

Finalizing – Back Up Everything!Do not pass GO, Do not collect $200, Stop Now and Back up!• SharePoint Content Database(s)

• Migration Database(s)

• Any virtual machines or physical servers used to execute the migration

These resources, particularly the migration databases containing the audit logs, are the key to any audit / compliance requests!

Also, if there are any documents that are problematic in the new system, the audit logs can be referenced to determine if there were any issues during the migration. Also the metadata can be used to re-migrate the document if necessary.

Finalizing – ValidationFor each migrated document…• Ideally, the migration solution should have a mechanism for

programmatically validating the documents that have been migrated. This should be the final status of any document.

• Some documents are modified by SharePoint after upload (Office Docs) and there can’t be compared programmatically (because SharePoint will change them immediately)

• We can disable SharePoint’s ability to change the document for a given site. But the change comes at the price of functionality. But it can be re-enabled after the migration!

Finalizing – Reconciliation ReportReconciliation Report• An accurate account of every document in the source system

and how it’s represented in the destination system

• Broken down by final document status (Validated, Failed, Excluded, etc.)

• Number should match EXACTLY with source system even if some numbers indicated “excluded” documents. The reason for exclusion should be indicated for audit purposes.

• Include the Reconcilation Report in the backup archive with the other migration resources.

Polling Question #4What is the #1 most important concept regarding migrations?

a) Accurately accounting for all documentsb) Validating that all documents and

metadata were successfully migratedc) Maintaining an accurate “chain of

custody” audit trail of the migrationd) All of the abovee) This all hurts my head. Is it time to go

home yet?

Finalizing – SharePoint CrawlsFinally…After the migration is complete and all content has been validated and reconciled…• Set up a full crawl to crawl all that content you

just migrated• Configure an incremental crawl to pick up any

new content.

Summary• Upgrade when possible, migration often necessary• PDF full text processing should be minimized whenever possible• Annotations and versions can be difficult (expensive) to migrate due to

proprietary storage of annotations• Source system COM Interop can cause problems• Know Your Data!• Plan your migration carefully.• Always ensure you have an audit trail for

compliance.• Reconcile EVERY document to prove

migration consistency.

Questions

Contact Informationsales@knowledgelake.com

Twitter @knowledgelake

https://www.facebook.com/knowledgelake

Thank you!