Migrating from Subversion to Perforce:

The methodology and the migration tool

Vitalii Pokrovskii and Mark FridrichBlue Coat Systems, Inc.http://www.bluecoat.com

Outline

►Why is migration necessary?► Subversion (SVN) to Perforce (P4) migration

objectives► The method► Implementation details► The migration tool: svn2p4► Practical recommendations►Q & A

The Need for Migration

► Three dozen of revision control tools available► One can choose from both open-source and proprietary

systems► Solutions are available to fit any taste and budget

► So, why is migration required?► Not all systems are created equal, engineering organizations

are always looking to improve their code management practices

► Requirements change, the organization outgrows the original system

► Need to unify two or more different systems as a result of corporate restructuring, merger, or an acquisition

Migration Requirements

Options:► Import a head snapshot of SVN repository (not a solution).► Migrate the entire repository with the full history of how it evolved,

from day 1 to present.Requirements:

► General-purpose (hopefully easy to follow) migration process capable of handling SVN repositories of arbitrary size and complexity.

► Preserve natural change order, branches, tags and integration history; no need to preserve original dates.

► Good performance, yet► Eliminate the need to do everything in one take (cutover weekend)

by supporting incremental (re-startable) migration.► Optimize and clean up the repository as it is being migrated by

supporting redundant node analysis and exclude lists to filter out unwanted contents.

► Extensibility, provide for handling of unique and unforeseen cases.

Migration Flow

Solution Outline

SVN and P4 side-by-side comparison:► Terminology► Functionality► What can and cannot be migrated

SVN to P4 conversion:► Use data presented in SVN revision log as basis for individual SVN

changeset migration.► “Sync-and-Replay” mechanism i.e.

► Check out single SVN revision into a local working copy► Translate SVN change into corresponding P4 change► Submit the change to Perforce

Mappings that are required to support this solution:► SVN to P4 functional mapping► SVN to P4 repository mapping► SVN to P4 revision mapping

Redundant node analysis and the exclude lists.

Comparing SVN and P4

Comparing SVN and P4 (cont.)

Comparing SVN and P4 (cont.)

Bottom line: Some of the properties can be automatically migrated or set after the migration. However, as a general case the list of SVN properties is site-specific and certain properties may have no direct analogs in P4.

SVN repositories used as case studies

*The* Subversion repository:► Complete migration, node dependency analysis, performance,

exclude lists

Apache Software Foundation (ASF) repository:► Node dependency analysis

► Processing the complete revision log (~450 MB of data)

Example of SVN revision log

Changes in this revision:- Edit file /README.- Add directory /developer-resources.- Branch difftools and guis folders to /developer-resources from

/trunk/notes @ revision 20617.- Replace some files in the folders with newer source @ revision 20619.- Delete folders difftools and guis at original location.

SVN P4 functional mapping

SVN P4A (add file)A (add directory)

p4 add <filespec>p4 add <filespec> (recursive)

M (edit file)M (edit directory)

p4 edit <filespec>No action required, the change only affects svn properties.

D (delete file)D (delete directory)

p4 delete <filespec>p4 delete <filespec> (recursive)

R (replace file)R (replace directory)

[ p4 delete <filespec>; p4 add <filespec>; ] p4 edit <filespec>

A (branch/tag from)A (copy from)R (replace from)

p4 integrate [–d –f] <from>@<p4-rev*> <to>;p4 resolve –at; p4 edit <to>

* Derived from svn/p4 revision map

SVN P4 functional mapping (cont.)

SVN P4A (tag from) p4 sync <filespec>@<p4-tag-rev*>

p4 label <tag-name>p4 labelsync –l <tag-name>

* Derived from svn/p4 revision map

A (add file to tag)M (edit file in tag)

A (add file to tag from)R (replace file in tag from)D (delete file from tag)

No solution (cannot add to label without adding to source tree first); same with edit.

p4 labelsync –a –l <tag-name> <filespec**>@<p4-rev*>p4 labelsync –l <tag-name> <filespec**>@<p4-rev*>p4 labelsync –d <tag-name> <filespec>@<p4-rev*>

** Potential problem: <filespec> may be outside of intended label view.

Example of SVN/P4 repository mapping:http://svn.collab.net/repos/svn

SVN P4

/trunk

/branches

/tags

/.svnignore

/COMMITTERS

/README

/modules

/developer-resources

/svn-logos

//depot/svn/main

//depot/svn/branches

//depot/svn/tags

//depot/svn/misc/.svnignore

//depot/svn/COMMITTERS

//depot/svn/README

//depot/svn/modules

//depot/svn/misc/dev-resources

//depot/svn/misc/svn-logos

SVN to P4 revision mapping

Practical rule:Every SVN revision converts to a single P4 revision (1:1 mapping)

How to handle special cases where migrating a SVN revision does not require P4 change?

Examples:► SVN revision is empty or it only modifies node properties.► Migrating SVN tag to P4 label does not require P4 check-in

Practical solution:Check into Perforce and modify a dummy file called .svn2p4 File location is defined as part of migration setup.

Summary of SVN changes that resulted in dummy P4 check-ins

Redundant Node Analysis

► Migration is a good opportunity to review existing repository contents and get rid of old “junk” such as dead branches, abandoned sandboxes, release tags with past expiration dates, etc.

NOTE: Consult with your Legal Department before permanently deleting anything.

► To make sure skipping certain contents during the migration doesnot adversely affect file integration history (and thus breaks the migration), the tool helps you identify redundant nodes. An entire revision log is processed to identify childless nodes, using thefollowing criteria:► Node is not a source of changes for other nodes► Node is only a source to itself► Node is a source to a deleted node that is not a source

► Skip the nodes that you think are redundant by adding them to the exclude lists:► Exclude nodes (files, directories, branches and tags)► Exclude revisions

svn2p4

► The migration tool is a single Perl script (svn2p4.pl) that contains both configuration data and actual code.

► Two execution modes:► Redundant node analysis (read and analyze the revision log, generate a

list of redundant nodes)► Migration proper (migrate SVN repository to P4 starting with revision 1)

► Runtime behavior:► The script writes its log to svn2p4.pl.log (old logs are moved to the logs

directory)► Maintains SVN/P4 revision map in svn2p4_rev_map.csv.► Incremental migration: The script can be stopped and re-started anytime.

► Availability: Download the script from public.perforce.com:1666//guest/vitalii_pokrovskii/svn2p4

Practical Recommendations

Consider two migration plans:► Gradual process (no rush, migration takes days or weeks),

incremental migration running as a background process. Things to consider:► Make sure there’s no performance degradation for your production

Perforce instance (migration process may generate a steady flow of check-ins committed in 2 second intervals)

► Decide on the cutover date when the last portion of SVN revisions is migrated and engineers permanently switch to Perforce.

► Cutover weekend. Migrate and switch to Perforce over a weekend. Things to consider: Performance! Two key factors:► Network latency:

► A latency < 1 ms for SVN and Perforce servers is ideal ► 5-10 ms is acceptable (migrates 10-15 revisions per minute)► >100 ms is likely too slow. Move SVN repository to local network.

► Subversion performance:► Upgrade Subversion (server and client) to version 1.4.3 or higher.

Q & A