Office 365 SharePoint Parallel High Speed Migration Powershell Example Scripts

Office 365 SharePoint Parallel High Speed Migration Powershell Example Scripts

Last week there was an announcement from the Office 365 team regarding the High Speed Migration Preview. It was first announced back in may at Ignite (First Look article here) and has been in trials by several ISVs as well as individuals (Find the announcement here). Directly linked from this announcement is an article talking abit more technically about the Migration Process that circumvents the CSOM throttling. This article (link) gives some observed performance numbers from .5GB/HR all the way up to 15GB/HR, I have seen these performance numbers and more (Performance Summary here) and what remains is the importance of parallelization of the migration jobs. The single threaded performance is impressive especially when migrating simple file shares however for complex migrations parallelization is key.

So I want to share the way I achieved parallelization during the migration tests I did, so the scripts can be modified/reused. The scripts I wrote point to a file share and gets a listing of the top level directories for that file share. Each Top Level directory then becomes its own migration package and then is submitted separately to the high speed migration ingestion service.

Parallel migration processFurthermore the migration process can go two ways, you have the option of doing all for steps at once (Package, Target, Upload, & Submit) or you have the option of delaying the migration job submission till later such as when all the packages have been uploaded Stage 1 (Package, Target, & Upload) Stage 2 (Submit). This can be useful if you have multiple packages that all need to be uploaded to azure for performance reason or if you want to run the jobs over the weekend for instance, the reason behind it depends.

With a basic understanding of the process,, please find the first stage powershell script below:

In addition to creating packages from each of the top level folders, the script will output the results from the upload of the files to Azure Storage. Its useful to capture the results of the upload in case the submission of the job needs to happen in a separate stage like this. Also good to keep a record of each of the packages which are uploaded, which is why you will find the transcript from the session as well which describes which package goes where in azure.

Well once everything is uploaded you can use the following script to submit the migration jobs to the High Speed Migration ingestion service:

This script takes input when run which is the prefix of the files output from the previous script, should be named something like 123456789.Package.csv. The prompt is looking for the “123456789” prefix from these files to identify which group of jobs to submit. If you don’t want to submit the packages, simply change the name of the file or add a prefix.

Hopefully these scripts can get someone started on their own migration to Office 365. You can find the download links below for the scripts.

Stage 1 – Upload Documents Parallel Generic

Stage 2  – Submit Migration Jobs Generic

Please rename the txt extensions to ps1

One more thing to note is that you should take advantage of the free storage offered by the Microsoft team via the import service, that way you wont have to eat the Azure storage fees. You can get your key and storage account via the Import option in the Office 365 Administrator Center.

2 thoughts on “Office 365 SharePoint Parallel High Speed Migration Powershell Example Scripts

  1. I am getting the error “No Files found with that prefix”, but I have copied the prefix from the CSV files created in my documents folder. Is there something special I need to do to make that work?

    1. Hi Bobby, Sent you an email with a question about your naming convention for the files. If you could send me a screenshot, it would be helpful.

      Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *