Last week I was researching throttling and performance issues for SharePoint Online migrations for a customer, which lead me to investigate if there were any updates to the High Speed Migration feature in Office 365. Since my last tests demonstrated that with parallelization is the way to maximize the performance of an Office 365 Migration, really though any migration benefits from this concept. What has changed since last summer is that now all Office 365 subscriptions now benefit from a baseline 1 Terabyte of storage plus the individual allotments per user. Looking to take advantage of the latest storage increase, I thought it would be suitable to test performance with a larger dataset. So that is how arrived at the 1TB Migration to SharePoint Online.
The methods I am using, haven’t changed since my last post about the parallel migration scripts, so I will not cover them here. Instead I’ll talk about the data I am using and the actual performance observed during the migration.
Where am I going to find 1TB of Documents?
This is the first question I asked myself, which lead me to explore available datasets to see if any would fit. For a moment I actually thought about converting the entire Wikipedia library to word format and uploading it to SharePoint Online. Since I’m not testing the readability of documents, Just the performance speed, I decided to leverage the same method I used before which generates large amount of “fake” documents. I still have Mikael Svenson (@mikaelsvenson) to thank for that little tool. It took me a full day to generate & copy 1TB of documents in my Azure VM, and eventually I filled up my data drive.
With 401k files across 95 Folders.
Each folder has just over 10GB of content with lots and lots of documents ranging from 1MB to 5MB. For this test, I chose to treat each folder as its own migration job and submit them independently at the same time.
I ran the packaging script which gave me a lot of CSV files which contain the upload details for each of the Migration Jobs. At this point I have all the files uploaded to azure, all the links with permissions to the different containers, all I need to do is submit the job to the Migration Service.
Which is what I did.
What did I expect?
At this point I started checking my SharePoint site to see how quickly the jobs would run against the server. I thought based on my previous tests that I would see a few jobs processing some of the data, each job performing on average the same as I could do with a 3rd party tool, just a bit better on the backend with the multi threading.
I was very wrong…
The import service ran an incredible 33 Parallel jobs against the same SharePoint Site Collection, the same folder really in the same document Library. This vastly exceeded my expectations of between 5-10 parallel jobs depending on the farm health.
AT my first checkpoint, I migrated almost 80GB of data in one hour… That’s about 80 x my experience using CSOM with one thread. That speed is incredible for an on premises migration let alone a cloud migration. In the end I migrated 788GB of content in just over 16 hours.
|End||4/1/2016 8:58 PM|
The SharePoint Online High Speed Migration Service is super fast. If you maximized your jobs by dispersing them across sites & libraries, you could migrate 2TB per day. Towards the ends of the test, performance dropped since there weren’t as many jobs running at the same time, and there were some errors during the migration. Several of the folders were missing content however, that’s likely due to the way i configured my packages. I had too many document per package and the service stops after 100 errors per job. I will try this again with some 3rd party tools to make sure the data has full fidelity.
This service is ready for prime time where throughput is concerned, speed will no longer be a limiting factor for SharePoint Online Migrations.