In a previous post, I conducted a terabyte migration to SharePoint Online. The premise was to discovery how fast you can perform a migration to SharePoint Online from a file share, or on premises data source. A byproduct of this migration though, is that it gave me a general idea of what a migration will entail when it comes to costs. I waited till the end of the billing cycle to see how much the storage, network, and transactional costs would be for the overall migration. One of these costs definitely surprised me… …
Please find a growing list of resources below, which I will update as information gets released about the Import Service for SharePoint Online. [Please click through to see the tables link]
Resources - Microsoft ArticlesList of Microsoft Articles about High Speed Migration
|SharePoint Online and OneDrive Migration Content Roadmap||Aug 7th, 2015||https://technet.microsoft.com/en-us/library/mt203955.aspx|
|SharePoint Online and OneDrive Migration User Guide||Dec 11th, 2015||https://technet.microsoft.com/en-us/library/mt203923.aspx|
|Import SharePoint Data to Office 365||Apr 4th, 2016||https://technet.microsoft.com/library/mt210445.aspx|
|SharePoint Online Migration API User Guide||n.d.||https://support.office.com/en-us/article/SharePoint-Online-Migration-API-User-Guide-555049c6-15ef-45a6-9a1f-a1ef673b867c?ui=en-US&rs=en-US&ad=US|
Yesterday a conversation came up on the Office 365 Network’s SharePoint Online Migration Group, stating that the import service had been disabled.
Sure enough when checking the link provided, it takes you to the the updated Import SharePoint data to Office 365 page (Updated April 4th), which up to a few days ago had instructions on how to ship your SharePoint Data to Microsoft (Updated March 1st).
Also visible in the Office 365 Administration Portal under Import, you can see the same message.
While the import service is suspended, Migrations using the API & PowerShell commands still work. It is likely that the product management team finally reached the speeds they need for network migrations and decided that the Drive Shipping Service/Front End Service was no longer necessary. We will find out more information when they publish updated information.
Last week I was researching throttling and performance issues for SharePoint Online migrations for a customer, which lead me to investigate if there were any updates to the High Speed Migration feature in Office 365. Since my last tests demonstrated that with parallelization is the way to maximize the performance of an Office 365 Migration, really though any migration benefits from this concept. What has changed since last summer is that now all Office 365 subscriptions now benefit from a baseline 1 Terabyte of storage plus the individual allotments per user. Looking to take advantage of the latest storage increase, I thought it would be suitable to test performance with a larger dataset. So that is how arrived at the 1TB Migration to SharePoint Online.
The methods I am using, haven’t changed since my last post about the parallel migration scripts, so I will not cover them here. Instead I’ll talk about the data I am using and the actual performance observed during the migration.
Where am I going to find 1TB of Documents?
This is the first question I asked myself, which lead me to explore available datasets to see if any would fit. For a moment I actually thought about converting the entire Wikipedia library to word format and uploading it to SharePoint Online. Since I’m not testing the readability of documents, Just the performance speed, I decided to leverage the same method I used before which generates large amount of “fake” documents. I still have Mikael Svenson (@mikaelsvenson) to thank for that little tool. It took me a full day to generate & copy 1TB of documents in my Azure VM, and eventually I filled up my data drive.
With 401k files across 95 Folders.
Each folder has just over 10GB of content with lots and lots of documents ranging from 1MB to 5MB. For this test, I chose to treat each folder as its own migration job and submit them independently at the same time.
I ran the packaging script which gave me a lot of CSV files which contain the upload details for each of the Migration Jobs. At this point I have all the files uploaded to azure, all the links with permissions to the different containers, all I need to do is submit the job to the Migration Service.
Which is what I did.
What did I expect?
At this point I started checking my SharePoint site to see how quickly the jobs would run against the server. I thought based on my previous tests that I would see a few jobs processing some of the data, each job performing on average the same as I could do with a 3rd party tool, just a bit better on the backend with the multi threading.
I was very wrong…
The import service ran an incredible 33 Parallel jobs against the same SharePoint Site Collection, the same folder really in the same document Library. This vastly exceeded my expectations of between 5-10 parallel jobs depending on the farm health.
AT my first checkpoint, I migrated almost 80GB of data in one hour… That’s about 80 x my experience using CSOM with one thread. That speed is incredible for an on premises migration let alone a cloud migration. In the end I migrated 788GB of content in just over 16 hours.
|End||4/1/2016 8:58 PM|
The SharePoint Online High Speed Migration Service is super fast. If you maximized your jobs by dispersing them across sites & libraries, you could migrate 2TB per day. Towards the ends of the test, performance dropped since there weren’t as many jobs running at the same time, and there were some errors during the migration. Several of the folders were missing content however, that’s likely due to the way i configured my packages. I had too many document per package and the service stops after 100 errors per job. I will try this again with some 3rd party tools to make sure the data has full fidelity.
This service is ready for prime time where throughput is concerned, speed will no longer be a limiting factor for SharePoint Online Migrations.