Updated Aug 19th: Quick Synopsis 25 hours @ 12.6GB/hr
Individual performance tests below,
Test 1 – Serial – 300GB of Large (video) files [here]
Test 2 – Serial – 300GB of Small (document) files [here]
Test 3 – Parallel – 300GB of Small (document) files [this post]
The last two tests I ran against the SharePoint Online High Speed Migration service were both Serial jobs, meaning that the files were processed one by one in the single solitary migration package. This allowed for simplicity and debugging of the jobs across the environment, there was a single files container in azure and a single package container as well. Even more simply there was one migration guid to track on the logs or in the migration queue.
What I decided to do following the last test was to create several migration packages from my file share and upload them all to Azure. I would then launch/submit all the jobs at once to get a gauge of how quickly the parallel jobs could work. The complexity comes from managing several containers per migration jobs, which can bog down any Azure storage explorer tool. The additional problem is that I needed to separate the upload & submission steps from each other so I could finish uploading all the packages prior to submitting the jobs for ingestion into SharePoint.
I ended up creating a Powershell script that would treat each top level folder of a file share as its own individual package and create its own file and package containers in azure for that folder to use. I will share this script in another post where I can explain how the batching process I used works in detail.
As you can see in the file properties view, I have 31 different folders in my file share, which means that I’ll have 62 different containers in azure that will be used. Also since this is a semi-controlled test I know that each of the folders has 10.2GB of data for a total of 316GB overall.
You can see here that the each of the folders now has a related package & file container. Please note though that the folder numbers do not necessarily correlate to the folders listed in the fileshare as seen below.
Current Path: \\hsmigration\Documents\Documents13
Package Path: \\hsmigration\MigrationPackage\PDocs\Documents13
Converted Package Path: \\hsmigration\MigrationOutput\PDocs\Documents13
Target Path: DocsParallel\Documents13
Running New Migration package
Converting Migration Package
Uploading Migration Package
Exporting Results from Upload
This is just a small result of one of the packaging processes, however it does let you track which folder links up to which containers in Azure in case you do need to troubleshoot any issues.
Another addition was the export of the results from the job upload. Its important to keep these results since they have the SAS URIs generated by the tool. Exporting these URIs to the files system lets me submit the jobs separately from the upload.
You can see that this exported Powershell object contains all the URI’s generated from the tool. You would think that if you could export the object to a file, that you could import the object as well from a CSV? unfortunately that doesn’t work with this specific type of object using Import-CSV/Export-CSV, however you can reference the URIs manually when submitting the Job for ingestion.
Its evident that it took a bit more thinking to create and submit these jobs in parallel, however not so much that the process is unrepeatable. Once I confirmed that all the packages were uploaded and ready I submitted the job and waited for it to finish and below are the results.
Start Time:8/5/2015 7:23 AM (PST)
End Time: 8/6/2015 8:31 AM (PST)
Elapsed: 25 hours and 8 minutes
Migration Rate 12.6GB/HR (316GB/25.133 hours)
Its important to note that this is more than ten times what I have seen in migrations for small files during migrations, and for 1TB of data it would only take 80 hours to migrate.