Browsed by
Category: Azure

Moving FTP Backup Files to SharePoint Online via Azure Logic Apps

Moving FTP Backup Files to SharePoint Online via Azure Logic Apps

I had the desire the other day to experiment with the Azure Web Jobs and see how they compare with some of the new functionality being rolled out. My idea was that I would take my site backups for the blog and copy them over to SharePoint for storage. Like many other techies, I host my own WordPress blog (which you’re reading hopefully). Luckily, I have the capability to generate periodic backups and I wanted to move these backups to a platform where I have more visibility. So, I though Azure Web Jobs would be the key to getting my datasets across to SharePoint Online. While this approach would have worked, when looking at the deployment options, I noticed that the available triggers and actions for the Azure Logic Apps have significantly increased. I already knew that it was easy to generate connections to SharePoint, however I was unsure if FTP connectivity was available. I dove in a little bit and noticed that not only is FTP available, SFTP (SSH) is available for Logic Apps. So now the “difficult” part, configuring the triggers and actions so my FTP backups would get copied. With a little bit of tinkering and luck, I was able to move the backups to SharePoint and I wanted to share that capability with anyone whom might need it.

So, if you have some files/backups available to you via FTP, here is the simple logic app to get your data moving to SharePoint Online. I assume at this point that you are familiar with the Azure Resource Manager, basic FTP functionality, and SharePoint Document Libraries.

First things first, let’s create a Logic App.

In the Azure Portal, add a new resource, and Logic Apps can be found under Enterprise Integration on the Marketplace menu

Give your Logic App a fancy name and reuse/create a resource group

And if all goes well, you’ll see the Logic App under your available resources

A quick note at this point, I chose the West US 2 region since it was close to the server being hosted. If you know where your server is hosted, the best practice recommendation is to create the resource in the closest data center.

Now, click into your newly created Logic App and you should be presented with a blank design surface option or one of many templates. In this case choose Blank Logic App.

We should have a blank template on which to create our new Logic Application.

Adding Some Logic

The first thing we need to do is got to the FTP Server and see what is in the directory, and if something has changed, we need to then trigger an action. So the first action we are going to insert in the blank design surface is the FTP action, if it is not one of the offered triggers by default, find it by simply typing FTP into the triggers and actions search box for the trigger to show up.

Another note, just because you choose FTP does not mean the connection will not use encryption. It specifies mainly the to be used by the API in the background. You’ll still be able to choose SSL encryption later.

On the FTP Configuration Screen, give the connection a name, and then specify the server details like URL/IP, Port, Username and Password. Then click Create.

You can see that there is still is an option to enable SSL even with the normal FTP protocol.

Use the FTP Folder Picker to choose the backup directory or the directory with the files you want to add. Make sure Include File content is set to true and then choose the interval you want your logic app to check the server. I chose three minutes so we’re not hammering the server checking for new files. This completely is dependent on your needs though. If you want to, this would be a great time to save your app before moving on to the next step. The save button is located on the toolbar in the top left corner.

Once saved, choose New Step and then Add an Action

In the list of actions, search for SharePoint and then choose create a file when it appears.

If you haven’t previously granted access to Azure to any SharePoint Sites, it will prompt you to allow the app to connect to SharePoint. So, click Sign in when prompted.

Once you sign in to Office 365 and grant the service access to your site, there should be a drop down where you choose your Site Collection and Subsite where you want to target the files. I chose my base Site Collection in the screenshot. In addition, “Folder Path” is the Document Library where you want to move the datasets, and it needs to be created in advance of the logic app executing for the first time.

The other two parts are inputs from the trigger, and will show as such in the dialog box when you click in File Name and File Content. Choose the same input options from the dialog on the right to keep the same filename and path. Alternatively, you could change the filename in an intermediate step, or depending on the filename, back it up to a different SharePoint site. However, we’re going to keep this example simple and only backup to one document library.

Now click save in the upper left and then click run to execute the logic app in a testing phase. Beware though, with FTP the trigger can run into situations with long running processes and get stuck. It may be better just to manually trigger the app in the Overview blade.

Once testing has begin, there will be a message that tells you to modify the content in your ftp server such as below.

Now, in order to see the application actually run, you need to put a file in the folder that the Logic App is checking. In my case, I just started another backup process that would place a new backup in the directory I specified.

Due to the interval that we set, we now have to wait and see that the process picks up the file from the directory, or that it doesn’t at all. Be careful that you don’t set the interval too long in the testing phase, since the logic app testing cycle has a timeout period. You’ll also see notifications like below, that demonstrate the app is “testing”

Once the application has successfully run, you should see your files begin to appear in the SharePoint Document Library

Now you can schedule this job to run however frequently you want, or you can always just trigger it manually in the portal if the files/backups aren’t automatically triggered.

Wrapping Up

The best part, is that not only can you create a file in SharePoint, you can target several other platforms to move the data. You can use Box, Dropbox, a File System, move it to another FTP, Google Drive, One Drive, or One Drive for business. If you have any other needs for platforms, you can create a custom activity and target the platform that you want.

The Azure Logic Apps platform is a powerful way to extend functionality without writing a ton of code. Logic Apps are flexible and practical, it makes Out of the Box configurations almost as powerful as custom coding. Please let me know what you think of this tutorial or if you have any questions on the steps within.

How Much Does a SharePoint High Speed Migration Cost?

How Much Does a SharePoint High Speed Migration Cost?

In a previous post, I conducted a terabyte migration to SharePoint Online. The premise was to discovery how fast you can perform a migration to SharePoint Online from a file share, or on premises data source. A byproduct of this migration though, is that it gave me a general idea of what a migration will entail when it comes to costs. I waited till the end of the billing cycle to see how much the storage, network, and transactional costs would be for the overall migration. One of these costs definitely surprised me…

Read More Read More

SharePoint Online: High Speed Migration Resources

SharePoint Online: High Speed Migration Resources

Please find a growing list of resources below, which I will update as information gets released about the Import Service for SharePoint Online. [Please click through to see the tables link]

Resources - Microsoft Articles

List of Microsoft Articles about High Speed Migration
TitleUpdatedLink
SharePoint Online and OneDrive Migration Content RoadmapAug 7th, 2015https://technet.microsoft.com/en-us/library/mt203955.aspx
SharePoint Online and OneDrive Migration User GuideDec 11th, 2015https://technet.microsoft.com/en-us/library/mt203923.aspx
Import SharePoint Data to Office 365Apr 4th, 2016https://technet.microsoft.com/library/mt210445.aspx
SharePoint Online Migration API User Guiden.d.https://support.office.com/en-us/article/SharePoint-Online-Migration-API-User-Guide-555049c6-15ef-45a6-9a1f-a1ef673b867c?ui=en-US&rs=en-US&ad=US

Read More Read More

300GB Test3 Migration Results via SharePoint Online High Speed Migration (Small Files and Parallel Jobs)

300GB Test3 Migration Results via SharePoint Online High Speed Migration (Small Files and Parallel Jobs)

Updated Aug 19th: Quick Synopsis 25 hours @ 12.6GB/hr

Individual performance tests below,

Test 1 – Serial – 300GB of Large (video) files [here]
Test 2 – Serial – 300GB of Small (document) files [here]
Test 3 – Parallel – 300GB of Small (document) files [this post]

The last two tests I ran against the SharePoint Online High Speed Migration service were both Serial jobs, meaning that the files were processed one by one in the single solitary migration package. This allowed for simplicity and debugging of the jobs across the environment, there was a single files container in azure and a single package container as well. Even more simply there was one migration guid to track on the logs or in the migration queue.

What I decided to do following the last test was to create several migration packages from my file share and upload them all to Azure. I would then launch/submit all the jobs at once to get a gauge of how quickly the parallel jobs could work. The complexity comes from managing several containers per migration jobs, which can bog down any Azure storage explorer tool. The additional problem is that I needed to separate the upload & submission steps  from each other so I could finish uploading all the packages prior to submitting the jobs for ingestion into SharePoint.

I ended up creating a Powershell script that would treat each top level folder of a file share as its own individual package and create its own file and package containers in azure for that folder to use. I will share this script in another post where I can explain how the batching process I used works in detail.

small file migration resultsAs you can see in the file properties view, I have 31 different folders in my file share, which means that I’ll have 62 different containers in azure that will be used. Also since this is a semi-controlled test I know that each of the folders has 10.2GB of data for a total of 316GB overall.

small file migration results parallel

You can see here that the each of the folders now has a related package & file container. Please note though that the folder numbers do not necessarily correlate to the folders listed in the fileshare as seen below.

This is just a small result of one of the packaging processes, however it does let you track which folder links up to which containers in Azure in case you do need to troubleshoot any issues.

small file migration results parallel2

Another addition was the export of the results from the job upload. Its important to keep these results since they have the SAS URIs generated by the tool. Exporting these URIs to the files system lets me submit the jobs separately from the upload.

 

small file migration results parallel3You can see that this exported Powershell object contains all the URI’s generated from the tool. You would think that if you could export the object to a file, that you could import the object as well from a CSV? unfortunately that doesn’t work with this specific type of object using Import-CSV/Export-CSV, however you can reference the URIs manually when submitting the Job for ingestion.

Its evident that it took a bit more thinking to create and submit these jobs in parallel, however not so much that the process is unrepeatable. Once I confirmed that all the packages were uploaded and ready I submitted the job and waited for it to finish and below are the results.

 

Start Time:8/5/2015 7:23 AM (PST)
End Time: 8/6/2015 8:31 AM (PST)
Elapsed: 25 hours and 8 minutes

Migration Rate 12.6GB/HR (316GB/25.133 hours)

Its important to note that this is more than ten times what I have seen in migrations for small files during migrations, and for 1TB of data it would only take 80 hours to migrate.

300GB Test2 Migration Results via SharePoint Online High Speed Migration (Small Files)

300GB Test2 Migration Results via SharePoint Online High Speed Migration (Small Files)

Updated Aug 19th: Quick Synopsis 97 hours @ 3.26GB/hr

As a followup from my last post regarding the migration speed during a high speed migration, Mikael Svenson (@mikaelsvenson) pointed out the dataset that I was using was comprised of very large video files. He postulated that that smaller files would slow the migration process down during the import.

The problem I had with this is I didnt have a good way to generate the test data for a test migration That is until he offered to create me a tool which would generate test data.

So without further ado, my new dataset for testing:

small file migration results

What you see here is 31 folders of documents with 131k files ranging in between 1Mb and 5MB with the overall dataset size of 316GB. A big factor that affects this test is that all of these folders and documents are part of one package (think serial) and thus would not take advantage of parallel processing. (more to come on that soon).

In addition I wont cover here the individual steps I took to package and ship this information to Office 365, you can find more information about that here.

The DataSet

  • Size: 316GB
  • Files: 130,862 Files and 31 Folders
  • Target: One Drive for Business on Office 365

The Results

After submitting the jobs for migration I saw documents start being created almost immediately after the job was submitted at 5:20 PM on 7/20/2015. The last file was migrated on 7/24/2015 at 6:24PM.

  • Duration: 4 days, 1 hour, and 4 minutes
  • Speed: 3.26GB/Hr (316GB/97.06666666666667

In the end the migration speed was cut in half from the large speed migration, however this is still more than 3GB an hour is not bad at all for a small file comprised dataset.