Uploading data to off-site URL

[MStangl]

I am looking for an way to automatically upload historic data values to a remote URL, ie an FTP server on a scheduled basis (weekly, monthly, etc.). I know that you can manually download from the Watchlist in JSON or CSV formats. Also that you can email or download xlsx reports from the Excel Reports module. How do you automatic this process so that the data get pushed to an off-site server on a routine basis?

[phildunlap]

Hi Mstangl,

Can you say more about what specifically you're trying to copy to another place? Just the data? To a webserver that's going to digest it or to a filesystem to store it?

The most general purpose answer is to invoke a script from the operating system's crontab. That script could FTP (or whatever else) to move run Excel Report instances from Mango/web/modules/excelReports/report-data for instance. It could invoke a client like IDrive to backup some particular set of directories (what you'll find on a Mango ES), or it could make API calls and then forward that data someplace.

You could alternatively have a script on some other machine SCP'ing (or whatever) or hitting the API on a cron job (Scheduled Tasks in Windows parlance).

Mango could also be sending the data somewhere as it's collecting it, via an HTTP Publisher, for instance.

[MStangl]

Phil,

I just trying to set up the MangoES to be an FTP server. Then I want to be able to upload the actual data files over the Internet using a remote FTP client. I am looking for two things, 1) a set of simple instructions on how to do this with my MangoES and 2) what will be the data format (CSV, XML, etc) of the uploaded files?

Thanks,

Mike

[Jared Wiltshire]

@MStangl you don't need to setup a FTP server, I would recommend against it. You can connect directly to the MangoES using SFTP which is built in. You can test out connecting to it using this great open source client - Filezilla.

Setup the connection like this-

That's just some general information on why not to setup a FTP server on your MangoES.

Your "offsite server", what OS is it running? Are you familiar with scripting and cron jobs? Are the MangoES devices publically accessible on the internet? Do you know their IP addresses at all times or do they have dynamic IP addresses?

[MStangl]

Jared,

The MangoES will be connected to the off-site server through a VPN over the Internet. The MangoES is on a private LAN at a remote site. The MangoES has a static IP address. The off-site server will also have a static IP address. The off-site server will be using Windows 10 Pro.

Complete understand your comments regarding FTP vs SFTP connections. I am not familiar with scripting and cron, so any references would be appreciated.

Thanks,

Mike

[Jared Wiltshire]

@mstangl said in Uploading data to off-site URL:

The MangoES will be connected to the off-site server through a VPN over the Internet. The MangoES is on a private LAN at a remote site. The MangoES has a static IP address. The off-site server will also have a static IP address. The off-site server will be using Windows 10 Pro.

OK so sounds like you would be best of running a script on the Windows machine which grabs the data from the REST API as a CSV. I'll get back to you with more details.

[Jared Wiltshire]

@MStangl

You can create a PowerShell script which you schedule on your Windows server.

I started with a snippet created by the Chrome DevTools, you go to the network tab and find the API request and copy the network request as shown below.

You just need to create an authentication token (on your users page) and add it as a header as shown below.

Invoke-WebRequest `
	-Uri "http://yourmangohostname:8080/rest/v2/point-values/single-array/time-period" `
	-Method "POST" `
	-Headers @{"Accept"="text/csv"; "Authorization"="Bearer yourlongtokenthatyougenerated"} `
	-ContentType "application/json;charset=UTF-8" `
	-Body "{`"dateTimeFormat`":`"yyyy-MM-dd'T'HH:mm:ss.SSSXXX`",`"from`":`"2018-10-23T21:54:58.951Z`",`"to`":`"2018-10-24T21:54:58.952Z`",`"bookend`":true,`"timezone`":`"UTC`",`"fields`":[`"TIMESTAMP`",`"VALUE`"],`"xids`":[`"DP_9004cb57-f1cb-4415-a078-b60999f06c4d`",`"DP_cd2587fa-93ab-492a-9963-e9f5c4e0a40d`",`"DP_915fb444-6133-45eb-a700-4f415662e108`",`"DP_506cab2d-8f10-46ca-90aa-ff9d8f6c0400`"]}" `
	-OutFile "myData.csv"

You will just need to modify the script so that it gets data relative to the time it runs at and names the output file accordingly. And also obviously make it get the XIDs you are interested in.