Retrieve HTTPS with HTTP retriever

jeremyh

Hi

I have a HTTPS datasource which I would like to use the HTTP retriever on.

Is it possible to retrieve a https:// url using this module?

I can use wget from the same Mango server to get the file successfully. If it's not possible to get Mango to retrieve the file then maybe I can write a script to do it or something.

Thanks!

terrypacker

I just tested it by retrieving data from https://google.com it seems to work fine for me. You may run into trouble accessing an un-trusted site. If that is the case we can add some extra code into the module to allow access to un-trusted sites.

jeremyh

HI Terry

Thanks for the reply. You are right, it does work with https. The site I am retrieving it from has an invalid SSL cert.

If it's not too much effort, it would be great if the module could be altered at a some point.

In the meantime I have used python to download the page/s I want and save them locally on a cron job, then I'm using lighttpd to serve them. Clunky but works. I am only updating every 5 minutes so this is not too taxing.

terrypacker

I've created a Git issue so we make sure to get this done:

https://github.com/infiniteautomation/ma-core-public/issues/421

jeremyh

Thanks Terry.

Just incase it helps anyone out, here is my python script which will iterate through an array of https URLs from Sunny Portal (useful for grabbing data from Speedwire/Webconnect-only PV plants):

#!/usr/bin/env python

import  urllib2

li = ['http://www.sunnyportal.com/Templates/PublicPage.aspx?page=xxx-xxx-xxx',     'https://www.sunnyportal.com/Templates/PublicPageOverview.aspx?page=xxx-xxx']

for s in li:

 file_name = "/home/mango/sunnyportal/"+ s.split('=')[-1] +".html"
 u = urllib2.urlopen(s)
 f = open(file_name, 'wb')
 meta = u.info()
 file_size = int(meta.getheaders("Content-Length")[0])
 print "Downloading: %s Bytes: %s" % (file_name, file_size)

 file_size_dl = 0
 block_sz = 8192
 while True:
   buffer = u.read(block_sz)
   if not buffer:
 break

   file_size_dl += len(buffer)
   f.write(buffer)
   status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
   status = status + chr(8)*(len(status)+1)
   print status,

f.close()

This is made executable (chmod a+x sunnyportal.py) and placed in /home/mango/sunnyportal.py and called by a Cron job every 5 minutes.

Lighttpd is installed and /etc/lighttpd/lighttpd.conf adjusted to serve the contents of the sunnyportal directory on an alternate port:

server.document-root        = "/home/mango/sunnyportal"
server.port                 = 8081

Finally the Mango datasource is then adjusted to point to: http://localhost:8081/xxx.html

JoelHaggar

Nice work, just as a thought you could probably use the ASCII file reader to parse the values out of the text file and then you don't have to service it up as a webpage.

jeremyh

Hi Joel,

I tried the ASCII source but couldn't get it to work easily (I have always struggled with it!) so gave up.

Anyway, at least this way when support is added for insecure HTTPs connections I can just change 'localhost' to 'sunnyportal' in the datasource, rather than migrating data out of and into different data sources.

Cheers

jeremyh

Actually, it seems that I overlooked something. The value that is found in the page's source code is not the same one that is displayed on the page when you view it in a browser, which is updated dynamically with JS.

Here is an example of a public page: https://www.sunnyportal.com/Templates/PublicPage.aspx?page=433f7377-fda7-4a12-a6dc-361b0aef84f3#

If you view-source you will see that the 'data-value' property of the span with class 'mainValueAmount' is not the same as the value shown on the website.

I ended up having to scale the recorded data-value by 3.27 to get it to match the actual value.

If you use a browser to inspect the span you will see that the real value is inserted in the span dynamically with Javascript.

I have no idea how to retrieve this value but would welcome any suggestions.

It would be a lot more useful to get this value as it updates about every 10 seconds, rather than every 30-60 minutes!

terrypacker

Jeremy,

I wasn't able to see the live data update, the value was just a dash and the gauge was at 0. But I did take a quick look at the javascript on that page. I'm pretty sure they are just doing a GET request to another URL and then using the information to update the page. Since the page wasn't updating I couldn't set a breakpoint and check what URL and parameters they are using.

If my assumption is correct you could probably access their API with that URL. I think the request should be getting made on line 231 in the LiveData.js file that is loaded for that page.

Let me know when that page comes back to life and I can take another quick look for you.

jeremyh

Hey Terry,

That was just a random public plant but it looks like it's been offline for some time.

Try this one:
https://www.sunnyportal.com/Templates/PublicPageOverview.aspx?plant=a3f3b48b-bb4d-4767-986b-e53b3cccf5c1&splang=

(It's in the Netherlands so is about 8 hours ahead of you)

Or this one if it's night-time there:

https://www.sunnyportal.com/Templates/PublicPageOverview.aspx?page=308c9591-e49c-40bd-a07b-f481ed0ad247&plant=3508870f-dda3-4a26-9350-a3378a869fb6&splang=en-US#

Not sure exactly how it works but it seems that when you load the page, the value is quite old, but if you leave the page open it will start to update very quickly, every 10 seconds or so.

If you can help at all that would be really appreciated!

Cheers
Jeremy

terrypacker

Jeremy,

I can't get either to go into 'Live Data' mode. Every time I go to the pages there is no live data available. I'm starting to wonder if you need to be logged in somehow?

I have traced the LiveData.js and powerWidget.js files to see how they work and can see what should be happening. Perhaps you can try this:

Once the page is loaded and the SMA module is installed with a user:

checkoutid: "5d87dfee-f5cb-4703-adce-24c916b9228e"
userid: "00000000-0000-0000-0000-000000000000"

It also always has the live object:

idleTimeoutMs: 3600000
isAuthorized: true
isAvailable: false
maxFailureCount: 10
requestIntervalMs: 5000

Since isAvailable is not true it won't ever make the live data request. I used a debugger to set it to true but I think since I'm not logged in the request doesn't actually return what it should (a JSON object), it instead returns a webpage.

The request looks like it should be to /Dashboard?<some parameters> and should return an object that looks like this:

Timestamp: some type of timetamp
PV: power value

My guess is that if you have the dashboard working and were to type this URL into your browser you should see a JSON object instead of the webpage I'm seeing.

Can you think of any reason that I would need to login for this?

jeremyh

Hey Terry

@terrypacker said:

Every time I go to the pages there is no live data available. I'm starting to wonder if you need to be logged in somehow?

It looks like you are right (once again). The URLs above weren't updating for me either, so I just tried logging in (with a private browser session) and navigating to one of our plants, then opening the 'PV system overview' page, and it does update every 10 seconds or so. When logged in, the 'PV system overview' page URL in the browser is https://www.sunnyportal.com/FixedPages/Dashboard.aspx

Without logging out, I can browse to the ../publicpageoverview.aspx/.. URL above and see the same page, but the data doesn't update and is ~30 minutes old. So it seems that you must be logged in to get the data dynamically updating, and even then, the page URL to access that is different. Sorry to send you on a wild goose chase.

Whilst logged in, I can see requests being made by sma.scripts.min.js to https://www.sunnyportal.com/Dashboard?t=1443694513741, and if I access that directly in the same browser session, this is returned:

{"__type":"LiveDataUI","Timestamp":{"__type":"DateTime","DateTime":"2015-10-01T10:20:24","Kind":"Unspecified"},"PV":25004,"FeedIn":null,"GridConsumption":null,"DirectConsumption":null,"SelfConsumption":null,"SelfSupply":null,"TotalConsumption":null,"DirectConsumptionQuote":null,"SelfConsumptionQuote":null,"AutarkyQuote":null,"BatteryIn":null,"BatteryOut":null,"BatteryChargeStatus":null,"OperationHealth":{"__type":"StateOfHealth","Ok":1,"Warning":0,"Error":0,"Unknown":0},"BatteryStateOfHealth":null,"InfoMessages":[],"WarningMessages":[],"ErrorMessages":[],"Info":{}}

For every couple of requests to t=nnnnnn, the nnnnn value increments by one, and always returns the current measurement. If I try https://www.sunnyportal.com/Dashboard?t=0 then I get the same JSON with the current measurement and timestamp.

I'm not really sure where to go from there - If you have any further ideas, I'd be happy to share my Sunny Portal login with you so you can poke around.

I found that someone has roughed up a node.js script: https://github.com/mkorthuis/sunnyportal-api
I might have a play with this to see if I can get it to write multiple plant data to a text file.

Kind regards
Jeremy

terrypacker

Jeremy,

Due to the 'login required before getting data' restriction you will have to create an intermediate step that would login, collect the data, then push it to Mango somehow. To my knowledge Mango doesn't have a module that would do this out of the box.

Some Ideas:

See if you can get/setup a public api for SunnyPortal
Write a small script/app in your favorite language that would retrieve the data as you mentioned above
Build a custom Mango Module to do this.