Import values from python script?

[Holzatelier]

Now this is may be a bit far out but I thought i give it a shot.
I have a Pythonscript that runs in the Linux-console. It reads values from a heatpump. The heat pump controller sends the values in the array [0,0,0.... ]. The values after that (in round brackets) are calculated according to the position of the value in the array.

Is it somehow possible to get these values automatically at an intervall of say 5 minutes into Mango datapoints?

(my-env) ubuntu-user@ubuntu-workstation:~/environments$ python test.py
Calculated
1
183
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 288, 274, 600, 295, 347, -56, -56, 750, 500, 255, 199, 750, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 24496366, 6981, 25406409, 6980, 2, 0, 0, 30438493, 30438493, 0, 0, 0, 0, 0, 0, 0, 0, 8665266, 354897, 0, 0, 0, 2, 3, 3, 86, 49, 46, 55, 48, 0, 0, 0, 0, 0, -1062731495, -256, -1062731265, 0, 1418949039, 1418809861, 1399258403, 1399258373, 1399258044, 752, 752, 757, 721, 715, 5, 3, 3, 3, 3, 3, 1526772832, 1526787615, 1526801426, 1526735908, 1526755311, 0, 1, 0, 4, 8665266, 0, 0, 0, 3, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1535466596, 1, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

('Temperatuur buiten : ', '-5.6')
('Temperatuur voorloop : ', '28.8')
('Temperatuur naloop : ', '27.4')
('Rucklauf Soll : ', '60.0')
('WQ Eingangstemp : ', '25.5')
('WA Ausgangstemp : ', '19.9')
('Betriebszeit Verdichter 1 : ', '283 days, 12:32:46')
('Impulse VD1 : ', '6981')
('Betriebsstunden WP : ', '352 days, 7:08:13')
('Betriebsstunden Heizung : ', '352 days, 7:08:13')
('Verbruik verwaming (kWh) : ', '0.0')
('Verbruik heet water (kWh) : ', '0.0')
('Betriebsstunden WW Erzeugung : ', '0:00:00')
('Status Zeile 1 (code) : ', '1')
('Status Zeile 2 (code) : ', '0')
('Status Zeile 3 (code) : ', '4')
('Status seit : ', '100 days, 7:01:06')
('WMZ Durchfluss l/h : ', '0.0')


Sorry can't figure out how to include a screen shot.
Thanks

[Holzatelier]

Ok so I've changed the python script and it has created a CSV file with the labels value1 , value2 , value3 in the first row and the recieved values 12,13,14 in the second row. Now I want to create a Data source from a "Data file" and put the file path in the "Path to data file or folder" and "Check"-> File exists.
Then I create a Data point and set " Identifier in import document" to say value1.
Save everything and activate.
Now I go to "Data point details" but I get no value.

What am I missing?

Thanks

[phildunlap]

Hi Holzatelier,

Sorry can't figure out how to include a screen shot.

No worries, it can sometimes give trouble and not finish linking in the uploaded image.

Is it somehow possible to get these values automatically at an intervall of say 5 minutes into Mango datapoints?

In so very many ways, yes, it is possible. You could,

1. Open a socket in your python script and take a connection from a TCP/IP data source, which could then query your python script with TCP/IP points.
2. Create a virtual serial server socket and let your python script submit messages to a Serial data source, which then is in the context of a script that parses out values (but for your protocol, you could probably just use a serial data source's points to regex parse out the appropriate values.
3. Go through some protocol wrapper on top of that, like HTTP, Modbus, BACnet, etc. A lot of options condensed into this one, including the Data File approach you have tried. It doesn't sound like you implemented your own CSV importer. Did you try to use one of the example importers for the CSV file?

But probably the way I would choose is to,

4. Create a user with set access to some virtual points set up to log all data that you've set up to receive these values. Create an authentication token on that user, and use that token in your script to POST API calls to the /rest/v2/point-values endpoint. You will submit the token in an HTTP header like, Authorization: Bearer <token> and the POST body would be

[
  {
    "xid": "pointXidHere",
    "value": 123.456,
    "timestamp": 1535475910355,
    "dataType": "NUMERIC"
  },
  {
    "xid": "pointXidHere",
    "value": 789.123,
    "timestamp": 1535476010355,
    "dataType": "NUMERIC"
  },
  ....
]

You can also do stuff like create points through the API. You can enable swagger in your env.properties, restart Mango, and then navigate to either /swagger/index.html (Mango 3.4.x and prior) or /swagger-ui.html (Mango 3.5 forward) to explore the API.

[phildunlap]

It looks like I should probably have advised the PUT /rest/v1/point-values endpoint. Everything else would be the same, but the v1 endpoint is subject to logging type and will generate point events to inform websockets or meta scripts about the points' having updated.

[Holzatelier]

Phil!

Thanks so much! Lots of options! I hate to admit it but I don't understand half of it ... Somuch to learn, but so little time.
Anyway after finding a CSV example file I was able to get the data import working. It's maybe a bit crude but it does the trick.

[phildunlap]

The data file is a good way to go, too!

I wonder which example you drew from though, as I know (and really should just fix) the SimpleCsvImporter and the MultiColumnCsvImporter have hh in their date format when they almost certainly meant HH.

[Holzatelier]

Phil, I simply searched the installation folder for "CSV" and found this example of a csv-file:
../web/modules/dataFile/web/CompilingGrounds/CSV/multiColumnCsv.csv
I then adjusted my python script to output the csv file in the appropriate format.

Now I wonder if there is a way to have mango start my python script automatically since I have not been able to get crontab to start it periodically for me....

[Holzatelier]

Ok, after a few hours of poking around and following various suggestions I actually managed to fumble my way to a working solution and my python script actually runs through cron.
So now I wonder if there is a way to change the time format in Mango to 24 hrs.
Is there some setting to display 2:00:00 PM as 14:00:00 ?

Thanks

[phildunlap]

Now I wonder if there is a way to have mango start my python script automatically since I have not been able to get crontab to start it periodically for me....

Certainly. I suspect you'll find the Process Event Handler, perhaps on something like the System startup event, can do this for you.

So now I wonder if there is a way to change the time format in Mango to 24 hrs.
Is there some setting to display 2:00:00 PM as 14:00:00 ?

Yes, it's a simple change. Go to Mango/web/modules/dataFile/web/CompilingGrounds/CSV/ and edit MultiColumnCsvImporter.java . Change,

private DateTimeFormatter dtf = DateTimeFormat.forPattern("MM/dd/yyyy hh:mm:ss");

to

private DateTimeFormatter dtf = DateTimeFormat.forPattern("MM/dd/yyyy HH:mm:ss");

Then edit the data source edit page to recompile the importer class:

And restart the data source.

[Holzatelier]

Phil, I was thinking of changing the time format for all data points in Mango. So for example Modbus and Bacnet. Can I change the Time formatter for those modules as well?

Thanks

[phildunlap]

I'm not really sure what you mean. There isn't time formatting in Modbus, for instance. The timestamps for a poll on Modbus is the time that the poll occurred measured in elapsed milliseconds, this is then converted for display using the locale set on your user. If you change your user's locale, you should see the new UI use an appropriate time format. It is not currently possible to explicitly control the time format.

[Holzatelier]

I meant the Times given in the UI. For example in the "data point details" page the windows "point history values", "point statistics" and "chart" all give the Time in 12 hrs AM/PM format. I tought maybe there is a setting to change that format to 24 hrs (military time format).
But now I'm nit picking ...

Thank again for the help.

[Holzatelier]

Just set my locale and bingo I'm a happy camper! There's just so much to explore!
Cool !! Thanks!

[phildunlap]

Sweet! Glad to hear it!

I'm not sure this would be easy to change on the data point details page or other pages we provided, but if you find yourself using <ma-point-value></ma-point-value> components in your custom dashboards and wanting to provide a time format, the API docs (enable in Administration --> Edit menu) suggest,

display-type
(optional)
string	
Changes how the data point value is rendered on the page. Options are:

rendered (Displays live value in point's text rendered format) *Default
converted (Displays live value in point's unit converted format)
dateTime (Displays the time the of the point's last update)

date-time-format
(optional)
string	
If dateTime is used with display-type then you can pass in a momentJs string to format the timestamp. (Defaults to Mango default date format set in system settings)

where the data point details page is displaying time in that element with the settings,

display-type="dateTime" flash-on-change="true" same-day-date-time-format="timeSeconds"

where timeSeconds is one of the shorthands for normal formats one may find around Mango, defined: https://github.com/infiniteautomation/ma-dashboards/blob/bf9f8ba49754a1a93d59d1f3bd562b69b8523680/UI/web-src/ngMango/ngMangoServices.js#L151

[Holzatelier]

I haven't yet started with the custom dashboards, so I will keep this in mind for future reference.
Another thing I having difficulty with is the spline setting for the charts.
I've set my data points "point properties/Plot type" to "Spline" but the chart in the "Data Point Page" is never a smooth spline. The Bar and Step entries in the dropdown menu in the Chart window work nicely, the Line and Smooth entries don't seem to change anything, there are always steps.
I'm obviously missing something here again but what?

[phildunlap]

I'm obviously missing something here again but what?

The plot type setting applies to the old UI pages, such as /data_point_details.shtm

On charts in the new UI, such as /ui/data-point-details/ , there is a selector in the top right portion of the chart to toggle between smooth, line, step, or bar styles.

[Holzatelier]

Phil, I' ll try to upload some screenshots. Whatever I choose in the selector in the top right portion of the chart the outcome seems the same to me. Except for "Bar", then the area below the graph is filled in solid:

[phildunlap]

Hmm. I was not able to replicate. Can you share the JSON for the point you're seeing that with? Are there any errors in the browser developer console?

[Holzatelier]

Hi, I've got myself a litte confused with this one. I only just realized how to zoom into the chart and it turns out that the steps I'm seeing are actually a result of the data aquisition. Although I poll the devices every minute the data sources do not seem to update the data that fast. So I might get back constant values for several minutes and then record a change from one poll to the next.
So then of course the spline is only 1 minute segment long and does not seem to be there at all when I'm looking at the whole chart.
Sorry about that, don't know where my brain was...

Nevertheless another question came up (of course). Please see next Post.

[Holzatelier]

I've been playing around with the dashboard designer and I can't delete any pages.
I'm admin and I think I've got all the permissions:

Then I try to delete this page called "Testpage" by clicking on the trashcan symbol but nothing happens.

I try the same thing in "edit pages". The page is empty but I can't delete it.

But I noticed this error in the Browser console:

So I checked Mango on my Windows installation and the exact same thing happens there.
This is the java version I'm running:

I hate to pepper you with questions, but I'm still hoping I just overlooked something really obvious.

Thanks for your patience.