Changing Data Source but Retaining Data Point History?

[mihairosu]

From a practical perspective, we will have to do these two kinds tasks:

• Replace a data source device, but we want to retain the same points.

• Move a data point from one data source to another (migrate to another device)

• Move BacNet point data source (all all respective points) from a MangoES to a Mango Enterprise server.

What do I need to do to make sure that we can make these changes while retaining all of our historical data.
Is it only necessary for the XID to stay the same?

[terrypacker]

@mihairosu we don't really support the concept of moving a data point between data sources. I just double checked and even if you try to update a data point via the REST api it won't allow you to change the data source it is attached to. We definitely do not support moving a point to a different data source type.

I don't really understand this?

Replace a data source device, but we want to retain the same points.

I have a few options for this:

Move a data point from one data source to another (migrate to another device)

A. It would be possible to copy the configuration of the point to a new data source by exporting the JSON and re-importing it onto another data source via the Import/Export feature of Mango however this would be a new point without any of the old point's history.

B. It would be possible to achieve what you want by terminating mango and performing some operations on the database to change the dataSourceId column of the data point to the new source's id. This is not recommended but is possible if you know what you are doing.

As for this:

Move BacNet point data source (all all respective points) from a MangoES to a Mango Enterprise server.

There are some migration tools with the NoSQL module assuming you use that for you point value store. Those tools coupled with the JSON Import/Export feature would likely work. The other option is to publish the data to the enterprise server assuming you are ok with the point becoming a Persistent TCP type data point and it won't have to be attached to a BACnet data source.

[mihairosu]

• Replace a data source device, but we want to retain the same points.

This means that we will have to replace a PLC or other data gathering device but want to retain the same point and history. For example, measuring temperature somewhere in a system. The new PLC may not have the same network address. However, if it did have the same address, I guess Mango won't know the difference, correct? However, if the new connect via another protocol: example, we want to upgrade from Modbus Serial to Bacnet PLC, and still want to retain data history.

As for the rest of your suggestions, unfortunately, we do not have the expertise to do that sort of work.

It would be ideal for us if we could do this type of task with the GUI. My interpretation is that the data points are too rigidly tied up with the device address or something like that.

I'm not sure what we can do going forward. We would like to upgrade our equipment, but this is making it very difficult for us.

[terrypacker]

@mihairosu said in Changing Data Source but Retaining Data Point History?:

However, if it did have the same address, I guess Mango won't know the difference, correct?

Correct.

I would suggest to look at the migration tools for NoSQL as you could export the data and re-import it on a new point using a combination of those tools and they are in the UI.

[mihairosu]

Looks to me like this is something we would be interested in pursuing in case we want to change the data source:

It would be possible to achieve what you want by terminating mango and performing some operations on the database to change the dataSourceId column of the data point to the new source's id. This is not recommended but is possible if you know what you are doing.

With regards to this:

There are some migration tools with the NoSQL module assuming you use that for you point value store. Those tools coupled with the JSON Import/Export feature would likely work. The other option is to publish the data to the enterprise server assuming you are ok with the point becoming a Persistent TCP type data point and it won't have to be attached to a BACnet data source.

I may not have explained myself well or misunderstood your instructions. I want to make sure we're on the sama page. We already publish this data point to the Enterprise server, so there's no need to move the actual data from the ES. I'll attach an image, it might work best:

Thanks

[MattFox]

Just my few cents here @mihairosu, if you're going to go direct and your server is in the cloud, sorry to jump in Terry

1. How feasible is it for you to set up a VPN?
   That will give you the ability to send data across the VPN into the server direct without firewalls posing an issue. This is also assuming you set up a bacnet server on a linux/windows box. //actually I think this isn't a concern here...
   1.a) ALSO, have you tested this with a standalone unit to see if it will talk to your server direct without the mango in between? no point otherwise
2. Export all of your pointValue data and your point configuration data so your xids and names all line up from the mango unit
3. export any additional data from the enterprise server as you may have used a publisher prefix on your XID, be sure to amend the point value data export XIDs to match your system (remove the prefix in other words)
4. create a new bacnet data source, amend your exported point configs datasourceXID to match the new one then import. (or simply import the old datasource)
5. import your point values.
6. assuming bacnet works over the web you should be able to update the connection and go for it.
7. Terry if there's anything I've missed lemme know!

[mihairosu]

Hey Matt,

Actually this is all on the same internal network, I only need to migrate the data source between the two devices. Network communication is not a problem.

Thanks for the very helpful instructions; as soon as I get a chance, I'll see if I can work it out.

[MattFox]

I hope they're helpful! Looking back it looks a little bit like gibberish! Gimme a bell if you need anything

Fox

[mihairosu]

So right now I'm trying to backup and data itself.

We got 23 data points collecting data over many years.

I can't fit enough data into one watch list graph even from one point to be able to download it all.

I can't download 5000 points at a time for 23 data points.

Is there another reliable way to download all of this data?

[MattFox]

Yep,I remember here being a post by Jared:

<ma-watch-list-get ng-model="designer.watchList" parameters="designer.parameters" on-points-change="designer.points = $points" id="29305bf3-cc2a-4bac-9650-3cbf64d738a9" watch-list-xid="WL_Flow"></ma-watch-list-get>
<div class="ma-designer-root" id="1a2a6980-f5bb-4acc-a41a-f8f69d5c12d2" style="width: 1366px; height: 768px; position: relative;">

    <ma-calc input="designer.points | filter:{name:'Flow'}:true | maFirst" output="point"></ma-calc>
    
    <md-button id="2585b1fa-a630-4255-8670-41884b8754e2" style="position: absolute; left: 260px; top: 70px;" class="md-raised" ng-href="/rest/v2/point-values/time-period/{{point.xid}}?fields=TIMESTAMP&fields=VALUE&fields=ANNOTATION&from={{dateBar.from.toISOString()}}&to={{dateBar.to.toISOString()}}&format=csv2" download="{{point.name}}.csv">Download values</md-button>
</div>

https://forum.infiniteautomation.com/topic/4050/datapoint-values-into-a-var-javascript/4

Set your time period and off you go!

[mihairosu]

Okay cool I'll give that a try and report back. Thanks Matt.

[MattFox]

np, look at the button, that's where you'll want to set your variables. If you get stuck lemme know.

[mihairosu]

I am getting these errors when trying to import data point configuration for every point:

Data point with XID 'boosterPumpsOutputPress' does not already exist and references a data source that does not exist. Ignored.

I'm probably not following your instructions 100% somehow, I'll try to understand better.

[MattFox]

Probably show me your entire markup.
If this is a data-point configuration your dataSourceXid property must match a datasource that already exists in your system.
The markup I gave you above exports point-values.

[mihairosu]

@mattfox

What parts of this script should I be changing to get the points I am looking for?

<ma-watch-list-get ng-model="designer.watchList" parameters="designer.parameters" on-points-change="designer.points = $points" id="29305bf3-cc2a-4bac-9650-3cbf64d738a9" watch-list-xid="WL_Flow"></ma-watch-list-get>
<div class="ma-designer-root" id="1a2a6980-f5bb-4acc-a41a-f8f69d5c12d2" style="width: 1366px; height: 768px; position: relative;">

    <ma-calc input="designer.points | filter:{name:'Flow'}:true | maFirst" output="point"></ma-calc>
    
    <md-button id="2585b1fa-a630-4255-8670-41884b8754e2" style="position: absolute; left: 260px; top: 70px;" class="md-raised" ng-href="/rest/v2/point-values/time-period/{{point.xid}}?fields=TIMESTAMP&fields=VALUE&fields=ANNOTATION&from={{dateBar.from.toISOString()}}&to={{dateBar.to.toISOString()}}&format=csv2" download="{{point.name}}.csv">Download values</md-button>
</div>

I'll re-create the data source because it's easy in bacnet, but I am still trying to figure out how to download the data easily.

I don't fully understand what XID I need to match to something in my own syste,

There's no point XID in there, as far as I can tell.

[MattFox]

I'll break it down for you. The code above is filtering out a single point from a watchlist using ma-calc, the first point with Flow as the point name.
I think for you, my focus was around the button itself. I'll throw together some markup for you here. Since it's 23 points we should be able to knock them out with an ng-repeat....

<!-- 1. Generate a watchlist with your desired points that you want data for -->
<ma-watch-list-get ng-model="designer.watchList" parameters="designer.parameters" on-points-change="designer.points = $points" watch-list-xid="[YOUR_WATCHLIST_XID_HERE]"></ma-watch-list-get>

<!-- 2. ng repeat the buttons for each point in the list -->
<md-button  ng-repeat="point in designer.points" id="{{point.xid}}_button" style="" class="md-raised" ng-href="/rest/v2/point-values/time-period/{{point.xid}}?fields=XID&fields=TIMESTAMP&fields=VALUE&fields=ANNOTATION&from={{dateBar.from.toISOString()}}&to={{dateBar.to.toISOString()}}&format=csv2" download="{{point.name}}.csv">Download {{point.name}} values</md-button>
</div>
<!-- 3. use the date bar at the top of the page to set your time span -->
<!-- 4. hit the download button! -->

Fox

[MattFox]

This post is deleted!

[mihairosu]

Ok I found the watch list for a test watchlist ID: WL_883061

{
         "xid":"WL_883061",
         "data":null,
         "query":null,
         "readPermission":null,
         "name":"Troubleshooting",
         "dataPoints":[
            "boosterPumpsFlowRate",
            "DHWBsmntReturnRate"
         ],
         "folderIds":null,
         "params":null,
         "type":"static",
         "user":"admin",
         "editPermission":null
      },

The page appears to work fine:

Data begins approximately Feb 7, 2017, so I set a test download from

Feb 7, 2017 to March 4, 2020 with 1 day rollup interval:

I was getting lost connectivity and restored notifications.

Anyway, after the server is finished doing whatever it is doing, with a high processor load, I get an error for the CSV download:

Any idea what the issue may be?

I tried a smaller time period for the download and that had no problems.

I assume it's some limit in some sub-system that I may have to adjust values for?

edit: I think I see what the issue is. ma/log shows this error:

ERROR 2020-03-04T12:36:28,716 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run:632) - Error in work item
java.lang.OutOfMemoryError: GC overhead limit exceeded

[MattFox]

Looks like you may need to break it up. Secondly, check your point limit count under the ui settings.

[mihairosu]

Okay thanks for everything, I can certainly handle it from here! If anything, I'll have to split it up, like you said, though I would have hoped to be able to get more out of this method. I hope it's more than I can get out of the watchlist page, haha.