-
-
M mihairosu
@mattfox said in Changing Data Source but Retaining Data Point History?:
&limit=10000
Since the last time we worked on this, this '&limit=10000" does not allow the page to load.
Anyway, the mango install just can't handle downloading an entire data set for even a single XID. The whole mango service comes to a slow crawl, as it pegs all cores to 100%. All screens can't reach the mango service, and in general is not ideal. And in the end, it still can't download it.
Plus how am I supposed to re-upload it. This download format is different than what you get using the watch list page downloads. Tn addition, the timestamp column appears messed up =( There are also not enough rows in excel for some of this data, I suspect, lol.
I will have to continue this conversation with IA.
As far as I can tell, there's no decent way, for the lay-man such as myself, to manipulate large amounts of data and other critical information. I suspect a lot of back-end work would have to be done to allow for such functionality.
My uneducated guess is that the whole structure is a bit rigid.
I'll keep putting this work off until a good solution appears, or I can get help directly from IA.
-
M mihairosu
Okay awesome, I'll play around with stuff, see what works well.
-
M mihairosu
Okay thanks for everything, I can certainly handle it from here! If anything, I'll have to split it up, like you said, though I would have hoped to be able to get more out of this method. I hope it's more than I can get out of the watchlist page, haha.
-
M mihairosu
Ok I found the watch list for a test watchlist ID: WL_883061
{ "xid":"WL_883061", "data":null, "query":null, "readPermission":null, "name":"Troubleshooting", "dataPoints":[ "boosterPumpsFlowRate", "DHWBsmntReturnRate" ], "folderIds":null, "params":null, "type":"static", "user":"admin", "editPermission":null },
The page appears to work fine:
Data begins approximately Feb 7, 2017, so I set a test download from
Feb 7, 2017 to March 4, 2020 with 1 day rollup interval:
I was getting lost connectivity and restored notifications.
Anyway, after the server is finished doing whatever it is doing, with a high processor load, I get an error for the CSV download:
Any idea what the issue may be?
I tried a smaller time period for the download and that had no problems.
I assume it's some limit in some sub-system that I may have to adjust values for?
edit: I think I see what the issue is. ma/log shows this error:
ERROR 2020-03-04T12:36:28,716 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run:632) - Error in work item java.lang.OutOfMemoryError: GC overhead limit exceeded
-
M mihairosu
What parts of this script should I be changing to get the points I am looking for?
<ma-watch-list-get ng-model="designer.watchList" parameters="designer.parameters" on-points-change="designer.points = $points" id="29305bf3-cc2a-4bac-9650-3cbf64d738a9" watch-list-xid="WL_Flow"></ma-watch-list-get> <div class="ma-designer-root" id="1a2a6980-f5bb-4acc-a41a-f8f69d5c12d2" style="width: 1366px; height: 768px; position: relative;"> <ma-calc input="designer.points | filter:{name:'Flow'}:true | maFirst" output="point"></ma-calc> <md-button id="2585b1fa-a630-4255-8670-41884b8754e2" style="position: absolute; left: 260px; top: 70px;" class="md-raised" ng-href="/rest/v2/point-values/time-period/{{point.xid}}?fields=TIMESTAMP&fields=VALUE&fields=ANNOTATION&from={{dateBar.from.toISOString()}}&to={{dateBar.to.toISOString()}}&format=csv2" download="{{point.name}}.csv">Download values</md-button> </div>
I'll re-create the data source because it's easy in bacnet, but I am still trying to figure out how to download the data easily.
I don't fully understand what XID I need to match to something in my own syste,
There's no point XID in there, as far as I can tell.
-
M mihairosu
I am getting these errors when trying to import data point configuration for every point:
Data point with XID 'boosterPumpsOutputPress' does not already exist and references a data source that does not exist. Ignored.
I'm probably not following your instructions 100% somehow, I'll try to understand better.
-
M mihairosu
Okay cool I'll give that a try and report back. Thanks Matt.
-
M mihairosu
So right now I'm trying to backup and data itself.
We got 23 data points collecting data over many years.
I can't fit enough data into one watch list graph even from one point to be able to download it all.
I can't download 5000 points at a time for 23 data points.
Is there another reliable way to download all of this data?
-
M mihairosu
Hey Terry,
This one is also giving a java out of bounds exception, but I don't understand why:
/* //Script by Phil Dunlap to automatically generate lost history if( my.time + 60000 < source.time ) { //We're at least a minute after var metaEditDwr = new com.serotonin.m2m2.meta.MetaEditDwr(); metaEditDwr.generateMetaPointHistory( my.getDataPointWrapper().getId(), my.time+1, CONTEXT.getRuntime(), false); //Arguments are, dataPointId, long from, long to, boolean deleteExistingData //my.time+1 because first argument is inclusive, and we have value there } //Your regular script here.*/ //Get daily run starts return runs.past(MINUTE, 1440)['data'].get(true).starts;