Using Persistent TCP points in script calculations

[psysak]

Hey Phil, I'm getting weird errors and behaviour from this approach. The errors in the system look like this;

Example, at 11:04

'MORGUARD': Script error in point "MORGUARD - EO00001SE_CBCKWH": Script returned null. Ignoring result. [Add comment]
'MORGUARD': Script error in point "MORGUARD - EO00001SE_CBCKWH": TypeError: Cannot get property "value" of null in at line number 32 in at line number 32 [Add comment]

Looking at that point, which is the META calculated point the values in the historical are as follows;
222.000 kW·h 11:06:00
216.000 kW·h 11:05:00
228.000 kW·h 10:59:00
270.000 kW·h 10:58:00

However the source point which triggers this, the sync'd history point, is as follows;
36.00 11:05:00
37.00 11:04:00
37.00 11:03:00
37.00 11:02:00
37.00 11:01:00
38.00 11:00:00
38.00 10:59:00
45.00 10:58:00
52.00 10:57:00
52.00 10:56:00

So the source point has the expected values and it's even calling the script, but upon running the script, which is performing a p.pointValueAt(CONTEXT.getTimestamp()).value can't find those values for some reason.

Any ideas?

Edit: Weird, it seems to happen to only the one site and it's always with the oldest data, so for example the sync is every 15mins, the data which seems to have issues is always the oldest in the sync. So at 11:15 the problems are with 11:00, 11:01, 11:02 etc. There's about 16 points worth of data which get's sync'd over, it's almost like the oldest values fall out of the end of some queue but the script is still triggered to run.

I'm wondering if instead of doing this the way that I am, when data is pushed quickly run through and calculate the values, I should just be triggering a script which goes to the source point and just manually calculates the values for the meta point. Kind of just say every 15 mins go and calculate the values for the latest 15 mins worth of data.

[phildunlap]

I would wager it's a timing issue. You can see in your NoSQL settings that there is a 5000ms (default) delay before writing backdated point data. This is to queue it up for a more efficient insert. Alas this is one of the reasons to do Meta points at the edge Mango.

You may see resolution lowering that NoSQL setting. Potentially also putting RuntimeManager.sleep(5000); in the Meta script would circumvent that (but not an execution delay of 5s, as execution delays cancel one another and the point would only run once, 5s from the last update).

Edit: Probably I led you astray in encouraging getting the live updates into the same point, and you were better off doing the aggressive syncs for this point (you could always set it up as its own publisher for just that point and sync every minute) and not recording the live data.

[psysak]

K that sounds fair. I'm seriously at the point where I may do it all at the edge and then just push, it would be a lot cleaner.

Out of curiosity, I'm just playing around with a script which would manually go back and backfill the history for the meta point, ie the script just returns an array of data values since it last ran and then attempts to backfill the historical for the meta. I don't think I'll end up doing this but curious how that would be accomplished. If I return say 15 values, one for each minute, and step through the calculation until the end of the array will the meta point just get one value at the time of script run or does it somehow understand that I am trying to insert values into history? IF that makes sense.

[psysak]

You know what, nevermind :) I'm going to do my calculations in the edge from now on.

[phildunlap]

With the set() function becoming available in all scripting environments, that's the pathway.

So, one could have a script like this:

if(p.time > my.time + 15000) { //if the source point is 15 seconds ahead or more
  var values = p.pointValuesSince(my.time);
  for(var k = 0; k < values.length; k+=1) {
    my.set(values[k].doubleValue, values[k].time); //need to be specific about
    // the data type as getPointValuesSince is an array of DataValue objects
  }
}
return p.value;

One needs to be mindful of logging types in using the set function. Only logging type 'ALL' on the point being set (the meta point) will work for backdates. Scripting data sources have the setting 'Saves historical' to avoid logging issues when using its set function, but there is no such option in Meta points.

[psysak]

You're awesome man, thanks for all the help :) I'll play around with that as well now that I have it

[phildunlap]

@psysak said in Using Persistent TCP points in script calculations:

You know what, nevermind :) I'm going to do my calculations in the edge from now on.

Ha!

Well, for posterity, the easiest setup for a meta point across a persistent TCP connection is to disable real time publishing (or just not save it), have frequent data syncs, and set the meta point to run on LOGGED events. The meta point can only be expected to update in that setup at each sync. But, computing things at the lowest level that has all the information is often a good distributed computing model.

You're awesome man, thanks for all the help :) I'll play around with that as well now that I have it

Certainly :D

[psysak]

@phildunlap said in Using Persistent TCP points in script calculations:

@psysak said in Using Persistent TCP points in script calculations:

You know what, nevermind :) I'm going to do my calculations in the edge from now on.

Ha!

Well, for posterity, the easiest setup for a meta point across a persistent TCP connection is to disable real time publishing (or just not save it), have frequent data syncs, and set the meta point to run on LOGGED events. The meta point can only be expected to update in that setup at each sync. But, computing things at the lowest level that has all the information is often a good distributed computing model.

You're awesome man, thanks for all the help :) I'll play around with that as well now that I have it

Certainly :D

Only issue I see with that Phil is communication loss and if you get an extended outage. If I were to go down for an hour then all the sync'd data would cause the same issue again. My thought was in that case run a script which doesn't rely on the LOGGED trigger but rather just manually steps through the source points values, keeping track of what the last one it processed was, which can just be accomplished by comparing what the last timestamps for each (meta and source) are.

[phildunlap]

The same issue being that point values were queued, then referred to before they were actually stored?

That should only have been an issue with backdated points, so when you're also saving the realtime data that backdate area exists because the realtime data got processed first and produced a current value. In a sync, it should go from the older to the newer, and it should log all those sync'ed values quite quickly. You didn't experience the timing issue when you had a separate point doing your realtime, I thought - only when I guided you to saving the real time data into the same meta point and publishing/saving on the LOGGED event did it become an issue. Right?

[psysak]

Hey, I'll have a think about what you asked in a bit I just have another question right now. In that META script, is there a way to prevent a value being pushed into the history for that point each time the script runs? Ie, the last statement, return p.value, will push a value in the database for that point. Can NOT have a value returned?

[phildunlap]

Yes. As of Mango 3.2 set point event handler scripts, point link scripts, and meta point scripts can all

return UNCHANGED;

to indicate no value should be set forth.

[psysak]

@phildunlap said in Using Persistent TCP points in script calculations:

return UNCHANGED;

At some point I will find a question you cannot answer!

[phildunlap]

Bring it on :P

[psysak]

Interesting, I'm still getting the same errors as before even though I tried changing the script so it doesn't rely on LOGGED or anything like that. I'm literally pulling the data right out of the database

if(pulse.lastValue().time > my.lastValue().time + 15000) {
   var list = pulse.pointValuesSince(my.lastValue().time + 5000);

   for (var i=0; i < list.length; i++) {
       my.set(list*.doubleValue * pfactor * tfactor, list*.time);
   }
}
return UNCHANGED;

Now I get this error, which is the same as before
'MORGUARD': Script error in point "MORGUARD - EO00026SW_BIERMARKTM3": TypeError: Cannot get property "time" of null in at line number 31 in at line number 31Add comment

Oh well... That was my last attempt, I'll move it all to the edge devices now.

[phildunlap]

The issue is that the meta point doesn't have any values, so my.lastValue() is null, and it cannot have a .time.

One can initialize a meta point like this by either setting a value to it, or putting a little handling in the point, like,

if(my.lastValue() === null) {
  //TIMESTAMP = new Date(2018, 1, 1).getTime(); //When does your point begin?
  //return 0; //What does it begin at?

  //Edit: I just remembered, you could also do,
  my.set(0, new Date(2018, 1, 1).getTime());
  //and then carry on with the script.
  //be sure your meta point is settable!
}

[psysak]

Noo way! If that's the issue I'm gonna have some kind of egg on my face haha. Classic rookie mistake not validating this.

K I'm going to poke this into all my points and see how it feels about it.

[psysak]

I did that on purpose just to see how well you knew your product.......

[phildunlap]

And I wrote getPointValuesSince instead of pointValuesSince the first time to ensure you had something to correct... yes... :D

[psysak]

Huh, I'm getting odd results Phil.

And there's time ranges with no issues at all. No errors now though. The only thing I can think of is a timestamp that I'm not aware of which affects the number of data points I get back.

I was hopeful!

My code looks like this now, triggered every 15mins.

if(pulse.lastValue() === null) {
   return UNCHANGED;
}

if(my.lastValue() === null) {
   my.set(0, new Date(2017, 1, 1).getTime());
}

if(pulse.lastValue().time > my.lastValue().time + 15000) {
   var list = pulse.pointValuesSince(my.lastValue().time + 5000);

   for (var i=0; i < list.length; i++) {
       my.set(list*.doubleValue * pfactor * tfactor, list*.time);
   }
}
return UNCHANGED;

I've turned on TRACE logging for this script, hopefully I can see what's going on.

Saga continues, Empire Strikes Back episode.

[phildunlap]

My first thought, pardon the asking, is if your source point has values for that time period.

Also, my offhand new Date() was somewhat off, the month field is zero indexed, so,
new Date(2017, 0, 1).getTime(); //Jan 1 2017 server timezone