var values = pvd.getLatestPointValues(dataPointId, 2);

I think you're right that your cache's contents have made it in to the statistics object.

var now = new Date();
var pv1 = p1.pointValueBefore(now.getTime() - 420000); //May need some slight adjusting of these offsets
var pv2 = p1.pointValueBefore(now.getTime() - 120000);
return pv2.value - pv1.value;

Unfortunately pointValueBefore() will also use the cache if the timestamp in the cache is prior to the requested time. But, you can probably handle that by triggering it on a context update and then having an execution delay (but know if your update interval is less than your execution delay, a meta point will never run. A script is the opposite, where it'll only schedule a timeout if it hasn't already)

Edit: The only way to truly guarantee they're logged is to get the values out via point value dao. So,

var pvd = com.serotonin.m2m2.Common.databaseProxy.newPointValueDao();
var pv1 = pvd.getPointValueBefore( dataPointId, now.getTime() - 420000);
...
//Note that it is dataPointId, not xid. In the next version of Mango you will be able to do something like p1.getDataPointWrapper().getId() 
// but for now it would require something like com.serotonin.m2m2.db.dao.DataPointDao.instance.getDataPointIdByXid(p1.getDataPointWrapper().getXid());
// if you need to do this programatically

pointValueBefore( CONTEXT.timestamp - 300000 );

Hey Phil, just a quick question on this again. Is there a way to pick specifically historical database entries when doing a calculation? For example, I log something every 5 mins and I'd like to take the delta between the last two historical database entries. I want it to ignore any cached values or anything else, I just want the last two entries in the history database.
I ask this cause I think I realized that the system interpolates values between historical entries, is that correct? For example, I log something now and again in exactly 5 mins. If I then wait 2 mins and say give me the delta past(MINUTE, 5), what values will it use to calculate that delta?

This is my dataset as downloaded from the mango
EO00002GE_1KWH 11-21-17 13:24:27.249 37922363 37922363.00
EO00002GE_1KWH 11-21-17 13:29:27.249 37922748 37922748.00
EO00002GE_1KWH 11-21-17 13:34:27.249 37923143 37923143.00
EO00002GE_1KWH 11-21-17 13:39:27.249 37923537 37923537.00
EO00002GE_1KWH 11-21-17 13:44:27.249 37923922 37923922.00
EO00002GE_1KWH 11-21-17 13:49:27.249 37924313 37924313.00
EO00002GE_1KWH 11-21-17 13:54:27.249 37924709 37924709.00
EO00002GE_1KWH 11-21-17 13:59:27.249 37925101 37925101.00
EO00002GE_1KWH 11-21-17 14:04:27.249 37925478 37925478.00

You can see it's doing a bang up job, every 5mins. I just want the delta between the last two entries and that's it :)

wa

Ya that's exactly what I was thinking too, I'd rather do it in the field and I will most likely do that there for the reasons exactly you mentioned. I would much rather just publish the computed values to a central mango system. Goofing around with this allows me to do something with "live data", pretty well like sitting at the mango on site. If I can figure this out on here then I can templatize it and deploy in the field. Plus ultimately what I'm working towards is some per site email notifications etc so I'd probably rather each site manage that on it's own.

Only downside is managing changes to a fleet of devices, that may become an issue.

You could indeed use var.past(MINUTE, 5).delta on a five minute interval or logged event. If you want to guarantee you're getting the value from the database at least five minutes ago, you could try var.pointValueBefore( CONTEXT.timestamp - 300000 );

Ultimately, I will never completely like any solution because I believe your data analysis meta point shouldn't be on the other side of a publisher from what it is analyzing. You should be computing that on the same device and publishing the result. The inevitable disconnects coupled with the interval logging still occurring guarantees junk data if the meta points are in the cloud.

Do you mind if I run an application implementation idea past you? This is just my experiment platform, I may end up pushing these applications down to the field mangos but I'll see how it goes, just experimenting with what I can do.

I have some Mango ES devices in the field. They poll local MODBUS devices for pulse counts (electric meter, water meter, gas meter) every minute. Currently I am goofing around with a cloud based Mango platform running on EC2. From the mangos on site, I push data to my central one using HTTP POST. The point updates are just an accumulated pulse count, so the number gets bigger and bigger all the time, and they are pushed every minute.

On the cloud Mango instance I have told the equivalent points to perform instant logs every 5 minutes. I then have a meta point which attempts to calculate the last five minute interval delta (it also applies a meter factor and does a couple other quick maths to the number). My initial implementation was to try and use
var.value - var.lastValue(1).value triggered at a 5 min interval, initially using CRON

This was causing some issues and what I just figured out is that lastValue(1) seems to return just any last value, so cached values are returned. I thought that lastValue(1) would return the last logged value.

I have now moved on to trying to use the following
var.past(MINUTE, 5) triggered update type logged

Since I am logging the context point at an interval of 5 minutes I would expect that the script is triggered only every 5 minutes and the delta should give me the difference in counts between now and 5 minutes ago. Do I understand this correctly?

Scripts and meta points provide the option for a point in its context to trigger execution with one of the three point events configurable in the context update event type drop down. "Context Update" is every single new value (so polls, sets) so long as they are new. "Context Logged" is for logged data, so this would fire when an interval logging point stores data as opposed to when it polls. And "Context Changed" is like "Context Update" but stipulates that the update must also be a different value.

Cron expressions link to this page for a tutorial: http://www.quartz-scheduler.org/documentation/quartz-2.1.x/tutorials/crontrigger

I believe the only possible idiosyncrasies there is that year support is limited, and in the vast majority of cases you shouldn't have a year in your Mango cron expressions. Also I believe the ? behaves as a wildcard in the Day of Week field.

Meta points have a few semantic options for 'Start of minute', 'Start of hour' etc.

Cron expressions and those updates are basically the same as if a point update were to have occurred: script execution is triggered.

I just looked into execution delays and it looks like they work slightly differently. On a meta point, if you have an execution delay of 5 seconds with a point updating every 3 seconds, your meta point will never run. Each update cancels the timer and schedules the update again, five seconds out.

Scripts work the opposite way, though, where if you execution delay is 5 seconds and you have a point updating every 3 seconds, the first update at time 0 will schedule the execution. The second update at time 3 will log an aborted poll message for the script, at time 5 the script will execute, at time 6 the third update occurs and the script is scheduled to execute at time 11.

Metadata scripts execute either on context update or on a cron pattern. This is set here
If you use context update then you may select which data points trigger the script.

Note: It's usually best to not select all of them as this can cause the script to execute too often and cause a performance problem. With power come responsibility :-)