History Generation Still Running or Finished??

[Phillip Weeks]

Just an update I have had no issue with the history generation today as the loop has now processed through 20 meta point recalculations with no issue creating complete sets of about 100000 values.

This metapoint logs every 5 minutes and it's the instant value, NaN isn't the culprit here..
ONLY when I check the delete beforehand or set that delete_first parameter to true does the issue of the zero's occur so it must be related to this somehow. Btw this checkbox is no big deal for me as I used the deletePointValuesBetween to purge all the point ranges.

[Phillip Weeks]

And after four more days I have finally finished the regeneration of 144 metapoints over a 10 month period. I have tried many suggestions about thread pool increases and NOSQL throttling techniques, monitoring values to be written, threads, memory. I did get a great improvement in that it only crashed 6 times over the weekend and each time I restarted I started on the point it crashed. I wish I could say I understood the processing better but I cannot. This is the recalculated data for the final unit 324. Thanks again Phil for your help in automating this process. I did notice that the console output did not print anything until the entire process had exited the validate run which made me wonder how the code gets executed? Does it all get stacked first? It seems to run better if I reduced the size of the run loop to less than 10 points.

[phildunlap]

The validate function on the meta point page won't return until the script finishes executing, correct. So, if you print a whole ton of stuff, it would be a large print buffer.

The meta history generation process is somewhat straightforward - all context points that produce context updates have all their values for the time period loaded into a list (so as not to hold database locks by iteratively fetching them), then a simulation timer is set to the beginning of the period, and time is replayed between values (so as to handle interval logging / execution), then values are submitted as context updates, and time is played to the period end.

[Phillip Weeks]

In addition to the primary updating source point, when I included a second point with a shorter history that the point did not use to trigger the update in the metapoint's context ... EVEN though this point without history was set not to update the context, the meta history recalculation would not go older than the history for this non updating point? Which seems to contradict what you say above wrt updating points.

[phildunlap]

Ah, you are correct. I have definitely thought it funny how the start time gets adjusted to the earliest inception time (first value) of its context points if those are after the submitted inception time. It's always been that way, but its merit does seem debatable.

[Phillip Weeks]

Yes because I wanted to include a point in the script to monitor its values changes and this point throws off the history recalc so I had to remove it. However I do believe the way you stated the process above, should be the way it works and non-updating points should be excluded from consideration. Or provide a parameter to enable this behaviour.

[phildunlap]

Sorry I misspoke, set to the latest inception time*

Why would it be adjusting what the user asks for at all, is my thought. I suspect when written meta points had some different characteristics, like every point always updated context unless the point was on cron, then none did (but I suspect this behavior still existed). Meta points have gotten more versatile since then. What makes you say it should adjust the user's request?

[Phillip Weeks]

I noticed that in 2.8.8 the script line below produces the full set of points for the DS with this ID but also adds the metapoint in context as the first element in this return list.
V3 only returns the DS.

pointsList = DataPointQuery.query("eq(dataSourceXid,DS_537506)&limit(500)&sort(name)");

Further to this I would like to use the same script in 2.8.8 that you suggested for deleting the period between in dates in V3.
pvd.deletePointValuesBetween(pointsList1[k].getId(),periodBegin.getTime(), now);

However getId() seem not to exist yet so can I just use pointsList1[k].xid
in 2.8.8 to pass in the pointId as this first param? OK just realized these are not the same identifiers.

[phildunlap]

The only way to get the Data Point's ID in the script in 2.8.8 is to get the whole data point and pull it out of that, like,

var pointId = com.serotonin.m2m2.db.dao.DataPointDao.instance.getDataPoint("YourXidHere").getId();

[Phillip Weeks]

thanks however in 2.8.8 deletePointValuesBetween is coming back as unknown method??
I assume it exits in this version? I would like to replicate the history recalc on our 2.8.8 cloud server. Unless you can suggest an easy method to transfer the regeneration data just completed over the weekend to another mango instance.