History Generation Still Running or Finished??

[Phillip Weeks]

I have had limited success with meta history generation, I wish I could have more control over this process but I feel like the metapoint history generation is an enigma. Sometimes it generates lots of history and other times I have no explanation for the complete lack of history generated. In fact for one meta source with 4 variables it calculated 4400 values for one point and just 7, 8, and 14 values respectively for the others and yet they all used similar source datapoints.

I think mango should have a better interface to configure the history generation parameters and express more control over what happens when data points are missing or values generated are outside specific minimum and maximum limits etcetera.

The good news is that I have mastered exporting troubled pont values to excel from the watchlist. Then using data import to delete/add corrected outliers and for finding the missing values as well as opposed to doing this in script which is much harder to see what is transpiring, This seems to work as expected.

[phildunlap]

We decided to add log messages at the debug level.

The history generation tool definitely has had similar recommendations made in the past. There are open git issues about there being little feedback and to add the ability to purge within the generating time range prior to generating. We hear you.

The history generation is a somewhat complex instrument, but it's essentially replaying time and the point values to a simulation of the regular meta point. This means that the normal things which affect a Meta point, like the log settings and discard extreme value settings, will affect Meta point's history generation. The logging settings for the points configured to update context is one thing that comes to my mind seeing the discrepancies in the point counts you offered.

As far as further customization, perhaps do that in the script body? That's what it's there for. If you wouldn't want a Meta point to be able to have a certain value range, throw and exception in the script body or discard the extreme values in the logging settings. I'm not sure what you mean "data points are missing" in regards to history generation.

As a taste of one subtlety in the history generation tool, if i have a context point with data only last month, and i do a history generation for a meta point that is interval logging unto the end of this month from the beginning of last, then it will repeatedly interval log the last value in the data set for this entire month! One can always experiment with shorter periods of sample data sets and get a grasp on how this feature works.

[Phillip Weeks]

Oh thank-you Phil that would be a great addition to debug messages. Should I update the core to get this? btw I'll script meta-point filtering from now on... keeping it light intentionally..

[phildunlap]

Hi Phillip,

I should have also included in this thread that you can tune Mango to handle the specific type of load generating massive Meta histories presents. To do this, you would navigate to the NoSQL settings on the System Settings page.

A history generation is a very fast accumulation of point values to write, but not as fast as a history import. If you're importing, everything is getting queued up together, batched and written to the disk. If you're using Meta points, the delay for the script can exceed the default batch delay and the batching won't do so well.

Experiment with bumping your settings for "Minimum batch size" to 1000 or more. Increase your "Minimum time to wait before flushing a small batch (ms)" to 1000 or more as well. This should speed up history generation considerably.

After you have finished, you can return these to their defaults or experiment with leaving them at 10 or so, or whatever seems to work best for your system.

Edit: The defaults are 1 and 1 to provide the highest quality of service in getting all point values one has supposed written when reading, and to provide the highest quality of service in a power loss situation (most data retained). As they get larger, situations like data pooling in the buffer not being returned in read requests can become commonplace for current point values. But, if throughput is the only ambition, this can be acceptable. Generally that only applies to short timescales, anyway, but is influenced by the minimum time to wait.

[Phillip Weeks]

Meta history update.. we ran history on 3 meta variables calculating over last 6 weeks at 1 minute slices. It started Saturday at 4pm and ran until Tuesday afternoon. Will the new debug log entry tell me all the details about how many pointValues were created for each point during this run?

BTW should we be using NoSQL, because I don't believe we are..

[phildunlap]

Without using the NoSQL, your system must maintain a small history or its performance will progressively degrade as the history grows. With NoSQL you should be able to have an arbitrary size of history and not suffer for it (obviously it does take longer to read a million samples than a thousand, but reading a thousand samples in NoSQL should be near constant time, where reading a thousand samples from a point values table will be influenced by the size of the table (which all point values are stored in together)).

The NoSQL module also can accept point writes much, much faster than using an SQL table. Where your whole system has to handle concurrency for the point values table in SQL, NoSQL only needs to address a single point's concurrency.

If your license permits it, we certainly encourage use of NoSQL over SQL for the point values.

[Phillip Weeks]

Wow ok then I will look into NoSQL further as we certainly do have large dataset histories. Thanks.

[Phillip Weeks]

Hi Phil; can you recall a discussion about history regeneration and the ability to also purge the specific range being regenerated? I believe in this thread we touched on this and I was wondering if it was ever implemented in later 2.8 or v3? It is very still useful to purge before we rebuild meta data because outliers cannot be removed by rebuilding the dataset using point limitations and the process leave the outlier values in place. A purge by start and end time range would be very helpful in resolving these issues.

[phildunlap]

I recall many such discussions - clearly and distinctly, no.

You will be happy to know this feature was added in 3.3

3.3 is currently released as a beta (if you change your "Upgrade release channel" to Development, Alpha, or Beta you will see it available) and we'll be finalizing the release next week (so if you wait for that, you don't need to change your release channel, or update twice).

[mihairosu]

I think Historical Generation would benefit from improving on the following points:

• Speed: It appears to run single threaded as it does only pegs one virtual CPU to max, while leaving much spare RAM and Disk IO.

• Stability: I'd love to be able to queue up historical generation for all time (many months or years worth of data); Unfortunately, some of my experiences have shown that queuing for too much time causes problems for Mango. For example, dashboards page would freeze and have to be reloaded. Many errors of queues being full show up in the logs. edit: I just had mango crash yesterday due to this exact problem. I only ran 2 months worth of history.

• Feedback on scripts: It would be nice to have a button which instantly calculates a point and also displays any errors right on the same page as where the code is written.

• Feedback on generation: I completely agree with this post. It would be pretty nice to know. The way I check is by occasionally refreshing a chart from the watch list. I can then see if additional data was created.

• Automatic historical calculation for missing times: After upgrades or other causes of downtime, it would be amazing if Mango can go back in time and fill in missing data.

[Phillip Weeks]

Yes indeed I agree with each point above. I have been employing the power of meta points and regenerating histories using scripts for the past year on hundreds of points. History regeneration is a very powerful tool in Mango's arsenal and I believe one not given enough attention with respect to functionality. I spend many hours dealing with issues surrounding execution of the script histories and when it fails it usually eats up all memory and brings mango down.

As mentioned above queuing up points so as not to overwhelm memory and system resources is a arduous admin task and should be better written to handle many simultaneous history recalculations with a dashboard style example to setup, control and monitor point history regeneration. There are many reasons we lose raw data at the source and this affects how the data and metadata looks in charting and presentations. Data manipulation tools such as meta point history generation and scripting data sources empower us with tools to fill these gaps back in using heuristic methodologies. to calculate appropriate values. So admin tools that queue up point history executions, and allow us to follow script progress and have inherent script support to detect errors would be very useful indeed and save many many hours of script supervision.

Thank-you to the team for the new history feature to zap the specific range before the script runs, it is a wonderful addition however there is something buggy with this new feature that causes millions of records created back to 1969 with 0 value under some circumstances.

[terrypacker]

With regards to values of 0 with the date set to 1969, I would think that could be due to an error in your script. This is the exact result you will get if one was to set the time stamp and value of a point to 0.

[Phillip Weeks]

Well one possible area of confusion might be that the point purge method requires dd/mm/yyyy and the purge before regenerate is mm/dd/yyyy so that alone is enough to be confusing and likely where I made the mistake resulting in the generation of the 1969 start date.

[phildunlap]

@phillip-weeks said in History Generation Still Running or Finished??:

Well one possible area of confusion might be that the point purge method requires dd/mm/yyyy and the purge before regenerate is mm/dd/yyyy so that alone is enough to be confusing and likely where I made the mistake resulting in the generation of the 1969 start date.

Huh? The purge before regenerate feature uses the time range from the history generation period to do the purge. It's the same date selectors.

[phildunlap]

@mihairosu said in History Generation Still Running or Finished??:

• Speed: It appears to run single threaded as it does only pegs one virtual CPU to max, while leaving much spare RAM and Disk IO.

This actually depends on how your invoke it. If you use the UI, then you run in a Jetty thread. You can run a few of these side by side. It's true that a particular point will only do a history generation in a single thread. If you were to invoke the meta history generation code, then it would run in the thread that invoked that (like the meta point or a scripting data source).

• Stability: I'd love to be able to queue up historical generation for all time (many months or years worth of data); Unfortunately, some of my experiences have shown that queuing for too much time causes problems for Mango. For example, dashboards page would freeze and have to be reloaded. Many errors of queues being full show up in the logs. edit: I just had mango crash yesterday due to this exact problem. I only ran 2 months worth of history.

Hmm. I know your system has quite a bit of memory, but that still sounds memory related. Did it supply an error?

• Feedback on scripts: It would be nice to have a button which instantly calculates a point and also displays any errors right on the same page as where the code is written.

I'm not sure I know what you mean. How is this different than the validate button?

• Feedback on generation: I completely agree with this post. It would be pretty nice to know. The way I check is by occasionally refreshing a chart from the watch list. I can then see if additional data was created.

I am sure there could be some traction internally here to make API endpoints for meta point histories. Then you'd have a temporary resource you could get the history generation information from. Currently you would have to set values out from the meta point, or check the meta point's values.

• Automatic historical calculation for missing times: After upgrades or other causes of downtime, it would be amazing if Mango can go back in time and fill in missing data.

Ask and ye may receive,

if( my.time + 60000 < source.time ) { //We're at least a minute after
  var metaEditDwr = new com.serotonin.m2m2.meta.MetaEditDwr();
  metaEditDwr.generateMetaPointHistory(
      my.getDataPointWrapper().getId(), my.time+1, CONTEXT.getRuntime(), false);
  //Arguments are, dataPointId, long from, long to, boolean deleteExistingData
  //my.time+1 because first argument is inclusive, and we have value there
}
//Your regular script here.

[phildunlap]

@terrypacker said in History Generation Still Running or Finished??:

With regards to values of 0 with the date set to 1969, I would think that could be due to an error in your script. This is the exact result you will get if one was to set the time stamp and value of a point to 0.

I would expect your start date was in 1969, and you have the point set to interval logging. So, it interval logged for forty years every minute, perhaps.

[Phillip Weeks]

@phildunlap

I believe I mixed up the days and the months so end date was before start date, not a script error because I didn't modify the script, just recalculating and it worked on the next run.

[phildunlap]

Ah, got it. You see the checkbox, "Delete existing data in range" in the date range dialog though, yes? That is how to purge the data before recalculating it for a meta point.

[Phillip Weeks]

@phildunlap yes apparently I was not paying attention and though both use similar date input format. It is difficult to remember sometimes since the date is autofilled and 5/1/2017 vs. 1/5/2017 looks similar. No problem if I pay attention.

[mihairosu]

This actually depends on how your invoke it. If you use the UI, then you run in a Jetty thread. You can run a few of these side by side. It's true that a particular point will only do a history generation in a single thread. If you were to invoke the meta history generation code, then it would run in the thread that invoked that (like the meta point or a scripting data source).

I know you've mentioned this to me before. I guess it would be nice if it split the load and spawned multiple instances, if I am only running 1 point.

Hmm. I know your system has quite a bit of memory, but that still sounds memory related. Did it supply an error?

These:

WARN  2018-01-22T17:12:25,907 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:12:28,709 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:28:02,061 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:28:03,709 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:36:32,351 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-22T17:36:32,352 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-22T17:36:33,708 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:39:49,070 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:39:49,071 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full

I still haven't able to pin down the problem.

I'm not sure I know what you mean. How is this different than the validate button?

I can't find a validate button on the point details page, but that sounds like what I've been looking for.

I am sure there could be some traction internally here to make API endpoints for meta point histories. Then you'd have a temporary resource you could get the history generation information from. Currently you would have to set values out from the meta point, or check the meta point's values.

It's not a big deal really, but I'm not a power user, so I wouldn't be doing anything but checking a GUI most of the time. Since event generation is time based, it could work quite well with a progress bar.

Ask and ye may receive,

if( my.time + 60000 < source.time ) { //We're at least a minute after
  var metaEditDwr = new com.serotonin.m2m2.meta.MetaEditDwr();
  metaEditDwr.generateMetaPointHistory(
      my.getDataPointWrapper().getId(), my.time+1, CONTEXT.getRuntime(), false);
  //Arguments are, dataPointId, long from, long to, boolean deleteExistingData
  //my.time+1 because first argument is inclusive, and we have value there
}
//Your regular script here.

Woah, amazing, thank you!

Purge now From: To: ---------------------screenshot

Woah, where is this screenshot from that I can purge a specific date range? I've been looking for something like this. I thought you could only purge "Purge data older than" or "Purge all".