History Generation Still Running or Finished??

Phillip Weeks

Yes indeed I agree with each point above. I have been employing the power of meta points and regenerating histories using scripts for the past year on hundreds of points. History regeneration is a very powerful tool in Mango's arsenal and I believe one not given enough attention with respect to functionality. I spend many hours dealing with issues surrounding execution of the script histories and when it fails it usually eats up all memory and brings mango down.

As mentioned above queuing up points so as not to overwhelm memory and system resources is a arduous admin task and should be better written to handle many simultaneous history recalculations with a dashboard style example to setup, control and monitor point history regeneration. There are many reasons we lose raw data at the source and this affects how the data and metadata looks in charting and presentations. Data manipulation tools such as meta point history generation and scripting data sources empower us with tools to fill these gaps back in using heuristic methodologies. to calculate appropriate values. So admin tools that queue up point history executions, and allow us to follow script progress and have inherent script support to detect errors would be very useful indeed and save many many hours of script supervision.

Thank-you to the team for the new history feature to zap the specific range before the script runs, it is a wonderful addition however there is something buggy with this new feature that causes millions of records created back to 1969 with 0 value under some circumstances.

terrypacker

With regards to values of 0 with the date set to 1969, I would think that could be due to an error in your script. This is the exact result you will get if one was to set the time stamp and value of a point to 0.

Phillip Weeks

Well one possible area of confusion might be that the point purge method requires dd/mm/yyyy and the purge before regenerate is mm/dd/yyyy so that alone is enough to be confusing and likely where I made the mistake resulting in the generation of the 1969 start date.

phildunlap

@phillip-weeks said in History Generation Still Running or Finished??:

Well one possible area of confusion might be that the point purge method requires dd/mm/yyyy and the purge before regenerate is mm/dd/yyyy so that alone is enough to be confusing and likely where I made the mistake resulting in the generation of the 1969 start date.

Huh? The purge before regenerate feature uses the time range from the history generation period to do the purge. It's the same date selectors.

phildunlap

@mihairosu said in History Generation Still Running or Finished??:

Speed: It appears to run single threaded as it does only pegs one virtual CPU to max, while leaving much spare RAM and Disk IO.

This actually depends on how your invoke it. If you use the UI, then you run in a Jetty thread. You can run a few of these side by side. It's true that a particular point will only do a history generation in a single thread. If you were to invoke the meta history generation code, then it would run in the thread that invoked that (like the meta point or a scripting data source).

Stability: I'd love to be able to queue up historical generation for all time (many months or years worth of data); Unfortunately, some of my experiences have shown that queuing for too much time causes problems for Mango. For example, dashboards page would freeze and have to be reloaded. Many errors of queues being full show up in the logs. edit: I just had mango crash yesterday due to this exact problem. I only ran 2 months worth of history.

Hmm. I know your system has quite a bit of memory, but that still sounds memory related. Did it supply an error?

Feedback on scripts: It would be nice to have a button which instantly calculates a point and also displays any errors right on the same page as where the code is written.

I'm not sure I know what you mean. How is this different than the validate button?

Feedback on generation: I completely agree with this post. It would be pretty nice to know. The way I check is by occasionally refreshing a chart from the watch list. I can then see if additional data was created.

I am sure there could be some traction internally here to make API endpoints for meta point histories. Then you'd have a temporary resource you could get the history generation information from. Currently you would have to set values out from the meta point, or check the meta point's values.

Automatic historical calculation for missing times: After upgrades or other causes of downtime, it would be amazing if Mango can go back in time and fill in missing data.

Ask and ye may receive,

if( my.time + 60000 < source.time ) { //We're at least a minute after
  var metaEditDwr = new com.serotonin.m2m2.meta.MetaEditDwr();
  metaEditDwr.generateMetaPointHistory(
      my.getDataPointWrapper().getId(), my.time+1, CONTEXT.getRuntime(), false);
  //Arguments are, dataPointId, long from, long to, boolean deleteExistingData
  //my.time+1 because first argument is inclusive, and we have value there
}
//Your regular script here.

phildunlap

@terrypacker said in History Generation Still Running or Finished??:

With regards to values of 0 with the date set to 1969, I would think that could be due to an error in your script. This is the exact result you will get if one was to set the time stamp and value of a point to 0.

I would expect your start date was in 1969, and you have the point set to interval logging. So, it interval logged for forty years every minute, perhaps.

Phillip Weeks

@phildunlap
0_1516642296626_8a63e2ff-d520-4b49-b112-bcea74256424-image.png

0_1516642366804_b09d491d-2b58-4be2-b244-20a709d75990-image.png
I believe I mixed up the days and the months so end date was before start date, not a script error because I didn't modify the script, just recalculating and it worked on the next run.

phildunlap

Ah, got it. You see the checkbox, "Delete existing data in range" in the date range dialog though, yes? That is how to purge the data before recalculating it for a meta point.

Phillip Weeks

@phildunlap yes apparently I was not paying attention and though both use similar date input format. It is difficult to remember sometimes since the date is autofilled and 5/1/2017 vs. 1/5/2017 looks similar. No problem if I pay attention.

mihairosu

This actually depends on how your invoke it. If you use the UI, then you run in a Jetty thread. You can run a few of these side by side. It's true that a particular point will only do a history generation in a single thread. If you were to invoke the meta history generation code, then it would run in the thread that invoked that (like the meta point or a scripting data source).

I know you've mentioned this to me before. I guess it would be nice if it split the load and spawned multiple instances, if I am only running 1 point.

Hmm. I know your system has quite a bit of memory, but that still sounds memory related. Did it supply an error?

These:

WARN  2018-01-22T17:12:25,907 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:12:28,709 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:28:02,061 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:28:03,709 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:36:32,351 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-22T17:36:32,352 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-22T17:36:33,708 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:39:49,070 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:39:49,071 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full

I still haven't able to pin down the problem.

I'm not sure I know what you mean. How is this different than the validate button?

I can't find a validate button on the point details page, but that sounds like what I've been looking for.

I am sure there could be some traction internally here to make API endpoints for meta point histories. Then you'd have a temporary resource you could get the history generation information from. Currently you would have to set values out from the meta point, or check the meta point's values.

It's not a big deal really, but I'm not a power user, so I wouldn't be doing anything but checking a GUI most of the time. Since event generation is time based, it could work quite well with a progress bar.

Ask and ye may receive,

if( my.time + 60000 < source.time ) { //We're at least a minute after
  var metaEditDwr = new com.serotonin.m2m2.meta.MetaEditDwr();
  metaEditDwr.generateMetaPointHistory(
      my.getDataPointWrapper().getId(), my.time+1, CONTEXT.getRuntime(), false);
  //Arguments are, dataPointId, long from, long to, boolean deleteExistingData
  //my.time+1 because first argument is inclusive, and we have value there
}
//Your regular script here.

Woah, amazing, thank you!

Purge now From: To: ---------------------screenshot

Woah, where is this screenshot from that I can purge a specific date range? I've been looking for something like this. I thought you could only purge "Purge data older than" or "Purge all".

phildunlap

The purge between is activated by the "Time range" checkbox on the edit data point page.

Where are you seeking the validate button? The validate button is the green checkmark next to the script composition box:

Do you also have an hs_err file in your Mango/ directory corresponding to the time of the crash?

mihairosu

The purge between is activated by the "Time range" checkbox on the edit data point page.

I'm blind again. I cannot see the any time range purge related checkbox.

0_1516713089259_time rage crop.jpg

The validate button is the green checkmark next to the script composition box:

You know I never knew that was a button! That's very helpful.

Do you also have an hs_err file in your Mango/ directory corresponding to the time of the crash?

Strange that I never noticed this before:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=916, tid=0x00007fd69a2eb700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimite$
#

It's 32-bit java!

We're running Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64).

I don't recall installing specifically 32-bit java. I should also have 64-bit java. Is this something that needs to be changed on Mango side?

mihairosu

Feels like I'm hijacking this thread, but I'm also getting this error on your script:

Script error: TypeError: my.getDataPointWrapper().getId is not a function

The only thing I did was substitute mypoint.time for source.time. Not sure that was correct. If I leave it as source.time, it shows this:

Script error: ReferenceError: "source" is not defined

phildunlap

No change is required in Mango to switch to the 64 bit JVM, you just need to install that Java.

Both the absence of the purge between checkbox and the my.getDataPointWrapper().getId() function were released in 3.3, so you would need to update to use those. In 3.2 you have to get the ID through the DataPointDao from my.getDataPointWapper().getXid()

You are correct in renaming "source" in what I posted.

It may be a minor hijack, but I think Phillip's issues were looked into. I tried to keep an eye on that.

phildunlap

I think you already had the 64 bit version?

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)

^from the hs_err file

mihairosu

@phildunlap said in History Generation Still Running or Finished??:

I think you already had the 64 bit version?
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
^from the hs_err file

java --version

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Any idea what's going on?

I'm going to upgrade the java version to the newest (162) 64 bit...

mihairosu

@phildunlap said in History Generation Still Running or Finished??:

No change is required in Mango to switch to the 64 bit JVM, you just need to install that Java.

Both the absence of the purge between checkbox and the my.getDataPointWrapper().getId() function were released in 3.3, so you would need to update to use those. In 3.2 you have to get the ID through the DataPointDao from my.getDataPointWapper().getXid()

You are correct in renaming "source" in what I posted.

It may be a minor hijack, but I think Phillip's issues were looked into. I tried to keep an eye on that.

I cannot update:

0_1516733706470_Mango cannot update.png

I could not find any other module named dashboards. Is that something I need to remove or can I replace it somehow?

phildunlap

There should be a module named 'dashboards' in the modules list. You can mark it for deletion and restart, then update. After 3.3 marking it for deletion would be sufficient, but alas. If it isn't there, there will be a Mango/web/modules/dashboards directory most likely that you can delete while Mango is off.

My suspicion on being OOM-killed is that you may have allocated too much memory to Java, and the operating system had another customer that it wished to supply memory to, and so it picked the giant, long-hanging memory that was your Mango and killed it for its memory. It's good to identify what this other process may have triggered that. I would guess you could lower your memory allocation to Mango since you're on MySQL now (I think), which will have its own process handling requests. With H2, the memory available to it is the same as the memory available to Mango.

Jared Wiltshire

@mihairosu dashboards has been replaced with a module named mangoUI. Remove dashboards and install mangoUI,

mihairosu

@jared-wiltshire said in History Generation Still Running or Finished??:

@mihairosu dashboards has been replaced with a module named mangoUI. Remove dashboards and install mangoUI,

Success. I thought I did this once before, which is what confused me in the back of my mind. Anyway thank you.

As far as the java issues are concerned, I'll spend more time later diagnosing, I don't have much time at the moment.