History Generation Still Running or Finished??

[phildunlap]

Ah, got it. You see the checkbox, "Delete existing data in range" in the date range dialog though, yes? That is how to purge the data before recalculating it for a meta point.

[Phillip Weeks]

@phildunlap yes apparently I was not paying attention and though both use similar date input format. It is difficult to remember sometimes since the date is autofilled and 5/1/2017 vs. 1/5/2017 looks similar. No problem if I pay attention.

[mihairosu]

This actually depends on how your invoke it. If you use the UI, then you run in a Jetty thread. You can run a few of these side by side. It's true that a particular point will only do a history generation in a single thread. If you were to invoke the meta history generation code, then it would run in the thread that invoked that (like the meta point or a scripting data source).

I know you've mentioned this to me before. I guess it would be nice if it split the load and spawned multiple instances, if I am only running 1 point.

Hmm. I know your system has quite a bit of memory, but that still sounds memory related. Did it supply an error?

These:

WARN  2018-01-22T17:12:25,907 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:12:28,709 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:28:02,061 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:28:03,709 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:36:32,351 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-22T17:36:32,352 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-22T17:36:33,708 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-22T17:39:49,070 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-22T17:39:49,071 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full

I still haven't able to pin down the problem.

I'm not sure I know what you mean. How is this different than the validate button?

I can't find a validate button on the point details page, but that sounds like what I've been looking for.

I am sure there could be some traction internally here to make API endpoints for meta point histories. Then you'd have a temporary resource you could get the history generation information from. Currently you would have to set values out from the meta point, or check the meta point's values.

It's not a big deal really, but I'm not a power user, so I wouldn't be doing anything but checking a GUI most of the time. Since event generation is time based, it could work quite well with a progress bar.

Ask and ye may receive,

if( my.time + 60000 < source.time ) { //We're at least a minute after
  var metaEditDwr = new com.serotonin.m2m2.meta.MetaEditDwr();
  metaEditDwr.generateMetaPointHistory(
      my.getDataPointWrapper().getId(), my.time+1, CONTEXT.getRuntime(), false);
  //Arguments are, dataPointId, long from, long to, boolean deleteExistingData
  //my.time+1 because first argument is inclusive, and we have value there
}
//Your regular script here.

Woah, amazing, thank you!

Purge now From: To: ---------------------screenshot

Woah, where is this screenshot from that I can purge a specific date range? I've been looking for something like this. I thought you could only purge "Purge data older than" or "Purge all".

[phildunlap]

The purge between is activated by the "Time range" checkbox on the edit data point page.

Where are you seeking the validate button? The validate button is the green checkmark next to the script composition box:

Do you also have an hs_err file in your Mango/ directory corresponding to the time of the crash?

[mihairosu]

The purge between is activated by the "Time range" checkbox on the edit data point page.

I'm blind again. I cannot see the any time range purge related checkbox.

The validate button is the green checkmark next to the script composition box:

You know I never knew that was a button! That's very helpful.

Do you also have an hs_err file in your Mango/ directory corresponding to the time of the crash?

Strange that I never noticed this before:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=916, tid=0x00007fd69a2eb700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimite$
#

It's 32-bit java!

We're running Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64).

I don't recall installing specifically 32-bit java. I should also have 64-bit java. Is this something that needs to be changed on Mango side?

[mihairosu]

Feels like I'm hijacking this thread, but I'm also getting this error on your script:

Script error: TypeError: my.getDataPointWrapper().getId is not a function 

The only thing I did was substitute mypoint.time for source.time. Not sure that was correct. If I leave it as source.time, it shows this:

Script error: ReferenceError: "source" is not defined

[phildunlap]

No change is required in Mango to switch to the 64 bit JVM, you just need to install that Java.

Both the absence of the purge between checkbox and the my.getDataPointWrapper().getId() function were released in 3.3, so you would need to update to use those. In 3.2 you have to get the ID through the DataPointDao from my.getDataPointWapper().getXid()

You are correct in renaming "source" in what I posted.

It may be a minor hijack, but I think Phillip's issues were looked into. I tried to keep an eye on that.

[phildunlap]

I think you already had the 64 bit version?

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)

^from the hs_err file

[mihairosu]

@phildunlap said in History Generation Still Running or Finished??:

I think you already had the 64 bit version?

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)

^from the hs_err file

java --version

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Any idea what's going on?

I'm going to upgrade the java version to the newest (162) 64 bit...

[mihairosu]

@phildunlap said in History Generation Still Running or Finished??:

No change is required in Mango to switch to the 64 bit JVM, you just need to install that Java.

Both the absence of the purge between checkbox and the my.getDataPointWrapper().getId() function were released in 3.3, so you would need to update to use those. In 3.2 you have to get the ID through the DataPointDao from my.getDataPointWapper().getXid()

You are correct in renaming "source" in what I posted.

It may be a minor hijack, but I think Phillip's issues were looked into. I tried to keep an eye on that.

I cannot update:

I could not find any other module named dashboards. Is that something I need to remove or can I replace it somehow?

[phildunlap]

There should be a module named 'dashboards' in the modules list. You can mark it for deletion and restart, then update. After 3.3 marking it for deletion would be sufficient, but alas. If it isn't there, there will be a Mango/web/modules/dashboards directory most likely that you can delete while Mango is off.

My suspicion on being OOM-killed is that you may have allocated too much memory to Java, and the operating system had another customer that it wished to supply memory to, and so it picked the giant, long-hanging memory that was your Mango and killed it for its memory. It's good to identify what this other process may have triggered that. I would guess you could lower your memory allocation to Mango since you're on MySQL now (I think), which will have its own process handling requests. With H2, the memory available to it is the same as the memory available to Mango.

[Jared Wiltshire]

@mihairosu dashboards has been replaced with a module named mangoUI. Remove dashboards and install mangoUI,

[mihairosu]

@jared-wiltshire said in History Generation Still Running or Finished??:

@mihairosu dashboards has been replaced with a module named mangoUI. Remove dashboards and install mangoUI,

Success. I thought I did this once before, which is what confused me in the back of my mind. Anyway thank you.

As far as the java issues are concerned, I'll spend more time later diagnosing, I don't have much time at the moment.

[Phillip Weeks]

Hi Phil I am tasked to regenerate 144 meta points over the past 3 months (approx, 10000 values on each point). I purge the data range beforehand Nov 1 2017 - Jan 22 2018 (most current) ...

tried the two ways to purge the data beforehand ..
First through the point definition window and successfully purged the time frame and regeneration works every time perfectly.

However, when using the new method of purging before the regeneration
(delete existing data in range checkbox) it always causes my 12gb heap server to run out of memory and when I restart the process has created 4 million records back to 1970. I purge these values back out in the point definition window and restart the history script leaving the delete existing data unchecked this time and it works.

It only creates these erroneous records and runs out of memory if I try to zap the data from this check box method.

:
ERROR 2018-01-23T09:33:07,929 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:728) - 1 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-23T09:33:42,398 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:728) - 1 BWB Task Failures, first is: Task Queue Full
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007b7480000, 5242880, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 5242880 bytes for committing reserved memory.
An error report file with more information is saved as:
/opt/mango/hs_err_pid15338.log
ma-start: no restart flag found, not restarting MA
ma-start: MA done

Native memory allocation (mmap) failed to map 5242880 bytes for committing reserved memory.
An error report file with more information is saved as:
/opt/mango/hs_err_pid15338.log
ma-start: no restart flag found, not restarting MA
ma-start: MA done

[phildunlap]

So it does. I was able to reproduce. It's odd to me that it depends on the delete, since the line that is causing the interval logging task to fire for those 30+ years has been in the generate history code for ages, but I too saw that. I would certainly say that's a bug, and the fix looks incredibly simple so I would think we will release a new Meta module pretty soon. The issue for this bug is here: https://github.com/infiniteautomation/ma-core-public/issues/1206

Thanks for bringing this to our attention!

[Phillip Weeks]

Well glad to help, while you are on this it would be also great if the checkbox delete existing data would remember the last date range setup over the calculated range making it easier to batch runs and of course also change it to the same date format as the point edit window for consistency.
..and our thanks is to you and the mango team for listening and improving this important tool.

[phildunlap]

Did you see the function I provided Mihai in the script? You could write a loop to call that function and set values out to some alphanumeric point along the way (just not between generations). Something like,

var pointsList = DataPointQuery.query("dataSourceXid=DS_XID");
var metaDwr = new com.serotonin.m2m2.meta.MetaEditDwr();
var now = new Date().getTime();
//var pvd = com.serotonin.m2m2.Common.databaseProxy.newPointValueDao(); 
for(var k = 0; k < pointsList.length; k+=1) {
    //Given the issue discussed here with the delete before checkbox, you may want to call,
    //pvd.deletePointValuesBetween(pointsList[k].getId(), 0, now);
    metaDwr.generateMetaPointHistory(pointsList[k].getId(), 0, now, true);
    //alphanum.set("Finished with point: " + pointsList[k].getId());
}

This will regenerate all meta points on data source "DS_XID" from 0 (1970) to now, deleting what exists already, even when you only press validate in the script window.

[Phillip Weeks]

ok sounds great and so can I replace the 0, now) with periodBegin , periodEnd right?
guess I also could also use the pvd.deletePointValuesBetween in other scripts where I want to delete before I manipulate original logged values in the same way before setting a new value correct? Not only for meta points I mean.

so Jan 3 - 5 2018

var periodBegin = new Date(2018, 0, 3);
periodBegin.setHours(0);
periodBegin.setMinutes(0);
periodBegin.setSeconds(0);
periodBegin.setMilliseconds(0);

var periodEnd = new Date(2018, 0, 5);
periodEnd.setHours(12);
periodEnd.setMinutes(0);
periodEnd.setSeconds(0);
periodEnd.setMilliseconds(0);

[phildunlap]

Yes all sounds good. I do believe Java will cast a Date to a long by using the getTime() but you may as well call it yourself, i.e. periodStart.getTime() and periodEnd.getTime() in the actual call to generateMetaPointHistory

To invoke the method on pvd you will need to create it as shown in that script.

[Phillip Weeks]

so where do I run this? .. I tried in a scripting source but is complains of meta data source being cast to the scripting data source ... doI create another metaDS for this to run inside of ? OK that worked in another meta source. Thanks