Mango System starts up and operates, but now when I log into the editor the mango system comes to a screeching halt? Any thoughts?

phildunlap

Hi srennhack,

Out of memory! Have you looked into increasing your Java Heap space with one of the Mango/bin/ext-avaiable/ scripts?

If it worked a few weeks ago, has anything changed? If nothing changed, is your events table large? Can you try purging it?

The stack trace would indicate a meta point is the thread that actually wasn't able to allocate more memory. There's nothing crazy going on like...

while(true) { //Cause memory errors!
  this[new Date().getTime()] = new Date();
}

right :D? Or, also dangerous could be

p.pointValuesSince(0);

if you have a lot of point values.

srennhack

Thank you for your response, unfortunately I did not initialize this system originally, and have only been working with it for 6-months or so. I think Java was updated on this machine about 6 weeks ago. How would I go about increasing Java Heap Space? Steps to do that? Where/how would I find the information about the stack trace that you show? I am not a programmer as such, and have limited knowledge about this system. We do have around 500 points or so that we are monitoring, and I am wanting to add about 100 more.

phildunlap

https://forum.infiniteautomation.com/topic/2884/java-out-of-memory You move a file from Mango/bin/ext-available to Mango/bin/ext-enabled and modify its parameters to your machine if need be, then restart Mango.

You posted the stack trace. The most important line for googling would have been the last line,

Caused by: java.lang.OutOfMemoryError: Java heap space

The other calls in the stack trace show it was in your MetaPointLocatorRT.execute method when this thread got that error. This means it is possible your meta script is allocating memory poorly (as in my example while loop) but it's also possible the meta point just experienced the system's memory starvation and didn't itself contribute too much. You could try running Mango with your Meta Data Source disabled, and see if something in the scripting is the cause.

I would expect that you updated Java from a Java 8 to a later Java 8 and so I wouldn't expect that to have caused it.

srennhack

OK, will work on that and let you know what happens.

srennhack

Tried increasing Java space, but still the issue persists. maybe the following error messages will help? I'm at a loss at this point.

srennhack

More stack trace information; I cant log on to the editor pages to make any changes or edit what is there.

phildunlap

Which Mango/bin/ext-available script did you move to Mango/bin/ext-enabled/ ? How much memory are you trying to allocate?

If it worked a few weeks ago, has anything changed? If nothing changed, is your events table large? Can you try purging it?

srennhack

Memory-Large script was move to Mango/bin/ext-enabled/
How do I purge the events table if I can't get into the editor?

srennhack

The last time I was able to get into the editor, I added several new data sources AB PLC ethernet IP links.

srennhack

That was probably 3-4 weeks ago.

phildunlap

I was trying to suss out and ensure you've used the memory-large.bat, not the .sh

You can purge the events by shutting Mango down, using Mango/bin/h2-web-console.bat (edit to have the -webAllowOthers if you're not working on the machine running this and you need to connect to the web console from another IP) and then running

delete from userEvents;
delete from events;

You can find your login information for the web console in your env.properties file. You may also be able to access the interface if you start Mango in safe mode (by creating the file Mango/SAFE and then starting Mango), as some of your stack traces indicate your scripts may be one of the offenders as well. Then you could purge your events from the Purge Settings section of the system settings page.

srennhack

OK, shutting down mango, I am not working on the machine running mango right now, but it is right here next to me.
so do i need to go through changing the h2-web-console.bat on the mango computer?

phildunlap

Yes.

srennhack

OK I have done that, now how do I connect to the web console from this computer? Remember I'm not a programmer, and am trying to learn as I go here.

phildunlap

No worries.

@echo off
rem
rem | Copyright (C) 2006-2015 //infintie Automation. All rights reserved.
rem | @author Phillip Dunlap
rem
rem | Runs H2 Web Console. (Script Version 1.0.0 - To be run from MA_HOME\bin)

rem | Define the port to start H2 on...
set H2_PORT=8085

rem | Check if MA_HOME is properly defined
if "%MA_HOME%" == "" goto useCD
if exist "%MA_HOME%\bin\ma-start.bat" goto okHome
echo The MA_HOME environment variable is not defined correctly: %MA_HOME%. Trying the current directory instead...

rem | Check if the current directory is ok to use
:useCD
set BIN_DIR=%CD%
pushd ..
set MA_HOME=%CD%
popd
if exist "%MA_HOME%\bin\ma-start.bat" goto okHome

rem | Don't know where home is.
echo Cannot determine the MA home directory
goto end

rem | Found a good home. Carry on...
:okHome
echo Using %MA_HOME% as MA_HOME

set EXECJAVA=java
if "%JAVA_HOME%" == "" goto gotJava
set EXECJAVA=%JAVA_HOME%\bin\java

:gotJava
echo Using Java at %EXECJAVA%

rem | Put the whole lib folder on CP because we cannot do h2*.jar...
rem | Add -webAllowOthers if you will be using the web console from an origin other than localhost
"%EXECJAVA%" -cp "%MA_HOME%\lib\*" org.h2.tools.Server -web -webPort "%H2_PORT%" -baseDir "%MA_HOME%\"

:end

So -webAllowOthers has been added somewhere on the "%EXECJAVA% line, probably best as ... -web -webAllowOthers -webPort ...

Now run the h2-web-console.bat script, and navigate to http://[the computer ip]:8085

You can see the port is defined in the script as H2_PORT

srennhack

ok in the H2 port.

srennhack

I have been out of the office for several weeks, and am just getting back to this. I am still unable to get into the developer without the mango system freezing up. Here is a screen shot of when I tried to go into the reports at 12:08 today. The hard drive seems to be searching constantly never stops.![alt text]( image url)

phildunlap

Hi srennhack,

Of the steps I have offered, which have you done? Did you purge your events tables?

Do you still have out of memory errors in your Mango/logs/ma.log file?

srennhack

The only step not tried is starting Mango in the safe mode. I did not see out of memory errors today.

phildunlap

Ah. I asked because there events didn't load in your header in the screenshot, which would lead me to believe you still have a very large events table (or you recently purged and have no events in the events table). The exception from the Meta point is unrelated to the performance issue beyond its being an event. How large is your events table currently?

You can try purging your audit table too, running DELETE FROM audit; either in the H2 web console or in the SQL console page.

There can be a caching issue on the excel reports page sometimes where the first time you load the page it doesn't pull up the list of templates. If you refresh this almost always brings up the list.

If the hard drive is going crazy, is it with reads or writes? If you have tons of meta points running often doing .past(MINUTE, 5) type operations, that would be a heavy read load. There are solutions to convert that over to using the cache. If you have lots of point values waiting to be written (check this on your /internal/status.shtm page), you can adjust the NoSQL settings to delay more for small batches and increase the small batch size.