Mango Using 100% CPU

dan

Re: [Mango consuming 100% CPU](web interface not responding)

Hey Guys, I reference whileoneloop's similar issues, although I'm having the same issues on a number of mango installations I have around the traps recently. They are all responding very slowly and timing out on the web interface and the CPU on the box running them is using over 100%.


  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2678 root      20   0  711504 365344  13468 S 110.6 17.6  73:34.81 java

I'm not quite sure where to go from here.
I can't find where whileoneloop was talking about putting the JAVAOPTS into the startup file, would this be ma-start.sh?

Regards
Dan

dan

This unit is an older MangoES hardware unit, it's running 2.4.2 still. I wonder if I should upgrade to the latest version. I guess the enterprise licence it came with won't upgrade?

phildunlap

Hi Dan,

From what I recall of 2.4.2, you very likely will see the problem resolved entirely by upgrading to 2.8.8. I can't recall if that means you're on a DERBY database, but if so I would advise converting to MySQL before updating.

I guess the enterprise licence it came with won't upgrade?

MangoES licenses are similar to Enterprise licenses in that they are unrestricted. A difference is that they cannot be transferred to other machines. Upgrade licenses for 2 to 3 on MangoES licenses are $500, I believe.

dan

Hey Phil, thanks for the info. I'll try the upgrade to 2.8.8.
I'm not quite sure how to tell which database it's using, I do see info about Mango NoSQL in the System Settings page. It doesn't seem very large, although I did purge a heap of historic data when I first noticed it going slow in the GUI.

0_1536095846731_8d0a68e5-346e-4298-bc06-b8386c595e82-image.png

Or is it the top part... H2? That's SQL I think?

0_1536095930518_edd784bc-3427-400c-bab6-dbc168218301-image.png

I'll give it a go, I've copied the whole folder sideways, I see the reports folder is massive, it doesn't seem to purge reports after it's sent them via email.

Cheers
Dan

phildunlap

see the reports folder is massive, it doesn't seem to purge reports after it's sent them via email.

I think that was fixed somewhere in the 2.7 series.

I'm not quite sure how to tell which database it's using,

Your last image reveals it to be the H2 database (so you have a Mango/databases/mah2.h2.db file, I would guess). This means you shouldn't have to worry about a database conversion before doing the upgrade.

That's SQL I think?

It is. Derby, H2, and MySQL are all SQL databases.

I'll give it a go, I've copied the whole folder sideways,

Sounds good. If that doesn't solve the issue, it could be caused by a few other things like,

A meta point with multiple points from the same data source updating context. This used to drive CPU usage up quite far.
An infinite event loop --> point A generates an event for B, B for A, etc. (less likely in 2.4 since fewer things had context update options)
The events table is large.

I'm not sure if that 2GB size reflects the NoSQL, the H2, or both, but you may also see performance improvements by doing a restore, via: create a backup from the SQL backup configuration section of the system settings page, stopping Mango, moving the Mango/databases/mah2.h2.db file for safe keeping, starting Mango and doing a restore. This should shrink your H2 database substantially, which should accelerate everything some. We can investigate that if necessary.

dan

Thanks for that I'll see how the upgrade goes..

Cheers
Dan

dan

hmm, looks like there's something on the wing...

It's cycling the startup script now, re-spawning over and over. There are a couple of issues I see in the ma.log

ERROR 2018-09-05 08:26:33,604 (org.springframework.web.context.ContextLoader.initWebApplicationContext:220) - Context initialization failed
org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Method 'delegatingApplicationListener' must not be private, final or static; change the method's modifiers to continue
Offending resource: class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]

ERROR 2018-09-05 08:26:26,624 (org.springframework.web.context.ContextLoader.initWebApplicationContext:220) - Context initialization failed
org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Method 'delegatingApplicationListener' must not be private, final or static; change the method's modifiers to continue
Offending resource: class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]
        at org.springframework.beans.factory.parsing.FailFastProblemReporter.error(FailFastProblemReporter.java:68)
        at org.springframework.context.annotation.ConfigurationClassMethod.validate(ConfigurationClassMethod.java:61)
        at org.springframework.context.annotation.ConfigurationClass.validate(ConfigurationClass.java:139)
        at org.springframework.context.annotation.ConfigurationClassParser.validate(ConfigurationClassParser.java:167)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:198)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:142)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:599)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:640)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:229)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1208)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:586)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:449)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:58)
        at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:89)
        at org.eclipse.jetty.server.Server.doStart(Server.java:258)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:58)
        at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:735)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:177)
        at com.serotonin.m2m2.Main.main(Main.java:106)
ERROR 2018-09-05 08:26:26,794 (com.serotonin.m2m2.Lifecycle.terminate:360) - com.serotonin.provider.InputStreamEPollProvider
ERROR 2018-09-05 08:26:26,796 (com.serotonin.m2m2.Lifecycle.terminate:368) - com.serotonin.provider.ProcessEPollProvider
ERROR 2018-09-05 08:26:26,818 (com.serotonin.m2m2.Main.main:111) - Error during initialization
java.lang.NullPointerException
        at com.serotonin.m2m2.Lifecycle.configureStartupDwr(Lifecycle.java:792)
        at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:741)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:177)
        at com.serotonin.m2m2.Main.main(Main.java:106)

Then in the dmesg output it looks like it's cycling the process. I might try a reboot, I'll also double check permissions haven't changed after the extraction of the new version?

[318213.951971] init: mango main process ended, respawning
[318220.950731] init: mango main process ended, respawning
[318227.939019] init: mango main process ended, respawning
[318235.510695] init: mango main process ended, respawning
[318242.501926] init: mango main process ended, respawning

phildunlap

Can you check if you have any override class files by doing a,

find /opt/mango/overrides/

and sharing any output. I also wonder if 2.4 is so long ago that you would have manually had to delete the contents of the Mango/lib/ directory before unzipping the update (the Mango/bin/ scripts didn't always perform the upgrade for you, I suspect 2.4 is that way) . I would check your Mango/lib/ directory for jars with duplicate names like ma-core-x.y.z.jar

I would expect you're running it under the root user, so permissions shouldn't be an issue.

dan

Ah looks like it hadn't finished deleting the lib folder.
File operations on these ES boxes seem a little sluggish at times.

I'll try again now.

Cheers

dan

OK. Now it's looking better, but it's not starting the modules for some reason. Should it be removing the old modules in favour of new ones?
https://pastebin.com/nFwmvse0

dan

this was the result of the overrides folder...

root@water:/opt/mango# find /opt/mango/overrides/
/opt/mango/overrides/
/opt/mango/overrides/properties
/opt/mango/overrides/properties/env.properties
/opt/mango/overrides/lib
/opt/mango/overrides/lib/libjdns_sd.so

phildunlap

Hmm. From your log it appears you maybe only downloaded the core, and not the bundle (which also has the modules). I would suggest downloading the 2.8.8 bundle from https://store.infiniteautomation.com/core and placing it in your Mango/ directory with the name m2m2-core-2.8.8.zip then trying to start Mango. It should reinstall 2.8.8 and have the 2.8.x versions of the modules in the Mango/web/modules directory (which you could also just copy out of the zip bundle).

dan

Ah I see, that makes sense ;)
I'll try that and come back to you.

Cheers

dan

Hey Phil, I can't seem to find a bundle pack with all the modules..

0_1536187014714_47eecce0-06f3-412c-9179-3a0852d67aa8-image.png

Am I looking in the right place?

phildunlap

Yeah, that's the right place. My mistake in using the world bundle, I thought it appeared there but it doesn't not. The full core downloads in the top section of the page (two for the latest 3.x enterprise and free, two for 2.8.8) contain modules. Where the cores in the section labelled 'Core Only - No Modules' do not.contain modules.

dan

Thanks for that, I'll try that one.

Cheers

dan

ok, so I went back to the original upgrade process, this time extracting the zip which contains the modules also.

now the mango process seems to be starting then restarting, looping..
one of the errors I see is:

ERROR 2018-09-07 11:39:27,544 (com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext:971) -
java.lang.NullPointerException
        at com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext(Lifecycle.java:967)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:324)
        at com.serotonin.m2m2.Main.main(Main.java:115)
ERROR 2018-09-07 11:39:27,555 (com.serotonin.m2m2.Main.main:119) - Error during initialization
java.lang.NullPointerException
        at com.serotonin.m2m2.Lifecycle.configureStartupDwr(Lifecycle.java:1023)
        at com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext(Lifecycle.java:974)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:324)
        at com.serotonin.m2m2.Main.main(Main.java:115)

phildunlap

Oof. I have seen that error before, but haven't found its exact cause. Something is usually off kilter with either the installation of the core or the Java installation. If you try running up 2.8.8 in a new directory (new installation) do you still see that? What version of Java is installed? I would guess it's an older on! You can download the latest ARM 32bit JVM and unpack it over the existing JVM (which you may need to ls -l /etc/alternatives/java to locate, but it's probably at /usr/lib/jvm/) or you can unpack it someplace else and set JAVA_HOME in your /etc/environment

dan

I can run up a fresh instance with no data points etc and it seems to run, I'll check what version JVM is on there.
Cheers

phildunlap

Weird!

I would try copying your databases folder into the new installation. If that works, I would stop Mango, move the old installation directory, move the new one where the old one was, and start Mango again, which will get the license working.