Mango Using 100% CPU
dan last edited by dan
Re: [Mango consuming 100% CPU](web interface not responding)
Hey Guys, I reference whileoneloop's similar issues, although I'm having the same issues on a number of mango installations I have around the traps recently. They are all responding very slowly and timing out on the web interface and the CPU on the box running them is using over 100%.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2678 root 20 0 711504 365344 13468 S 110.6 17.6 73:34.81 java
I'm not quite sure where to go from here.
I can't find where whileoneloop was talking about putting the JAVAOPTS into the startup file, would this be ma-start.sh?
This unit is an older MangoES hardware unit, it's running 2.4.2 still. I wonder if I should upgrade to the latest version. I guess the enterprise licence it came with won't upgrade?
From what I recall of 2.4.2, you very likely will see the problem resolved entirely by upgrading to 2.8.8. I can't recall if that means you're on a DERBY database, but if so I would advise converting to MySQL before updating.
I guess the enterprise licence it came with won't upgrade?
MangoES licenses are similar to Enterprise licenses in that they are unrestricted. A difference is that they cannot be transferred to other machines. Upgrade licenses for 2 to 3 on MangoES licenses are $500, I believe.
Hey Phil, thanks for the info. I'll try the upgrade to 2.8.8.
I'm not quite sure how to tell which database it's using, I do see info about Mango NoSQL in the System Settings page. It doesn't seem very large, although I did purge a heap of historic data when I first noticed it going slow in the GUI.
Or is it the top part... H2? That's SQL I think?
I'll give it a go, I've copied the whole folder sideways, I see the reports folder is massive, it doesn't seem to purge reports after it's sent them via email.
phildunlap last edited by phildunlap
see the reports folder is massive, it doesn't seem to purge reports after it's sent them via email.
I think that was fixed somewhere in the 2.7 series.
I'm not quite sure how to tell which database it's using,
Your last image reveals it to be the H2 database (so you have a Mango/databases/mah2.h2.db file, I would guess). This means you shouldn't have to worry about a database conversion before doing the upgrade.
That's SQL I think?
It is. Derby, H2, and MySQL are all SQL databases.
I'll give it a go, I've copied the whole folder sideways,
Sounds good. If that doesn't solve the issue, it could be caused by a few other things like,
- A meta point with multiple points from the same data source updating context. This used to drive CPU usage up quite far.
- An infinite event loop --> point A generates an event for B, B for A, etc. (less likely in 2.4 since fewer things had context update options)
- The events table is large.
I'm not sure if that 2GB size reflects the NoSQL, the H2, or both, but you may also see performance improvements by doing a restore, via: create a backup from the SQL backup configuration section of the system settings page, stopping Mango, moving the Mango/databases/mah2.h2.db file for safe keeping, starting Mango and doing a restore. This should shrink your H2 database substantially, which should accelerate everything some. We can investigate that if necessary.
Thanks for that I'll see how the upgrade goes..
hmm, looks like there's something on the wing...
It's cycling the startup script now, re-spawning over and over. There are a couple of issues I see in the ma.log
ERROR 2018-09-05 08:26:33,604 (org.springframework.web.context.ContextLoader.initWebApplicationContext:220) - Context initialization failed org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Method 'delegatingApplicationListener' must not be private, final or static; change the method's modifiers to continue Offending resource: class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class] ERROR 2018-09-05 08:26:26,624 (org.springframework.web.context.ContextLoader.initWebApplicationContext:220) - Context initialization failed org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Method 'delegatingApplicationListener' must not be private, final or static; change the method's modifiers to continue Offending resource: class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class] at org.springframework.beans.factory.parsing.FailFastProblemReporter.error(FailFastProblemReporter.java:68) at org.springframework.context.annotation.ConfigurationClassMethod.validate(ConfigurationClassMethod.java:61) at org.springframework.context.annotation.ConfigurationClass.validate(ConfigurationClass.java:139) at org.springframework.context.annotation.ConfigurationClassParser.validate(ConfigurationClassParser.java:167) at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:198) at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:142) at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:599) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407) at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:640) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:229) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1208) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:586) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:449) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:58) at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:89) at org.eclipse.jetty.server.Server.doStart(Server.java:258) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:58) at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:735) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:177) at com.serotonin.m2m2.Main.main(Main.java:106) ERROR 2018-09-05 08:26:26,794 (com.serotonin.m2m2.Lifecycle.terminate:360) - com.serotonin.provider.InputStreamEPollProvider ERROR 2018-09-05 08:26:26,796 (com.serotonin.m2m2.Lifecycle.terminate:368) - com.serotonin.provider.ProcessEPollProvider ERROR 2018-09-05 08:26:26,818 (com.serotonin.m2m2.Main.main:111) - Error during initialization java.lang.NullPointerException at com.serotonin.m2m2.Lifecycle.configureStartupDwr(Lifecycle.java:792) at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:741) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:177) at com.serotonin.m2m2.Main.main(Main.java:106)
Then in the dmesg output it looks like it's cycling the process. I might try a reboot, I'll also double check permissions haven't changed after the extraction of the new version?
[318213.951971] init: mango main process ended, respawning [318220.950731] init: mango main process ended, respawning [318227.939019] init: mango main process ended, respawning [318235.510695] init: mango main process ended, respawning [318242.501926] init: mango main process ended, respawning
phildunlap last edited by phildunlap
Can you check if you have any override class files by doing a,
and sharing any output. I also wonder if 2.4 is so long ago that you would have manually had to delete the contents of the Mango/lib/ directory before unzipping the update (the Mango/bin/ scripts didn't always perform the upgrade for you, I suspect 2.4 is that way) . I would check your Mango/lib/ directory for jars with duplicate names like ma-core-x.y.z.jar
I would expect you're running it under the root user, so permissions shouldn't be an issue.
Ah looks like it hadn't finished deleting the lib folder.
File operations on these ES boxes seem a little sluggish at times.
I'll try again now.
OK. Now it's looking better, but it's not starting the modules for some reason. Should it be removing the old modules in favour of new ones?
this was the result of the overrides folder...
root@water:/opt/mango# find /opt/mango/overrides/ /opt/mango/overrides/ /opt/mango/overrides/properties /opt/mango/overrides/properties/env.properties /opt/mango/overrides/lib /opt/mango/overrides/lib/libjdns_sd.so
Hmm. From your log it appears you maybe only downloaded the core, and not the bundle (which also has the modules). I would suggest downloading the 2.8.8 bundle from https://store.infiniteautomation.com/core and placing it in your Mango/ directory with the name m2m2-core-2.8.8.zip then trying to start Mango. It should reinstall 2.8.8 and have the 2.8.x versions of the modules in the Mango/web/modules directory (which you could also just copy out of the zip bundle).
Ah I see, that makes sense ;)
I'll try that and come back to you.
Hey Phil, I can't seem to find a bundle pack with all the modules..
Am I looking in the right place?
Yeah, that's the right place. My mistake in using the world bundle, I thought it appeared there but it doesn't not. The full core downloads in the top section of the page (two for the latest 3.x enterprise and free, two for 2.8.8) contain modules. Where the cores in the section labelled 'Core Only - No Modules' do not.contain modules.
Thanks for that, I'll try that one.
ok, so I went back to the original upgrade process, this time extracting the zip which contains the modules also.
now the mango process seems to be starting then restarting, looping..
one of the errors I see is:
ERROR 2018-09-07 11:39:27,544 (com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext:971) - java.lang.NullPointerException at com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext(Lifecycle.java:967) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:324) at com.serotonin.m2m2.Main.main(Main.java:115) ERROR 2018-09-07 11:39:27,555 (com.serotonin.m2m2.Main.main:119) - Error during initialization java.lang.NullPointerException at com.serotonin.m2m2.Lifecycle.configureStartupDwr(Lifecycle.java:1023) at com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext(Lifecycle.java:974) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:324) at com.serotonin.m2m2.Main.main(Main.java:115)
Oof. I have seen that error before, but haven't found its exact cause. Something is usually off kilter with either the installation of the core or the Java installation. If you try running up 2.8.8 in a new directory (new installation) do you still see that? What version of Java is installed? I would guess it's an older on! You can download the latest ARM 32bit JVM and unpack it over the existing JVM (which you may need to
ls -l /etc/alternatives/javato locate, but it's probably at
/usr/lib/jvm/) or you can unpack it someplace else and set JAVA_HOME in your
I can run up a fresh instance with no data points etc and it seems to run, I'll check what version JVM is on there.
I would try copying your databases folder into the new installation. If that works, I would stop Mango, move the old installation directory, move the new one where the old one was, and start Mango again, which will get the license working.