Mango Using 100% CPU

[dan]

hmm, looks like there's something on the wing...

It's cycling the startup script now, re-spawning over and over. There are a couple of issues I see in the ma.log

ERROR 2018-09-05 08:26:33,604 (org.springframework.web.context.ContextLoader.initWebApplicationContext:220) - Context initialization failed
org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Method 'delegatingApplicationListener' must not be private, final or static; change the method's modifiers to continue
Offending resource: class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]

ERROR 2018-09-05 08:26:26,624 (org.springframework.web.context.ContextLoader.initWebApplicationContext:220) - Context initialization failed
org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Method 'delegatingApplicationListener' must not be private, final or static; change the method's modifiers to continue
Offending resource: class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]
        at org.springframework.beans.factory.parsing.FailFastProblemReporter.error(FailFastProblemReporter.java:68)
        at org.springframework.context.annotation.ConfigurationClassMethod.validate(ConfigurationClassMethod.java:61)
        at org.springframework.context.annotation.ConfigurationClass.validate(ConfigurationClass.java:139)
        at org.springframework.context.annotation.ConfigurationClassParser.validate(ConfigurationClassParser.java:167)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:198)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:142)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:599)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:640)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:229)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1208)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:586)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:449)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:58)
        at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:89)
        at org.eclipse.jetty.server.Server.doStart(Server.java:258)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:58)
        at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:735)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:177)
        at com.serotonin.m2m2.Main.main(Main.java:106)
ERROR 2018-09-05 08:26:26,794 (com.serotonin.m2m2.Lifecycle.terminate:360) - com.serotonin.provider.InputStreamEPollProvider
ERROR 2018-09-05 08:26:26,796 (com.serotonin.m2m2.Lifecycle.terminate:368) - com.serotonin.provider.ProcessEPollProvider
ERROR 2018-09-05 08:26:26,818 (com.serotonin.m2m2.Main.main:111) - Error during initialization
java.lang.NullPointerException
        at com.serotonin.m2m2.Lifecycle.configureStartupDwr(Lifecycle.java:792)
        at com.serotonin.m2m2.Lifecycle.webServerInitialize(Lifecycle.java:741)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:177)
        at com.serotonin.m2m2.Main.main(Main.java:106)

Then in the dmesg output it looks like it's cycling the process. I might try a reboot, I'll also double check permissions haven't changed after the extraction of the new version?

[318213.951971] init: mango main process ended, respawning
[318220.950731] init: mango main process ended, respawning
[318227.939019] init: mango main process ended, respawning
[318235.510695] init: mango main process ended, respawning
[318242.501926] init: mango main process ended, respawning

[phildunlap]

Can you check if you have any override class files by doing a,

find /opt/mango/overrides/

and sharing any output. I also wonder if 2.4 is so long ago that you would have manually had to delete the contents of the Mango/lib/ directory before unzipping the update (the Mango/bin/ scripts didn't always perform the upgrade for you, I suspect 2.4 is that way) . I would check your Mango/lib/ directory for jars with duplicate names like ma-core-x.y.z.jar

I would expect you're running it under the root user, so permissions shouldn't be an issue.

[dan]

Ah looks like it hadn't finished deleting the lib folder.
File operations on these ES boxes seem a little sluggish at times.

I'll try again now.

Cheers

[dan]

OK. Now it's looking better, but it's not starting the modules for some reason. Should it be removing the old modules in favour of new ones?
https://pastebin.com/nFwmvse0

[dan]

this was the result of the overrides folder...

root@water:/opt/mango# find /opt/mango/overrides/
/opt/mango/overrides/
/opt/mango/overrides/properties
/opt/mango/overrides/properties/env.properties
/opt/mango/overrides/lib
/opt/mango/overrides/lib/libjdns_sd.so

[phildunlap]

Hmm. From your log it appears you maybe only downloaded the core, and not the bundle (which also has the modules). I would suggest downloading the 2.8.8 bundle from https://store.infiniteautomation.com/core and placing it in your Mango/ directory with the name m2m2-core-2.8.8.zip then trying to start Mango. It should reinstall 2.8.8 and have the 2.8.x versions of the modules in the Mango/web/modules directory (which you could also just copy out of the zip bundle).

[dan]

Ah I see, that makes sense ;)
I'll try that and come back to you.

Cheers

[dan]

Hey Phil, I can't seem to find a bundle pack with all the modules..

Am I looking in the right place?

[phildunlap]

Yeah, that's the right place. My mistake in using the world bundle, I thought it appeared there but it doesn't not. The full core downloads in the top section of the page (two for the latest 3.x enterprise and free, two for 2.8.8) contain modules. Where the cores in the section labelled 'Core Only - No Modules' do not.contain modules.

[dan]

Thanks for that, I'll try that one.

Cheers

[dan]

ok, so I went back to the original upgrade process, this time extracting the zip which contains the modules also.

now the mango process seems to be starting then restarting, looping..
one of the errors I see is:

ERROR 2018-09-07 11:39:27,544 (com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext:971) -
java.lang.NullPointerException
        at com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext(Lifecycle.java:967)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:324)
        at com.serotonin.m2m2.Main.main(Main.java:115)
ERROR 2018-09-07 11:39:27,555 (com.serotonin.m2m2.Main.main:119) - Error during initialization
java.lang.NullPointerException
        at com.serotonin.m2m2.Lifecycle.configureStartupDwr(Lifecycle.java:1023)
        at com.serotonin.m2m2.Lifecycle.fullyInitializeWebContext(Lifecycle.java:974)
        at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:324)
        at com.serotonin.m2m2.Main.main(Main.java:115)

[phildunlap]

Oof. I have seen that error before, but haven't found its exact cause. Something is usually off kilter with either the installation of the core or the Java installation. If you try running up 2.8.8 in a new directory (new installation) do you still see that? What version of Java is installed? I would guess it's an older on! You can download the latest ARM 32bit JVM and unpack it over the existing JVM (which you may need to ls -l /etc/alternatives/java to locate, but it's probably at /usr/lib/jvm/) or you can unpack it someplace else and set JAVA_HOME in your /etc/environment

[dan]

I can run up a fresh instance with no data points etc and it seems to run, I'll check what version JVM is on there.
Cheers

[phildunlap]

Weird!

I would try copying your databases folder into the new installation. If that works, I would stop Mango, move the old installation directory, move the new one where the old one was, and start Mango again, which will get the license working.