<![CDATA[Mango 3 will not start after recent upgrade]]>Hello All,
I had Mango 3.3.1 running on a Centos 7.4 web server with Java 1.8.0_161 until yesterday.
I started a mango update and now cannot get Mango to start. I have restarted the server several times in my attempts to get things going again. but I am going in ever decreasing circles with the problem.

Here are the last few lines from ma.log from the time of the upgrade attempt:

INFO 2018-03-05T16:22:45,983 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:407) - UpgradeDownloader started
INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:411) - UpgradeDownloader: BACKUP
INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:424) - UpgradeDownloader: DOWNLOAD
INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.execute:109) - Starting database backup WorkItem.
INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem.
ERROR 2018-03-05T16:22:46,045 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
INFO 2018-03-05T16:22:46,139 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.backupMysqlWithOutDatabase:442) - Backup created successfully forMango3 in localhost:3306
ERROR 2018-03-05T16:22:47,132 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:48,204 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:49,263 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:50,322 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:51,387 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:52,448 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:53,511 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:54,570 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:55,646 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:56,705 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:57,770 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:58,829 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:22:59,887 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:23:00,947 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:23:02,008 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:23:03,068 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:23:04,126 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:23:05,188 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
ERROR 2018-03-05T16:23:06,248 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
INFO 2018-03-05T16:23:07,259 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:497) - UpgradeDownloader: INSTALL
INFO 2018-03-05T16:23:07,260 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:548) - UpgradeDownloader: RESTART
ERROR 2018-03-05T16:23:07,307 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
INFO 2018-03-05T16:23:12,267 (com.serotonin.m2m2.Lifecycle.terminate:382) - Mango Lifecycle terminating...
INFO 2018-03-05T16:23:13,741 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:72) - Terminating 1 LAST priority data sources in 8 threads.
INFO 2018-03-05T16:23:13,766 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:450) - Data source '- Demo 2 Meta' stopped
INFO 2018-03-05T16:23:13,841 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:102) - Termination of 1 LAST priority data sources took 100ms
INFO 2018-03-05T16:23:13,842 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:72) - Terminating 1 NORMAL priority data sources in 8 threads.
INFO 2018-03-05T16:23:13,861 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:450) - Data source '- Demo 2 raw' stopped
INFO 2018-03-05T16:23:13,942 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:102) - Termination of 1 NORMAL priority data sources took 100ms
INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:548) - All high priority tasks exited gracefully.
INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:554) - All medium priority tasks exited gracefully.
INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:559) - All low priority tasks exited gracefully.
INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.db.BasePooledProxy.terminateImpl:117) - Stopping database
INFO 2018-03-05T16:23:13,947 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:115) - Terminating NoSQL Batch Write Manager.
INFO 2018-03-05T16:23:13,947 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:242) - Terminating NoSQL Point Value Mover.
INFO 2018-03-05T16:23:13,947 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:249) - Terminating 1 Batch Writer Tasks.
INFO 2018-03-05T16:23:13,948 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:258) - 1 Batch Writer Tasks awaiting termination.
INFO 2018-03-05T16:23:14,446 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.joinTermination:173) - Batch Writer Task 0 Terminated.
INFO 2018-03-05T16:23:14,446 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:266) - Terminating NoSQL Stale Point Cleaner.
INFO 2018-03-05T16:23:14,446 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:269) - Terminating NoSQL Status Provider.
WARN 2018-03-05T16:23:14,447 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:274) - 10 queued point values discarded due to shutdown.
INFO 2018-03-05T16:23:14,447 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:123) - Closing NoSQL Database: Point Values
INFO 2018-03-05T16:23:14,458 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:131) - Closing NoSQL Database: Point Data
INFO 2018-03-05T16:23:14,466 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:145) - Closing Auxilary NoSQL Database: com.infiniteautomation.tsdb.impl.IasTsdbImpl
INFO 2018-03-05T16:23:14,473 (com.serotonin.m2m2.Lifecycle.terminate:491) - Mango Lifecycle terminated.

There were no further entries in that log since Mango would not restart.

I have copied here the lines from ma-script log since it is the only log updating from my attempts to restart:

Mon 5 Mar 16:23:14 GMT 2018 ma-start: upgrading core...
Mon 5 Mar 16:34:51 GMT 2018 ma.stop: No PID file found. Java process not stopped...
Mon 5 Mar 16:43:53 GMT 2018 ma-start: upgrading core...
Mon 5 Mar 17:05:47 GMT 2018 ma-start: upgrading core...
Tue 6 Mar 09:16:18 GMT 2018 ma-start: upgrading core...
Tue 6 Mar 11:43:37 GMT 2018 ma-start: upgrading core...
Tue 6 Mar 12:25:28 GMT 2018 ma-start: upgrading core...
Tue 6 Mar 12:37:49 GMT 2018 ma-start: upgrading core...

When I looked at this log early this morning, I thought maybe the Java process is causing the problem. I have been trying all morning here to verify if the Java process is running properly. Which is weird because I have another Java app running on this server just fine.

I went back to the basics of setting up Java as described under the IA Help-Linux Installation.

All the commands and check for the java install seem to be fine.

java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

echo $JAVA_HOME
/usr/local/java/jdk1.8.0_161

But if I check with systemctl:

systemctl status java
Unit java.service could not be found.

If I check the Mango service:

ystemctl status mango
รข mango.service - Mango Daemon Service
Loaded: loaded (/etc/systemd/system/mango.service; enabled; vendor preset: disabled)
Active: failed (Result: resources) since Tue 2018-03-06 12:25:28 GMT; 11min ago
Process: 1079 ExecStart=/opt/mango/bin/ma.sh start (code=exited, status=0/SUCCESS)

Mar 06 12:25:27 m-2-m.com systemd[1]: Starting Mango Daemon Service...
Mar 06 12:25:27 m-2-m.com ma.sh[1079]: MA_HOME is /opt/mango
Mar 06 12:25:28 m-2-m.com systemd[1]: PID file /opt/mango/bin/ma.pid not readable (yet?) after start.
Mar 06 12:25:28 m-2-m.com ma.sh[1079]: /opt/mango/bin/ma-start.sh: line 25: /opt/mango/bin/upgrade.sh: Permission denied
Mar 06 12:25:28 m-2-m.com systemd[1]: mango.service never wrote its PID file. Failing.
Mar 06 12:25:28 m-2-m.com systemd[1]: Failed to start Mango Daemon Service.
Mar 06 12:25:28 m-2-m.com systemd[1]: Unit mango.service entered failed state.
Mar 06 12:25:28 m-2-m.com systemd[1]: mango.service failed.

When I try to restart Mango I get:

systemctl restart mango
Job for mango.service failed because a configured resource limit was exceeded. See "systemctl status mango.service" and "journalctl -xe" for details.

Any advice would be greatly appreciated.

Cheers

Brian

]]>https://forum.mango-os.com/topic/3328/mango-3-will-not-start-after-recent-upgradeRSS for NodeTue, 12 May 2026 14:51:57 GMTTue, 06 Mar 2018 12:58:59 GMT60<![CDATA[Reply to Mango 3 will not start after recent upgrade on Tue, 06 Mar 2018 14:43:51 GMT]]>Hello Phil,
Thank you very much for your advice. The first command:

sudo chmod +x /opt/mango/bin/*.sh

is the one that did the trick. It is strange that this is not the first time that I have performed the updates on the Mango 3 server.

But that is up and running again./

Cheers :)

Brian

]]>https://forum.mango-os.com/post/17583https://forum.mango-os.com/post/17583Tue, 06 Mar 2018 14:43:51 GMT<![CDATA[Reply to Mango 3 will not start after recent upgrade on Tue, 06 Mar 2018 14:29:59 GMT]]>Hi Brian,

That,
Mar 06 12:25:28 m-2-m.com ma.sh[1079]: /opt/mango/bin/ma-start.sh: line 25: /opt/mango/bin/upgrade.sh: Permission denied

probably reveals the issue. Somehow, /opt/mango/bin/upgrade.sh must be owned by the wrong user or not executable for your mango user. So, you should either

sudo chmod +x /opt/mango/bin/*.sh or
sudo chown -R mango-user:group /opt/mango

You will also need unzip installed.

]]>https://forum.mango-os.com/post/17582https://forum.mango-os.com/post/17582Tue, 06 Mar 2018 14:29:59 GMT