• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    Mango 3 will not start after recent upgrade

    User help
    2
    3
    801
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • BGB
      BG
      last edited by BG

      Hello All,
      I had Mango 3.3.1 running on a Centos 7.4 web server with Java 1.8.0_161 until yesterday.
      I started a mango update and now cannot get Mango to start. I have restarted the server several times in my attempts to get things going again. but I am going in ever decreasing circles with the problem.

      Here are the last few lines from ma.log from the time of the upgrade attempt:

      INFO 2018-03-05T16:22:45,983 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:407) - UpgradeDownloader started
      INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:411) - UpgradeDownloader: BACKUP
      INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:424) - UpgradeDownloader: DOWNLOAD
      INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.execute:109) - Starting database backup WorkItem.
      INFO 2018-03-05T16:22:45,984 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem.
      ERROR 2018-03-05T16:22:46,045 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      INFO 2018-03-05T16:22:46,139 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.backupMysqlWithOutDatabase:442) - Backup created successfully forMango3 in localhost:3306
      ERROR 2018-03-05T16:22:47,132 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:48,204 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:49,263 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:50,322 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:51,387 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:52,448 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:53,511 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:54,570 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:55,646 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:56,705 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:57,770 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:58,829 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:22:59,887 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:23:00,947 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:23:02,008 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:23:03,068 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:23:04,126 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:23:05,188 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      ERROR 2018-03-05T16:23:06,248 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      INFO 2018-03-05T16:23:07,259 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:497) - UpgradeDownloader: INSTALL
      INFO 2018-03-05T16:23:07,260 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:548) - UpgradeDownloader: RESTART
      ERROR 2018-03-05T16:23:07,307 (com.serotonin.m2m2.web.dwr.util.BlabberConverterManager.convertOutbound:188) - No converter found for 'com.serotonin.m2m2.module.ModuleNotificationListener$UpgradeState'
      INFO 2018-03-05T16:23:12,267 (com.serotonin.m2m2.Lifecycle.terminate:382) - Mango Lifecycle terminating...
      INFO 2018-03-05T16:23:13,741 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:72) - Terminating 1 LAST priority data sources in 8 threads.
      INFO 2018-03-05T16:23:13,766 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:450) - Data source '- Demo 2 Meta' stopped
      INFO 2018-03-05T16:23:13,841 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:102) - Termination of 1 LAST priority data sources took 100ms
      INFO 2018-03-05T16:23:13,842 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:72) - Terminating 1 NORMAL priority data sources in 8 threads.
      INFO 2018-03-05T16:23:13,861 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:450) - Data source '- Demo 2 raw' stopped
      INFO 2018-03-05T16:23:13,942 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.terminate:102) - Termination of 1 NORMAL priority data sources took 100ms
      INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:548) - All high priority tasks exited gracefully.
      INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:554) - All medium priority tasks exited gracefully.
      INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:559) - All low priority tasks exited gracefully.
      INFO 2018-03-05T16:23:13,946 (com.serotonin.m2m2.db.BasePooledProxy.terminateImpl:117) - Stopping database
      INFO 2018-03-05T16:23:13,947 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:115) - Terminating NoSQL Batch Write Manager.
      INFO 2018-03-05T16:23:13,947 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:242) - Terminating NoSQL Point Value Mover.
      INFO 2018-03-05T16:23:13,947 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:249) - Terminating 1 Batch Writer Tasks.
      INFO 2018-03-05T16:23:13,948 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:258) - 1 Batch Writer Tasks awaiting termination.
      INFO 2018-03-05T16:23:14,446 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.joinTermination:173) - Batch Writer Task 0 Terminated.
      INFO 2018-03-05T16:23:14,446 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:266) - Terminating NoSQL Stale Point Cleaner.
      INFO 2018-03-05T16:23:14,446 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:269) - Terminating NoSQL Status Provider.
      WARN 2018-03-05T16:23:14,447 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager.terminate:274) - 10 queued point values discarded due to shutdown.
      INFO 2018-03-05T16:23:14,447 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:123) - Closing NoSQL Database: Point Values
      INFO 2018-03-05T16:23:14,458 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:131) - Closing NoSQL Database: Point Data
      INFO 2018-03-05T16:23:14,466 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:145) - Closing Auxilary NoSQL Database: com.infiniteautomation.tsdb.impl.IasTsdbImpl
      INFO 2018-03-05T16:23:14,473 (com.serotonin.m2m2.Lifecycle.terminate:491) - Mango Lifecycle terminated.

      There were no further entries in that log since Mango would not restart.

      I have copied here the lines from ma-script log since it is the only log updating from my attempts to restart:

      Mon 5 Mar 16:23:14 GMT 2018 ma-start: upgrading core...
      Mon 5 Mar 16:34:51 GMT 2018 ma.stop: No PID file found. Java process not stopped...
      Mon 5 Mar 16:43:53 GMT 2018 ma-start: upgrading core...
      Mon 5 Mar 17:05:47 GMT 2018 ma-start: upgrading core...
      Tue 6 Mar 09:16:18 GMT 2018 ma-start: upgrading core...
      Tue 6 Mar 11:43:37 GMT 2018 ma-start: upgrading core...
      Tue 6 Mar 12:25:28 GMT 2018 ma-start: upgrading core...
      Tue 6 Mar 12:37:49 GMT 2018 ma-start: upgrading core...

      When I looked at this log early this morning, I thought maybe the Java process is causing the problem. I have been trying all morning here to verify if the Java process is running properly. Which is weird because I have another Java app running on this server just fine.

      I went back to the basics of setting up Java as described under the IA Help-Linux Installation.

      All the commands and check for the java install seem to be fine.

      java -version
      java version "1.8.0_161"
      Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
      Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

      echo $JAVA_HOME
      /usr/local/java/jdk1.8.0_161

      But if I check with systemctl:

      systemctl status java
      Unit java.service could not be found.

      If I check the Mango service:

      ystemctl status mango
      â mango.service - Mango Daemon Service
      Loaded: loaded (/etc/systemd/system/mango.service; enabled; vendor preset: disabled)
      Active: failed (Result: resources) since Tue 2018-03-06 12:25:28 GMT; 11min ago
      Process: 1079 ExecStart=/opt/mango/bin/ma.sh start (code=exited, status=0/SUCCESS)

      Mar 06 12:25:27 m-2-m.com systemd[1]: Starting Mango Daemon Service...
      Mar 06 12:25:27 m-2-m.com ma.sh[1079]: MA_HOME is /opt/mango
      Mar 06 12:25:28 m-2-m.com systemd[1]: PID file /opt/mango/bin/ma.pid not readable (yet?) after start.
      Mar 06 12:25:28 m-2-m.com ma.sh[1079]: /opt/mango/bin/ma-start.sh: line 25: /opt/mango/bin/upgrade.sh: Permission denied
      Mar 06 12:25:28 m-2-m.com systemd[1]: mango.service never wrote its PID file. Failing.
      Mar 06 12:25:28 m-2-m.com systemd[1]: Failed to start Mango Daemon Service.
      Mar 06 12:25:28 m-2-m.com systemd[1]: Unit mango.service entered failed state.
      Mar 06 12:25:28 m-2-m.com systemd[1]: mango.service failed.

      When I try to restart Mango I get:

      systemctl restart mango
      Job for mango.service failed because a configured resource limit was exceeded. See "systemctl status mango.service" and "journalctl -xe" for details.

      Any advice would be greatly appreciated.

      Cheers

      Brian

      1 Reply Last reply Reply Quote 0
      • phildunlapP
        phildunlap
        last edited by phildunlap

        Hi Brian,

        That,
        Mar 06 12:25:28 m-2-m.com ma.sh[1079]: /opt/mango/bin/ma-start.sh: line 25: /opt/mango/bin/upgrade.sh: Permission denied

        probably reveals the issue. Somehow, /opt/mango/bin/upgrade.sh must be owned by the wrong user or not executable for your mango user. So, you should either

        sudo chmod +x /opt/mango/bin/*.sh or
        sudo chown -R mango-user:group /opt/mango

        You will also need unzip installed.

        1 Reply Last reply Reply Quote 0
        • BGB
          BG
          last edited by

          Hello Phil,
          Thank you very much for your advice. The first command:

          sudo chmod +x /opt/mango/bin/*.sh

          is the one that did the trick. It is strange that this is not the first time that I have performed the updates on the Mango 3 server.

          But that is up and running again./

          Cheers :)

          Brian

          1 Reply Last reply Reply Quote 0
          • First post
            Last post