Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website

  • I have a Mango installation (Centos 7) that has been running fine for a couple of years on a customers site. When I logged in yesterday to check on it the UI had locked up. I managed to restart Mango but it was still very slow to respond and screens would not populate. I tried a few things including installing the 8 or so updates that were outstanding to no avail. When I run the System Monitor it shows JAVA using almost 100% processor and several GB of RAM. I updated the Mango config to use large memory java init with 10G RAM available but in a matter of a couple of minutes Java used all that up.
    I was using Java SE 1.8.152 and I considered updating to the latest but I see Oracle have changed their licensing and I haven't figured out the consequences of that yet. Instead I updated to OpenJDK 1.8.0.232 but this made no difference.
    Any suggestions on what I should check next?

    BTW this is the console log:

    [petermcs@localhost ~]$ cd /opt/mango/e*/bin
    [petermcs@localhost bin]$ sudo ./ma.sh start
    [sudo] password for petermcs: 
    MA_HOME is /opt/mango/enterprise-m2m2-core-3.2.2
    [petermcs@localhost bin]$ MA removing dead pid file for PID:  5322
    ma-start: MA started with Process ID:  12604
    INFO  2019-11-06T10:24:04,596 (com.serotonin.m2m2.Main.main:105) - Starting Mango 3.6.5 with Schema Version: 28 
    INFO  2019-11-06T10:24:06,314 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'ssh', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,401 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'Haystack', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,490 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'loggingConsole', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,499 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'advancedComponents', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,535 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/advancedComponents/web/dox/manifest.xml 
    INFO  2019-11-06T10:24:06,575 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dnp3', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,634 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'excelReports', v3.6.2 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,765 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'openv4j', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,834 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dataPointDetailsView', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,905 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mqttClientDataSource', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,952 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'scripting', v3.6.2 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,955 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'opcda', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:06,956 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'reports', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,013 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sstTheme', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,014 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'measurlogicDTSCell', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,019 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mbus', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,069 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'twilio', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,163 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'asciiFile', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,211 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'controlcore', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,298 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'meta', v3.6.2 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,382 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mangoUI', v3.6.7 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,450 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/mangoUI/web/dox/manifest.xml 
    INFO  2019-11-06T10:24:07,835 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'BACnet', v3.6.3 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,906 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mangoApi', v3.6.4 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,981 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'log4JReset', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:07,981 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pointLinks', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,050 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sstGraphics', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,195 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'virtualDS', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,233 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mangoNoSqlDatabase', v3.6.4 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,276 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'http', v3.6.2 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,305 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'persistent', v3.6.2 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,320 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dashboardDesigner', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,322 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/dashboardDesigner/web/dox/manifest.xml 
    INFO  2019-11-06T10:24:08,338 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'graphicalViews', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,393 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'serial', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,395 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'maintenanceEvents', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,398 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'onewire', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,400 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'scheduledEvents', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,441 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'vmstat', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,455 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dataFile', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,488 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'egauge', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,521 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'jsonFileImport', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,595 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'watchlists', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,648 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'advancedScheduler', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,649 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dataImport', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,665 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'envcands', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,666 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'log4jDS', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,668 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pachube', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,671 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'snmp', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,673 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'TCPIP', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,689 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'deviceConfig', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,690 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/deviceConfig/web/dox/manifest.xml 
    INFO  2019-11-06T10:24:08,708 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'jmxds', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,712 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pakbus', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,802 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sqlConsole', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,804 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'templateConfig', v3.6.0 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,846 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sstGlobalScripts', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,855 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'modbus', v3.6.6 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,895 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pid', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,960 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sqlds', v3.6.1 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:08,995 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pop3', v3.6.2 by Infinite Automation Systems Inc. 
    INFO  2019-11-06T10:24:09,012 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'internal', v3.6.0 by Infinite Automation Systems Inc. 
    WARN  2019-11-06T10:24:13,790 (com.serotonin.m2m2.Lifecycle.webServerInitialize:1058) - BROWSE action not supported on your platform, unable to open browser at startup. 
    INFO  2019-11-06T10:24:13,979 (com.serotonin.m2m2.Lifecycle.loadLic:611) - Checking license... 
    INFO  2019-11-06T10:24:14,154 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:61) - Initializing H2 connection manager 
    INFO  2019-11-06T10:24:18,884 (com.serotonin.m2m2.db.upgrade.DBUpgrade.checkUpgrade:38) - Starting instance with core version 3.6.5, schema v28 
    INFO  2019-11-06T10:24:18,884 (com.infiniteautomation.nosql.MangoNoSqlProxy.initialize:77) - Starting MangoNoSql Database 
    INFO  2019-11-06T10:24:18,994 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:144) - Database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDB started 
    INFO  2019-11-06T10:24:22,897 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:173) - Empty dir delete for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDB took 3874ms 
    INFO  2019-11-06T10:24:22,901 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:234) - Corruption scan for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDB skipped because the database is configured not to run scan. 
    INFO  2019-11-06T10:24:22,945 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:144) - Database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDBAux started 
    INFO  2019-11-06T10:24:22,971 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:173) - Empty dir delete for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDBAux took 0ms 
    INFO  2019-11-06T10:24:22,972 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:234) - Corruption scan for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDBAux skipped because the database is configured not to run scan. 
    INFO  2019-11-06T10:24:25,577 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'runtimeContext' refreshed: org.springframework.context.annotation.AnnotationConfigApplicationContext@6f044c58 
    INFO  2019-11-06T10:24:25,581 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextStarted:139) - Spring context 'runtimeContext' started: org.springframework.context.annotation.AnnotationConfigApplicationContext@6f044c58 
    INFO  2019-11-06T10:24:25,777 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:77) - Initializing 5 NORMAL priority data sources in 8 threads. 
    INFO  2019-11-06T10:24:26,276 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Schedule Points took 497ms to start 
    INFO  2019-11-06T10:24:26,351 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Mango Internal took 571ms to start 
    INFO  2019-11-06T10:24:27,119 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Fast BACnet took 1341ms to start 
    INFO  2019-11-06T10:24:28,951 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source HVAC took 3172ms to start 
    INFO  2019-11-06T10:24:29,261 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source site took 3480ms to start 
    INFO  2019-11-06T10:24:29,327 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:109) - Initialization of 5 NORMAL priority data sources took 3550ms 
    INFO  2019-11-06T10:24:29,327 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:77) - Initializing 5 LAST priority data sources in 8 threads. 
    INFO  2019-11-06T10:24:30,377 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source landlord total electricity took 1048ms to start 
    INFO  2019-11-06T10:24:30,377 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source innovate total electricity took 1048ms to start 
    INFO  2019-11-06T10:24:30,393 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source wcc total electricity took 1065ms to start 
    INFO  2019-11-06T10:24:30,560 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source site total electricity took 1232ms to start 
    INFO  2019-11-06T10:24:30,664 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Window Control took 1336ms to start 
    INFO  2019-11-06T10:24:30,704 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:109) - Initialization of 5 LAST priority data sources took 1377ms 
    INFO  2019-11-06T10:24:31,102 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initialize:176) - Starting 0 Publishers... 
    INFO  2019-11-06T10:24:31,102 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initialize:188) - 0 Publisher's started in 3ms 
    WARN  2019-11-06T10:24:32,572 (org.eclipse.jetty.webapp.StandardDescriptorProcessor.addServletMapping:1220) - Duplicate mapping from *.jsp to jsp 
    INFO  2019-11-06T10:24:33,190 (com.serotonin.m2m2.web.M2M2ContextListener.contextInitialized:32) - Mango Automation context starting 
    INFO  2019-11-06T10:24:33,190 (com.serotonin.m2m2.web.M2M2ContextListener.contextInitialized:40) - Mango Automation context started 
    INFO  2019-11-06T10:24:34,149 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'rootWebContext' refreshed: Root WebApplicationContext 
    INFO  2019-11-06T10:24:34,892 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'jspDispatcherContext' refreshed: WebApplicationContext for namespace 'JSP_DISPATCHER-servlet' 
    INFO  2019-11-06T10:24:35,212 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'rootRestDispatcherContext' refreshed: WebApplicationContext for namespace 'ROOT_REST_DISPATCHER-servlet' 
    INFO  2019-11-06T10:24:36,706 (com.serotonin.m2m2.web.mvc.rest.v1.BacnetDiscoveryRestController.<init>:54) - Creating Discovery Rest Controller 
    INFO  2019-11-06T10:24:37,042 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'restV1DispatcherContext' refreshed: WebApplicationContext for namespace 'restV1DispatcherServlet-servlet' 
    INFO  2019-11-06T10:24:38,847 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'restV2DispatcherContext' refreshed: WebApplicationContext for namespace 'restV2DispatcherServlet-servlet' 
    INFO  2019-11-06T10:24:38,850 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MeasurlogicDTSCellDataSourceDwr 
    INFO  2019-11-06T10:24:38,850 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: EnvCanEditDwr 
    INFO  2019-11-06T10:24:38,852 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: HttpDataSourceDwr 
    INFO  2019-11-06T10:24:38,853 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: TcpIpDataSourceEditDwr 
    INFO  2019-11-06T10:24:38,854 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: Pop3EditDwr 
    INFO  2019-11-06T10:24:38,855 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: HaystackDataSourceDwr 
    INFO  2019-11-06T10:24:38,855 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
    INFO  2019-11-06T10:24:38,856 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: OPCDAEditDwr 
    INFO  2019-11-06T10:24:38,858 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: DataFileDataSourceEditDwr 
    INFO  2019-11-06T10:24:38,860 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: BACnetMSTPEditDwr 
    INFO  2019-11-06T10:24:38,862 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PakBusEditDwr 
    INFO  2019-11-06T10:24:38,863 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MBusEditDwr 
    INFO  2019-11-06T10:24:38,863 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.infiniteautomation.mango.pakbus.dwr.PakBusEditDwr 
    INFO  2019-11-06T10:24:38,863 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SNMPEditDwr 
    INFO  2019-11-06T10:24:38,864 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: InternalEditDwr 
    INFO  2019-11-06T10:24:38,865 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SerialEditDwr 
    INFO  2019-11-06T10:24:38,866 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MetaEditDwr 
    INFO  2019-11-06T10:24:38,867 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PersistentDataSourceDwr 
    INFO  2019-11-06T10:24:38,868 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: DnpEditDwr 
    INFO  2019-11-06T10:24:38,869 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: ModbusEditDwr 
    INFO  2019-11-06T10:24:38,870 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SshDataSourceEditDwr 
    INFO  2019-11-06T10:24:38,872 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: ScriptingEditDwr 
    INFO  2019-11-06T10:24:38,873 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: Log4jEditDwr 
    INFO  2019-11-06T10:24:38,875 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PidDataSourceEditDwr 
    INFO  2019-11-06T10:24:38,876 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: AsciiFileEditDwr 
    INFO  2019-11-06T10:24:38,876 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
    INFO  2019-11-06T10:24:38,878 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: OneWireEditDwr 
    INFO  2019-11-06T10:24:38,881 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PachubeDataSourceDwr 
    INFO  2019-11-06T10:24:38,882 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: VMStatEditDwr 
    INFO  2019-11-06T10:24:38,884 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SqlEditDwr 
    INFO  2019-11-06T10:24:38,884 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MqttClientEditDwr 
    INFO  2019-11-06T10:24:38,886 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: EgaugeDataSourceDwr 
    INFO  2019-11-06T10:24:38,886 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
    INFO  2019-11-06T10:24:38,887 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: JmxEditDwr 
    INFO  2019-11-06T10:24:38,889 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: BACnetIPEditDwr 
    INFO  2019-11-06T10:24:38,890 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: OpenVEditDwr 
    INFO  2019-11-06T10:24:38,891 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: br.org.scadabr.dnp3.dwr.DnpEditDwr 
    INFO  2019-11-06T10:24:38,892 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: ControlCoreEditDwr 
    INFO  2019-11-06T10:24:38,892 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.modbus.dwr.ModbusEditDwr 
    INFO  2019-11-06T10:24:38,893 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: VirtualEditDwr 
    INFO  2019-11-06T10:24:38,893 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
    WARN  2019-11-06T10:31:38,863 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:150) - Data Source Fast BACnet aborted 1 polls since it started. 
    WARN  2019-11-06T10:35:51,712 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:150) - Data Source Mango Internal aborted 1 polls since it started. 
    WARN  2019-11-06T10:37:46,812 (org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask:361) -  
    java.lang.OutOfMemoryError: GC overhead limit exceeded
    	at java.util.ArrayDeque.iterator(ArrayDeque.java:611) ~[?:1.8.0_152]
    	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processUpdates(ManagedSelector.java:376) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:355) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
    WARN  2019-11-06T10:40:07,183 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:150) - Data Source site aborted 1 polls since it started. 
    
    

  • @petermcs the best way to analyze OOM problems is by taking a heap dump and loading it into an analysis tool such as JVisualVM or MAT. However that file will generally be GB in size on a system like yours so before we get into that I would suggest that you take some samples of the Java process while it is running. From that we may be able to see what is eating up all the memory.

    By using the jmap tool packaged with the JDK you should be able to get the necessary info:

    https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html

    Step 1. Find the PID of Mango while it is running. This can be by something like ps aux | grep java or looking into the ma.pid file in MA_HOME\bin

    Step 2. Run jmap -histo <mango pid> every so often until Mango crashes. Write the output to a file for reference later obviously.

    By looking at the classes that are using the most memory you should be able to get an idea of what the problem is. Please post a summary of your findings here and I can try to see what the problem is.

    NOTE: likely jmap won't be on your path so you may need to execute it from within the JDK installation directory directly.


  • @terrypacker I got jmap running (seems that even though I changed JAVA_HOME to the OpenJDK folder Mango still runs using the Oracle Java so I need to look at the again... Anyway when I run jmap before and after the memory blows up, the results are identical!
    Also a bit of further info, If I start Mango and don't connect (locally via 127.0.0.1 as I am Teamviewering in to check things) the browser the memory does not blow up. It only happens a few minutes after I connect and log in. If I connect but don't login I.e. stay at the login screen, the memory usage does not increase.

    Here is the after jmap result

    sudo /usr/java/jdk1.8.0_152/bin/jmap 18673
    Attaching to process ID 18673, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.152-b16
    0x0000000000400000	7K	/usr/java/jdk1.8.0_152/bin/java
    0x00007f9eb46f8000	30K	/usr/lib64/libnss_dns-2.17.so
    0x00007f9eb6118000	259K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libjpeg.so
    0x00007f9ec7168000	11K	/usr/lib64/libfreebl3.so
    0x00007f9ec736b000	39K	/usr/lib64/libcrypt-2.17.so
    0x00007f9ec75a2000	40K	/usr/lib64/libltdl.so.7.3.0
    0x00007f9ec77ac000	93K	/usr/lib64/libtdb.so.1.3.16
    0x00007f9ec79c3000	27K	/usr/lib64/libogg.so.0.8.0
    0x00007f9ec7bca000	180K	/usr/lib64/libvorbis.so.0.4.6
    0x00007f9ec7df7000	35K	/usr/lib64/libvorbisfile.so.3.3.5
    0x00007f9ecc091000	68K	/usr/lib64/libcanberra.so.0.2.5
    0x00007f9ecc2a2000	19K	/usr/lib64/libcanberra-gtk.so.0.1.9
    0x00007f9ecc4a7000	23K	/usr/lib64/gtk-2.0/modules/libcanberra-gtk-module.so
    0x00007f9ecc6ad000	6K	/usr/lib64/gtk-2.0/modules/libpk-gtk-module.so
    0x00007f9ecc8af000	43K	/usr/lib64/gtk-2.0/2.10.0/engines/libpixmap.so
    0x00007f9eccaba000	15K	/usr/lib64/gtk-2.0/2.10.0/engines/libadwaita.so
    0x00007f9ecccbe000	6K	/usr/lib64/libgthread-2.0.so.0.5600.1
    0x00007f9eccec0000	188K	/usr/lib64/libgraphite2.so.3.0.1
    0x00007f9ecd0ee000	259K	/usr/lib64/libblkid.so.1.1.0
    0x00007f9ecd32e000	73K	/usr/lib64/libGLX.so.0.0.0
    0x00007f9ecd560000	625K	/usr/lib64/libGLdispatch.so.0.0.0
    0x00007f9ecd816000	66K	/usr/lib64/libbz2.so.1.0.6
    0x00007f9ecda26000	19K	/usr/lib64/libuuid.so.1.3.0
    0x00007f9ecdc2b000	169K	/usr/lib64/libexpat.so.1.6.0
    0x00007f9ecde55000	111K	/usr/lib64/libfribidi.so.0.4.0
    0x00007f9ece071000	50K	/usr/lib64/libthai.so.0.1.6
    0x00007f9ece27d000	637K	/usr/lib64/libharfbuzz.so.0.10705.0
    0x00007f9ece51a000	271K	/usr/lib64/libmount.so.1.1.0
    0x00007f9ece75d000	103K	/usr/lib64/libresolv-2.17.so
    0x00007f9ece976000	152K	/usr/lib64/libselinux.so.1
    0x00007f9eceb9d000	31K	/usr/lib64/libffi.so.6.0.1
    0x00007f9eceda5000	568K	/usr/lib64/libGL.so.1.7.0
    0x00007f9ecf031000	88K	/usr/lib64/libz.so.1.2.7
    0x00007f9ecf247000	55K	/usr/lib64/libxcb-render.so.0.0.0
    0x00007f9ecf455000	15K	/usr/lib64/libxcb-shm.so.0.0.0
    0x00007f9ecf659000	175K	/usr/lib64/libpng15.so.15.13.0
    0x00007f9ecf884000	82K	/usr/lib64/libEGL.so.1.1.0
    0x00007f9ecfa98000	682K	/usr/lib64/libpixman-1.so.0.34.0
    0x00007f9ecfd41000	776K	/usr/lib64/libfreetype.so.6.14.0
    0x00007f9ed8044000	392K	/usr/lib64/libpcre.so.1.2.0
    0x00007f9ed82a6000	10K	/usr/lib64/libXdamage.so.1.1.0
    0x00007f9ed84a9000	10K	/usr/lib64/libXcomposite.so.1.0.0
    0x00007f9ed86ac000	43K	/usr/lib64/libXrandr.so.2.2.0
    0x00007f9ed88b7000	11K	/usr/lib64/libXinerama.so.1.0.0
    0x00007f9ed8aba000	1129K	/usr/lib64/libglib-2.0.so.0.5600.1
    0x00007f9ed8dd0000	331K	/usr/lib64/libgobject-2.0.so.0.5600.1
    0x00007f9ed9021000	270K	/usr/lib64/libfontconfig.so.1.11.1
    0x00007f9ed9263000	281K	/usr/lib64/libpango-1.0.so.0.4200.3
    0x00007f9ed94a9000	88K	/usr/lib64/libpangoft2-1.0.so.0.4200.3
    0x00007f9ed96bf000	1682K	/usr/lib64/libgio-2.0.so.0.5600.1
    0x00007f9ed9a5e000	162K	/usr/lib64/libgdk_pixbuf-2.0.so.0.3612.0
    0x00007f9ed9c86000	1263K	/usr/lib64/libcairo.so.2.11512.0
    0x00007f9ed9fbd000	153K	/usr/lib64/libatk-1.0.so.0.22810.1
    0x00007f9eda1e3000	55K	/usr/lib64/libpangocairo-1.0.so.0.4200.3
    0x00007f9eda3f1000	15K	/usr/lib64/libgmodule-2.0.so.0.5600.1
    0x00007f9eda5f5000	789K	/usr/lib64/libgdk-x11-2.0.so.0.2400.31
    0x00007f9eda8b6000	4705K	/usr/lib64/libgtk-x11-2.0.so.0.2400.31
    0x00007f9edb039000	23K	/usr/lib64/libXfixes.so.3.1.0
    0x00007f9edb23f000	44K	/usr/lib64/libXcursor.so.1.0.2
    0x00007f9edb54b000	513K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libfontmanager.so
    0x00007f9edb9c4000	15K	/usr/lib64/libXau.so.6.0.0
    0x00007f9edbbc8000	162K	/usr/lib64/libxcb.so.1.1.0
    0x00007f9edbdf0000	64K	/usr/lib64/libXi.so.6.1.0
    0x00007f9f18023000	23K	/usr/lib64/libXtst.so.6.1.0
    0x00007f9f18229000	43K	/usr/lib64/libXrender.so.1.3.0
    0x00007f9f18434000	1287K	/usr/lib64/libX11.so.6.3.0
    0x00007f9f18772000	74K	/usr/lib64/libXext.so.6.4.0
    0x00007f9f189a3000	432K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libawt_xawt.so
    0x00007f9f18bfc000	754K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libawt.so
    0x00007f9f507e9000	86K	/usr/lib64/libgcc_s-4.8.5-20150702.so.1
    0x00007f9f50a1e000	251K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libsunec.so
    0x00007f9f51168000	91K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libnio.so
    0x00007f9f51394000	112K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libnet.so
    0x00007f9f517ab000	49K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libmanagement.so
    0x00007f9f6f1f3000	125K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libzip.so
    0x00007f9f6f40f000	60K	/usr/lib64/libnss_files-2.17.so
    0x00007f9f6f622000	221K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libjava.so
    0x00007f9f6f84e000	64K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libverify.so
    0x00007f9f6fa5d000	42K	/usr/lib64/librt-2.17.so
    0x00007f9f6fc65000	1110K	/usr/lib64/libm-2.17.so
    0x00007f9f6ff67000	16640K	/usr/java/jdk1.8.0_152/jre/lib/amd64/server/libjvm.so
    0x00007f9f70f63000	2105K	/usr/lib64/libc-2.17.so
    0x00007f9f71331000	18K	/usr/lib64/libdl-2.17.so
    0x00007f9f71535000	101K	/usr/java/jdk1.8.0_152/lib/amd64/jli/libjli.so
    0x00007f9f7174b000	138K	/usr/lib64/libpthread-2.17.so
    0x00007f9f71967000	159K	/usr/lib64/ld-2.17.so
    

  • @petermcs please run jmap -histo pid to get the object count.


  • Okay:

    Here's some stuff related to this after going through this:

    I just upgraded to Mango 3.6.5 running on our production server. We use persistent TCP to haul data back from outstations monitoring solar assets, and it works Very Well.

    However, there's a couple of things that popped up after the upgrade. We've identified (2) separate issues:

    Issue (1): Reporting tool causes Mango to run out of memory and crash itself. Mango apparently doesn't check these things internally, so if you run out of Ram, it just merrily gobbles it all up and blows up. We have a spreadsheet that's about 8 megs, that loads an additional meg of data from Mango, and then reports it out again. Running this spreadsheet uses 1500 megs of ram, according to the Mango's internal JVM free memory.

    We fixed this by setting the -Xms5g and -Xmx5g via the "memory-medium.sh". However, if we have more than one of two of those reports ran at the same time, the system will still Blow Up. So, I'd like to see Mango either (1) do something to manage these reports, if it means slowing them down or something to not eat that much memory, or (2) at least crash elegantly: When this occurs with our system, the thing completely shuts down, and then does not re-start itself. I'd rather the reporting tool not crash the system, but if it does, it'd be nice to recover gracefully. As a solution to this, we're going to keep jacking up -Xms and -Xmx until we run out of memory on our servers, then we'll do something else.

    Issue (2), And maybe this is a different thread:
    Mango opens LOTS of files. At night, when the outstations are set to synchronize historic back to our cloud server, we see >100,000 files open, as reported by pmap. This normally is ~1000, but at night, when the historical syncs happen, that number jumps by a lot. This causes a Java fault, as the thing bumps against the available files.

    We increased the max file limit by doing
    vm.max_map_count=131060
    to /etc/sysctl.conf

    and restarting.

    We're still regularly getting >100k open files, but at least now Java doesn't crash out. We have the tsdb running out our instance, and something must have changed between 3.5 and 3.6 to how this is done, as we haven't changed any settings that would cause additional data to be written via the persistent publisher.

    Just letting people know here about the work we did on this. Is there some way in Mango to limit the number of open files it does on a historical sync? My concern here is that as I add more sites (and keep historical sync active, to force the system to catch up on any weird outages that happen during the day as is common with cell-connected systems), this file open issue is going to increasingly get bigger and bigger. Did I miss something here about the best way to do this?

    Cheers,
    -Greg Linder


  • Also:

    For those of you running into memory problems in Mango:

    Since the JVM doesn't ever "give memory back" to the operating system, whatever you set those -Xms and -Xmx to will be about what the JVM will take. once it reaches that Max of that, you will NEVER see that memory returned to the system unless you restart the JVM.

    Our system, when running reports, cranks right up to the limit, and then sits there. If you see CPU utilization pegged high, but the system in unresponsive, it's possible that the JVM has maxed out it's available memory (partially defined by -Xms and -Xmx settings), and the JVM is cranking around it's jobs trying to get your work done without anything getting actually complete.


  • Hi Greg

    I think it will be helpful to know how large your system is. How many points and which database are you using. H2 or MySQL?

    @turbo said in Out of memory problem:

    Just letting people know here about the work we did on this. Is there some way in Mango to limit the number of open files it does on a historical sync? My concern here is that as I add more sites (and keep historical sync active, to force the system to catch up on any weird outages that happen during the day as is common with cell-connected systems), this file open issue is going to increasingly get bigger and bigger. Did I miss something here about the best way to do this?

    I would consider running the sync more often, I have been using a cron pattern of every 5 minutes
    and have not run into any issue with it. However, I have no idea how much data you are logging.
    Also, have you enabled "save real-time data" on the data source? This could be why the sync needs to open so many files.

    The current recommended way to run mango as a service does lack the ability to restart after crashing. Are you running Ubuntu or debian? Some work has been done on a new service script which you could use.


  • @Turbo

    Here is some general information on what you see, and I have been tracking these known problems for some time trying to find a solution.

    1. Excel Reports Module can use up all the JVM memory.

    This is mostly out of our control and a known problem with the Apache POI library that is the backbone of that module. They have recently released a more performant version that will eventually be part of our Excel Reports module. We have reworked our REST api using a new algorithm to manage memory that will in the not-to-distant future be ported over to the ExcelReports module and should solve all of the problems outside of the POI library limitations.

    1. Open Files - MangoNoSQL
      The time series database will open many files and leave them open for some time, they eventually do get closed but for performance reasons we don't close them immediately after use. As @CraigWeb mentioned better tuning can help during intense syncs. As for the problem being worse in 3.6, I was not aware of that and can confirm that the logic for opening and closing the files has not changes. It is likely that you system is just getting larger and you are now aware that this happens. There are a whole slew of settings for very large systems in the env.properties file with the prefix: db.nosql.*. I don't have time to go into them all here but I would suggest experimenting with them and posting questions about specific properties later.

    As for the JVM general memory use:

    We are starting to use Adopt OpenJDK 13 and are finding that it is much better at memory management. As for the -Xms and -Xmx settings, depending what you are trying to achieve they may not be ideal. By setting both to the same value you get a fully allocated JVM at startup which on a machine only running Mango is likely ideal. But for a system with other processes that need memory at other times the JVM cannot release memory with that configuration. We have found that on a very large machine using Adopt OpenJDK it is quite good expanding and contracting the JVM memory use for Mango's needs. But we are still evaluating the best configuration for it at this point.


  • Greetings, all:

    Thanks for the answers here. I appreciate the clarity on this.

    As for "Sync every five minutes" vs "save real-time data"-- I'm a bit unclear on how these options interact with each other.

    I'd like to get all the data that my systems in the field are logging. The points out there are set with different poll and logging settings. So if I set to "sync every 5 minutes" (or something similar), does that mean that my data points will update every 5 minutes with the logged data? Vs the "save real time data" that saves all the data that comes in from every poll?

    Is that the proper way of using those options?

    Cheers,
    -Greg


  • @terrypacker Hi Terry, I am just continuing @petermcs's work and have the jmap -hist outputs.

    Before:

     num     #instances         #bytes  class name
    ----------------------------------------------
       1:        173771      715822488  [I
       2:       3748110      508063720  [C
       3:       1945314      443221056  [B
       4:       1733728       41609472  java.lang.String
       5:       1973991       37379904  [Ljava.lang.Class;
       6:        478710       23272440  [Ljava.lang.Object;
       7:             1       16777232  [Lorg.h2.util.CacheObject;
       8:        151493       11321608  [Ljava.util.HashMap$Node;
       9:        228571       10971408  java.util.HashMap
      10:        168383       10776512  java.net.URL
      11:        318951       10206432  java.util.HashMap$Node
      12:         99723        9573408  java.util.jar.JarFile$JarFileEntry
      13:        101097        8896536  java.lang.reflect.Method
      14:         99443        7955440  java.util.zip.ZipEntry
      15:        194720        7788800  java.util.LinkedHashMap$Entry
      16:        262144        6291456  org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper
      17:        107562        6023472  java.util.LinkedHashMap
      18:        211916        5085984  java.lang.StringBuilder
      19:        155322        4970304  java.util.LinkedHashMap$LinkedEntryIterator
      20:        132436        4237952  java.util.ArrayList$Itr
      21:        106631        4096384  [Ljava.lang.String;
      22:         48305        3915312  [S
      23:        103984        3635760  [Ljava.lang.reflect.Method;
      24:        148571        3565704  java.util.ArrayList
      25:        145630        3495120  org.springframework.core.annotation.AnnotationUtils$AnnotationCacheKey
      26:        213448        3415168  java.util.HashSet
      27:         57295        3208520  org.springframework.asm.Label
      28:          5081        3195656  [J
      29:         37779        3022320  java.lang.reflect.Constructor
      30:         89701        2870432  java.io.File
      31:         59175        2840400  java.nio.HeapByteBuffer
      32:        168665        2698640  java.lang.Integer
      33:         16617        2641896  [Lorg.springframework.asm.Label;
      34:         73858        2363456  java.util.concurrent.ConcurrentHashMap$Node
      35:         89469        2147256  java.lang.Class$MethodArray
      36:         44728        2146944  org.springframework.core.ResolvableType
      37:         81522        1956528  java.util.LinkedList$Node
      38:         32363        1812328  org.springframework.asm.SymbolTable$Entry
      39:         15886        1783136  java.lang.Class
      40:         52571        1682272  java.util.LinkedHashMap$LinkedKeyIterator
      41:         22174        1596528  org.springframework.core.annotation.AnnotationAttributes
      42:         32146        1543008  org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference
      43:         57998        1391952  java.util.concurrent.CopyOnWriteArrayList$COWIterator
      44:         42207        1350624  org.springframework.asm.Type
      45:         29777        1191080  java.lang.ref.Finalizer
      46:         36791        1177312  java.util.AbstractList$Itr
      47:         65801        1052816  sun.reflect.ConstantPool
      48:          9550         993200  sun.net.www.protocol.file.FileURLConnection
      49:         19600         940800  org.springframework.asm.Frame
      50:         22090         883600  com.serotonin.bacnet4j.util.sero.ByteQueue
      51:         37028         864280  [Ljava.lang.reflect.Type;
      52:         35583         853992  org.springframework.asm.Edge
      53:          1968         826856  [Ljava.util.concurrent.ConcurrentHashMap$Node;
      54:         25594         819008  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
      55:         50928         814848  java.lang.Object
      56:         25147         804704  java.util.LinkedHashMap$LinkedValueIterator
      57:         24671         789472  java.util.LinkedList
      58:         49091         785456  org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder
      59:         13853         775768  org.springframework.core.type.classreading.MethodMetadataReadingVisitor
      60:         13740         769440  jdk.internal.org.objectweb.asm.Item
      61:         31486         755664  java.util.Arrays$ArrayList
      62:         46717         747472  java.util.LinkedHashSet
      63:         17178         687120  java.lang.invoke.MethodType
      64:          9400         676800  java.lang.reflect.Field
      65:         27985         671640  sun.reflect.generics.tree.SimpleClassTypeSignature
      66:         26545         637080  sun.reflect.generics.visitor.Reifier
      67:          2806         628544  org.springframework.asm.MethodWriter
      68:         11198         627088  java.lang.invoke.LambdaFormEditor$Transform
      69:         24021         576504  sun.reflect.generics.factory.CoreReflectionFactory
      70:         23902         573648  sun.reflect.generics.parser.SignatureParser
      71:         10093         565208  java.util.zip.ZipFile$ZipFileInputStream
      72:         10089         564984  java.util.zip.ZipFile$ZipFileInflaterInputStream
      73:         23307         559368  sun.reflect.generics.scope.ClassScope
      74:         17324         554368  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
      75:         13514         540560  java.util.HashMap$KeyIterator
      76:         16637         532384  java.lang.ref.WeakReference
      77:         13288         531520  java.util.HashMap$EntryIterator
      78:          6026         530288  sun.net.www.protocol.jar.JarURLConnection
      79:         16281         520992  java.util.RegularEnumSet
      80:         31992         511872  java.util.LinkedHashMap$LinkedKeySet
      81:         18654         507960  [Ljava.lang.annotation.Annotation;
      82:         15760         504320  org.springframework.util.ConcurrentReferenceHashMap$1
      83:         30116         498104  [Lsun.reflect.generics.tree.TypeArgument;
      84:          7673         491072  org.springframework.core.MethodParameter
      85:          5066         486336  sun.net.www.protocol.jar.URLJarFile$URLJarFileEntry
      86:         14914         477248  java.util.LinkedList$ListItr
      87:          5921         476640  [Ljava.util.WeakHashMap$Entry;
      88:          7171         458944  com.serotonin.timer.OrderedTaskInfo
      89:           394         452912  [Lorg.springframework.asm.SymbolTable$Entry;
      90:         18617         446808  org.springframework.core.annotation.AnnotatedElementUtils$MergedAnnotationAttributesProcessor
      91:         27787         444592  sun.reflect.generics.tree.ClassTypeSignature
      92:         11069         442760  sun.nio.cs.UTF_8$Decoder
      93:          6897         441408  java.util.regex.Matcher
      94:         13780         440960  org.springframework.core.annotation.AnnotationUtils$$Lambda$48/1194398327
      95:          5356         428480  sun.net.www.protocol.jar.URLJarFile
      96:          7588         424928  com.serotonin.bacnet4j.npdu.NPCI
      97:         19494         420832  [Lorg.springframework.asm.Type;
      98:         12977         415264  java.lang.reflect.WeakCache$CacheKey
      99:         17180         412320  org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey
     100:         12269         392608  java.util.Hashtable$Entry
    
    Total      16594285     2034054440
    
    

    After opening a browser window and logging in, greeted by the admin page and an immediate "connectivity lost, restoring connection" toast message that never permenantly resolves.
    It then takes about 3-5 minutes to max out

    
     num     #instances         #bytes  class name
    ----------------------------------------------
       1:      51017979     3130737016  [C
       2:      51017618     1224422832  java.lang.String
       3:      10175891      895478408  com.serotonin.m2m2.rt.event.EventInstance
       4:      20380459      550396632  [Ljava.lang.Object;
       5:      20352135      488451240  com.serotonin.m2m2.i18n.TranslatableMessage
       6:      10175862      325627584  com.serotonin.m2m2.rt.event.type.DataSourceEventType
       7:        469887      198080400  [B
       8:       1599229       87706096  [Lorg.h2.value.Value;
       9:       1248203       39942496  org.h2.result.RowImpl
      10:        425490       30635280  org.h2.value.ValueLobDb
      11:        828920       26525440  org.h2.result.SimpleRowValue
      12:             1       16777232  [Lorg.h2.util.CacheObject;
      13:         45454       15596376  [I
      14:         31155       12509192  [J
      15:        438464        7015424  org.h2.value.ValueInt
      16:        265342        6368208  org.h2.value.ValueLong
      17:        262144        6291456  org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper
      18:         30244        5475296  [Lorg.h2.result.Row;
      19:         85336        3413440  java.util.LinkedHashMap$Entry
      20:          2874        3363200  [Lorg.h2.result.SearchRow;
      21:         30243        3145272  org.h2.index.PageDataLeaf
      22:         28393        2498584  java.lang.reflect.Method
      23:         28704        2297592  [Ljava.util.HashMap$Node;
      24:         66713        2134816  java.util.HashMap$Node
      25:         60656        1940992  java.util.concurrent.ConcurrentHashMap$Node
      26:         16839        1888904  java.lang.Class
      27:         23457        1313592  java.util.LinkedHashMap
      28:         33277        1064864  org.h2.store.Data
      29:           608         741064  [Ljava.util.concurrent.ConcurrentHashMap$Node;
      30:          8489         611208  org.springframework.core.annotation.AnnotationAttributes
      31:         26206         593480  [Ljava.lang.Class;
      32:          7131         513432  java.lang.reflect.Field
      33:         31170         498720  java.lang.Object
      34:         10792         476208  [Ljava.lang.String;
      35:         18951         454824  java.util.ArrayList
      36:          9363         449424  java.util.HashMap
      37:         15410         369840  org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey
      38:          4273         341840  java.lang.reflect.Constructor
      39:          9479         303328  java.util.LinkedList
      40:          9008         288256  java.util.Hashtable$Entry
      41:         11579         277896  java.util.LinkedList$Node
      42:         17155         274480  org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder
      43:         11080         265920  org.h2.value.ValueRow
      44:          2851         228080  org.h2.index.PageBtreeLeaf
      45:          4539         181560  java.lang.ref.SoftReference
      46:          2998         167888  java.lang.Class$ReflectionData
      47:         10089         161424  java.util.LinkedHashMap$LinkedEntrySet
      48:          9949         159184  java.util.LinkedHashSet
      49:          3796         151840  javax.servlet.jsp.tagext.TagAttributeInfo
      50:          2614         146384  java.lang.invoke.MemberName
      51:          6080         145920  java.util.jar.Attributes$Name
      52:          5878         141072  sun.reflect.generics.tree.SimpleClassTypeSignature
      53:          3514         140560  java.util.TreeMap$Entry
      54:          2734         140264  [Ljava.lang.reflect.Method;
      55:             4         131136  [Ljava.util.concurrent.ForkJoinTask;
      56:          1807         130104  org.springframework.web.method.HandlerMethod$HandlerMethodParameter
      57:          3164         126560  jdk.nashorn.internal.runtime.CompiledFunction
      58:          2488         119424  org.springframework.core.ResolvableType
      59:          1796         114944  java.net.URL
      60:          5878         109584  [Lsun.reflect.generics.tree.TypeArgument;
      61:          1890         105840  jdk.nashorn.internal.runtime.ScriptFunction
      62:           488         105720  [Ljava.util.Hashtable$Entry;
      63:          4046          97104  sun.reflect.annotation.AnnotationInvocationHandler
      64:          1483          94912  java.util.concurrent.ConcurrentHashMap
      65:          3898          93696  [Lorg.springframework.core.annotation.AnnotationAttributes;
      66:          1056          92928  org.h2.table.Column
      67:          5702          91232  sun.reflect.generics.tree.ClassTypeSignature
      68:           543          91224  org.springframework.beans.factory.support.RootBeanDefinition
      69:          5543          88688  java.lang.Integer
      70:          1485          83160  java.security.Provider$Service
      71:          1904          76160  java.lang.invoke.BoundMethodHandle$Species_LL
      72:          4608          73728  java.util.LinkedHashMap$LinkedKeySet
      73:           973          70056  org.springframework.core.type.classreading.AnnotationMetadataReadingVisitor
      74:          2944          70008  [Ljava.lang.reflect.Type;
      75:          2890          69360  java.security.Provider$ServiceKey
      76:           650          67600  org.h2.mvstore.Chunk
      77:          4151          66416  java.util.jar.Attributes
      78:          2063          66016  java.lang.ref.WeakReference
      79:          2036          65152  com.serotonin.bacnet4j.obj.PropertyTypeDefinition
      80:          2002          64064  jdk.nashorn.internal.runtime.FinalScriptFunctionData
      81:          1099          61544  java.lang.Package
      82:           162          61088  [Ljdk.nashorn.internal.runtime.PropertyHashMap$Element;
      83:          1507          60280  java.lang.invoke.MethodType
      84:          1497          59880  java.lang.ref.Finalizer
      85:           710          58992  [Ljava.util.WeakHashMap$Entry;
      86:            41          58216  [Ljava.nio.ByteBuffer;
      87:          2464          56256  [Ljava.lang.reflect.Constructor;
      88:           308          54208  org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$ConfigurationClassBeanDefinition
      89:          1123          53904  org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference
      90:          1648          52736  java.lang.invoke.DirectMethodHandle
      91:          1638          52416  jdk.nashorn.internal.runtime.PropertyHashMap$Element
      92:          1624          51968  java.lang.invoke.LambdaForm$Name
      93:           205          50840  com.serotonin.m2m2.vo.DataPointVO
      94:           880          49280  org.springframework.core.type.classreading.MethodMetadataReadingVisitor
      95:          2036          48864  com.serotonin.bacnet4j.obj.ObjectPropertyTypeDefinition
      96:          1519          48608  java.util.concurrent.locks.ReentrantLock$NonfairSync
      97:           432          48384  org.springframework.context.annotation.ScannedGenericBeanDefinition
      98:          1511          48352  java.lang.ref.ReferenceQueue
      99:          1508          48256  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
     100:          1147          45880  java.math.BigInteger
    
    Total     169664483     7108600472
    

    I've limited it to the first 100 and totals, just to save post space. The full outputs ~7k can be sent on if needed.

    htop when things max out:
    0_1576156299362_htop.png


  • @cbyrne the second jmap -hist output looks like the majority of the memory in use is from events. I'm just guessing here but it looks like you have a large number of events in the database that are being loaded and processed on login. Mango has known issues when the events table becomes large (which we are working to fix in Mango 4.0).

    To confirm this I would restart Mango with the h2 web console enabled so you can look at the database without logging into Mango and causing it to eat up all the memory. This can be done via the env properties:

    db.web.start=true
    db.web.port=8091
    

    See also here: https://help.infiniteautomation.com/about-the-h2-database

    Then login to the console make sure you set the JDBC url to point to the mango database on your file system, on a MangoES this would be jdbc:h2:/opt/mango/databases/mah2 basically the path to the db without the .h2.db on the end.

    Then run this query: SELECT COUNT(*) FROM events;

    If the number is very large and you are ok with removing all events from the system I suggest you delete all event data. Before doing this I would make a copy of the h2 database file as a backup. You can simply copy the file if Mango is down.

    Then this post describes how to remove events from the database:
    https://forum.infiniteautomation.com/topic/2266/full-storage-issue-on-mangoes
    Basically run:

    SET FOREIGN_KEY_CHECKS=0; 
    CREATE TABLE eventsNewTable LIKE events; 
    DROP TABLE events; 
    RENAME TABLE eventsNewTable TO events; 
    CREATE TABLE userEventsNewTable LIKE userEvents; 
    DROP TABLE userEvents; 
    RENAME TABLE userEventsNewTable TO userEvents; 
    SET FOREIGN_KEY_CHECKS=1;
    

    If the events were the problem I suggest you take a look at this to better manage your events:
    https://help.infiniteautomation.com/managing-disk-space

    Another possibility that we can look at if my first assumption is not true is that something is generating many events on login. But since a big chunk of memory is being held by DataSourceEventTypes I expect this is from them being stored in the database.


  • Perfect, thanks @terrypacker! I'll try that and update this post.

    update 1:
    The H2 database shell doesn't recognise LIKE in CREATE TABLE. It may not be supported, The H2 command info for CREATE DATABASE doesn't show LIKE - https://www.h2database.com/html/commands.html#create_table

    update 2
    I just did a DELETE FROM EVENTS; instead.

    So yeah, there might have been a few too many events... 12 million. The database has gone from ~6 Gb to 76 Mb.


  • @terrypacker Sorry for dredging up an old thread but I ran the same DELETE FROM EVENTS; as before but this time mango has failed to restart. It gets an out of bounds error when connecting to H2.

    INFO  2020-06-10T18:38:06,912 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:61) - Initializing H2 connection manager 
    INFO  2020-06-10T18:38:06,953 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:83) - Initializing H2 web server 
    FATAL 2020-06-10T18:38:07,527 (com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize:163) - Unable to connect to database of type H2 
    org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:611) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) ~[mango-3.6.5.jar:?]
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:405) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.AbstractDatabaseProxy.newDatabaseCheck(AbstractDatabaseProxy.java:181) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize(AbstractDatabaseProxy.java:82) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:760) ~[ma-priv-3.6.5.jar:?]
    	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:312) ~[ma-priv-3.6.5.jar:?]
    	at com.serotonin.m2m2.Main.main(Main.java:143) ~[ma-priv-3.6.5.jar:?]
    Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	... 12 more
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    	at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	... 12 more
    ERROR 2020-06-10T18:38:07,527 (com.serotonin.m2m2.Main.main:146) - Error during initialization 
    org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:611) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) ~[mango-3.6.5.jar:?]
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:405) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.AbstractDatabaseProxy.newDatabaseCheck(AbstractDatabaseProxy.java:181) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize(AbstractDatabaseProxy.java:82) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:760) ~[ma-priv-3.6.5.jar:?]
    	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:312) ~[ma-priv-3.6.5.jar:?]
    	at com.serotonin.m2m2.Main.main(Main.java:143) ~[ma-priv-3.6.5.jar:?]
    Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	... 12 more
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    	at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	... 12 more
    INFO  2020-06-10T18:38:07,527 (com.serotonin.m2m2.Lifecycle.terminate:427) - Mango Lifecycle terminating... 
    INFO  2020-06-10T18:38:07,559 (com.serotonin.m2m2.db.H2Proxy.terminateImpl:394) - Terminating database. 
    ERROR 2020-06-10T18:38:07,823 (com.serotonin.m2m2.Lifecycle.terminate:563) - Failed to terminate database. 
    org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:370) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:417) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:280) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.H2Proxy.terminateImpl(H2Proxy.java:395) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.db.AbstractDatabaseProxy.terminate(AbstractDatabaseProxy.java:211) ~[mango-3.6.5.jar:?]
    	at com.serotonin.m2m2.Lifecycle.databaseTerminate(Lifecycle.java:765) ~[ma-priv-3.6.5.jar:?]
    	at com.serotonin.m2m2.Lifecycle.terminate(Lifecycle.java:561) ~[ma-priv-3.6.5.jar:?]
    	at com.serotonin.m2m2.Main.main(Main.java:147) ~[ma-priv-3.6.5.jar:?]
    Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	... 8 more
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    	at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
    	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	... 8 more
    INFO  2020-06-10T18:38:07,826 (com.serotonin.m2m2.Lifecycle.terminate:578) - Mango Lifecycle terminated. 
    

    Running the web console manually gives the same out of bounds error when the connection is tested.

    General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] HY000/50000
    

    Update

    Managed to get it fixed (after a database restore) by:

    1. SET REFERENTIAL_INTEGRITY FALSE;
    2. TRUNCATE TABLE EVENTS;
    3. SET REFERENTIAL_INTEGRITY TRUE;
    4. TRUNCATE TABLE USEREVENTS;
    5. Restart Mango

    Database went from 6Gb to 17Mb.