Out of memory problem

petermcs

I have a Mango installation (Centos 7) that has been running fine for a couple of years on a customers site. When I logged in yesterday to check on it the UI had locked up. I managed to restart Mango but it was still very slow to respond and screens would not populate. I tried a few things including installing the 8 or so updates that were outstanding to no avail. When I run the System Monitor it shows JAVA using almost 100% processor and several GB of RAM. I updated the Mango config to use large memory java init with 10G RAM available but in a matter of a couple of minutes Java used all that up.
I was using Java SE 1.8.152 and I considered updating to the latest but I see Oracle have changed their licensing and I haven't figured out the consequences of that yet. Instead I updated to OpenJDK 1.8.0.232 but this made no difference.
Any suggestions on what I should check next?

BTW this is the console log:

[petermcs@localhost ~]$ cd /opt/mango/e*/bin
[petermcs@localhost bin]$ sudo ./ma.sh start
[sudo] password for petermcs: 
MA_HOME is /opt/mango/enterprise-m2m2-core-3.2.2
[petermcs@localhost bin]$ MA removing dead pid file for PID:  5322
ma-start: MA started with Process ID:  12604
INFO  2019-11-06T10:24:04,596 (com.serotonin.m2m2.Main.main:105) - Starting Mango 3.6.5 with Schema Version: 28 
INFO  2019-11-06T10:24:06,314 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'ssh', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,401 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'Haystack', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,490 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'loggingConsole', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,499 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'advancedComponents', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,535 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/advancedComponents/web/dox/manifest.xml 
INFO  2019-11-06T10:24:06,575 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dnp3', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,634 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'excelReports', v3.6.2 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,765 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'openv4j', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,834 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dataPointDetailsView', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,905 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mqttClientDataSource', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,952 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'scripting', v3.6.2 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,955 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'opcda', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:06,956 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'reports', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,013 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sstTheme', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,014 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'measurlogicDTSCell', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,019 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mbus', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,069 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'twilio', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,163 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'asciiFile', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,211 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'controlcore', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,298 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'meta', v3.6.2 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,382 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mangoUI', v3.6.7 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,450 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/mangoUI/web/dox/manifest.xml 
INFO  2019-11-06T10:24:07,835 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'BACnet', v3.6.3 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,906 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mangoApi', v3.6.4 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,981 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'log4JReset', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:07,981 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pointLinks', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,050 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sstGraphics', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,195 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'virtualDS', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,233 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'mangoNoSqlDatabase', v3.6.4 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,276 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'http', v3.6.2 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,305 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'persistent', v3.6.2 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,320 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dashboardDesigner', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,322 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/dashboardDesigner/web/dox/manifest.xml 
INFO  2019-11-06T10:24:08,338 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'graphicalViews', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,393 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'serial', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,395 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'maintenanceEvents', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,398 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'onewire', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,400 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'scheduledEvents', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,441 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'vmstat', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,455 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dataFile', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,488 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'egauge', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,521 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'jsonFileImport', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,595 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'watchlists', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,648 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'advancedScheduler', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,649 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'dataImport', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,665 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'envcands', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,666 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'log4jDS', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,668 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pachube', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,671 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'snmp', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,673 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'TCPIP', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,689 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'deviceConfig', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,690 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /opt/mango/enterprise-m2m2-core-3.2.2/web/modules/deviceConfig/web/dox/manifest.xml 
INFO  2019-11-06T10:24:08,708 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'jmxds', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,712 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pakbus', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,802 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sqlConsole', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,804 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'templateConfig', v3.6.0 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,846 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sstGlobalScripts', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,855 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'modbus', v3.6.6 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,895 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pid', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,960 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'sqlds', v3.6.1 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:08,995 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'pop3', v3.6.2 by Infinite Automation Systems Inc. 
INFO  2019-11-06T10:24:09,012 (com.serotonin.m2m2.Main.loadModules:586) - Loading module 'internal', v3.6.0 by Infinite Automation Systems Inc. 
WARN  2019-11-06T10:24:13,790 (com.serotonin.m2m2.Lifecycle.webServerInitialize:1058) - BROWSE action not supported on your platform, unable to open browser at startup. 
INFO  2019-11-06T10:24:13,979 (com.serotonin.m2m2.Lifecycle.loadLic:611) - Checking license... 
INFO  2019-11-06T10:24:14,154 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:61) - Initializing H2 connection manager 
INFO  2019-11-06T10:24:18,884 (com.serotonin.m2m2.db.upgrade.DBUpgrade.checkUpgrade:38) - Starting instance with core version 3.6.5, schema v28 
INFO  2019-11-06T10:24:18,884 (com.infiniteautomation.nosql.MangoNoSqlProxy.initialize:77) - Starting MangoNoSql Database 
INFO  2019-11-06T10:24:18,994 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:144) - Database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDB started 
INFO  2019-11-06T10:24:22,897 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:173) - Empty dir delete for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDB took 3874ms 
INFO  2019-11-06T10:24:22,901 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:234) - Corruption scan for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDB skipped because the database is configured not to run scan. 
INFO  2019-11-06T10:24:22,945 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:144) - Database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDBAux started 
INFO  2019-11-06T10:24:22,971 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:173) - Empty dir delete for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDBAux took 0ms 
INFO  2019-11-06T10:24:22,972 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:234) - Corruption scan for database at /opt/mango/enterprise-m2m2-core-3.2.2/databases/mangoTSDBAux skipped because the database is configured not to run scan. 
INFO  2019-11-06T10:24:25,577 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'runtimeContext' refreshed: org.springframework.context.annotation.AnnotationConfigApplicationContext@6f044c58 
INFO  2019-11-06T10:24:25,581 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextStarted:139) - Spring context 'runtimeContext' started: org.springframework.context.annotation.AnnotationConfigApplicationContext@6f044c58 
INFO  2019-11-06T10:24:25,777 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:77) - Initializing 5 NORMAL priority data sources in 8 threads. 
INFO  2019-11-06T10:24:26,276 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Schedule Points took 497ms to start 
INFO  2019-11-06T10:24:26,351 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Mango Internal took 571ms to start 
INFO  2019-11-06T10:24:27,119 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Fast BACnet took 1341ms to start 
INFO  2019-11-06T10:24:28,951 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source HVAC took 3172ms to start 
INFO  2019-11-06T10:24:29,261 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source site took 3480ms to start 
INFO  2019-11-06T10:24:29,327 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:109) - Initialization of 5 NORMAL priority data sources took 3550ms 
INFO  2019-11-06T10:24:29,327 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:77) - Initializing 5 LAST priority data sources in 8 threads. 
INFO  2019-11-06T10:24:30,377 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source landlord total electricity took 1048ms to start 
INFO  2019-11-06T10:24:30,377 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source innovate total electricity took 1048ms to start 
INFO  2019-11-06T10:24:30,393 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source wcc total electricity took 1065ms to start 
INFO  2019-11-06T10:24:30,560 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source site total electricity took 1232ms to start 
INFO  2019-11-06T10:24:30,664 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:408) - Data source Window Control took 1336ms to start 
INFO  2019-11-06T10:24:30,704 (com.serotonin.m2m2.rt.DataSourceGroupInitializer.initialize:109) - Initialization of 5 LAST priority data sources took 1377ms 
INFO  2019-11-06T10:24:31,102 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initialize:176) - Starting 0 Publishers... 
INFO  2019-11-06T10:24:31,102 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initialize:188) - 0 Publisher's started in 3ms 
WARN  2019-11-06T10:24:32,572 (org.eclipse.jetty.webapp.StandardDescriptorProcessor.addServletMapping:1220) - Duplicate mapping from *.jsp to jsp 
INFO  2019-11-06T10:24:33,190 (com.serotonin.m2m2.web.M2M2ContextListener.contextInitialized:32) - Mango Automation context starting 
INFO  2019-11-06T10:24:33,190 (com.serotonin.m2m2.web.M2M2ContextListener.contextInitialized:40) - Mango Automation context started 
INFO  2019-11-06T10:24:34,149 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'rootWebContext' refreshed: Root WebApplicationContext 
INFO  2019-11-06T10:24:34,892 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'jspDispatcherContext' refreshed: WebApplicationContext for namespace 'JSP_DISPATCHER-servlet' 
INFO  2019-11-06T10:24:35,212 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'rootRestDispatcherContext' refreshed: WebApplicationContext for namespace 'ROOT_REST_DISPATCHER-servlet' 
INFO  2019-11-06T10:24:36,706 (com.serotonin.m2m2.web.mvc.rest.v1.BacnetDiscoveryRestController.<init>:54) - Creating Discovery Rest Controller 
INFO  2019-11-06T10:24:37,042 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'restV1DispatcherContext' refreshed: WebApplicationContext for namespace 'restV1DispatcherServlet-servlet' 
INFO  2019-11-06T10:24:38,847 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:125) - Spring context 'restV2DispatcherContext' refreshed: WebApplicationContext for namespace 'restV2DispatcherServlet-servlet' 
INFO  2019-11-06T10:24:38,850 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MeasurlogicDTSCellDataSourceDwr 
INFO  2019-11-06T10:24:38,850 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: EnvCanEditDwr 
INFO  2019-11-06T10:24:38,852 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: HttpDataSourceDwr 
INFO  2019-11-06T10:24:38,853 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: TcpIpDataSourceEditDwr 
INFO  2019-11-06T10:24:38,854 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: Pop3EditDwr 
INFO  2019-11-06T10:24:38,855 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: HaystackDataSourceDwr 
INFO  2019-11-06T10:24:38,855 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
INFO  2019-11-06T10:24:38,856 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: OPCDAEditDwr 
INFO  2019-11-06T10:24:38,858 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: DataFileDataSourceEditDwr 
INFO  2019-11-06T10:24:38,860 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: BACnetMSTPEditDwr 
INFO  2019-11-06T10:24:38,862 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PakBusEditDwr 
INFO  2019-11-06T10:24:38,863 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MBusEditDwr 
INFO  2019-11-06T10:24:38,863 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.infiniteautomation.mango.pakbus.dwr.PakBusEditDwr 
INFO  2019-11-06T10:24:38,863 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SNMPEditDwr 
INFO  2019-11-06T10:24:38,864 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: InternalEditDwr 
INFO  2019-11-06T10:24:38,865 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SerialEditDwr 
INFO  2019-11-06T10:24:38,866 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MetaEditDwr 
INFO  2019-11-06T10:24:38,867 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PersistentDataSourceDwr 
INFO  2019-11-06T10:24:38,868 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: DnpEditDwr 
INFO  2019-11-06T10:24:38,869 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: ModbusEditDwr 
INFO  2019-11-06T10:24:38,870 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SshDataSourceEditDwr 
INFO  2019-11-06T10:24:38,872 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: ScriptingEditDwr 
INFO  2019-11-06T10:24:38,873 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: Log4jEditDwr 
INFO  2019-11-06T10:24:38,875 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PidDataSourceEditDwr 
INFO  2019-11-06T10:24:38,876 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: AsciiFileEditDwr 
INFO  2019-11-06T10:24:38,876 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
INFO  2019-11-06T10:24:38,878 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: OneWireEditDwr 
INFO  2019-11-06T10:24:38,881 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: PachubeDataSourceDwr 
INFO  2019-11-06T10:24:38,882 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: VMStatEditDwr 
INFO  2019-11-06T10:24:38,884 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: SqlEditDwr 
INFO  2019-11-06T10:24:38,884 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: MqttClientEditDwr 
INFO  2019-11-06T10:24:38,886 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: EgaugeDataSourceDwr 
INFO  2019-11-06T10:24:38,886 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
INFO  2019-11-06T10:24:38,887 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: JmxEditDwr 
INFO  2019-11-06T10:24:38,889 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: BACnetIPEditDwr 
INFO  2019-11-06T10:24:38,890 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: OpenVEditDwr 
INFO  2019-11-06T10:24:38,891 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: br.org.scadabr.dnp3.dwr.DnpEditDwr 
INFO  2019-11-06T10:24:38,892 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: ControlCoreEditDwr 
INFO  2019-11-06T10:24:38,892 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.modbus.dwr.ModbusEditDwr 
INFO  2019-11-06T10:24:38,893 (com.serotonin.m2m2.Lifecycle.configureDwr:1284) - Added DWR definition for: VirtualEditDwr 
INFO  2019-11-06T10:24:38,893 (com.serotonin.m2m2.Lifecycle.configureDwr:1277) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.http.dwr.HttpDataSourceDwr 
WARN  2019-11-06T10:31:38,863 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:150) - Data Source Fast BACnet aborted 1 polls since it started. 
WARN  2019-11-06T10:35:51,712 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:150) - Data Source Mango Internal aborted 1 polls since it started. 
WARN  2019-11-06T10:37:46,812 (org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask:361) -  
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.ArrayDeque.iterator(ArrayDeque.java:611) ~[?:1.8.0_152]
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processUpdates(ManagedSelector.java:376) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:355) ~[jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
WARN  2019-11-06T10:40:07,183 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:150) - Data Source site aborted 1 polls since it started.

terrypacker

@petermcs the best way to analyze OOM problems is by taking a heap dump and loading it into an analysis tool such as JVisualVM or MAT. However that file will generally be GB in size on a system like yours so before we get into that I would suggest that you take some samples of the Java process while it is running. From that we may be able to see what is eating up all the memory.

By using the jmap tool packaged with the JDK you should be able to get the necessary info:

https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html

Step 1. Find the PID of Mango while it is running. This can be by something like ps aux | grep java or looking into the ma.pid file in MA_HOME\bin

Step 2. Run jmap -histo <mango pid> every so often until Mango crashes. Write the output to a file for reference later obviously.

By looking at the classes that are using the most memory you should be able to get an idea of what the problem is. Please post a summary of your findings here and I can try to see what the problem is.

NOTE: likely jmap won't be on your path so you may need to execute it from within the JDK installation directory directly.

petermcs

@terrypacker I got jmap running (seems that even though I changed JAVA_HOME to the OpenJDK folder Mango still runs using the Oracle Java so I need to look at the again... Anyway when I run jmap before and after the memory blows up, the results are identical!
Also a bit of further info, If I start Mango and don't connect (locally via 127.0.0.1 as I am Teamviewering in to check things) the browser the memory does not blow up. It only happens a few minutes after I connect and log in. If I connect but don't login I.e. stay at the login screen, the memory usage does not increase.

Here is the after jmap result

sudo /usr/java/jdk1.8.0_152/bin/jmap 18673
Attaching to process ID 18673, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.152-b16
0x0000000000400000	7K	/usr/java/jdk1.8.0_152/bin/java
0x00007f9eb46f8000	30K	/usr/lib64/libnss_dns-2.17.so
0x00007f9eb6118000	259K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libjpeg.so
0x00007f9ec7168000	11K	/usr/lib64/libfreebl3.so
0x00007f9ec736b000	39K	/usr/lib64/libcrypt-2.17.so
0x00007f9ec75a2000	40K	/usr/lib64/libltdl.so.7.3.0
0x00007f9ec77ac000	93K	/usr/lib64/libtdb.so.1.3.16
0x00007f9ec79c3000	27K	/usr/lib64/libogg.so.0.8.0
0x00007f9ec7bca000	180K	/usr/lib64/libvorbis.so.0.4.6
0x00007f9ec7df7000	35K	/usr/lib64/libvorbisfile.so.3.3.5
0x00007f9ecc091000	68K	/usr/lib64/libcanberra.so.0.2.5
0x00007f9ecc2a2000	19K	/usr/lib64/libcanberra-gtk.so.0.1.9
0x00007f9ecc4a7000	23K	/usr/lib64/gtk-2.0/modules/libcanberra-gtk-module.so
0x00007f9ecc6ad000	6K	/usr/lib64/gtk-2.0/modules/libpk-gtk-module.so
0x00007f9ecc8af000	43K	/usr/lib64/gtk-2.0/2.10.0/engines/libpixmap.so
0x00007f9eccaba000	15K	/usr/lib64/gtk-2.0/2.10.0/engines/libadwaita.so
0x00007f9ecccbe000	6K	/usr/lib64/libgthread-2.0.so.0.5600.1
0x00007f9eccec0000	188K	/usr/lib64/libgraphite2.so.3.0.1
0x00007f9ecd0ee000	259K	/usr/lib64/libblkid.so.1.1.0
0x00007f9ecd32e000	73K	/usr/lib64/libGLX.so.0.0.0
0x00007f9ecd560000	625K	/usr/lib64/libGLdispatch.so.0.0.0
0x00007f9ecd816000	66K	/usr/lib64/libbz2.so.1.0.6
0x00007f9ecda26000	19K	/usr/lib64/libuuid.so.1.3.0
0x00007f9ecdc2b000	169K	/usr/lib64/libexpat.so.1.6.0
0x00007f9ecde55000	111K	/usr/lib64/libfribidi.so.0.4.0
0x00007f9ece071000	50K	/usr/lib64/libthai.so.0.1.6
0x00007f9ece27d000	637K	/usr/lib64/libharfbuzz.so.0.10705.0
0x00007f9ece51a000	271K	/usr/lib64/libmount.so.1.1.0
0x00007f9ece75d000	103K	/usr/lib64/libresolv-2.17.so
0x00007f9ece976000	152K	/usr/lib64/libselinux.so.1
0x00007f9eceb9d000	31K	/usr/lib64/libffi.so.6.0.1
0x00007f9eceda5000	568K	/usr/lib64/libGL.so.1.7.0
0x00007f9ecf031000	88K	/usr/lib64/libz.so.1.2.7
0x00007f9ecf247000	55K	/usr/lib64/libxcb-render.so.0.0.0
0x00007f9ecf455000	15K	/usr/lib64/libxcb-shm.so.0.0.0
0x00007f9ecf659000	175K	/usr/lib64/libpng15.so.15.13.0
0x00007f9ecf884000	82K	/usr/lib64/libEGL.so.1.1.0
0x00007f9ecfa98000	682K	/usr/lib64/libpixman-1.so.0.34.0
0x00007f9ecfd41000	776K	/usr/lib64/libfreetype.so.6.14.0
0x00007f9ed8044000	392K	/usr/lib64/libpcre.so.1.2.0
0x00007f9ed82a6000	10K	/usr/lib64/libXdamage.so.1.1.0
0x00007f9ed84a9000	10K	/usr/lib64/libXcomposite.so.1.0.0
0x00007f9ed86ac000	43K	/usr/lib64/libXrandr.so.2.2.0
0x00007f9ed88b7000	11K	/usr/lib64/libXinerama.so.1.0.0
0x00007f9ed8aba000	1129K	/usr/lib64/libglib-2.0.so.0.5600.1
0x00007f9ed8dd0000	331K	/usr/lib64/libgobject-2.0.so.0.5600.1
0x00007f9ed9021000	270K	/usr/lib64/libfontconfig.so.1.11.1
0x00007f9ed9263000	281K	/usr/lib64/libpango-1.0.so.0.4200.3
0x00007f9ed94a9000	88K	/usr/lib64/libpangoft2-1.0.so.0.4200.3
0x00007f9ed96bf000	1682K	/usr/lib64/libgio-2.0.so.0.5600.1
0x00007f9ed9a5e000	162K	/usr/lib64/libgdk_pixbuf-2.0.so.0.3612.0
0x00007f9ed9c86000	1263K	/usr/lib64/libcairo.so.2.11512.0
0x00007f9ed9fbd000	153K	/usr/lib64/libatk-1.0.so.0.22810.1
0x00007f9eda1e3000	55K	/usr/lib64/libpangocairo-1.0.so.0.4200.3
0x00007f9eda3f1000	15K	/usr/lib64/libgmodule-2.0.so.0.5600.1
0x00007f9eda5f5000	789K	/usr/lib64/libgdk-x11-2.0.so.0.2400.31
0x00007f9eda8b6000	4705K	/usr/lib64/libgtk-x11-2.0.so.0.2400.31
0x00007f9edb039000	23K	/usr/lib64/libXfixes.so.3.1.0
0x00007f9edb23f000	44K	/usr/lib64/libXcursor.so.1.0.2
0x00007f9edb54b000	513K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libfontmanager.so
0x00007f9edb9c4000	15K	/usr/lib64/libXau.so.6.0.0
0x00007f9edbbc8000	162K	/usr/lib64/libxcb.so.1.1.0
0x00007f9edbdf0000	64K	/usr/lib64/libXi.so.6.1.0
0x00007f9f18023000	23K	/usr/lib64/libXtst.so.6.1.0
0x00007f9f18229000	43K	/usr/lib64/libXrender.so.1.3.0
0x00007f9f18434000	1287K	/usr/lib64/libX11.so.6.3.0
0x00007f9f18772000	74K	/usr/lib64/libXext.so.6.4.0
0x00007f9f189a3000	432K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libawt_xawt.so
0x00007f9f18bfc000	754K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libawt.so
0x00007f9f507e9000	86K	/usr/lib64/libgcc_s-4.8.5-20150702.so.1
0x00007f9f50a1e000	251K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libsunec.so
0x00007f9f51168000	91K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libnio.so
0x00007f9f51394000	112K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libnet.so
0x00007f9f517ab000	49K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libmanagement.so
0x00007f9f6f1f3000	125K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libzip.so
0x00007f9f6f40f000	60K	/usr/lib64/libnss_files-2.17.so
0x00007f9f6f622000	221K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libjava.so
0x00007f9f6f84e000	64K	/usr/java/jdk1.8.0_152/jre/lib/amd64/libverify.so
0x00007f9f6fa5d000	42K	/usr/lib64/librt-2.17.so
0x00007f9f6fc65000	1110K	/usr/lib64/libm-2.17.so
0x00007f9f6ff67000	16640K	/usr/java/jdk1.8.0_152/jre/lib/amd64/server/libjvm.so
0x00007f9f70f63000	2105K	/usr/lib64/libc-2.17.so
0x00007f9f71331000	18K	/usr/lib64/libdl-2.17.so
0x00007f9f71535000	101K	/usr/java/jdk1.8.0_152/lib/amd64/jli/libjli.so
0x00007f9f7174b000	138K	/usr/lib64/libpthread-2.17.so
0x00007f9f71967000	159K	/usr/lib64/ld-2.17.so

terrypacker

@petermcs please run jmap -histo pid to get the object count.

Turbo

Okay:

Here's some stuff related to this after going through this:

I just upgraded to Mango 3.6.5 running on our production server. We use persistent TCP to haul data back from outstations monitoring solar assets, and it works Very Well.

However, there's a couple of things that popped up after the upgrade. We've identified (2) separate issues:

Issue (1): Reporting tool causes Mango to run out of memory and crash itself. Mango apparently doesn't check these things internally, so if you run out of Ram, it just merrily gobbles it all up and blows up. We have a spreadsheet that's about 8 megs, that loads an additional meg of data from Mango, and then reports it out again. Running this spreadsheet uses 1500 megs of ram, according to the Mango's internal JVM free memory.

We fixed this by setting the -Xms5g and -Xmx5g via the "memory-medium.sh". However, if we have more than one of two of those reports ran at the same time, the system will still Blow Up. So, I'd like to see Mango either (1) do something to manage these reports, if it means slowing them down or something to not eat that much memory, or (2) at least crash elegantly: When this occurs with our system, the thing completely shuts down, and then does not re-start itself. I'd rather the reporting tool not crash the system, but if it does, it'd be nice to recover gracefully. As a solution to this, we're going to keep jacking up -Xms and -Xmx until we run out of memory on our servers, then we'll do something else.

Issue (2), And maybe this is a different thread:
Mango opens LOTS of files. At night, when the outstations are set to synchronize historic back to our cloud server, we see >100,000 files open, as reported by pmap. This normally is ~1000, but at night, when the historical syncs happen, that number jumps by a lot. This causes a Java fault, as the thing bumps against the available files.

We increased the max file limit by doing
vm.max_map_count=131060
to /etc/sysctl.conf

and restarting.

We're still regularly getting >100k open files, but at least now Java doesn't crash out. We have the tsdb running out our instance, and something must have changed between 3.5 and 3.6 to how this is done, as we haven't changed any settings that would cause additional data to be written via the persistent publisher.

Just letting people know here about the work we did on this. Is there some way in Mango to limit the number of open files it does on a historical sync? My concern here is that as I add more sites (and keep historical sync active, to force the system to catch up on any weird outages that happen during the day as is common with cell-connected systems), this file open issue is going to increasingly get bigger and bigger. Did I miss something here about the best way to do this?

Cheers,
-Greg Linder

Turbo

Also:

For those of you running into memory problems in Mango:

Since the JVM doesn't ever "give memory back" to the operating system, whatever you set those -Xms and -Xmx to will be about what the JVM will take. once it reaches that Max of that, you will NEVER see that memory returned to the system unless you restart the JVM.

Our system, when running reports, cranks right up to the limit, and then sits there. If you see CPU utilization pegged high, but the system in unresponsive, it's possible that the JVM has maxed out it's available memory (partially defined by -Xms and -Xmx settings), and the JVM is cranking around it's jobs trying to get your work done without anything getting actually complete.

CraigWeb

Hi Greg

I think it will be helpful to know how large your system is. How many points and which database are you using. H2 or MySQL?

@turbo said in Out of memory problem:

Just letting people know here about the work we did on this. Is there some way in Mango to limit the number of open files it does on a historical sync? My concern here is that as I add more sites (and keep historical sync active, to force the system to catch up on any weird outages that happen during the day as is common with cell-connected systems), this file open issue is going to increasingly get bigger and bigger. Did I miss something here about the best way to do this?

I would consider running the sync more often, I have been using a cron pattern of every 5 minutes
and have not run into any issue with it. However, I have no idea how much data you are logging.
Also, have you enabled "save real-time data" on the data source? This could be why the sync needs to open so many files.

The current recommended way to run mango as a service does lack the ability to restart after crashing. Are you running Ubuntu or debian? Some work has been done on a new service script which you could use.

terrypacker

@Turbo

Here is some general information on what you see, and I have been tracking these known problems for some time trying to find a solution.

Excel Reports Module can use up all the JVM memory.

This is mostly out of our control and a known problem with the Apache POI library that is the backbone of that module. They have recently released a more performant version that will eventually be part of our Excel Reports module. We have reworked our REST api using a new algorithm to manage memory that will in the not-to-distant future be ported over to the ExcelReports module and should solve all of the problems outside of the POI library limitations.

Open Files - MangoNoSQL
The time series database will open many files and leave them open for some time, they eventually do get closed but for performance reasons we don't close them immediately after use. As @CraigWeb mentioned better tuning can help during intense syncs. As for the problem being worse in 3.6, I was not aware of that and can confirm that the logic for opening and closing the files has not changes. It is likely that you system is just getting larger and you are now aware that this happens. There are a whole slew of settings for very large systems in the env.properties file with the prefix: db.nosql.*. I don't have time to go into them all here but I would suggest experimenting with them and posting questions about specific properties later.

As for the JVM general memory use:

We are starting to use Adopt OpenJDK 13 and are finding that it is much better at memory management. As for the -Xms and -Xmx settings, depending what you are trying to achieve they may not be ideal. By setting both to the same value you get a fully allocated JVM at startup which on a machine only running Mango is likely ideal. But for a system with other processes that need memory at other times the JVM cannot release memory with that configuration. We have found that on a very large machine using Adopt OpenJDK it is quite good expanding and contracting the JVM memory use for Mango's needs. But we are still evaluating the best configuration for it at this point.

Turbo

Greetings, all:

Thanks for the answers here. I appreciate the clarity on this.

As for "Sync every five minutes" vs "save real-time data"-- I'm a bit unclear on how these options interact with each other.

I'd like to get all the data that my systems in the field are logging. The points out there are set with different poll and logging settings. So if I set to "sync every 5 minutes" (or something similar), does that mean that my data points will update every 5 minutes with the logged data? Vs the "save real time data" that saves all the data that comes in from every poll?

Is that the proper way of using those options?

Cheers,
-Greg

cbyrne

@terrypacker Hi Terry, I am just continuing @petermcs's work and have the jmap -hist outputs.

Before:

 num     #instances         #bytes  class name
----------------------------------------------
   1:        173771      715822488  [I
   2:       3748110      508063720  [C
   3:       1945314      443221056  [B
   4:       1733728       41609472  java.lang.String
   5:       1973991       37379904  [Ljava.lang.Class;
   6:        478710       23272440  [Ljava.lang.Object;
   7:             1       16777232  [Lorg.h2.util.CacheObject;
   8:        151493       11321608  [Ljava.util.HashMap$Node;
   9:        228571       10971408  java.util.HashMap
  10:        168383       10776512  java.net.URL
  11:        318951       10206432  java.util.HashMap$Node
  12:         99723        9573408  java.util.jar.JarFile$JarFileEntry
  13:        101097        8896536  java.lang.reflect.Method
  14:         99443        7955440  java.util.zip.ZipEntry
  15:        194720        7788800  java.util.LinkedHashMap$Entry
  16:        262144        6291456  org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper
  17:        107562        6023472  java.util.LinkedHashMap
  18:        211916        5085984  java.lang.StringBuilder
  19:        155322        4970304  java.util.LinkedHashMap$LinkedEntryIterator
  20:        132436        4237952  java.util.ArrayList$Itr
  21:        106631        4096384  [Ljava.lang.String;
  22:         48305        3915312  [S
  23:        103984        3635760  [Ljava.lang.reflect.Method;
  24:        148571        3565704  java.util.ArrayList
  25:        145630        3495120  org.springframework.core.annotation.AnnotationUtils$AnnotationCacheKey
  26:        213448        3415168  java.util.HashSet
  27:         57295        3208520  org.springframework.asm.Label
  28:          5081        3195656  [J
  29:         37779        3022320  java.lang.reflect.Constructor
  30:         89701        2870432  java.io.File
  31:         59175        2840400  java.nio.HeapByteBuffer
  32:        168665        2698640  java.lang.Integer
  33:         16617        2641896  [Lorg.springframework.asm.Label;
  34:         73858        2363456  java.util.concurrent.ConcurrentHashMap$Node
  35:         89469        2147256  java.lang.Class$MethodArray
  36:         44728        2146944  org.springframework.core.ResolvableType
  37:         81522        1956528  java.util.LinkedList$Node
  38:         32363        1812328  org.springframework.asm.SymbolTable$Entry
  39:         15886        1783136  java.lang.Class
  40:         52571        1682272  java.util.LinkedHashMap$LinkedKeyIterator
  41:         22174        1596528  org.springframework.core.annotation.AnnotationAttributes
  42:         32146        1543008  org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference
  43:         57998        1391952  java.util.concurrent.CopyOnWriteArrayList$COWIterator
  44:         42207        1350624  org.springframework.asm.Type
  45:         29777        1191080  java.lang.ref.Finalizer
  46:         36791        1177312  java.util.AbstractList$Itr
  47:         65801        1052816  sun.reflect.ConstantPool
  48:          9550         993200  sun.net.www.protocol.file.FileURLConnection
  49:         19600         940800  org.springframework.asm.Frame
  50:         22090         883600  com.serotonin.bacnet4j.util.sero.ByteQueue
  51:         37028         864280  [Ljava.lang.reflect.Type;
  52:         35583         853992  org.springframework.asm.Edge
  53:          1968         826856  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  54:         25594         819008  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
  55:         50928         814848  java.lang.Object
  56:         25147         804704  java.util.LinkedHashMap$LinkedValueIterator
  57:         24671         789472  java.util.LinkedList
  58:         49091         785456  org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder
  59:         13853         775768  org.springframework.core.type.classreading.MethodMetadataReadingVisitor
  60:         13740         769440  jdk.internal.org.objectweb.asm.Item
  61:         31486         755664  java.util.Arrays$ArrayList
  62:         46717         747472  java.util.LinkedHashSet
  63:         17178         687120  java.lang.invoke.MethodType
  64:          9400         676800  java.lang.reflect.Field
  65:         27985         671640  sun.reflect.generics.tree.SimpleClassTypeSignature
  66:         26545         637080  sun.reflect.generics.visitor.Reifier
  67:          2806         628544  org.springframework.asm.MethodWriter
  68:         11198         627088  java.lang.invoke.LambdaFormEditor$Transform
  69:         24021         576504  sun.reflect.generics.factory.CoreReflectionFactory
  70:         23902         573648  sun.reflect.generics.parser.SignatureParser
  71:         10093         565208  java.util.zip.ZipFile$ZipFileInputStream
  72:         10089         564984  java.util.zip.ZipFile$ZipFileInflaterInputStream
  73:         23307         559368  sun.reflect.generics.scope.ClassScope
  74:         17324         554368  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
  75:         13514         540560  java.util.HashMap$KeyIterator
  76:         16637         532384  java.lang.ref.WeakReference
  77:         13288         531520  java.util.HashMap$EntryIterator
  78:          6026         530288  sun.net.www.protocol.jar.JarURLConnection
  79:         16281         520992  java.util.RegularEnumSet
  80:         31992         511872  java.util.LinkedHashMap$LinkedKeySet
  81:         18654         507960  [Ljava.lang.annotation.Annotation;
  82:         15760         504320  org.springframework.util.ConcurrentReferenceHashMap$1
  83:         30116         498104  [Lsun.reflect.generics.tree.TypeArgument;
  84:          7673         491072  org.springframework.core.MethodParameter
  85:          5066         486336  sun.net.www.protocol.jar.URLJarFile$URLJarFileEntry
  86:         14914         477248  java.util.LinkedList$ListItr
  87:          5921         476640  [Ljava.util.WeakHashMap$Entry;
  88:          7171         458944  com.serotonin.timer.OrderedTaskInfo
  89:           394         452912  [Lorg.springframework.asm.SymbolTable$Entry;
  90:         18617         446808  org.springframework.core.annotation.AnnotatedElementUtils$MergedAnnotationAttributesProcessor
  91:         27787         444592  sun.reflect.generics.tree.ClassTypeSignature
  92:         11069         442760  sun.nio.cs.UTF_8$Decoder
  93:          6897         441408  java.util.regex.Matcher
  94:         13780         440960  org.springframework.core.annotation.AnnotationUtils$$Lambda$48/1194398327
  95:          5356         428480  sun.net.www.protocol.jar.URLJarFile
  96:          7588         424928  com.serotonin.bacnet4j.npdu.NPCI
  97:         19494         420832  [Lorg.springframework.asm.Type;
  98:         12977         415264  java.lang.reflect.WeakCache$CacheKey
  99:         17180         412320  org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey
 100:         12269         392608  java.util.Hashtable$Entry

Total      16594285     2034054440

After opening a browser window and logging in, greeted by the admin page and an immediate "connectivity lost, restoring connection" toast message that never permenantly resolves.
It then takes about 3-5 minutes to max out


 num     #instances         #bytes  class name
----------------------------------------------
   1:      51017979     3130737016  [C
   2:      51017618     1224422832  java.lang.String
   3:      10175891      895478408  com.serotonin.m2m2.rt.event.EventInstance
   4:      20380459      550396632  [Ljava.lang.Object;
   5:      20352135      488451240  com.serotonin.m2m2.i18n.TranslatableMessage
   6:      10175862      325627584  com.serotonin.m2m2.rt.event.type.DataSourceEventType
   7:        469887      198080400  [B
   8:       1599229       87706096  [Lorg.h2.value.Value;
   9:       1248203       39942496  org.h2.result.RowImpl
  10:        425490       30635280  org.h2.value.ValueLobDb
  11:        828920       26525440  org.h2.result.SimpleRowValue
  12:             1       16777232  [Lorg.h2.util.CacheObject;
  13:         45454       15596376  [I
  14:         31155       12509192  [J
  15:        438464        7015424  org.h2.value.ValueInt
  16:        265342        6368208  org.h2.value.ValueLong
  17:        262144        6291456  org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper
  18:         30244        5475296  [Lorg.h2.result.Row;
  19:         85336        3413440  java.util.LinkedHashMap$Entry
  20:          2874        3363200  [Lorg.h2.result.SearchRow;
  21:         30243        3145272  org.h2.index.PageDataLeaf
  22:         28393        2498584  java.lang.reflect.Method
  23:         28704        2297592  [Ljava.util.HashMap$Node;
  24:         66713        2134816  java.util.HashMap$Node
  25:         60656        1940992  java.util.concurrent.ConcurrentHashMap$Node
  26:         16839        1888904  java.lang.Class
  27:         23457        1313592  java.util.LinkedHashMap
  28:         33277        1064864  org.h2.store.Data
  29:           608         741064  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  30:          8489         611208  org.springframework.core.annotation.AnnotationAttributes
  31:         26206         593480  [Ljava.lang.Class;
  32:          7131         513432  java.lang.reflect.Field
  33:         31170         498720  java.lang.Object
  34:         10792         476208  [Ljava.lang.String;
  35:         18951         454824  java.util.ArrayList
  36:          9363         449424  java.util.HashMap
  37:         15410         369840  org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey
  38:          4273         341840  java.lang.reflect.Constructor
  39:          9479         303328  java.util.LinkedList
  40:          9008         288256  java.util.Hashtable$Entry
  41:         11579         277896  java.util.LinkedList$Node
  42:         17155         274480  org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder
  43:         11080         265920  org.h2.value.ValueRow
  44:          2851         228080  org.h2.index.PageBtreeLeaf
  45:          4539         181560  java.lang.ref.SoftReference
  46:          2998         167888  java.lang.Class$ReflectionData
  47:         10089         161424  java.util.LinkedHashMap$LinkedEntrySet
  48:          9949         159184  java.util.LinkedHashSet
  49:          3796         151840  javax.servlet.jsp.tagext.TagAttributeInfo
  50:          2614         146384  java.lang.invoke.MemberName
  51:          6080         145920  java.util.jar.Attributes$Name
  52:          5878         141072  sun.reflect.generics.tree.SimpleClassTypeSignature
  53:          3514         140560  java.util.TreeMap$Entry
  54:          2734         140264  [Ljava.lang.reflect.Method;
  55:             4         131136  [Ljava.util.concurrent.ForkJoinTask;
  56:          1807         130104  org.springframework.web.method.HandlerMethod$HandlerMethodParameter
  57:          3164         126560  jdk.nashorn.internal.runtime.CompiledFunction
  58:          2488         119424  org.springframework.core.ResolvableType
  59:          1796         114944  java.net.URL
  60:          5878         109584  [Lsun.reflect.generics.tree.TypeArgument;
  61:          1890         105840  jdk.nashorn.internal.runtime.ScriptFunction
  62:           488         105720  [Ljava.util.Hashtable$Entry;
  63:          4046          97104  sun.reflect.annotation.AnnotationInvocationHandler
  64:          1483          94912  java.util.concurrent.ConcurrentHashMap
  65:          3898          93696  [Lorg.springframework.core.annotation.AnnotationAttributes;
  66:          1056          92928  org.h2.table.Column
  67:          5702          91232  sun.reflect.generics.tree.ClassTypeSignature
  68:           543          91224  org.springframework.beans.factory.support.RootBeanDefinition
  69:          5543          88688  java.lang.Integer
  70:          1485          83160  java.security.Provider$Service
  71:          1904          76160  java.lang.invoke.BoundMethodHandle$Species_LL
  72:          4608          73728  java.util.LinkedHashMap$LinkedKeySet
  73:           973          70056  org.springframework.core.type.classreading.AnnotationMetadataReadingVisitor
  74:          2944          70008  [Ljava.lang.reflect.Type;
  75:          2890          69360  java.security.Provider$ServiceKey
  76:           650          67600  org.h2.mvstore.Chunk
  77:          4151          66416  java.util.jar.Attributes
  78:          2063          66016  java.lang.ref.WeakReference
  79:          2036          65152  com.serotonin.bacnet4j.obj.PropertyTypeDefinition
  80:          2002          64064  jdk.nashorn.internal.runtime.FinalScriptFunctionData
  81:          1099          61544  java.lang.Package
  82:           162          61088  [Ljdk.nashorn.internal.runtime.PropertyHashMap$Element;
  83:          1507          60280  java.lang.invoke.MethodType
  84:          1497          59880  java.lang.ref.Finalizer
  85:           710          58992  [Ljava.util.WeakHashMap$Entry;
  86:            41          58216  [Ljava.nio.ByteBuffer;
  87:          2464          56256  [Ljava.lang.reflect.Constructor;
  88:           308          54208  org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$ConfigurationClassBeanDefinition
  89:          1123          53904  org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference
  90:          1648          52736  java.lang.invoke.DirectMethodHandle
  91:          1638          52416  jdk.nashorn.internal.runtime.PropertyHashMap$Element
  92:          1624          51968  java.lang.invoke.LambdaForm$Name
  93:           205          50840  com.serotonin.m2m2.vo.DataPointVO
  94:           880          49280  org.springframework.core.type.classreading.MethodMetadataReadingVisitor
  95:          2036          48864  com.serotonin.bacnet4j.obj.ObjectPropertyTypeDefinition
  96:          1519          48608  java.util.concurrent.locks.ReentrantLock$NonfairSync
  97:           432          48384  org.springframework.context.annotation.ScannedGenericBeanDefinition
  98:          1511          48352  java.lang.ref.ReferenceQueue
  99:          1508          48256  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
 100:          1147          45880  java.math.BigInteger

Total     169664483     7108600472

I've limited it to the first 100 and totals, just to save post space. The full outputs ~7k can be sent on if needed.

htop when things max out:

terrypacker

@cbyrne the second jmap -hist output looks like the majority of the memory in use is from events. I'm just guessing here but it looks like you have a large number of events in the database that are being loaded and processed on login. Mango has known issues when the events table becomes large (which we are working to fix in Mango 4.0).

To confirm this I would restart Mango with the h2 web console enabled so you can look at the database without logging into Mango and causing it to eat up all the memory. This can be done via the env properties:

db.web.start=true
db.web.port=8091

See also here: https://help.infiniteautomation.com/about-the-h2-database

Then login to the console make sure you set the JDBC url to point to the mango database on your file system, on a MangoES this would be jdbc:h2:/opt/mango/databases/mah2 basically the path to the db without the .h2.db on the end.

Then run this query: SELECT COUNT(*) FROM events;

If the number is very large and you are ok with removing all events from the system I suggest you delete all event data. Before doing this I would make a copy of the h2 database file as a backup. You can simply copy the file if Mango is down.

Then this post describes how to remove events from the database:
https://forum.infiniteautomation.com/topic/2266/full-storage-issue-on-mangoes
Basically run:

SET FOREIGN_KEY_CHECKS=0; 
CREATE TABLE eventsNewTable LIKE events; 
DROP TABLE events; 
RENAME TABLE eventsNewTable TO events; 
CREATE TABLE userEventsNewTable LIKE userEvents; 
DROP TABLE userEvents; 
RENAME TABLE userEventsNewTable TO userEvents; 
SET FOREIGN_KEY_CHECKS=1;

If the events were the problem I suggest you take a look at this to better manage your events:
https://help.infiniteautomation.com/managing-disk-space

Another possibility that we can look at if my first assumption is not true is that something is generating many events on login. But since a big chunk of memory is being held by DataSourceEventTypes I expect this is from them being stored in the database.

cbyrne

Perfect, thanks @terrypacker! I'll try that and update this post.

update 1:
The H2 database shell doesn't recognise LIKE in CREATE TABLE. It may not be supported, The H2 command info for CREATE DATABASE doesn't show LIKE - https://www.h2database.com/html/commands.html#create_table

update 2
I just did a DELETE FROM EVENTS; instead.

So yeah, there might have been a few too many events... 12 million. The database has gone from ~6 Gb to 76 Mb.

cbyrne

@terrypacker Sorry for dredging up an old thread but I ran the same DELETE FROM EVENTS; as before but this time mango has failed to restart. It gets an out of bounds error when connecting to H2.

INFO  2020-06-10T18:38:06,912 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:61) - Initializing H2 connection manager 
INFO  2020-06-10T18:38:06,953 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:83) - Initializing H2 web server 
FATAL 2020-06-10T18:38:07,527 (com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize:163) - Unable to connect to database of type H2 
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:611) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) ~[mango-3.6.5.jar:?]
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:405) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.AbstractDatabaseProxy.newDatabaseCheck(AbstractDatabaseProxy.java:181) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize(AbstractDatabaseProxy.java:82) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:760) ~[ma-priv-3.6.5.jar:?]
	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:312) ~[ma-priv-3.6.5.jar:?]
	at com.serotonin.m2m2.Main.main(Main.java:143) ~[ma-priv-3.6.5.jar:?]
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	... 12 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
	at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	... 12 more
ERROR 2020-06-10T18:38:07,527 (com.serotonin.m2m2.Main.main:146) - Error during initialization 
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:611) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) ~[mango-3.6.5.jar:?]
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:405) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.AbstractDatabaseProxy.newDatabaseCheck(AbstractDatabaseProxy.java:181) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize(AbstractDatabaseProxy.java:82) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:760) ~[ma-priv-3.6.5.jar:?]
	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:312) ~[ma-priv-3.6.5.jar:?]
	at com.serotonin.m2m2.Main.main(Main.java:143) ~[ma-priv-3.6.5.jar:?]
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	... 12 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
	at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	... 12 more
INFO  2020-06-10T18:38:07,527 (com.serotonin.m2m2.Lifecycle.terminate:427) - Mango Lifecycle terminating... 
INFO  2020-06-10T18:38:07,559 (com.serotonin.m2m2.db.H2Proxy.terminateImpl:394) - Terminating database. 
ERROR 2020-06-10T18:38:07,823 (com.serotonin.m2m2.Lifecycle.terminate:563) - Failed to terminate database. 
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:370) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:417) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:280) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.H2Proxy.terminateImpl(H2Proxy.java:395) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.db.AbstractDatabaseProxy.terminate(AbstractDatabaseProxy.java:211) ~[mango-3.6.5.jar:?]
	at com.serotonin.m2m2.Lifecycle.databaseTerminate(Lifecycle.java:765) ~[ma-priv-3.6.5.jar:?]
	at com.serotonin.m2m2.Lifecycle.terminate(Lifecycle.java:561) ~[ma-priv-3.6.5.jar:?]
	at com.serotonin.m2m2.Main.main(Main.java:147) ~[ma-priv-3.6.5.jar:?]
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	... 8 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
	at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199]
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
	... 8 more
INFO  2020-06-10T18:38:07,826 (com.serotonin.m2m2.Lifecycle.terminate:578) - Mango Lifecycle terminated.

Running the web console manually gives the same out of bounds error when the connection is tested.

General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] HY000/50000

Update

Managed to get it fixed (after a database restore) by:

SET REFERENTIAL_INTEGRITY FALSE;
TRUNCATE TABLE EVENTS;
SET REFERENTIAL_INTEGRITY TRUE;
TRUNCATE TABLE USEREVENTS;
Restart Mango

Database went from 6Gb to 17Mb.