Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
Out of memory problem
-
@petermcs the best way to analyze OOM problems is by taking a heap dump and loading it into an analysis tool such as JVisualVM or MAT. However that file will generally be GB in size on a system like yours so before we get into that I would suggest that you take some samples of the Java process while it is running. From that we may be able to see what is eating up all the memory.
By using the
jmaptool packaged with the JDK you should be able to get the necessary info:https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html
Step 1. Find the PID of Mango while it is running. This can be by something like
ps aux | grep javaor looking into the ma.pid file in MA_HOME\binStep 2. Run
jmap -histo <mango pid>every so often until Mango crashes. Write the output to a file for reference later obviously.By looking at the classes that are using the most memory you should be able to get an idea of what the problem is. Please post a summary of your findings here and I can try to see what the problem is.
NOTE: likely
jmapwon't be on your path so you may need to execute it from within the JDK installation directory directly. -
@terrypacker I got jmap running (seems that even though I changed JAVA_HOME to the OpenJDK folder Mango still runs using the Oracle Java so I need to look at the again... Anyway when I run jmap before and after the memory blows up, the results are identical!
Also a bit of further info, If I start Mango and don't connect (locally via 127.0.0.1 as I am Teamviewering in to check things) the browser the memory does not blow up. It only happens a few minutes after I connect and log in. If I connect but don't login I.e. stay at the login screen, the memory usage does not increase.Here is the after jmap result
sudo /usr/java/jdk1.8.0_152/bin/jmap 18673 Attaching to process ID 18673, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.152-b16 0x0000000000400000 7K /usr/java/jdk1.8.0_152/bin/java 0x00007f9eb46f8000 30K /usr/lib64/libnss_dns-2.17.so 0x00007f9eb6118000 259K /usr/java/jdk1.8.0_152/jre/lib/amd64/libjpeg.so 0x00007f9ec7168000 11K /usr/lib64/libfreebl3.so 0x00007f9ec736b000 39K /usr/lib64/libcrypt-2.17.so 0x00007f9ec75a2000 40K /usr/lib64/libltdl.so.7.3.0 0x00007f9ec77ac000 93K /usr/lib64/libtdb.so.1.3.16 0x00007f9ec79c3000 27K /usr/lib64/libogg.so.0.8.0 0x00007f9ec7bca000 180K /usr/lib64/libvorbis.so.0.4.6 0x00007f9ec7df7000 35K /usr/lib64/libvorbisfile.so.3.3.5 0x00007f9ecc091000 68K /usr/lib64/libcanberra.so.0.2.5 0x00007f9ecc2a2000 19K /usr/lib64/libcanberra-gtk.so.0.1.9 0x00007f9ecc4a7000 23K /usr/lib64/gtk-2.0/modules/libcanberra-gtk-module.so 0x00007f9ecc6ad000 6K /usr/lib64/gtk-2.0/modules/libpk-gtk-module.so 0x00007f9ecc8af000 43K /usr/lib64/gtk-2.0/2.10.0/engines/libpixmap.so 0x00007f9eccaba000 15K /usr/lib64/gtk-2.0/2.10.0/engines/libadwaita.so 0x00007f9ecccbe000 6K /usr/lib64/libgthread-2.0.so.0.5600.1 0x00007f9eccec0000 188K /usr/lib64/libgraphite2.so.3.0.1 0x00007f9ecd0ee000 259K /usr/lib64/libblkid.so.1.1.0 0x00007f9ecd32e000 73K /usr/lib64/libGLX.so.0.0.0 0x00007f9ecd560000 625K /usr/lib64/libGLdispatch.so.0.0.0 0x00007f9ecd816000 66K /usr/lib64/libbz2.so.1.0.6 0x00007f9ecda26000 19K /usr/lib64/libuuid.so.1.3.0 0x00007f9ecdc2b000 169K /usr/lib64/libexpat.so.1.6.0 0x00007f9ecde55000 111K /usr/lib64/libfribidi.so.0.4.0 0x00007f9ece071000 50K /usr/lib64/libthai.so.0.1.6 0x00007f9ece27d000 637K /usr/lib64/libharfbuzz.so.0.10705.0 0x00007f9ece51a000 271K /usr/lib64/libmount.so.1.1.0 0x00007f9ece75d000 103K /usr/lib64/libresolv-2.17.so 0x00007f9ece976000 152K /usr/lib64/libselinux.so.1 0x00007f9eceb9d000 31K /usr/lib64/libffi.so.6.0.1 0x00007f9eceda5000 568K /usr/lib64/libGL.so.1.7.0 0x00007f9ecf031000 88K /usr/lib64/libz.so.1.2.7 0x00007f9ecf247000 55K /usr/lib64/libxcb-render.so.0.0.0 0x00007f9ecf455000 15K /usr/lib64/libxcb-shm.so.0.0.0 0x00007f9ecf659000 175K /usr/lib64/libpng15.so.15.13.0 0x00007f9ecf884000 82K /usr/lib64/libEGL.so.1.1.0 0x00007f9ecfa98000 682K /usr/lib64/libpixman-1.so.0.34.0 0x00007f9ecfd41000 776K /usr/lib64/libfreetype.so.6.14.0 0x00007f9ed8044000 392K /usr/lib64/libpcre.so.1.2.0 0x00007f9ed82a6000 10K /usr/lib64/libXdamage.so.1.1.0 0x00007f9ed84a9000 10K /usr/lib64/libXcomposite.so.1.0.0 0x00007f9ed86ac000 43K /usr/lib64/libXrandr.so.2.2.0 0x00007f9ed88b7000 11K /usr/lib64/libXinerama.so.1.0.0 0x00007f9ed8aba000 1129K /usr/lib64/libglib-2.0.so.0.5600.1 0x00007f9ed8dd0000 331K /usr/lib64/libgobject-2.0.so.0.5600.1 0x00007f9ed9021000 270K /usr/lib64/libfontconfig.so.1.11.1 0x00007f9ed9263000 281K /usr/lib64/libpango-1.0.so.0.4200.3 0x00007f9ed94a9000 88K /usr/lib64/libpangoft2-1.0.so.0.4200.3 0x00007f9ed96bf000 1682K /usr/lib64/libgio-2.0.so.0.5600.1 0x00007f9ed9a5e000 162K /usr/lib64/libgdk_pixbuf-2.0.so.0.3612.0 0x00007f9ed9c86000 1263K /usr/lib64/libcairo.so.2.11512.0 0x00007f9ed9fbd000 153K /usr/lib64/libatk-1.0.so.0.22810.1 0x00007f9eda1e3000 55K /usr/lib64/libpangocairo-1.0.so.0.4200.3 0x00007f9eda3f1000 15K /usr/lib64/libgmodule-2.0.so.0.5600.1 0x00007f9eda5f5000 789K /usr/lib64/libgdk-x11-2.0.so.0.2400.31 0x00007f9eda8b6000 4705K /usr/lib64/libgtk-x11-2.0.so.0.2400.31 0x00007f9edb039000 23K /usr/lib64/libXfixes.so.3.1.0 0x00007f9edb23f000 44K /usr/lib64/libXcursor.so.1.0.2 0x00007f9edb54b000 513K /usr/java/jdk1.8.0_152/jre/lib/amd64/libfontmanager.so 0x00007f9edb9c4000 15K /usr/lib64/libXau.so.6.0.0 0x00007f9edbbc8000 162K /usr/lib64/libxcb.so.1.1.0 0x00007f9edbdf0000 64K /usr/lib64/libXi.so.6.1.0 0x00007f9f18023000 23K /usr/lib64/libXtst.so.6.1.0 0x00007f9f18229000 43K /usr/lib64/libXrender.so.1.3.0 0x00007f9f18434000 1287K /usr/lib64/libX11.so.6.3.0 0x00007f9f18772000 74K /usr/lib64/libXext.so.6.4.0 0x00007f9f189a3000 432K /usr/java/jdk1.8.0_152/jre/lib/amd64/libawt_xawt.so 0x00007f9f18bfc000 754K /usr/java/jdk1.8.0_152/jre/lib/amd64/libawt.so 0x00007f9f507e9000 86K /usr/lib64/libgcc_s-4.8.5-20150702.so.1 0x00007f9f50a1e000 251K /usr/java/jdk1.8.0_152/jre/lib/amd64/libsunec.so 0x00007f9f51168000 91K /usr/java/jdk1.8.0_152/jre/lib/amd64/libnio.so 0x00007f9f51394000 112K /usr/java/jdk1.8.0_152/jre/lib/amd64/libnet.so 0x00007f9f517ab000 49K /usr/java/jdk1.8.0_152/jre/lib/amd64/libmanagement.so 0x00007f9f6f1f3000 125K /usr/java/jdk1.8.0_152/jre/lib/amd64/libzip.so 0x00007f9f6f40f000 60K /usr/lib64/libnss_files-2.17.so 0x00007f9f6f622000 221K /usr/java/jdk1.8.0_152/jre/lib/amd64/libjava.so 0x00007f9f6f84e000 64K /usr/java/jdk1.8.0_152/jre/lib/amd64/libverify.so 0x00007f9f6fa5d000 42K /usr/lib64/librt-2.17.so 0x00007f9f6fc65000 1110K /usr/lib64/libm-2.17.so 0x00007f9f6ff67000 16640K /usr/java/jdk1.8.0_152/jre/lib/amd64/server/libjvm.so 0x00007f9f70f63000 2105K /usr/lib64/libc-2.17.so 0x00007f9f71331000 18K /usr/lib64/libdl-2.17.so 0x00007f9f71535000 101K /usr/java/jdk1.8.0_152/lib/amd64/jli/libjli.so 0x00007f9f7174b000 138K /usr/lib64/libpthread-2.17.so 0x00007f9f71967000 159K /usr/lib64/ld-2.17.so -
@petermcs please run
jmap -histo pidto get the object count. -
Okay:
Here's some stuff related to this after going through this:
I just upgraded to Mango 3.6.5 running on our production server. We use persistent TCP to haul data back from outstations monitoring solar assets, and it works Very Well.
However, there's a couple of things that popped up after the upgrade. We've identified (2) separate issues:
Issue (1): Reporting tool causes Mango to run out of memory and crash itself. Mango apparently doesn't check these things internally, so if you run out of Ram, it just merrily gobbles it all up and blows up. We have a spreadsheet that's about 8 megs, that loads an additional meg of data from Mango, and then reports it out again. Running this spreadsheet uses 1500 megs of ram, according to the Mango's internal JVM free memory.
We fixed this by setting the -Xms5g and -Xmx5g via the "memory-medium.sh". However, if we have more than one of two of those reports ran at the same time, the system will still Blow Up. So, I'd like to see Mango either (1) do something to manage these reports, if it means slowing them down or something to not eat that much memory, or (2) at least crash elegantly: When this occurs with our system, the thing completely shuts down, and then does not re-start itself. I'd rather the reporting tool not crash the system, but if it does, it'd be nice to recover gracefully. As a solution to this, we're going to keep jacking up -Xms and -Xmx until we run out of memory on our servers, then we'll do something else.
Issue (2), And maybe this is a different thread:
Mango opens LOTS of files. At night, when the outstations are set to synchronize historic back to our cloud server, we see >100,000 files open, as reported by pmap. This normally is ~1000, but at night, when the historical syncs happen, that number jumps by a lot. This causes a Java fault, as the thing bumps against the available files.We increased the max file limit by doing
vm.max_map_count=131060
to /etc/sysctl.confand restarting.
We're still regularly getting >100k open files, but at least now Java doesn't crash out. We have the tsdb running out our instance, and something must have changed between 3.5 and 3.6 to how this is done, as we haven't changed any settings that would cause additional data to be written via the persistent publisher.
Just letting people know here about the work we did on this. Is there some way in Mango to limit the number of open files it does on a historical sync? My concern here is that as I add more sites (and keep historical sync active, to force the system to catch up on any weird outages that happen during the day as is common with cell-connected systems), this file open issue is going to increasingly get bigger and bigger. Did I miss something here about the best way to do this?
Cheers,
-Greg Linder -
Also:
For those of you running into memory problems in Mango:
Since the JVM doesn't ever "give memory back" to the operating system, whatever you set those -Xms and -Xmx to will be about what the JVM will take. once it reaches that Max of that, you will NEVER see that memory returned to the system unless you restart the JVM.
Our system, when running reports, cranks right up to the limit, and then sits there. If you see CPU utilization pegged high, but the system in unresponsive, it's possible that the JVM has maxed out it's available memory (partially defined by -Xms and -Xmx settings), and the JVM is cranking around it's jobs trying to get your work done without anything getting actually complete.
-
Hi Greg
I think it will be helpful to know how large your system is. How many points and which database are you using. H2 or MySQL?
@turbo said in Out of memory problem:
Just letting people know here about the work we did on this. Is there some way in Mango to limit the number of open files it does on a historical sync? My concern here is that as I add more sites (and keep historical sync active, to force the system to catch up on any weird outages that happen during the day as is common with cell-connected systems), this file open issue is going to increasingly get bigger and bigger. Did I miss something here about the best way to do this?
I would consider running the sync more often, I have been using a cron pattern of every 5 minutes
and have not run into any issue with it. However, I have no idea how much data you are logging.
Also, have you enabled "save real-time data" on the data source? This could be why the sync needs to open so many files.The current recommended way to run mango as a service does lack the ability to restart after crashing. Are you running Ubuntu or debian? Some work has been done on a new service script which you could use.
-
Here is some general information on what you see, and I have been tracking these known problems for some time trying to find a solution.
- Excel Reports Module can use up all the JVM memory.
This is mostly out of our control and a known problem with the Apache POI library that is the backbone of that module. They have recently released a more performant version that will eventually be part of our Excel Reports module. We have reworked our REST api using a new algorithm to manage memory that will in the not-to-distant future be ported over to the ExcelReports module and should solve all of the problems outside of the POI library limitations.
- Open Files - MangoNoSQL
The time series database will open many files and leave them open for some time, they eventually do get closed but for performance reasons we don't close them immediately after use. As @CraigWeb mentioned better tuning can help during intense syncs. As for the problem being worse in 3.6, I was not aware of that and can confirm that the logic for opening and closing the files has not changes. It is likely that you system is just getting larger and you are now aware that this happens. There are a whole slew of settings for very large systems in the env.properties file with the prefix:db.nosql.*. I don't have time to go into them all here but I would suggest experimenting with them and posting questions about specific properties later.
As for the JVM general memory use:
We are starting to use Adopt OpenJDK 13 and are finding that it is much better at memory management. As for the
-Xmsand-Xmxsettings, depending what you are trying to achieve they may not be ideal. By setting both to the same value you get a fully allocated JVM at startup which on a machine only running Mango is likely ideal. But for a system with other processes that need memory at other times the JVM cannot release memory with that configuration. We have found that on a very large machine using Adopt OpenJDK it is quite good expanding and contracting the JVM memory use for Mango's needs. But we are still evaluating the best configuration for it at this point. -
Greetings, all:
Thanks for the answers here. I appreciate the clarity on this.
As for "Sync every five minutes" vs "save real-time data"-- I'm a bit unclear on how these options interact with each other.
I'd like to get all the data that my systems in the field are logging. The points out there are set with different poll and logging settings. So if I set to "sync every 5 minutes" (or something similar), does that mean that my data points will update every 5 minutes with the logged data? Vs the "save real time data" that saves all the data that comes in from every poll?
Is that the proper way of using those options?
Cheers,
-Greg -
@terrypacker Hi Terry, I am just continuing @petermcs's work and have the jmap -hist outputs.
Before:
num #instances #bytes class name ---------------------------------------------- 1: 173771 715822488 [I 2: 3748110 508063720 [C 3: 1945314 443221056 [B 4: 1733728 41609472 java.lang.String 5: 1973991 37379904 [Ljava.lang.Class; 6: 478710 23272440 [Ljava.lang.Object; 7: 1 16777232 [Lorg.h2.util.CacheObject; 8: 151493 11321608 [Ljava.util.HashMap$Node; 9: 228571 10971408 java.util.HashMap 10: 168383 10776512 java.net.URL 11: 318951 10206432 java.util.HashMap$Node 12: 99723 9573408 java.util.jar.JarFile$JarFileEntry 13: 101097 8896536 java.lang.reflect.Method 14: 99443 7955440 java.util.zip.ZipEntry 15: 194720 7788800 java.util.LinkedHashMap$Entry 16: 262144 6291456 org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper 17: 107562 6023472 java.util.LinkedHashMap 18: 211916 5085984 java.lang.StringBuilder 19: 155322 4970304 java.util.LinkedHashMap$LinkedEntryIterator 20: 132436 4237952 java.util.ArrayList$Itr 21: 106631 4096384 [Ljava.lang.String; 22: 48305 3915312 [S 23: 103984 3635760 [Ljava.lang.reflect.Method; 24: 148571 3565704 java.util.ArrayList 25: 145630 3495120 org.springframework.core.annotation.AnnotationUtils$AnnotationCacheKey 26: 213448 3415168 java.util.HashSet 27: 57295 3208520 org.springframework.asm.Label 28: 5081 3195656 [J 29: 37779 3022320 java.lang.reflect.Constructor 30: 89701 2870432 java.io.File 31: 59175 2840400 java.nio.HeapByteBuffer 32: 168665 2698640 java.lang.Integer 33: 16617 2641896 [Lorg.springframework.asm.Label; 34: 73858 2363456 java.util.concurrent.ConcurrentHashMap$Node 35: 89469 2147256 java.lang.Class$MethodArray 36: 44728 2146944 org.springframework.core.ResolvableType 37: 81522 1956528 java.util.LinkedList$Node 38: 32363 1812328 org.springframework.asm.SymbolTable$Entry 39: 15886 1783136 java.lang.Class 40: 52571 1682272 java.util.LinkedHashMap$LinkedKeyIterator 41: 22174 1596528 org.springframework.core.annotation.AnnotationAttributes 42: 32146 1543008 org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference 43: 57998 1391952 java.util.concurrent.CopyOnWriteArrayList$COWIterator 44: 42207 1350624 org.springframework.asm.Type 45: 29777 1191080 java.lang.ref.Finalizer 46: 36791 1177312 java.util.AbstractList$Itr 47: 65801 1052816 sun.reflect.ConstantPool 48: 9550 993200 sun.net.www.protocol.file.FileURLConnection 49: 19600 940800 org.springframework.asm.Frame 50: 22090 883600 com.serotonin.bacnet4j.util.sero.ByteQueue 51: 37028 864280 [Ljava.lang.reflect.Type; 52: 35583 853992 org.springframework.asm.Edge 53: 1968 826856 [Ljava.util.concurrent.ConcurrentHashMap$Node; 54: 25594 819008 java.util.concurrent.locks.AbstractQueuedSynchronizer$Node 55: 50928 814848 java.lang.Object 56: 25147 804704 java.util.LinkedHashMap$LinkedValueIterator 57: 24671 789472 java.util.LinkedList 58: 49091 785456 org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder 59: 13853 775768 org.springframework.core.type.classreading.MethodMetadataReadingVisitor 60: 13740 769440 jdk.internal.org.objectweb.asm.Item 61: 31486 755664 java.util.Arrays$ArrayList 62: 46717 747472 java.util.LinkedHashSet 63: 17178 687120 java.lang.invoke.MethodType 64: 9400 676800 java.lang.reflect.Field 65: 27985 671640 sun.reflect.generics.tree.SimpleClassTypeSignature 66: 26545 637080 sun.reflect.generics.visitor.Reifier 67: 2806 628544 org.springframework.asm.MethodWriter 68: 11198 627088 java.lang.invoke.LambdaFormEditor$Transform 69: 24021 576504 sun.reflect.generics.factory.CoreReflectionFactory 70: 23902 573648 sun.reflect.generics.parser.SignatureParser 71: 10093 565208 java.util.zip.ZipFile$ZipFileInputStream 72: 10089 564984 java.util.zip.ZipFile$ZipFileInflaterInputStream 73: 23307 559368 sun.reflect.generics.scope.ClassScope 74: 17324 554368 java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry 75: 13514 540560 java.util.HashMap$KeyIterator 76: 16637 532384 java.lang.ref.WeakReference 77: 13288 531520 java.util.HashMap$EntryIterator 78: 6026 530288 sun.net.www.protocol.jar.JarURLConnection 79: 16281 520992 java.util.RegularEnumSet 80: 31992 511872 java.util.LinkedHashMap$LinkedKeySet 81: 18654 507960 [Ljava.lang.annotation.Annotation; 82: 15760 504320 org.springframework.util.ConcurrentReferenceHashMap$1 83: 30116 498104 [Lsun.reflect.generics.tree.TypeArgument; 84: 7673 491072 org.springframework.core.MethodParameter 85: 5066 486336 sun.net.www.protocol.jar.URLJarFile$URLJarFileEntry 86: 14914 477248 java.util.LinkedList$ListItr 87: 5921 476640 [Ljava.util.WeakHashMap$Entry; 88: 7171 458944 com.serotonin.timer.OrderedTaskInfo 89: 394 452912 [Lorg.springframework.asm.SymbolTable$Entry; 90: 18617 446808 org.springframework.core.annotation.AnnotatedElementUtils$MergedAnnotationAttributesProcessor 91: 27787 444592 sun.reflect.generics.tree.ClassTypeSignature 92: 11069 442760 sun.nio.cs.UTF_8$Decoder 93: 6897 441408 java.util.regex.Matcher 94: 13780 440960 org.springframework.core.annotation.AnnotationUtils$$Lambda$48/1194398327 95: 5356 428480 sun.net.www.protocol.jar.URLJarFile 96: 7588 424928 com.serotonin.bacnet4j.npdu.NPCI 97: 19494 420832 [Lorg.springframework.asm.Type; 98: 12977 415264 java.lang.reflect.WeakCache$CacheKey 99: 17180 412320 org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey 100: 12269 392608 java.util.Hashtable$Entry Total 16594285 2034054440
After opening a browser window and logging in, greeted by the admin page and an immediate "connectivity lost, restoring connection" toast message that never permenantly resolves.
It then takes about 3-5 minutes to max outnum #instances #bytes class name ---------------------------------------------- 1: 51017979 3130737016 [C 2: 51017618 1224422832 java.lang.String 3: 10175891 895478408 com.serotonin.m2m2.rt.event.EventInstance 4: 20380459 550396632 [Ljava.lang.Object; 5: 20352135 488451240 com.serotonin.m2m2.i18n.TranslatableMessage 6: 10175862 325627584 com.serotonin.m2m2.rt.event.type.DataSourceEventType 7: 469887 198080400 [B 8: 1599229 87706096 [Lorg.h2.value.Value; 9: 1248203 39942496 org.h2.result.RowImpl 10: 425490 30635280 org.h2.value.ValueLobDb 11: 828920 26525440 org.h2.result.SimpleRowValue 12: 1 16777232 [Lorg.h2.util.CacheObject; 13: 45454 15596376 [I 14: 31155 12509192 [J 15: 438464 7015424 org.h2.value.ValueInt 16: 265342 6368208 org.h2.value.ValueLong 17: 262144 6291456 org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper 18: 30244 5475296 [Lorg.h2.result.Row; 19: 85336 3413440 java.util.LinkedHashMap$Entry 20: 2874 3363200 [Lorg.h2.result.SearchRow; 21: 30243 3145272 org.h2.index.PageDataLeaf 22: 28393 2498584 java.lang.reflect.Method 23: 28704 2297592 [Ljava.util.HashMap$Node; 24: 66713 2134816 java.util.HashMap$Node 25: 60656 1940992 java.util.concurrent.ConcurrentHashMap$Node 26: 16839 1888904 java.lang.Class 27: 23457 1313592 java.util.LinkedHashMap 28: 33277 1064864 org.h2.store.Data 29: 608 741064 [Ljava.util.concurrent.ConcurrentHashMap$Node; 30: 8489 611208 org.springframework.core.annotation.AnnotationAttributes 31: 26206 593480 [Ljava.lang.Class; 32: 7131 513432 java.lang.reflect.Field 33: 31170 498720 java.lang.Object 34: 10792 476208 [Ljava.lang.String; 35: 18951 454824 java.util.ArrayList 36: 9363 449424 java.util.HashMap 37: 15410 369840 org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource$DefaultCacheKey 38: 4273 341840 java.lang.reflect.Constructor 39: 9479 303328 java.util.LinkedList 40: 9008 288256 java.util.Hashtable$Entry 41: 11579 277896 java.util.LinkedList$Node 42: 17155 274480 org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder 43: 11080 265920 org.h2.value.ValueRow 44: 2851 228080 org.h2.index.PageBtreeLeaf 45: 4539 181560 java.lang.ref.SoftReference 46: 2998 167888 java.lang.Class$ReflectionData 47: 10089 161424 java.util.LinkedHashMap$LinkedEntrySet 48: 9949 159184 java.util.LinkedHashSet 49: 3796 151840 javax.servlet.jsp.tagext.TagAttributeInfo 50: 2614 146384 java.lang.invoke.MemberName 51: 6080 145920 java.util.jar.Attributes$Name 52: 5878 141072 sun.reflect.generics.tree.SimpleClassTypeSignature 53: 3514 140560 java.util.TreeMap$Entry 54: 2734 140264 [Ljava.lang.reflect.Method; 55: 4 131136 [Ljava.util.concurrent.ForkJoinTask; 56: 1807 130104 org.springframework.web.method.HandlerMethod$HandlerMethodParameter 57: 3164 126560 jdk.nashorn.internal.runtime.CompiledFunction 58: 2488 119424 org.springframework.core.ResolvableType 59: 1796 114944 java.net.URL 60: 5878 109584 [Lsun.reflect.generics.tree.TypeArgument; 61: 1890 105840 jdk.nashorn.internal.runtime.ScriptFunction 62: 488 105720 [Ljava.util.Hashtable$Entry; 63: 4046 97104 sun.reflect.annotation.AnnotationInvocationHandler 64: 1483 94912 java.util.concurrent.ConcurrentHashMap 65: 3898 93696 [Lorg.springframework.core.annotation.AnnotationAttributes; 66: 1056 92928 org.h2.table.Column 67: 5702 91232 sun.reflect.generics.tree.ClassTypeSignature 68: 543 91224 org.springframework.beans.factory.support.RootBeanDefinition 69: 5543 88688 java.lang.Integer 70: 1485 83160 java.security.Provider$Service 71: 1904 76160 java.lang.invoke.BoundMethodHandle$Species_LL 72: 4608 73728 java.util.LinkedHashMap$LinkedKeySet 73: 973 70056 org.springframework.core.type.classreading.AnnotationMetadataReadingVisitor 74: 2944 70008 [Ljava.lang.reflect.Type; 75: 2890 69360 java.security.Provider$ServiceKey 76: 650 67600 org.h2.mvstore.Chunk 77: 4151 66416 java.util.jar.Attributes 78: 2063 66016 java.lang.ref.WeakReference 79: 2036 65152 com.serotonin.bacnet4j.obj.PropertyTypeDefinition 80: 2002 64064 jdk.nashorn.internal.runtime.FinalScriptFunctionData 81: 1099 61544 java.lang.Package 82: 162 61088 [Ljdk.nashorn.internal.runtime.PropertyHashMap$Element; 83: 1507 60280 java.lang.invoke.MethodType 84: 1497 59880 java.lang.ref.Finalizer 85: 710 58992 [Ljava.util.WeakHashMap$Entry; 86: 41 58216 [Ljava.nio.ByteBuffer; 87: 2464 56256 [Ljava.lang.reflect.Constructor; 88: 308 54208 org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$ConfigurationClassBeanDefinition 89: 1123 53904 org.springframework.util.ConcurrentReferenceHashMap$SoftEntryReference 90: 1648 52736 java.lang.invoke.DirectMethodHandle 91: 1638 52416 jdk.nashorn.internal.runtime.PropertyHashMap$Element 92: 1624 51968 java.lang.invoke.LambdaForm$Name 93: 205 50840 com.serotonin.m2m2.vo.DataPointVO 94: 880 49280 org.springframework.core.type.classreading.MethodMetadataReadingVisitor 95: 2036 48864 com.serotonin.bacnet4j.obj.ObjectPropertyTypeDefinition 96: 1519 48608 java.util.concurrent.locks.ReentrantLock$NonfairSync 97: 432 48384 org.springframework.context.annotation.ScannedGenericBeanDefinition 98: 1511 48352 java.lang.ref.ReferenceQueue 99: 1508 48256 java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry 100: 1147 45880 java.math.BigInteger Total 169664483 7108600472I've limited it to the first 100 and totals, just to save post space. The full outputs ~7k can be sent on if needed.
htop when things max out:
-
@cbyrne the second
jmap -histoutput looks like the majority of the memory in use is from events. I'm just guessing here but it looks like you have a large number of events in the database that are being loaded and processed on login. Mango has known issues when the events table becomes large (which we are working to fix in Mango 4.0).To confirm this I would restart Mango with the h2 web console enabled so you can look at the database without logging into Mango and causing it to eat up all the memory. This can be done via the env properties:
db.web.start=true db.web.port=8091See also here: https://help.infiniteautomation.com/about-the-h2-database
Then login to the console make sure you set the JDBC url to point to the mango database on your file system, on a MangoES this would be
jdbc:h2:/opt/mango/databases/mah2basically the path to the db without the .h2.db on the end.Then run this query:
SELECT COUNT(*) FROM events;If the number is very large and you are ok with removing all events from the system I suggest you delete all event data. Before doing this I would make a copy of the h2 database file as a backup. You can simply copy the file if Mango is down.
Then this post describes how to remove events from the database:
https://forum.infiniteautomation.com/topic/2266/full-storage-issue-on-mangoes
Basically run:SET FOREIGN_KEY_CHECKS=0; CREATE TABLE eventsNewTable LIKE events; DROP TABLE events; RENAME TABLE eventsNewTable TO events; CREATE TABLE userEventsNewTable LIKE userEvents; DROP TABLE userEvents; RENAME TABLE userEventsNewTable TO userEvents; SET FOREIGN_KEY_CHECKS=1;If the events were the problem I suggest you take a look at this to better manage your events:
https://help.infiniteautomation.com/managing-disk-spaceAnother possibility that we can look at if my first assumption is not true is that something is generating many events on login. But since a big chunk of memory is being held by
DataSourceEventTypes I expect this is from them being stored in the database. -
Perfect, thanks @terrypacker! I'll try that and update this post.
update 1:
The H2 database shell doesn't recognise LIKE in CREATE TABLE. It may not be supported, The H2 command info for CREATE DATABASE doesn't show LIKE - https://www.h2database.com/html/commands.html#create_tableupdate 2
I just did aDELETE FROM EVENTS;instead.So yeah, there might have been a few too many events... 12 million. The database has gone from ~6 Gb to 76 Mb.
-
@terrypacker Sorry for dredging up an old thread but I ran the same
DELETE FROM EVENTS;as before but this time mango has failed to restart. It gets an out of bounds error when connecting to H2.INFO 2020-06-10T18:38:06,912 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:61) - Initializing H2 connection manager INFO 2020-06-10T18:38:06,953 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:83) - Initializing H2 web server FATAL 2020-06-10T18:38:07,527 (com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize:163) - Unable to connect to database of type H2 org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:611) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) ~[mango-3.6.5.jar:?] at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:405) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.AbstractDatabaseProxy.newDatabaseCheck(AbstractDatabaseProxy.java:181) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize(AbstractDatabaseProxy.java:82) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:760) ~[ma-priv-3.6.5.jar:?] at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:312) ~[ma-priv-3.6.5.jar:?] at com.serotonin.m2m2.Main.main(Main.java:143) ~[ma-priv-3.6.5.jar:?] Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199] at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199] at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] ... 12 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199] at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199] at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199] at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199] at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199] at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] ... 12 more ERROR 2020-06-10T18:38:07,527 (com.serotonin.m2m2.Main.main:146) - Error during initialization org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:611) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) ~[mango-3.6.5.jar:?] at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:405) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.AbstractDatabaseProxy.newDatabaseCheck(AbstractDatabaseProxy.java:181) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.AbstractDatabaseProxy.initialize(AbstractDatabaseProxy.java:82) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:760) ~[ma-priv-3.6.5.jar:?] at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:312) ~[ma-priv-3.6.5.jar:?] at com.serotonin.m2m2.Main.main(Main.java:143) ~[ma-priv-3.6.5.jar:?] Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199] at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199] at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] ... 12 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199] at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199] at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199] at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199] at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199] at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] ... 12 more INFO 2020-06-10T18:38:07,527 (com.serotonin.m2m2.Lifecycle.terminate:427) - Mango Lifecycle terminating... INFO 2020-06-10T18:38:07,559 (com.serotonin.m2m2.db.H2Proxy.terminateImpl:394) - Terminating database. ERROR 2020-06-10T18:38:07,823 (com.serotonin.m2m2.Lifecycle.terminate:563) - Failed to terminate database. org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:370) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:417) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:280) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.H2Proxy.terminateImpl(H2Proxy.java:395) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.db.AbstractDatabaseProxy.terminate(AbstractDatabaseProxy.java:211) ~[mango-3.6.5.jar:?] at com.serotonin.m2m2.Lifecycle.databaseTerminate(Lifecycle.java:765) ~[ma-priv-3.6.5.jar:?] at com.serotonin.m2m2.Lifecycle.terminate(Lifecycle.java:561) ~[ma-priv-3.6.5.jar:?] at com.serotonin.m2m2.Main.main(Main.java:147) ~[ma-priv-3.6.5.jar:?] Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.get(DbException.java:194) ~[h2-1.4.199.jar:1.4.199] at org.h2.message.DbException.convert(DbException.java:347) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.openDatabase(Database.java:345) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199] at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199] at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] ... 8 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at org.h2.index.PageDataLeaf.getRowAt(PageDataLeaf.java:329) ~[h2-1.4.199.jar:1.4.199] at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:67) ~[h2-1.4.199.jar:1.4.199] at org.h2.index.PageDataCursor.next(PageDataCursor.java:40) ~[h2-1.4.199.jar:1.4.199] at org.h2.table.PageStoreTable.addIndex(PageStoreTable.java:222) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.open(Database.java:827) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Database.<init>(Database.java:313) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2-1.4.199.jar:1.4.199] at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2-1.4.199.jar:1.4.199] at org.h2.Driver.connect(Driver.java:69) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234) ~[h2-1.4.199.jar:1.4.199] at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) ~[h2-1.4.199.jar:1.4.199] at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] ... 8 more INFO 2020-06-10T18:38:07,826 (com.serotonin.m2m2.Lifecycle.terminate:578) - Mango Lifecycle terminated.Running the web console manually gives the same out of bounds error when the connection is tested.
General error: "java.lang.ArrayIndexOutOfBoundsException: 0" [50000-199] HY000/50000
Update
Managed to get it fixed (after a database restore) by:
SET REFERENTIAL_INTEGRITY FALSE;TRUNCATE TABLE EVENTS;SET REFERENTIAL_INTEGRITY TRUE;TRUNCATE TABLE USEREVENTS;- Restart Mango
Database went from 6Gb to 17Mb.
