MangoGT running out of memory?
-
Catching a few MangoGT units hanging, to find Out Of Memory errors in the logs:
Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]: Exception in thread "high-pool-2-thread-5894" com.infiniteautomation.tsdb.IasTsdbException: java.io.IOException: Map failed Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.IasTsdbImpl.count(IasTsdbImpl.java:1124) Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.nosql.InternalGenericDao.rangeCount(InternalGenericDao.java:393) Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]: at [snipped] Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.checkRangeImpl(SyncHandler.java:514) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.checkRange(SyncHandler.java:345) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.checkPoint(SyncHandler.java:326) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.run(SyncHandler.java:276) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.timer.sync.Synchronizer$TaskWrapper.run(Synchronizer.java:150) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.timer.Task.runTask(Task.java:179) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at java.base/java.lang.Thread.run(Thread.java:830) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: Caused by: java.io.IOException: Map failed Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1017) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.ChecksumMappedByteBufferInputStream.<init>(ChecksumMappedByteBufferInputStream.java:41) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.ReversibleShard.openDataIn(ReversibleShard.java:250) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.Shard.query(Shard.java:473) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.CompressibleShard.query(CompressibleShard.java:391) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.Series.query(Series.java:532) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at com.infiniteautomation.tsdb.impl.IasTsdbImpl.count(IasTsdbImpl.java:1095) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: ... 48 more Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: Caused by: java.lang.OutOfMemoryError: Map failed Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1014) Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: ... 54 more
Suggestions? Adjust thread pools? Swap?
-
Was this ever fixed?
-
Either get more memory or try this env property:
#Set the number of files the database can have open at one time db.nosql.maxOpenFiles=500
It will close/unmap files at the expense of cpu use. If this problem persists then try another stream type:
#Query Performance Tuning, File Access Type:Available[INPUT_STREAM,FILE_CHANNEL,RANDOM_ACCESS_FILE,MAPPED_BYTE_BUFFER] db.nosql.shardStreamType=MAPPED_BYTE_BUFFER