P.S. I also purged the /work directory, just to eliminate variables

MangoGT unit that had previously run out of memory (see MangoGT running out of memory? restarted, but the UI failed to present a login page. After several attempts, I moved the databases directory and recovered from a recent backup. I have tried this several times now, going further back in the backup history, but with no success.

Then I noticed a very unexpected behavior: one browser window which was logged in to do the SQL restore DID let me log in, somehow, and I am able to operate normally in that browser, for now. However, other browsers just hang on a /ui/ URL; or if I access the device using a different URL (e.g. IP address) it hangs similarly.

I am sending the URL separately so you can try for yourselves, see the HTTP conversation.

Catching a few MangoGT units hanging, to find Out Of Memory errors in the logs:

Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]: Exception in thread "high-pool-2-thread-5894" com.infiniteautomation.tsdb.IasTsdbException: java.io.IOException: Map failed
Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.IasTsdbImpl.count(IasTsdbImpl.java:1124)
Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.nosql.InternalGenericDao.rangeCount(InternalGenericDao.java:393)
Dec 06 01:00:43 mangoGT5176 start-mango.sh[529]:         at 
[snipped]
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.checkRangeImpl(SyncHandler.java:514)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.checkRange(SyncHandler.java:345)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.checkPoint(SyncHandler.java:326)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.m2m2.persistent.pub.SyncHandler$PointSync.run(SyncHandler.java:276)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.timer.sync.Synchronizer$TaskWrapper.run(Synchronizer.java:150)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.timer.Task.runTask(Task.java:179)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at java.base/java.lang.Thread.run(Thread.java:830)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: Caused by: java.io.IOException: Map failed
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1017)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.ChecksumMappedByteBufferInputStream.<init>(ChecksumMappedByteBufferInputStream.java:41)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.ReversibleShard.openDataIn(ReversibleShard.java:250)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.Shard.query(Shard.java:473)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.CompressibleShard.query(CompressibleShard.java:391)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.Series.query(Series.java:532)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at com.infiniteautomation.tsdb.impl.IasTsdbImpl.count(IasTsdbImpl.java:1095)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         ... 48 more
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]: Caused by: java.lang.OutOfMemoryError: Map failed
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1014)
Dec 06 01:02:22 mangoGT5176 start-mango.sh[529]:         ... 54 more

Suggestions? Adjust thread pools? Swap?

Has anyone been able to duplicate this? Any other thoughts or theories? Suggestions, what should I try?

Here is the original datapoint on the MangoGT, reading from a meter via MODBUS (stripped down):

{
   "dataPoints":[
      {
         "dataSourceXid":"DS_MODBUS_RS485",
                "enabled":true,
         "xid":"DP_CW_flow_inlet",
         "textRenderer":{
            "useUnitAsSuffix":true,
            "format":"0.0",
            "type":"ANALOG"
         },
         "loggingType":"ON_CHANGE_INTERVAL",
         "overrideIntervalLoggingSamples":false,
         "defaultCacheSize":1,
         "renderedUnit":"gal\/min",
         "setPermission":"",
         "tags":{
            "varName":"flowIn",
            "systemID":"115",
            "system":"Creekwood",
            "chart":"true",
            "flow":"true"
         },
         "unit":"m³\/h",
         "pointLocator":{
            "charset":"ASCII",
            "writeType":"NOT_SETTABLE",
            "offset":0,
            "slaveId":2,
            "slaveMonitor":false,
            "multiplier":1,
            "multistateNumeric":false,
            "registerCount":0,
            "range":"HOLDING_REGISTER",
            "modbusDataType":"FOUR_BYTE_FLOAT_SWAPPED",
            "bit":0,
            "additive":0
         },
         "name":"Flow Rate (Inlet)",
      }
   ]
}

And here is the data point on the cloud instance (Intel), also stripped:

{
   "dataPoints":[
      {
         "intervalLoggingType":"INSTANT",
         "dataSourceXid":"DS_persistent_TCP",
         "enabled":true,
         "xid":"115DP_CW_flow_inlet",
         "textRenderer":{
            "useUnitAsSuffix":true,
            "format":"0.0",
            "type":"ANALOG"
         },
         "loggingType":"ALL",
         "renderedUnit":"gal\/min",
         "tags":{
            "varName":"flowIn",
            "systemID":"115",
            "system":"Creekwood",
            "EdgeServerID":"115",
            "chart":"true",
            "flow":"true"
         },
         "unit":"m³\/h",
         "pointLocator":{
            "settable":false,
            "dataTypeId":3
         },
         "name":"Flow Rate (Inlet)",
      }
   ]
}

After some more experience, going through this several more times, I seem to have found a pattern.

Apparently, this kind of problem arises when I reboot the MangoGT without explicitly stopping Mango first.

Having examined the output from systemctl show mango.service -p TimeoutStopUSec which shows TimeoutStopUSec=1min 30s it appears the service file is correctly configured; however, my experience suggests there is a failure somewhere in there. Additionally, a reboot command seems to execute immediately, too soon to properly shut down.

What do you think?

The obvious question is that the MangoGT has an ARM processor, while the cloud has an Intel processor. Is the Java math off on the ARM processor?

This should be easy to duplicate, just a data point with the units as I've described, with a manually-entered point value.

I don't quite follow your question/suggestion.

Let me reiterate that the data point is configured as m³/h with the display unit as gal/min

On the MangoGT, the display unit is off, both in comparison to the device as well as according to the math. However, it is published to the cloud, where the configuration is identical, the display unit is correct, matching the device and the math.

I have come across an odd situation with unit conversion. We have flow meters which report in m³/h, and Mango is configured to convert to gal/min for display. The meter itself can be configured to display in gal/min, even though the data cannot be reported in any other units. This was previously addressed in another post (https://forum.mango-os.com/topic/4388/are-values-in-converted-unit-available-to-scripts), and the workaround provided by @phildunlap has been working fine, until recently when we replaced the unit with a MangoGT. Then we noticed the display (and more critically, our scripts) was off.

The result is the local display is wrong, along with scripts which use the converted units. However, the data is published (persistent TCP) and the receiving unit (cloud instance) is displaying the gal/min correctly, in agreement with the display on the device.

For example, as I debug one of the scripts, it reads the metric flow at 2.2167255878448486 m³/h; the device and the Mango instance receiving the published data display an accurate 9.76 gal/min. However, Mango's rendered value is a mysterious 8.1 gal/min.

FWIW the gateway device is a MangoGT running version 3.7.7
The cloud instance is also running Mango version 3.7.7

Any other details that might be useful?

Aldo

@MattFox actually, that brings to mind a very important question: how do I restore from the H2 backup? Easy to do with MySQL/MariaDB, but how does one import data into H2?

Update:
Never mind; I found this: https://docs-v3.mango-os.com/how-to-restore-a-h2-database-backup

Hadn't used that before; didn't realize that was an option.

Restoring now.