P.S. I also purged the /work directory, just to eliminate variables

Either get more memory or try this env property:

#Set the number of files the database can have open at one time db.nosql.maxOpenFiles=500

It will close/unmap files at the expense of cpu use. If this problem persists then try another stream type:

#Query Performance Tuning, File Access Type:Available[INPUT_STREAM,FILE_CHANNEL,RANDOM_ACCESS_FILE,MAPPED_BYTE_BUFFER] db.nosql.shardStreamType=MAPPED_BYTE_BUFFER

Has anyone been able to duplicate this? Any other thoughts or theories? Suggestions, what should I try?

After some more experience, going through this several more times, I seem to have found a pattern.

Apparently, this kind of problem arises when I reboot the MangoGT without explicitly stopping Mango first.

Having examined the output from systemctl show mango.service -p TimeoutStopUSec which shows TimeoutStopUSec=1min 30s it appears the service file is correctly configured; however, my experience suggests there is a failure somewhere in there. Additionally, a reboot command seems to execute immediately, too soon to properly shut down.

What do you think?

I purged the work directory, and the 302 error went away. Turns out the fact that it was the last message didn't connect it to the cause of the crash.

I have sent the database and log files separately. Hopefully we can learn what's going on.

Aldo

That's about what I expected. Glad it was simple and isolated.

@AldoRamos it looks like the database was corrupted. I would restore from a backup if possible. If not there are h2 tools to attempt recovery. Might want to just check you disk isn't full.

Here is one such post by Phillip,

https://forum.infiniteautomation.com/topic/4073/corrupt-h2-database

@AldoRamos Services prefixed with maUi were originally not intended for public use, this is why it is not documented.

The service code is actually located here - https://github.com/infiniteautomation/ma-dashboards/blob/main/UI/web-src/ui/services/dateBar.js

This should do the trick:

maUiDateBar.autoUpdate = true; maUiDateBar.updateIntervals = 10; maUiDateBar.updateIntervalPeriod = 'MINUTES';

Yes, that's what I'd figured I would do if needed, but this is good enough for now to solve my issue, move forward and be productive.

This instance is running Mango 3.5.6; will be upgraded soon.

@phildunlap Excellent! That's really useful information.

I could see how on my.pointValueBefore(my.time) could run into issues in a history generation without use of the cache control, so my.pointValueBefore(my.time, true)

If the logic you want is return my.value + (power.past(MINUTE,1).average/60); I think we should approach from why this would not be working. Have you tried generating history for return (power.past(MINUTE,1).average/60); to ensure you're getting the positive values? That simple form is what I think would be the right way to write it.

That will likely be it, thanks Phil!

I didn't find this earlier when searching, but this seems to be similar to unable to discover a bacnet device on my network, specifically in the brief window of apparently successful operation when saving or enabling the publisher.

Bingo! Good call!

Yeah, sorry, that was a bug. Got the fix in, though.

The good news: I finally migrated the data. I think.

The bad news: I still don't know why java crashes

In frustration, I finally ran the migration with 1 low-priority thread, and in about 15 minutes, it finally completed — sort of.

I tried updating java to 1.8.0_152; no effect
Added 4GB swap (in addition to the 1GB already configured): no effect
I executed the migration on a VirtualBox VM: no problem, success in about 3 minutes

Finally, migrated on the server in question with 1 low-priority thread, It executed for 923 seconds — then stopped. I feared it had crashed again, but I noticed mango was still alive, so I checked a few things.

That's when it got interesting.

The migration page was seemingly stalled at 19648161 of 38879054
0_1513389336077_migrate-success.png
But in the log file, it claimed to have successfully completed:

INFO 2017-12-15T16:03:36,024 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... INFO 2017-12-15T16:18:59,072 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 19648161 point values. INFO 2017-12-15T16:18:59,072 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 19648161 point values migrated to Time Series Datastore took 923.048s

Eventually my eye caught that the 19648161 was (approximately) the same number I had seen in my successful simulation on the VirtualBox instance:
0_1513391609589_migrate-aldarch.png
But a quick query on the original db shows:

MariaDB [mango]> SELECT COUNT(*) FROM pointValues; +----------+ | COUNT(*) | +----------+ | 38879054 | +----------+

Curiously, the “successful” migrations show about half the total. I also notice (especially on the actual server) that the total changes a few times during the migration.

So please help me understand: did the data migrate? Or only half? What am I not understanding?

The crashing seems obvious to be a java problem, but I can't glean any diagnostic information to pursue further. Note that this instance is running on an Arm architecture, so that might be a factor as well.

I meant to say it will use the NoSQL if installed. That's what I meant by prefer. I don't believe we created any tools to covert existing reportInstanceData tables to the NoSQL data store. So, I would expect any old report instances you have don't have their data available.

Understood, but there was no such file. There is a file named com.serotonin.m2m2.db.upgrade.Upgrade17.log but it's empty

-rw-r--r-- 1 mango mango 0 Nov 22 14:44 com.infiniteautomation.mango.scheduling.db.upgrade.Upgrade4.log -rw-r--r-- 1 mango mango 0 Nov 22 12:58 com.serotonin.m2m2.db.upgrade.Upgrade13.log -rw-r--r-- 1 mango mango 35 Nov 22 12:58 com.serotonin.m2m2.db.upgrade.Upgrade14.log -rw-r--r-- 1 mango mango 0 Nov 22 12:58 com.serotonin.m2m2.db.upgrade.Upgrade15.log -rw-r--r-- 1 mango mango 0 Nov 22 12:58 com.serotonin.m2m2.db.upgrade.Upgrade16.log -rw-r--r-- 1 mango mango 0 Nov 22 12:58 com.serotonin.m2m2.db.upgrade.Upgrade17.log

The one file which isn't completely empty only reads:

Updated 0 user comments with XIDs.