ma.log filling up with NoSQL Batch Writer Fatal Error
I could use some help/guidance on an issue that is causing our ma.log to fill up the 5M max about every 10 minutes.
I'm seeing the following error:
FATAL 2022-01-24T15:59:31,354 [high-pool-2-thread-247181 --> NoSQL Batch Writer 0] - Should never happen, data loss for unknown reason java.lang.RuntimeException: java.io.FileNotFoundException: C:\Databases\mangoTSDB\71\20306\764.data.rev (Access is denied)
I've looked into the file in question and its permissions do seem messed up. Even from an admin command prompt I cannot copy or move the file.
Any suggestions on what could be done with the file to help get this error resolved would be greatly appreciated.
After clicking Continue:
terrypacker last edited by
@cmusselm that is one of the shard files that Mango stores its time series data in. It would have been created by Mango. I don't use Windows but a similar situation in Linux will occur if you started Mango as root, then stopped it and restarted it as a user with less permissions.
Thanks for the info, Terry. We stop and start Mango with the same Windows service each time, so it should have the same permissions each time.
I've done some digging to see if there's other things on the Windows side we can do to try and fix this. We'll give them a try and post something here if we have any luck.
cmusselm last edited by cmusselm
Our instance of Mango has been up for 41 days (last restarted on 12/15/21), and if Windows correct, the problem file was created on 12/28/21, which was after Mango was started.
It seems the issue has appeared while Mango was running and isn't related to a reboot.
After trying various things with the file; elevating my privileges in Windows to System and still failing to do anything with it, we stopped Mango and tried to manipulate the file with no luck.
The good news is that when we restarted Mango, it looks like it removed the trouble file for us and all is good now.
I wanted to provide another round of updates as we've run into this issue two more times.
The most recent time we had the issue an app restart did stop the error from being written to the ma.log, but the corrupt file was still hanging around. After several days it did go away, so it seems whatever process Mango uses to clean up the files eventually worked.
While working with the support team we were pointed to look for other programs that could possibly lock one of the data files while NoSQL is trying to write to it. If that happens, this issue could occur.
I checked with our infrastructure team and they had an antivirus program that was performing checks on the NoSQL database directory. The antivirus has been disabled for that folder now.
I also found this post about Windows Indexing locking files: https://techcommunity.microsoft.com/t5/windows-blog-archive/the-case-of-the-mysterious-locked-file/ba-p/723349 Because of this I made sure to uncheck the "Allow files in this folder to have contents indexed..." option as that could have also contributed to the file being locked when NoSQL needed to write to it.
So far things are good since making those updates. Hopefully this helps anyone that may run into this.