Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
(SOLVED) Recent System Updates Cannot Backup Stale File Handle
-
Hello all, I just recently did some system updates, but not Mango (run 2x MangoES (Debian 10) and 1x Mango Enterprise Server Ubuntu 22.04) and some backups are now failing and I'm not sure why.
The files are created with 0 size.
Backups happen to a networked Samba share (TrueNAS system 12 to 13). The location mounts via autofs and I set to mount as root with permission 777 just to test and make sure it's not a permission error. I can write to it using any other user (previously it was mounted as mango:mango, but that doesn't work either)
I am seeing the following in the logs:
ERROR 2022-09-09T14:48:46,409 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.backupMysqlWithOutDatabase:453) - java.io.IOException: Cannot run program "/usr/bin/mysqldump": /mnt/backups> java.io.FileNotFoundException: /mnt/backups/sqlbackups/core-database-MYSQL-Sep-09-2022_144846.sql (Stale file handle) at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_162] at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_162] at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[?:1.8.0_162] at java.lang.ProcessImpl.start(ProcessImpl.java:115) ~[?:1.8.0_162] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_162] at com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.backupMysqlWithOutDatabase(DatabaseBackupWorkItem.java:414) ~[mango-3.7.12.jar:?] at com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.execute(DatabaseBackupWorkItem.java:161) ~[mango-3.7.12.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:629) ~[mango-3.7.12.jar:?] at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.7.12.jar:?] at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.7.12.jar:?] at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:314) ~[mango-3.7.12.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] WARN 2022-09-09T14:48:46,500 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.execute:176) - Unable to create backup file: /mnt/backups/sqlbackups/core-database-MYSQL-Sep-09-2022_144846 INFO 2022-09-09T14:50:04,689 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem. WARN 2022-09-09T14:50:09,919 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:174) - java.io.FileNotFoundException: /mnt/backups/configurationbackups/Mango-Configuration-Sep-09-2022_1450> INFO 2022-09-09T14:50:10,005 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:181) - Finished backup WorkItem. INFO 2022-09-09T14:50:44,830 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem. INFO 2022-09-09T14:50:49,796 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:181) - Finished backup WorkItem. INFO 2022-09-09T14:52:32,283 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem. WARN 2022-09-09T14:52:37,333 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:174) - java.io.FileNotFoundException: /mnt/backups/configurationbackups/Mango-Configuration-Sep-09-2022_1452> INFO 2022-09-09T14:52:37,452 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:181) - Finished backup WorkItem. INFO 2022-09-09T14:54:34,713 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem. WARN 2022-09-09T14:54:39,775 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:174) - java.io.FileNotFoundException: /mnt/backups/configurationbackups/Mango-Configuration-Sep-09-2022_1454> INFO 2022-09-09T14:54:39,871 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:181) - Finished backup WorkItem. INFO 2022-09-09T14:55:32,981 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:101) - Starting backup WorkItem. WARN 2022-09-09T14:55:37,478 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:174) - java.io.FileNotFoundException: /mnt/backups/configurationbackups/Mango-Configuration-Sep-09-2022_1455> INFO 2022-09-09T14:55:37,568 (com.serotonin.m2m2.rt.maint.work.BackupWorkItem.execute:181) - Finished backup WorkItem.
Weird thing is, on the MangoES, Configuration and SQL database backups work fine, but NoSQL backups fail due to stale file handle.
On the Mango Enterprise, ALL backups fail.
All backup to the same device and share.
Here are some examples of the mount options I used:
/mnt/backups -fstype=cifs,username=smbusername,password=smbpassword,uid=root,gid=root,file_mode=0777,dir_mode=0777 ://backups.internal.sheridanpoint.org/backups/historian
There's no memory issues...
I tried Smb version 1.0 without any luck.Any ideas on what might be going on?
-
This post is deleted! -
I was able to solve it by adding "noserverino" to the mount options.
-
@mihairosu Thanks for sharing!
Fox