Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
image filedata not deleted when points purged
-
I have enabled logging on two image points and set it to log all data and purge after 1 days however there are still images in the filedata directory from 4 days ago. There are no errors in ma.log that would indicate the deletion of the image files failed. I did purge all data on the httpimage source and all of the files are still there. Are images deleted when the point data is purged?
-
There are a few things going on here that may factor into when files get deleted.
First if you are using MangoNoSQL as your database the purge resolution is roughly 14 days based on the boundaries of the database shards. So in the worst case you will have data that is about 14 days old.
The second part is when you manually purge an Image value the images themselves are not removed until the nightly purge. Every night Mango will check to see if there are any orphaned image files that do not correspond to a stored point value and delete them.
There may be some bugs in there seeing as the Image data type have only recently been added back into Mango, but this is how it is supposed to work.
-
So with the default mangoES database (mangoNoSQL) any data point can't be purged in less than 14 days, and if the purge time is 14-27 days it is equivalent to 14 day purge time, and 28-41 days is all equivalent to 28 days, and so on? Allowing any purge period (days, weeks, months, years) is a bit misleading in that case.
Therefore any image points with logging enabled on mangoES all of the images will be kept for a minimum of 14 days?
I don't really care if the point value records are in the database (and marked as deleted?) but I do need the image files gone because otherwise the mangoES will run out of disk space.
-
@craig said in image filedata not deleted when points purged:
So with the default mangoES database (mangoNoSQL) any data point can't be purged in less than 14 days, and if the purge time is 14-27 days it is equivalent to 14 day purge time, and 28-41 days is all equivalent to 28 days, and so on? Allowing any purge period (days, weeks, months, years) is a bit misleading in that case.
The purge settings are for all database types, and other types can purge to exact times. We have considered allowing purging to exact dates for NoSQL, but it would come at a performance cost so we haven't implemented it. Previously images were not used much so this hasn't been a problem. We could implement the functionality where if the purge time is set to < 14 days we can purge the exact values from the database.
Therefore any image points with logging enabled on mangoES all of the images will be kept for a minimum of 14 days?
Yes.
I don't really care if the point value records are in the database (and marked as deleted?) but I do need the image files gone because otherwise the mangoES will run out of disk space.
If you want you could run a script that would delete images out of the filedata directory nightly. The script could parse the filenames using this format:
imgNT.jpg
Where N is the data point ID from the database and T is the Java timestamp in ms since Jan 1 1970.
-
Can the file-delete script be part of mango or does it need to be a cron job?
Seems like the 8gb mangoES is too small for me with ~50 data points and 34 days historical data (logged every 5s) I am running usually around 1 GB free before I started logging the image points. I only need the image points logged so they can be included in the reports, and once the reports are sent the images can be deleted.
As an alternative to a cron job I will see if I can make the image only log immediately before the report is sent and then there should be enough disk space for 14 days worth of images.
thanks for the assistance
-
You could write a bash script that is called by a process event handler that is set on a Maintenance Event that is scheduled. But the script would need to be pretty smart and either pull the list of data points from MySQL every time it runs or use a hand-coded list of point IDs.
The other, much easier option, is to wait for us to fix the problem for you. I just opened a git issue around this where we can provide an updated NoSQL module that allows finer grained purge resolution if the purge time is less than our shard period (which it turns out is closer to 28 days, not the 14 I mentioned above).
https://github.com/infiniteautomation/ma-core-public/issues/930
I've marked this as high priority and will try and get to it later this week or next with the goal of having it done before the holidays.
-
We have just released a new version of the MangoNoSQL module that has finer grained purging. So basically you can now accurately purge down to a one day level. This setting is not enabled by default for periods larger than about 25 days but can be enabled by checking the Purge within shards checkbox on the System settings page in the Mango NoSQL Information area.
Excerpt from help file:
Purge within shards
This setting governs if the purge task will delete values from within a shard file. With this unchecked, purges are much faster because data is purged in whole shards as opposed to excising individual samples. If the purge period is smaller than a single shard (about 25 days), however, the data older than the purge configuration is deleted from all shards, regardless of this setting.
