Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
Unable to purge points
-
Hello Everyone,
I am having some issues getting some points to purge on my MangoES. I've had the purge policy set to remove anything older than 12 days but it isn't actually removing the points.
If I force the purge using the configured settings by going to System Settings > Purge Settings > "Purge now using Purge Settings" it shows in the log that it is deleting points before a certain time stamp and it ends with "Data purge ended, 30 million point samples deleted"
Database Before Purge
Purge Settings
Purge Complete
However all of those points are actually still there and it still shows the database as having 60+ million history count.
Database After Purge
If I try purging by going to Data Sources > MyDataSource > Purge Now it shows "31 million point values were purged" but they are still shown in the database afterwards.
Purge Settings
Purge Complete
Database After Purge
Do I have something configured wrong?
Any help would be much appreciated.
Ed
-
Hello,
I just wanted to let you know we are investigating this issue. What Mango Core Version are you running on your MangoES?
Moving to Mango feedback category.
Thanks,
-
Hi Woody,
It is showing 2.7.10 as the core version.
Ed
-
It looks like the purge has acutally completed at some point over the weekend.
Point count is roughly 25 million less then Friday and my history starts on March 30th rather the 18th. I'd still really like to home in on what would make it run in this fashion.
-
Ed,
We had an internal discussion on this and found that the number of values that Mango says are being deleted is inaccurate if you are using the NoSQL module to store your point values.
The important thing to realize here is that purging data is a rough way to free up space but is not supposed to be used to remove an exact number of point values.
The reason for this is that purging with the NoSQL module is implemented to be highly efficient but this results in a loss of accuracy. When a purge is performed the NoSQL module will remove blocks of data without having to take time to find the exact date range to purge.
The NoSQL shard files (blocks of data) are roughly 12 days long and purging removes entire files. This means purges have a resolution of roughly 12 days, Mango will never purge more data than your purge period so if you have less than 12 days of data nothing will be purged. This also means that all of the data in the shard must be older than 12 days before that shard is removed.
Lastly the count of values deleted is an exact count of the values between the range you want to purge, but not necessary the number that have been deleted. I am entering a bug for that into our bug tracking system. I will also add in a note about improving the purge resolution to allow purge periods of less than our shard period to actually work.
Thanks for your help in identifying this.
Terry
-
This issue can be tracked on GitHub https://github.com/infiniteautomation/ma-core-public/issues/764
