None Events with MangoNoSqlBatchWriteBehindManager

---

I am getting 20-40 None level events a day since I upgraded to the latest MangoNoSql module and enabled the new purge within shards feature to delete the old image points and files nightly.

High priority task: com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$PointDataMover was rejected because it is already running.

This is on a v2 mangoES. Mostly the event timestamps coincide when reports or purge are running.

Are there any settings that can be modified to stop these None Events from being raised?

Generally I want the alarms and events raised in Mango to be about the process and not mango itself. I would prefer if this type of event was just in the ma.log so we can review periodically to know when there are problems but any process operator can't be bothered with this many notifications that they can't do anything about or they will tune them all out and miss the important ones.

Also I am getting these events, but they were occurring before the mangoNoSql module upgrade:
High priority task: com.serotonin.m2m2.rt.dataImage.DataPointRT was rejected because it is already running.
The data retrieval failure, data save exception, and poll aborted events are all set to Do Not Log on the httpimage data source.

Hi Craig,

Yes! I wonder, are these 20-40 occurring all clumped together, or is it one every few seconds/minutes during a purge or report period? The period of the data mover is configurable in the "Mango NoSQL Information" section of the system settings as the "Batch process manager period (ms)" but you probably don't need to adjust this.

The log level is controlled by the "Rejected Work Item" in the "System event alarm levels" section of the System settings page. This likely governs the DataPointRT rejection's level as well. So, one path might be to set this to "Do not log" and then creating an email event handler to dump these messages to an account.

To the DataPointRT rejection, I would suspect that is an interval logging task, so likely the data point that it applies to didn't interval log in that period. I suspect you could eliminate that message entirely by changing this setting in your env.properties file and restarting Mango:

runtime.realTimeTimer.defaultTaskQueueSize=0

to

runtime.realTimeTimer.defaultTaskQueueSize=2

which I would probably do first, before adjusting the alarm levels to hide them. If the rejections of the data mover come somewhat spaces out, permitting some task queuing may prevent those messages as well.

I just wanted to add that our own Mangos also get these exact same notifications, but only a few a week.

Which? The point data mover? I wouldn't worry about that at all.

I set defaultTaskQueueSize=5 and Rejected Work Item log level none and I still get two DataPointRT rejections every night around 3 AM.

Anything else I can do?

Is it a point of major concern? I wouldn't be alarmed at that, personally.

But assuming it is, it's likely the result of either 1) the nightly purge or 2) garbage collection. To see which, you could disable the nightly purge for a night and observe if they still occur. I would expect them to. I would think it's from a period of 'stop the world' garbage collection, and a possible path would be to experiment with using the concurrent-mark-sweep collector instead of the parallel collector.

From an investigatory standpoint, I would look for one of your points that is doing interval logging and check if it is missing a point of data around the time of the message.

It isn't a point of major concern, but it would be better for us if it were possible to make it so the events raised in mango were only related to the process that an operator would be able to deal with. Timeouts, reconnects, retries etc should be in a log file so that they can be reviewed to diagnose faulty equipment, improve settings, etc but for us they don't belong in the same interface as events generated from the process since they are for a different audience.

Shouldn't Rejected Work Item Log Level = None make it so DataPointRT rejections don't show up in mango?

With Log Level = None do the events still get recorded in ma.log?

thanks for the help

Yes None level events do get logged. There is a new event level in 2.8 called Ignore which prevents them from being handled or logged. There is also a Do Not Log level that still allows them to be handled just not logged.

In Mango 3.0 with the new UI it's much easier to switch between system events and user defined event detectors and I think your point is well taken about separating system events from process events.

Hi Craig,

To add to what Joel said, None level events are recorded in the database. Whether or not anything will appear in ma.log will depend on the Log4j configuration you are using, more likely.