High priority tasks rejected after latest upgrade

[sjobbe]

Hello

Updated yesterday to the latest Mango version and started getting this error:
High priority task: com.serotonin.m2m2.util.timeout.TimeoutTask@32bc5080 was rejected because it is already running.

Did the latest version get slower or what is going on? How should I troubleshoot this?

Regards,
Jan-Erik

[JoelHaggar]

@sjobbe we don't make new versions of Mango that are slower, it's quite the opposite. Do you want help on this issue or are you just trying to upset our developers who have put in long hours and great effort to bring you a greatly improved product all for a free upgrade?

This event can mean a lot of different things so it depends on if you are having any issues or not. If you can provide more information about any symptoms your Mango has we can probably point you in the right direction.

[sjobbe]

I'm not trying to attack anyone. I'm just saying that I've never seen this before and immediately after upgrading Mango started to send me e-mail notifications like this.

I guess I can turn off the e-mails and then I do not have any problems with it since it runs fine from what I can understand.

Keep up the good work and I was delighted to see that the upgrade notes listen "performance enhancements" for several different modules.

I guess my question was: is there a way to see why it is still running and what is it?

[stensma]

Hi I see the same Alerts! Why?
Pictures from web interface attached!

// Per

[terrypacker]

Per,

In the later 2.7 releases we added a feature that enforces time ordering on Mango's tasks. Previously tasks could run in parallel to other tasks of the same type. Data source polls were the exception because they had a special mechanism to 'abort' their polls if they tried to run in parallel to another poll.

In heavily loaded systems, or for tasks that take long time to run Mango will now 'Reject' the any additional task of the same type that tries to run in parallel. Unfortunately due to the limitations of what we can change in minor releases we weren't able to implement all of the tracking so you can't know exactly what task is trying to run in parallel to another instance of itself. In 2.8 we will have a full featured tracking mechanism to help you tune Mango better. For now all you can know is that some task is being run more often than is possible (and it is NOT a data source poll).

I was recently doing some testing on a heavily loaded Mango and found that I could reduce task rejections almost completely for periodic tasks by changing the type of Garbage Collector that the JVM uses. I can see from your system output that you are using around 75% of your memory which would likely force the Garbage collector to work hard to free up memory. When this happens the main execution thread of the JVM will hang momentarily and when it restarts there is the potential for many tasks to run at the same time.

You can add this to your JVM parameters to enable a more suitable collector algorithm:

-XX:+UseConcMarkSweepGC

We already have this available to you in the ext-available directory in the bin directory. To enable this you can make a simlink (ln -s) for this file into the ext-enabled directory. Such that:

/mango/bin/ext-enabled/concurrent-garbage-collector.sh --> /mango/bin/ext-available/concurrent-garbage-collector.sh

Give this a try and let me know if it helps.

Thanks,
Terry

[stensma]

I will try this !
Have just created the symbolic link, when I did this I saw that the memory usage between my two similar systems was significant different.
The MangoES system seems much more loaded that the installed free system on a raspberry Pi (V3)
Even when I have all my temp readings on the latter one(Pi) and publish them over my network as Modbus TCP to the Mango ES instance to read and present, could I optimize MangoES to use less memory? I notice that MangoES has no swap space!

[ThomasEinasto]

As I could see you start your MangoES with 1200 mb memory usage but on your rasperry you are not starting mango with it. My guess is when you have a small project on the MangoES then you could change the -Xms1200m -Xmx1200m parameters so that you allocate less memory to mango than you have before but I guess MangoES comes with the most optimal memory allocation.

Right now I have my MangoES which is using 1.7 GB constantly and that way the restarts on Mango occur more often because memory is insufficient and the same errors occur more often.

[stensma]

Terry
I have now used the new garbage collector, for 24h
it seems to work better only one timeout alarm and that was when the backup of db had started.
The memory usage is down to 50% now.

/Per

[stensma]

Hi!
I still have alarms but they are not as many.
I found that I get them when the system do backup.
see picture

//Per

[jpstaub]

On a new installation of Mango I'm having the same problem. The fix above regarding the adjustment of the JVM parameter was put in place. However, urgent messages were generated at approximately the same frequency as before the JVM adjustment.

The urgent event message is as follows:

High priority task: com.serotonin.m2m2.util.timeout.TimeoutTask@653124b was rejected because it is 
already running. 

I was able to strangle the urgent notices on the Mango dashboard by going to: System settings >> System event alarm levels >> Rejected work item >> None. However, this fix ignores that there's a and persistent problem as reported by the system.

Why does Mango keep tapping on a task that is already running and is there a way to prevent it? Is there a way to find out what the task does?

Configuration
OS: Ubuntu server 16.04
Java version: 1.8.0_101
Mango: 2.7.12

Regards,
Jake

[terrypacker]

jpstaub,

In the next minor release of Mango you will have access to more information on the tasks being rejected which should clear up the confusion. In the meantime if you are not seeing any problems with your data or event handling you can safely ignore these messages. There are many types of tasks that do not cause problems if they are rejected due to another instance already doing the job.

If you still want to investigate further you can try to increase the global setting for task queue size. In your env.properties file there should be a line like this:

runtime.realTimeTimer.defaultTaskQueueSize=0

This prevents multiple tasks from being queued up, to allow multiple tasks of the same type to be queued instead of rejected just increase this number. More than likely increasing this number by 1 or 2 will reduce and possibly eliminate the rejections.

The downside of this is that the tasks can now be run at a later date than expected. This is fine for most task types and as I said in later versions of Mango there will be a much better reporting system on all the tasks.

It is likely that the messages you are seeing do not effect any of your data and in the future Mango will have a very informative view into its task subsystem that will tell you if there are any problems.

[jpstaub]

Terry,

Thanks for the tip on:

runtime.realTimeTimer.defaultTaskQueueSize

The setting was adjusted to a max of 2. The FATAL message still occurs. Periodicity is around a minute between occurrences. It seems better than it was but still nagging away.

If you're not seeing the same thing on your end there must be a configuration difference on my end. Dealing with configuration problems is tiresome for all involved. With that in mind, I highly recommend Docker. If Mango were available on the Docker hub I'd definitely use it.