Mango polling problem

stefan

I'm using 2.x version of Mango SCADA with enterprise license + DGLux 2.5.

The system has 240 Modbus I/P sources created, in two different groups of 120 each. The first group has 348 points per source and a polling frequency of 24 hours, and the second one has 62 points per source and a polling frequency of 5 minutes. 120 devices with 120 different IP addresses are connected to Mango and each device has one source from both groups polling it for data. 60 devices are connected to the internet with cable and the other 60 are connected with 2G GPRS. Along with that, I have one Meta source with 120 points that calculate something every minute for each of those devices using their data. That's the setup.

Here's the problem. every now and then some Mango source stops poling some of these devices (not discriminating them by their internet connection type) even though they are available and responsive. If I turn off that source and turn it on again, it'll start working immediately. It also happens for Meta points, some of those points would just stop doing new calculations. If I turn the Meta source off and on, they'll start working orderly.

Best regards, Stefan Radunovic

phildunlap

Hi Stefan,

There is some variance in the 2.x series based on the x. What is your Mango core version?

When you've witnessed it stop polling for a modbus data source, does the data source abort polls after that? Or, does it report a timeout? If you visit the data source's page, do the read tools work?

It does sound like it may be a Mango issue. I have seen a meta data point stop updating when it truly seemed to me (and the administrator) that it should have been, but the customer was unwilling to update which created an impasse and I never resolved the root cause. Theirs was the only system I've observed it on.

It's possible that taking thread dumps by GET'ing the /rest/v1/threads?asFile=true&stackDepth=40 endpoint a few times and sharing that information here or in an email could shed some light.

What sort of logs exist at and one poll period before/after the time one of these points stopped updating? If this env.properties entry exists in your version runtime.realTimeTimer.defaultTaskQueueSize you could see if changing it from 0 to 2 has an effect.

Also, running Mango on the command line or with stderr directed to a file may prove useful is some error is escaping the log.

stefan

Hi Phil,

Well, my Mango core is 2.7.10.

We were somewhat lucky and unlucky at the same time, it didn't happen again since the last time I turned those sources off and on some 10 days ago, but it will happen again. I'll notify you then, and provide you with answer to your questions.

I've checked the logging page and I see a lot of this

This is my OS (Ubuntu) log, Mango has eaten all the RAM

It seems to me that this is the same issue like in this thread https://forum.infiniteautomation.com/topic/2483/high-cpu-usage/27

I can't send you the log file here, says I don't have permissions to do that. Can you give me your email so I could send it there?

phildunlap

The task rejection message from a full core pool makes me think you may see this resolved if you increase your maximum high priority thread pool size in the thread pools section of the system settings. That definitely would result in what you were seeing.

Have you explicitly constrained / expanded Mango's memory using an ext-enabled script?

stefan

Nope, haven't done anything to mango externally. So, I'll increase it on Monday and update you with the results.

stefan

Hi phil,

It happened again. There is nothing on logs page about this event, the read tools do work. My thread pool size is 100, I'll increase it to 200 and see what happens. It has eaten up all the RAM again. What's your email, so I can send you log files?

phildunlap

You can email them to support@infiniteautomation.com but please do look and if your high priority thread pool is full in the log.

You can also upload them to the forum. I find it most effective to use the button above the posting body, then wait for the (uploading 100%) transform into the link to the file.

stefan

Yeah, I've tried to upload it here with the "Upload File" option but it says I have no permission to do that. I've sent logs to the support address.