threadpools are full

[atkins.chrisw]

Is thre a good way to understand threadpools? I keep getting this error.

High priority task: com.serotonin.m2m2.util.timeout.TimeoutTask@6a5ebeb was rejected because the thread pool is full.

Im reading about 80 devices by Modbus TCP. I want to make sure Im getting quality data and not losing packets.

[phildunlap]

What are your threadpools currently set to? You can increase the limits on the System Settings page, in the Thread Pools section.

[atkins.chrisw]

[atkins.chrisw]

What would be a good size for the the threadpools?

[phildunlap]

You could go ahead and set the high priority pool size to 10 or so and the maximum high priority pool to 200.

[atkins.chrisw]

great. Thanks Phil

[atkins.chrisw]

My mango has stopped logging any data. Could this have been the problem? How do I get Mango storing the data again.

[JoelHaggar]

When you say it stopped logging data did it also stop polling data? You don't see the timestamps on the Watch List updating?

If you run out of threads then yes that would be the cause. You can increase your thread pools and then you could try disabling and enabling the data sources or you might need to reboot Mango.

[atkins.chrisw]

I see them in the Watchlists. I go to the datapoints and I can see the small graph. where it shows no data for that time.

I also cant see them in my angular charts.

How big of threads can I make for this. I really can't lose all of this data.

[JoelHaggar]

That totally depends on the type of computer it is running on and what the CPU can handle. Under normal operation you really shouldn't run out of threads. What type of data sources do you have and how many? What type of computer is it on?

[atkins.chrisw]

Its running on a Windows 2012 Server.
4GB RAM

80GB hard drive

We have 25 Modbus IP connections reading roughly 30 points each
And 25 Metadata sources with around 10 points each

[JoelHaggar]

That doesn't make sense to me that it's running out of threads. You might want to get a thread dump and send it to us to look at.

One thing to check with your Meta Data Points is that you only have 1 data point updating its context from the same Data Source. You don't want two points that both poll at the same time updating the same meta data point.

If you run out of threads again to get a list of all the threads running use this URL after you are logged in /rest/v1/threads?stackDepth=30

Copy all the JSON into a text file and email to support@infiniteautomation.com and we'll look it over. Thanks. Joel.

[atkins.chrisw]

Thanks. Should I reboot Mango then

[JoelHaggar]

Probably, just to be sure you got things back to normal.

[atkins.chrisw]

Could I make a no_update error to send ot my cloud version so I can make sure I don't lose data?

Do we have a no update error for the database?

[JoelHaggar]

You could use the Lo4J Data Source to capture errors and then send those to your cloud Mango. This is not a normal situation so once you figure out the cause I wouldn't worry about it too much.

[atkins.chrisw]

ok. Ill do that. Thanks

[atkins.chrisw]

Ok. So as I am watching the Internal Diagnostics. I see two java.exe running. One PID 1704 and one PID 1220.

Are both of these for Mango?

[phildunlap]

Hi Chris,

Probably the right tool to resolve what is going on there is jstack, a thread dumping utility in Java. To use (if jstack is on your path. otherwise would be in jdk/bin/ ), you can type:

jstack -l 1704 > C:\Users\You\MangoStuff\stack-1704
jstack -l 1220 > C:\Users\You\MangoStuff\stack-1220

It could also be useful to know which is the parent PID, if one is the parent pid of the other (maybe you have two Java processes, maybe you have started two instances of Mango on different ports). It looks like you can get PPID numbers via this command at the command prompt:

wmic process where (processid=1704) get parentprocessid

My only familiarity with causing these forks to happen and hang uses Process Event Handlers, so if you have some I will invariably ask something about if that script runs and works.