Server error timeout

sky_watcher

Hello, I have a MangoES and I've noticed that after 2 or 3 days of running starts to go very slow and when I'm trying to access the data sources give's me this error: Server error timeout.

After a reboot every thing comes back to normal. But after a while the error is back.

Also I've noticed that when gives me that error the RAM memory is almost full.

Any idea why gives me that error, and is a way to build a script to auto-reboot when the error occurs?

Thanks!

phildunlap

Hi sky_watcher,

Hmm. Do you have any large reports being generated on a timer or anything like that? Do you have an internal point for the free memory, and if so does it show it getting very low? It would be pretty easy to restart Mango when that point gets below a low limit, but it isn't really the right solution. A script set point event handler that calls,

com.serotonin.provider.Providers.get( com.serotonin.m2m2.IMangoLifecycle.class ).scheduleShutdown(10, true, null);

Would shutdown Mango (with restart) after 10 seconds. But, it still may not be the right idea to do that (since it'll be an annoyance in quality of service).

When you notice the slowdown, have you checked the internal monitoring pages, /internal/status.shtm and /internal/queueStats.shtm and /internal/workItems.shtm for instance, to see if there are tasks building up faster than they're running?

What version of Mango are you using?

sky_watcher

Hi Phil, I'm using core version 3.4.5.

Next time when the mango starts to go slow I'll be watch the pages that you suggested and I'll come with an update of the problem than.

Thanks.

sky_watcher

@phildunlap I had again that error when I tried to access my data sources and MangoES runs very slow. I've watched the pages that you suggested but I didn't see anything suspicious.

Also I've noticed that if I close my custom dashboard page the Mango starts to run better and than I can open my page again... I don't know if this is relevant or not.

phildunlap

Also I've noticed that if I close my custom dashboard page the Mango starts to run better and than I can open my page again... I don't know if this is relevant or not.

Certainly sounds relevant! One can definitely make too many REST requests and slow the system down, for instance. Maybe you can open your browser developer tools and watch the network tab's traffic, to see if one request seems to be made faster than it is responded to.

You could navigate to the http://host:port/rest/v1/threads?asFile=true&stackDepth=40 endpoint to download a JSON file of the current state of threads. This would help us to identify any long running threads that would be responsible for the slowdown.

sky_watcher

@phildunlap Hi Phil, I was able to go to the specified path and I've downloaded some JSON files. How can I send them to you?

Thanks!

phildunlap

support@infiniteautomation.com

phildunlap

Hi sky_watcher,

I'm 99% sure I received thread dumps from you. But, the email address is slightly different than the email your forum user is registered under. How strange!

From those thread dumps, I must wonder the size of the Mango/databases/mah2.h2.db file. No thread in the thread dumps was an obvious runaway, but there are a lot of threads that are getting blocked trying to access the database going through this route of code:

	  org.h2.command.Command:executeQuery:195
	  org.h2.jdbc.JdbcPreparedStatement:executeQuery:111
	  org.springframework.jdbc.core.JdbcTemplate$1:doInPreparedStatement:698
	  org.springframework.jdbc.core.JdbcTemplate:execute:639
	  org.springframework.jdbc.core.JdbcTemplate:query:690
	  org.springframework.jdbc.core.JdbcTemplate:query:722
	  org.springframework.jdbc.core.JdbcTemplate:query:732
	  com.serotonin.db.spring.ExtendedJdbcTemplate:queryForObject:39
	  com.serotonin.db.DaoUtils:queryForObject:406
	  com.serotonin.m2m2.db.dao.AbstractDao:getByXid:228
	  com.serotonin.m2m2.web.mvc.rest.v1.PointValueRestController:latestPointValuesForMultiplePointsAsMultipleArrays:725
	  com.serotonin.m2m2.web.mvc.rest.v1.PointValueRestController:getLatestPointValuesForMultiplePointsAsMultipleArrays:671

which is getting data points from the data points table to supply any needed columns in the values response. But, there are a lot of these happening at the same time. I wonder how you are querying for values on this dashboard? I suspect it's either too frequent, or you need to bundle some requests together. Or your mah2 file is huge, in which case you need to do a backup, start clean and restore to shrink it I would say. I can link you to a thread with details on that if need be.

sky_watcher

@phildunlap Sorry about email address, I've sent the email from another one....

The mah2.h2.db is 172 MB, I don't know if it is a big size or not.

The points are updated in the dashboard every 2 seconds, is this to frequent?

Thank you!

phildunlap

Sorry about email address, I've sent the email from another one....

No worries! It just was unusual, so I figured maybe your forum email address isn't correct anymore or maybe you have multiple. I was still able to be 99% sure!

The mah2.h2.db is 172 MB, I don't know if it is a big size or not.

It's definitely not small. If you have a core-database-H2-date.zip file in your Mango/backup/ directory, you could try restoring that into a fresh Mango to see how large it is. I would expect the restored database to be much smaller, and that you may get a much snappier performance if you do.

The points are updated in the dashboard every 2 seconds, is this to frequent?

Maybe. I would try to shrink the database first, and if that doesn't alleviate the issue I would try slowing the update rate. I would expect that to depend on your device and your internet connection to that device.

sky_watcher

@phildunlap Hi, I don't have a second MangoES, but I've installed mango local, on my PC and restored the core-database-H2-date.zip file and the size of the mah2.h2.db file is only 20 MB.

Also, over the night the mango crashed and I had to reboot it from PuTTY in order to be able to login in the web interface, I will send the logs over the email, this time the right email :).

Thank you!

phildunlap

I replied via email to check for /opt/mango/hs_err* files.