Potential Memory Leak

[adamlevy]

Well it is locking up. REST API response times have increased above 2 seconds and are now not responding. And now: Exception .... Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

How exactly should I go about tuning the MangoNoSQL database?

Should I start with the publishers? There are a number of them and that will require me to login to multiple mango instances to adjust them as I don't have JWT tokens set up for all of them yet.

I'm still watching iotop to see what is going on with mango when it starts to error out.

[phildunlap]

I would try lowering your persistent point value throttle threshhold on the receiver's side. This setting is in the system settings:
0_1519859772450_persistentSynchronizationSettings.png

I would lower it from 37500000 to maybe 5000000 but it's hard to know offhand. Do you have your "Point values waiting to be written" getting logged? Is the number of values waiting getting very high before the GC errors? I would also consider lowering the persistent sync threads count, but that would be on the publisher. The old default is 10, and that's definitely a little high. The new default is 3. Are you using a different minimum overlap than the default on the publishers? You may also see performance improvements if you increase this, but the receiver needs to be using NoSQL to set this value > 1.

To tuning the NoSQL, I hesitate to guess. But, you could try (on the receiver, which is the one suffering the crashes, yes?),

1. increase minimum small batch wait time, perhaps 50ms
2. decrease batch write behind spawn threshhold, perhaps 100
3. decrease batch write behind max instances, maybe 4
4. increase small batch size to 100

[adamlevy]

The system crashes about every 40 minutes with the update, which appears to be a little longer than it was lasting before. So something was improved by moving to 3.3.1 but not everything... Also I don't start getting errors until the system hits about 69% memory usage.

I just started collecting points waiting to be written. Here is the graph for the last 30 minutes. There were around 15k points before the crash. Now it is down below 100 which is hard to see on the graph in the picture. I just restarted it though so we'll see if it fails again.

I lowered the persistent point value throttle to 5,000,000 as you suggested and I increased the small batch wait time to 20ms and decreased the batch write behind spawn threshold by an order of magnitude to 10,000. I can lower that further if you think it would help but it was previously set at 100,000 so I didn't want to drop it so drastically all at once. I lowered max batch write behind to 6.

What's strange to me is that I don't think I am seeing abnormal io wait % time.

You can see the memory usage hit a cap at 70% and then drop down to around 40% when I restarted mango. Also the usage % spiked as well. The blue line is the user usage %. The system and io usage % is also graphed but it is all well below 10%. Weighted IO Time is maybe a little high but its not spiking with the increased load. so that seems strange. The load average, DB write per second, point value database rate, number of open files, and MangoNoSQL open shards graphs are coming from Mango's /v2/server/system-info endpoint.

[adamlevy]

The issue persists. I am really not sure what to do here. I can't move forward with any other work while our server is failing every 40 minutes.

I am willing to share some access to our instance of Mango or get on the phone to talk this through if that helps.

This is Adam from iA3 BTW.

[adamlevy]

I just realized that the memory-small.sh extension was enabled. I'm bumping it up to medium to see if that helps.

[terrypacker]

Adam,

I'd be a little weary of hitting the /v2/server/system-info endpoint frequently, some of the data returned is computationally intensive for Mango to calculate. For example it will compute the database size by recursively accessing every file to get its size. For NoSQL there will be 1 file for every 2 week period a data point has data.

I would strip down the request to only get what you want:

GET /rest/v2/server/system-info/noSqlPointValueDatabaseStatistics
GET /rest/v2/server/system-info/loadAverage

I would avoid requesting noSqlPointValueDatabaseSize because of the intensity of the request on the server.

[adamlevy]

Thanks for the advice! I will reduce how frequently I hit that endpoint and make the query more specific.

[terrypacker]

In addition to those metrics you can also request all of the information found on the InternalMetrics page via the /rest/v1/system-metrics/ and /rest/v1/system-metrics/{id} endpoints.

The most useful of these for your current problem would be the id of com.serotonin.m2m2.db.dao.PointValueDao$BatchWriteBehind.ENTRIES_MONITOR which will show you how many values are currently waiting to be written to the database (cached in memory).

This information can also be logged by the Internal Metrics data source.

[adamlevy]

Awesome! I will check that out. Is there anyway to view the swagger interface for both v1 and v2 without restarting? or can the swagger interface only be enable for one version at a time?

[terrypacker]

To see both in swagger just set:

swagger.mangoApiVersion=v[12]

You must restart to see the changes. Also Swagger isn't really designed for use in production environments especially if you are running thin on memory as it will eat up some of your precious ram.

[adamlevy]

Good to know. Thank you again. So far java hasn't run out of memory with the memory-medium ext-enabled but I'm also hitting 98% system memory usage and starting to use swap. But the response times are still OK.

I increased my query interval from 10s to 90s. I don't think this is the cause of the issue at all but it won't hurt to hit that endpoint less frequently. I will need to reconfigure telegraf to just grab the metrics I want.

The points waiting to be written are high but are still staying a tad lower than they were before. I think they'll be high as long as mango is catching up on historical point values for awhile. They are peaking around 10k whereas before they were hitting upwards of 15k.

I intend to disable swagger for production but I have been experimenting with it there as I was instrumenting Mango. Thanks for the reminder though.

[terrypacker]

It definitely sounds like you don't have enough memory for your configuration. If you allocate the JVM too much memory you run the risk of having the process get killed by the OS.

If you intend to run with 4GB of system memory I would take a look at throttling the Persistent publishers via the setting on the receiving Mango. Phillip suggested setting it to 5 million but it seems like your system would run out of memory before there are 5 million values waiting to be written. I would keep an eye on that value and see when you start to experience GC thrashing (High CPU and OOM errors in the logs). Then set the throttle threshold to below that number of values waiting.

From the graph you posted you could set it to 10,000 (but that was with less memory so the value is going to be higher now).

[adamlevy]

Yeah I was just coming to this myself. I wanted to let it run and see how it handled it but I can see that I just need more memory. I'm bumping it up to a t2.large with 8G of memory. It was actually not crashing even though it was at 99% memory usage. But swap was increasing to 50% of the 2G of swap. We'll see how this performs now...

Thanks for your continued help with this.

[adamlevy]

So while increasing the size of the ec2 instance and switching to the memory-medium option has allowed us to catch up on the historical points, I am still noticing a significant memory leak. Here are graphs of our system stats over the past 5 days since I started running mango on a t2.large instance.

We are now working with 8G and the memory-medium option tells java it can use 5G. I have been watching the memory usage steadily climb with periodic jumps once a day around the time when our persistent TCP data sync is scheduled and we get a surge of points.

Why does the memory consistently grow? This made the system unresponsive again for me at a critical time when I had to demo the system for a potential client. Are we doing something wrong here?

Thank you

Adam

[JoelHaggar]

What version of Mango is this? We released Mango 3.3 about a week ago that might improve this.

[adamlevy]

This is running core 3.3.1 ATM. I see that latest core is 3.3.3 and I will upgrade tomorrow.