Dreaded out of memory issue

[v8dave]

We are running into an issue with the Mango server where the client tries to view the data in a chart for 1 weeks worth of data and we get the out of memory error. The client then loses the connection and we can no longer access Mango. Checking with ps aux on the Linux server shows the service as still running. I have a cron job that checks for the service not running and this then restarts it but in this case the server is still running but it's not responsive. The field units can't access it either. I have to force stop it with kill -KILL pid.

The server is currently running Centos with 1GB of RAM and we wonder if we should increase this to 2 or 4GB

REQUEST ATTRIBUTES
   javax.servlet.forward.context_path=
   javax.servlet.error.status_code=500
   javax.servlet.forward.servlet_path=/chart

   javax.servlet.jsp.jstl.fmt.localizationContext.request=org.springframework.web.servlet.support.JstlUtils$SpringLocalizationContext@401afa58
   javax.servlet.forward.request_uri=/chart/1470071635821_1814400000_85_w370_h140.png
   javax.servlet.forward.path_info=/1470071635821_1814400000_85_w370_h140.png
   org.eclipse.jetty.server.error_page=/exception/error.jsp
   javax.servlet.error.servlet_name=imageChart
   javax.servlet.error.message=java.lang.OutOfMemoryError: Java heap space
   javax.servlet.error.exception_type=class java.lang.OutOfMemoryError
   javax.servlet.error.request_uri=/chart/1470071635821_1814400000_85_w370_h140.png
   javax.servlet.jsp.jstl.fmt.locale.request=en_US
   javax.servlet.error.exception=java.lang.OutOfMemoryError: Java heap space

How can we increase this or at least recover from this short of me removing the ability to try and chart data for max 4 days at the most.

[Woody Beverley]

Hey Dave,

Are you using rollups or are you getting every datapoint for that time period?

Try using 5 or 15 minute rollups to see if that improves you situation.

Thanks,

[v8dave]

Hi Woody,

I am using the following to get the data points. I might have to add something to change the interval. Need to see how to do this in the code without the user having to select this option.

<ma-point-values point="pressure1" values="pressure1Values" from="from" to="to" rollup="AVERAGE" rollup-interval="1 minutes"></ma-point-values>

[v8dave]

How do I implement this without having to show the input containers? I just want the 4 buttons and these set the values for getting the points. Right now, the rollup-interval doesn't work.

<div layout="column" style="width:350px">
    <md-input-container class="no-errors-spacer">
       <label>Date preset</label>
       <ma-date-range-picker from="from" to="to" preset="LAST_1_DAYS" update-interval="1 minutes"></ma-date-range-picker>
    </md-input-container>
    <md-input-container class="no-errors-spacer">
       <label>From date</label>
       <ma-date-picker ng-model="from"></ma-date-picker>
    </md-input-container>
    <md-input-container class="no-errors-spacer">
        <label>To date</label>
        <ma-date-picker ng-model="to"></ma-date-picker>
    </md-input-container>
</div>

<ma-now update-interval="1 minutes" output="to"></ma-now>
<ma-calc input="now | moment:'subtract':subtractX:subtractUnit" output="from"></ma-calc>

<md-button class="md-raised" ng-click="subtractX=4;subtractUnit='day'; rollupInterval='1 minutes'">Last 4 days</md-button>
<md-button class="md-raised" ng-click="subtractX=2;subtractUnit='day'; rollupInterval='1 minutes'">Last 2 days</md-button>
<md-button class="md-raised" ng-click="subtractX=7;subtractUnit='day'; rollupInterval='5 minutes'">Last week</md-button>
<md-button class="md-raised" ng-click="subtractX=30;subtractUnit='day'; rollupInterval='10 minutes'">Last month</md-button>

<ma-point-values point="oilbbl1" values="oilbbl1Values" from="from" to="to" rollup="AVERAGE" rollup-interval="rollupInterval"></ma-point-values>
<ma-point-values point="oilbbl2" values="oilbbl2Values" from="from" to="to" rollup="AVERAGE" rollup-interval="rollupInterval"></ma-point-values>
<ma-point-values point="oilbbl3" values="oilbbl3Values" from="from" to="to" rollup="AVERAGE" rollup-interval="rollupInterval"></ma-point-values>```

[Jared Wiltshire]

@v8dave said in Dreaded out of memory issue:

The server is currently running Centos with 1GB of RAM and we wonder if we should increase this to 2 or 4GB

1GB is not much, Mango will work but you might run into issues like these. Its probably worth checking your Java heap size settings, I'd set the maximum size (-Xmx flag) to around 75% of your memory (768MB) provided you are not running anything else. Let us know if you don't know how to do this.

@v8dave said in Dreaded out of memory issue:

Hi Woody,

I am using the following to get the data points. I might have to add something to change the interval. Need to see how to do this in the code without the user having to select this option.

<ma-point-values point="pressure1" values="pressure1Values" from="from" to="to" rollup="AVERAGE" rollup-interval="1 minutes"></ma-point-values>

This is definitely something we can add into the directive so that it does auto rollup, I'll look at it for the next release.

@v8dave said in Dreaded out of memory issue:

How do I implement this without having to show the input containers? I just want the 4 buttons and these set the values for getting the points. Right now, the rollup-interval doesn't work.

I think your issue is just that you don't have the curly braces around your variable, try this rollup-interval="{{rollupInterval}}"
Some attributes are designed to take strings and you must use an interpolated expression (i.e. the curly braces) to turn your variable into a string. The next version of Dashboards will have complete documentation of the directives and their attribute types, its being worked on as I speak.

[JoelHaggar]

I just added this article about using Mango's start up extensions to set memory usage:
https://help.infiniteautomation.com/support/solutions/articles/14000039297-using-startup-extensions

This is the easiest way to provide custom memory settings. You can also check the Internal metrics page to see what the memory usage looks like while Mango is running. You can also create data point using the Internal Data Source to track this over time.

[v8dave]

This is the current settings so the memory needs to be increased. We are also going to look at moving up to 4GB of memory on the server itself.

How do I make the changes to increase the current JVM memory as it is set low as you can see from this image.

[Woody Beverley]

Hi Dave,

I think if you go to our help site http://help.infiniteautomation.com/support/home and search for "JVM memory" you will find the information you need. If not, let us know and we will track it down.

Thanks,

[JoelHaggar]

@v8dave That's what this article is about: https://help.infiniteautomation.com/support/solutions/articles/14000039297-using-startup-extensions

Let me know if it needs any clarification.

[v8dave]

Yeah, I found it and set the extension and change the values to be max 768 and min 512 and will see how that goes.

[v8dave]

It does like the changes. Mango service has shutdown twice now so I am backing off the memory max to 512MB just to see if that is the issue. The last error in the log is this.

ERROR 2016-08-02 17:13:52,639 (org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException:367) - Failed to invoke @ExceptionHandler method: protected org.springframework.http.ResponseEntity<java.lang.Object> com.serotonin.m2m2.web.mvc.rest.v1.exception.RestExceptionHandler.handleMangoError(java.lang.Exception,org.springframework.web.context.request.WebRequest) 
java.lang.IllegalStateException: Compressed output stream is already assigned.

[JoelHaggar]

You have to be careful to not use too much memory or the OS will shut down the Mango Java process. I would think you'll be ok with a Max of 512 if you have a total of 1GB but Your real issue will be to increase the total memory on the server. It's also not ideal to give Java too much more memory than it needs as the OS can also use extra memory as system cache which seems to improve overall performance. I don't remember how many total Data Points you have but 4GB should be pretty good.

[v8dave]

That should have said DOESN'T LIKE.

Seems more stable with 512.

We have 20 data points with one system and 15 on another. I've already suggested to the client we upgrade to the 4GB and as we are looking to get more clients, we will probably move this from the current virtual server to a dedicated real one soon and that comes with 8GB of RAM.