Hi,

Have noticed a problem where a mango instance stops displaying point values (by XID). This causes intermittent failure of a dashboard app (as it cannot present current values). Restarting the mango server instance fixes the issue (until it occurs again). The problem can be reproduced via the 'mdAdmin' Custom Dashboard example:

http://mangohost/modules/dashboards/web/mdAdmin/#/dashboard/examples/basics/get-point-by-xid

When selecting XID for points which do exist in the instance, where mango shows a value when you view it in the data points view, no value is presented by the angular material dashboard. The point name and details can be seen, but the value doesn't appear. The following network calls can be seen:

http://mangohost/rest/v1/data-points/DP_HP-POD-004_9867690 (status 200)
ws://pod.energymadeclean.com:8080/rest/v1/websocket/point-value (status 101)
then another entry to indicate the following is finished:
ws://pod.energymadeclean.com:8080/rest/v1/websocket/point-value (status 'finished')

However no value is rendered.

Have tried to log out and back into mango (using the admin account) but the issue persists. Restarting the mango instance fixes the problem (for a while) - until it occurs again.

Is there a known problem with the mango instance and the web-socket point values not being returned after some time? There is no entry in the ma.log file which corresponds with the failure. Are there other locations to look for an error message?

Snippet which reproduces the problem (when it is occurring):

--
<!-- Display a point value using only its XID -->
<!-- Get a point (and value) using its XID and assign it to an output variable "myPoint" -->
<!-- example XIDs for points which do display current values in mango -->

R06-LI0073
DP_HP-POD-004_9867690

<ma-get-point-value point-xid="DP_HP-POD-004_9867690" point="myPoint"></ma-get-point-value>

<p>The point name is "{{myPoint.name}}" and its value is "{{myPoint.renderedValue}}".</p>

<!--
<span>
{{myPoint | json}}
</span>
-->

Regards,
Tim

Hi All,

We would like to transfer the nosql data store module license from a mango instance running on machine (A) to mango running on machine (B) . Machine (A) is does not require speedy statistics and point value results, unlike machine (B) which incorporates the dashboard module and needs faster API query response times. We need to restore data already in the nosql store on (A) back into a sql data store on (A) - so historic data can still be referenced in Mango (and DGLux).

Machine (A) is using mango core 2.5.2
Machine (B) is running mango core 2.7.10

Steps

1. read data out of nosql db on (A) and load into PointValues table on (A)

2. count the rows in the h2 embedded pointvalues table on (A). If the row count is the same as the nosql store, disable the nosql module on (A)

3. enable the nosql module on (B)

4. go to portal and associate nosql license with (B) instead of (A), so regular shutdown no longer triggers

5. import data from PointValues table in (B) into new nosql backend structure on (B)

Questions:

Q1. Is there a simpler process?

Q2. For Step 1 above, Is there a guide for migrating from data in nosql back into the embedded H2 pointvalues table?

Have reviewed the database conversion doc here, but it doesn't mention how to move from point values stored in the NoSQL module back into H2 (or another db):

https://help.infiniteautomation.com/support/solutions/articles/14000022650-database-conversions

Q3. For Step 2 above, if we disable the nosql module first, can we still read the data out to migrate into the h2 embedded? This would allow data arriving in the meantime to flow into the pointvalues table. Alternately we could disable the publishers while the migration is taking place, and re-enable and sync when the migration is complete.

Cheers,
Tim

Hi All,

Am using the latest custom dashboards module, receiving:

WebSocket connection to 'ws://our-server-here:8080/rest/v1/websocket/point-value' failed: Error during WebSocket handshake: Incorrect 'Sec-WebSocket-Accept' header value

When attempting to use a point value directive. The error is thrown by EventManager.js line 81:

var socket = this.socket = new WebSocket(protocol + '//' + host + this.url);

To reproduce, am using the following javascript snippet:

var user = User.login({
username: username.value,
password: password.value
});

        user.$promise.then(function() {
            console.log('doLogin, user promise resolved');
            /*
            var redirect = 'dashboard.home';
            if ($state.loginRedirect) {
                redirect = $state.loginRedirect;
                delete $state.loginRedirect;
            }
            $rootScope.clearErrors();
            $state.go(redirect);*/

            // reload the page.

            setTimeout(
                function () {
                    console.log('doLogin, attempt refresh');
                    //location.href=location.href
                    location.reload(true);
                },
                2000);

        }, function(error) {
            console.log('doLogin, error');
            console.error(error);
            /*
            if (error.status === 406) {
                $scope.errors.invalidLogin = true;
            }
            else {
                $scope.errors.otherError = error.statusText || 'Connection refused';
            }*/
        });

After a few minutes the dashboard appears to retry the websocket connect, and everything is ok. If I reload immediately the error is not thrown. The problem I have is detecting the initial websocket failure and retrying. Has anyone seen the same? How is Sec-WebSocket-Accept being set by the client? Am I missing a step to initialise the client after the resolve of User.login().$promise?

Cheers,
Tim

@Jared-Wiltshire the link https://github.com/infiniteautomation/ma-dashboards/tree/master/Custom Dashboards/web/loginPageTemplate isn't working atm :)

Using the mangoNoSqlDatabase has fixed the performance issue we were seeing when retrieving point values and statistics via the API. The client is unable to justify the cost of this module however, and is asking if an alternate option exists for improving the performance of retrieving data from the built-in pointValues table . The stack trace and log file appears to suggest database locking is in place which slows down concurrent requests to point values. Any ideas?

A few things have been done. Will wait to see if it fixes the problem.

1. enabled the mangoNoSqlDatabase module
2. configured the process to start with 6GB heap size via:

#!/bin/bash
case "$1" in
init)
# Startup with Java Memory setup for Medium size installation
# The heap is set to non-expanding for increased performance. If memory use is a factor
# set the Minimum heap size to a lower number and let the JVM adjust when necessary.
JAVAOPTS=' -Xms6g -Xmx6g'
;;
esac

3. imported the old points from H2
4. API calls for stats are completing in 1/20th the time

Will reply if the CPU load issue happens again.

Regards,
Tim

Re the java heap space issue, going to give the Mango VM another 4GB of RAM, plus swap, and try again.

Hi,

Thanks for the fast response. The ma.log file had entries similar to the following at the time of the problem:

--
ERROR 2016-05-17 14:22:37,909 (com.serotonin.m2m2.web.mvc.rest.v1.exception.RestExceptionHandler.handleMangoError:58) - Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Database may be already in use: "L
ocked by another process". Possible solutions: close all other connection(s); use the server mode [90020-181]
ERROR 2016-05-17 14:22:43,606 (com.serotonin.m2m2.web.mvc.rest.v1.exception.RestExceptionHandler.handleMangoError:58) - Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Database may be already in use: "L
ocked by another process". Possible solutions: close all other connection(s); use the server mode [90020-181]
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Database may be already in use: "Locked by another process". Possible solutions: close all othe
r connection(s); use the server mode [90020-181]
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)

The high CPU core utilisation activity appears to be correlated to API calls from the dashboard module (when retrieving various stats and the current/live values for 6 points). There are 19 million rows in the pointValues table. After logging this post, noticed the mangoNoSqlDatabase module was not installed - so have just installed it.

So migrating data from the old H2 store pointValue table into the new mangoNoSqlDatabase store may help. This however may have hit a wall, as I have received the following error: "GC overhead limit exceeded" twice (perhaps a few minutes apart), after processing 2.1 million rows.

I can now see entries like this while the migration is failing:

--
Caused by: java.lang.OutOfMemoryError: Java heap space
ERROR 2016-05-17 15:42:02,996 (org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException:367) - Failed to invoke @ExceptionHandler method: protected org.springframework.http
.ResponseEntity<java.lang.Object> com.serotonin.m2m2.web.mvc.rest.v1.exception.RestExceptionHandler.handleMangoError(java.lang.Exception,org.springframework.web.context.request.WebRequest)
java.lang.ClassCastException: java.lang.OutOfMemoryError cannot be cast to java.lang.Exception
at com.serotonin.m2m2.web.mvc.rest.v1.exception.RestExceptionHandler.handleExceptionInternal(RestExceptionHandler.java:78)
at com.serotonin.m2m2.web.mvc.rest.v1.exception.RestExceptionHandler.handleMangoError(RestExceptionHandler.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

--

So a few further questions:

1. how can I solve the out of memory error while migrating data from the h2 instance into the new mangonosqlmodule?

2. is it safe to retry the import of data from pointValue, or will duplication occur in the destination?

3. would prefer the server to not shut down in 8 hours (and then drop data), I suspect we have the license (but not sure which account it is associated with. Is there a temporary way I can keep this server up? If there is no way to stop the shutdown, I can switch off the module, but then what will happen to data which was received into the modules nosql store, will that need to be re-sent from the child MangoES devices?

Regards,
Tim

Hi,

We are encountering unusual behaviour of our Mango process (running on a ubuntu 14.04 VM). CPU on the VM jumps up to consume all (4) cores. Disk and memory utilisation are low. This is a lightly loaded installation, with < 7 mangoES publishers pushing data to the instance. When this happens we are unable to view the management web interface, and javascript API calls are timing out (e.g. via the dashboard module). Attached are the stack traces from the two times this happened today, prior to today the issue occurred two weeks ago. The problem has occurred while running the latest version of the software and modules, as well as the prior version. The only way to bring it back is to kill the java process (with kill -9), then start Mango (service mango start) . What options do we have to track down the cause and stop this from happening?

Output of jstack when the CPU cores are busy and the web interface is timing out:

0_1463468117135_stack-trace.log

There seem to be a large number of threads working with JdbcConnectionPool when the problem occurs, does this suggest the embedded DB is running slowly or locked in some way?

Cheers,
Tim