Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
How to clean a very large H2 Database
-
according to your suggestion now the env.properties is the following:
db.type=mysql
db.url=jdbc:mysql://127.0.0.1:3306/mango
db.username=solergy
db.password=secret
db.mysqldump=
db.mysql=convert.db.type=h2
convert.db.url=jdbc:h2:${ma.home}/databases/mah2
convert.db.username=
convert.db.password=the mango database for user solergy is created, granted and up and running:
but still mango doesn't start:
INFO 2018-10-04 10:27:16,771 (com.serotonin.m2m2.db.BasePooledProxy.initializeI mpl:36) - Initializing pooled connection manager FATAL 2018-10-04 10:27:18,016 (com.serotonin.m2m2.db.DatabaseProxy.initialize:19 0) - Unable to connect to database of type MYSQL org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Co nnection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not create connection to database serve r.) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(Dat aSourceUtils.java:80) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java: 615) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:68 0) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:71 2) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:72 2) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJ dbcTemplate.java:39) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJ dbcTemplate.java:63) at com.serotonin.db.DaoUtils.queryForObject(DaoUtils.java:336) at com.serotonin.m2m2.db.dao.SystemSettingsDao.getValue(SystemSettingsDa o.java:158) at com.serotonin.m2m2.db.dao.SystemSettingsDao.getIntValue(SystemSetting sDao.java:178) at com.serotonin.m2m2.db.upgrade.DBUpgrade.checkUpgrade(DBUpgrade.java:5 1) at com.serotonin.m2m2.db.upgrade.DBUpgrade.checkUpgrade(DBUpgrade.java:3 4) at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:180 ) at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:609) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:287) at com.serotonin.m2m2.Main.main(Main.java:115) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableCon nectionFactory (Could not create connection to database server.) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFacto ry(BasicDataSource.java:1549) at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSou rce.java:1388) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource .java:1044) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(D ataSourceUtils.java:111) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(Dat aSourceUtils.java:77) ... 15 more Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou rce) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2 568) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou rce) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java :346) at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(Driv erConnectionFactory.java:38) at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(Poolable ConnectionFactory.java:582) at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(Bas icDataSource.java:1556) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFacto ry(BasicDataSource.java:1545) ... 19 more Caused by: java.lang.NullPointerException at com.mysql.jdbc.ConnectionImpl.getServerCharacterEncoding(ConnectionIm pl.java:3279) at com.mysql.jdbc.MysqlIO.sendConnectionAttributes(MysqlIO.java:1940) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(My sqlIO.java:1866) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1252) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2 519) ... 33 more ERROR 2018-10-04 10:27:18,400 (com.serotonin.m2m2.Main.main:119) - Error during initialization org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Co nnection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not create connection to database serve r.) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(Dat aSourceUtils.java:80) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java: 615) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:68 0) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:71 2) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:72 2) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJ dbcTemplate.java:39) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJ dbcTemplate.java:63) at com.serotonin.db.DaoUtils.queryForObject(DaoUtils.java:336) at com.serotonin.m2m2.db.dao.SystemSettingsDao.getValue(SystemSettingsDa o.java:158) at com.serotonin.m2m2.db.dao.SystemSettingsDao.getIntValue(SystemSetting sDao.java:178) at com.serotonin.m2m2.db.upgrade.DBUpgrade.checkUpgrade(DBUpgrade.java:5 1) at com.serotonin.m2m2.db.upgrade.DBUpgrade.checkUpgrade(DBUpgrade.java:3 4) at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:180 ) at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:609) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:287) at com.serotonin.m2m2.Main.main(Main.java:115) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableCon nectionFactory (Could not create connection to database server.) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFacto ry(BasicDataSource.java:1549) at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSou rce.java:1388) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource .java:1044) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(D ataSourceUtils.java:111) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(Dat aSourceUtils.java:77) ... 15 more Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou rce) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2 568) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou rce) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java :346) at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(Driv erConnectionFactory.java:38) at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(Poolable ConnectionFactory.java:582) at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(Bas icDataSource.java:1556) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFacto ry(BasicDataSource.java:1545) ... 19 more Caused by: java.lang.NullPointerException at com.mysql.jdbc.ConnectionImpl.getServerCharacterEncoding(ConnectionIm pl.java:3279) at com.mysql.jdbc.MysqlIO.sendConnectionAttributes(MysqlIO.java:1940) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(My sqlIO.java:1866) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1252) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2 519) ... 33 more INFO 2018-10-04 10:27:18,448 (com.serotonin.m2m2.Lifecycle.terminate:353) - Man go Lifecycle terminating... Exception in thread "high-pool-1-thread-1" java.lang.NullPointerException at com.serotonin.m2m2.CoreLicenseDefinition$2.run(CoreLicenseDefinition. java:149) at com.serotonin.timer.sync.Synchronizer$TaskWrapper.run(Synchronizer.ja va:155) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) INFO 2018-10-04 10:27:18,625 (com.serotonin.m2m2.db.BasePooledProxy.terminateIm pl:110) - Stopping database INFO 2018-10-04 10:27:18,634 (com.serotonin.m2m2.Lifecycle.terminate:453) - Man go Lifecycle terminated. -
Did you perchance change the
db.streamproperty totrue?I don't believe we've done much testing against MySQL 8 (thanks for posting the version information!), so it's possible there are issues related to that. You could try and older version and see if that works out of the box. I know 5.5 has been used a great deal.
-
Hy,
db.stream is unchanged, this is remaining properties for db on env.properties:db.forceUseIndex=true
db.fetchSize=-1
db.stream=falsemaybe I've to choose a certain character encoding in mysql?
Caused by: java.lang.NullPointerException
at com.mysql.jdbc.ConnectionImpl.getServerCharacterEncoding(ConnectionIm
pl.java:3279) -
No, I wouldn't think so. It's never come up in the past, but I think few people were using the MySQL server version released last April and they may be new issues. I did search the error a little and find something in the code that may have something to do with it, but I do intend to look into this more and try to run on MySQL 8.
-
I installed MySQL server 8 on my Windows machine and found I could convert an H2 database without issue. I also tried telling Java that it was in Italy (in case locale was affecting the getServerCharacterEncoding method) but it still converted without issue.
maybe I've to choose a certain character encoding in mysql?
This is done for you in the MySQLProxy class, it modifies the db.url for UTF-8:
https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/serotonin/m2m2/db/MySQLProxy.java#L28A suggestion I saw is to try the latest JDBC. This is a 2.8 system? I would guess you have a very outdated Mango/lib/mysql-connector-java-x.y.z.jar file. You could try the version bundled with the latest Mango 3, or you could download the latest MySQL jdbc jar and replace the outdated with the new. You can download the latest here: https://dev.mysql.com/downloads/connector/j/5.1.html
-
Ok,
after driver change, installation is started but after a long while during migration I had this error,
can I do something to avoid it and continue migration?Using Java at java INFO 2018-10-05 08:57:59,936 (com.serotonin.m2m2.Main.main:82) - Starting Mango 2.8.4 with Schema Version: 13 INFO 2018-10-05 08:58:01,596 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'asciiFile', v1.3.0 by Infinite Automation INFO 2018-10-05 08:58:01,656 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'BACnet', v2.3.0 by Infinite Automation Software INFO 2018-10-05 08:58:01,666 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'controlcore', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:01,676 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'dashboards', v3.3.0 by Infinite Automation Systems Inc. INFO 2018-10-05 08:58:01,826 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'dataFile', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:01,846 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'dataImport', v1.5.0 by Infinite Automation Software INFO 2018-10-05 08:58:01,856 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'dataPointDetailsView', v1.1.0 by Infinite Automation INFO 2018-10-05 08:58:01,866 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'dnp3', v1.7.0 by Infinite Automation Software INFO 2018-10-05 08:58:01,876 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'egauge', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:01,886 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'excelReports', v1.1.1 by Infinite Automation Software INFO 2018-10-05 08:58:02,006 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'graphicalViews', v1.7.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,316 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'http', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,366 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'internal', v1.6.1 by Infinite Automation Software INFO 2018-10-05 08:58:02,386 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'log4jDS', v1.2.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,396 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'log4JReset', v1.2.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,406 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'loggingConsole', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:02,426 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'maintenanceEvents', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,436 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'mangoApi', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:02,446 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'mbus', v1.2.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,456 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'meta', v2.3.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,476 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'modbus', v1.7.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,536 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'onewire', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,606 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'opcda', v1.7.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,616 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'pakbus', v1.1.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,676 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'persistent', v1.7.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,706 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'pointLinks', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,726 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'pop3', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,736 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'reports', v2.3.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,796 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'scheduledEvents', v1.6.0 by Infinite Automation INFO 2018-10-05 08:58:02,886 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'scripting', v1.2.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,906 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'serial', v1.3.0 by Infinite Automation INFO 2018-10-05 08:58:02,926 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'snmp', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,966 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'sqlConsole', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,976 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'sqlds', v1.7.0 by Infinite Automation Software INFO 2018-10-05 08:58:02,996 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'ssh', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:03,006 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'sstGlobalScripts', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:03,016 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'sstGraphics', v1.2.0 by Infinite Automation Software INFO 2018-10-05 08:58:03,056 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'sstTheme', v1.2.0 by Infinite Automation Software INFO 2018-10-05 08:58:03,066 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'TCPIP', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:03,076 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'templateConfig', v1.1.0 by Infinite Automation Software INFO 2018-10-05 08:58:03,106 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'twilio', v1.2.0 by Infinite Automation INFO 2018-10-05 08:58:03,116 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'virtualDS', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:03,126 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'vmstat', v1.6.0 by Infinite Automation Software INFO 2018-10-05 08:58:03,136 (com.serotonin.m2m2.Main.loadModules:416) - Loadin g module 'watchlists', v1.8.0 by Infinite Automation Software WARN 2018-10-05 08:58:05,118 (com.serotonin.m2m2.Main.loadModules:623) - Unused classes entry: com.infiniteautomation.mango.pakbus.dwr.PakBusEditDwr INFO 2018-10-05 08:58:05,368 (com.serotonin.m2m2.Lifecycle.initialize:216) - In itializing Spring Object Mapper INFO 2018-10-05 08:58:08,428 (com.serotonin.m2m2.Lifecycle.loadLic:471) - Check ing license... INFO 2018-10-05 08:58:09,298 (com.serotonin.m2m2.db.BasePooledProxy.initializeI mpl:36) - Initializing pooled connection manager Fri Oct 05 08:58:10 CEST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ a nd 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the v erifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. Fri Oct 05 08:58:11 CEST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ a nd 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the v erifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. INFO 2018-10-05 09:02:03,183 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:42) - Initializing H2 connection manager WARN 2018-10-05 09:02:03,718 (com.serotonin.m2m2.db.DBConvert.execute:41) - Run ning database conversion from H2 to MYSQL WARN 2018-10-05 09:02:08,243 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table systemSettings... WARN 2018-10-05 09:02:08,843 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table systemSettings. 78 records copied. WARN 2018-10-05 09:02:08,844 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table users... WARN 2018-10-05 09:02:09,074 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table users. 5 records copied. WARN 2018-10-05 09:02:09,080 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table userComments... WARN 2018-10-05 09:02:09,089 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table userComments. 0 records copied. WARN 2018-10-05 09:02:09,095 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table mailingLists... WARN 2018-10-05 09:02:09,427 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table mailingLists. 1 records copied. WARN 2018-10-05 09:02:09,434 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table mailingListInactive... WARN 2018-10-05 09:02:09,442 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table mailingListInactive. 0 records copied. WARN 2018-10-05 09:02:09,445 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table mailingListMembers... WARN 2018-10-05 09:02:09,770 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table mailingListMembers. 1 records copied. WARN 2018-10-05 09:02:09,776 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table templates... WARN 2018-10-05 09:02:10,642 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table templates. 4 records copied. WARN 2018-10-05 09:02:10,648 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table dataSources... WARN 2018-10-05 09:02:11,113 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table dataSources. 12 records copied. WARN 2018-10-05 09:02:11,117 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table dataPoints... WARN 2018-10-05 09:02:38,963 (com.serotonin.m2m2.db.DBConvert.copyTable:136) - --> Finished converting table dataPoints. 201 records copied. WARN 2018-10-05 09:02:38,972 (com.serotonin.m2m2.db.DBConvert.copyTable:87) - --> Converting table pointValues... ERROR 2018-10-05 10:07:39,219 (com.serotonin.m2m2.Main.main:119) - Error during initialization com.serotonin.ShouldNeverHappenException: java.sql.SQLException: 'NaN' is not a valid numeric or approximate numeric value at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:134 ) at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:609) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:287) at com.serotonin.m2m2.Main.main(Main.java:115) Caused by: java.sql.SQLException: 'NaN' is not a valid numeric or approximate nu meric value at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861) at com.mysql.jdbc.PreparedStatement.setDouble(PreparedStatement.java:330 0) at com.mysql.jdbc.PreparedStatement.setNumericObject(PreparedStatement.j ava:3531) at com.mysql.jdbc.PreparedStatement.setObject(PreparedStatement.java:369 5) at com.mysql.jdbc.JDBC42PreparedStatement.setObject(JDBC42PreparedStatem ent.java:99) at com.mysql.jdbc.PreparedStatement.setObject(PreparedStatement.java:362 0) at com.mysql.jdbc.JDBC42PreparedStatement.setObject(JDBC42PreparedStatem ent.java:83) at org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(Delegat ingPreparedStatement.java:166) at org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(Delegat ingPreparedStatement.java:166) at com.serotonin.m2m2.db.DBConvert.copyTable(DBConvert.java:119) at com.serotonin.m2m2.db.DBConvert.execute(DBConvert.java:54) at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:131 ) ... 3 more INFO 2018-10-05 10:07:39,224 (com.serotonin.m2m2.Lifecycle.terminate:353) - Man go Lifecycle terminating... Exception in thread "high-pool-1-thread-1" java.lang.NullPointerException at com.serotonin.m2m2.CoreLicenseDefinition$2.run(CoreLicenseDefinition. java:149) at com.serotonin.timer.sync.Synchronizer$TaskWrapper.run(Synchronizer.ja va:155) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) INFO 2018-10-05 10:07:39,279 (com.serotonin.m2m2.db.BasePooledProxy.terminateIm pl:110) - Stopping database INFO 2018-10-05 10:07:39,281 (com.serotonin.m2m2.Lifecycle.terminate:453) - Man go Lifecycle terminated.I think the problem could be that pointvalue is nullable
you suggest me to clean values with an sql like
"delete from pointvalues where pointvalue is null;"
? -
That looks like it would be caused by MySQL not supporting NaN, Infinity, or -Infinity as valid values in a column described as a double. If one has Mango already running on MySQL and attempts to set the value of a point to NaN, you get 0, if you do Infinity, you get Double.MAX_VALUE
But for converting, it doesn't look like there's any handling for the differences in what values are acceptable in one database or another. You could fix this by removing or changing these values before doing the conversion, by running this either through the Mango/bin/h2-web-console script or on the /sqlConsole.shtm page of Mango running on the old database:
-- for use to remove values from the H2 pointValues table that cannot be brought into MySQL delete from pointValues where pointValue=CAST('NaN' AS DOUBLE PRECISION) delete from pointValues where pointValue=CAST('Infinity' AS DOUBLE PRECISION); delete from pointValues where pointValue=CAST('-Infinity' AS DOUBLE PRECISION)Most commercial users use the NoSQL database to avoid problems with larger point values tables (since in Mango 3 all licensed cores can use the NoSQL database, unlike in 2.x where it is a separate license). That's probably why the issue has not been significant in the past. If you want to use SQL to store your point values, you definitely have to keep that table size under control or Mango will begin to slow down.
-
The db convertion from H2 to mysql is running (10 hours just to convert table pointValues) but I think there's something strange, at the moment I have only data from last 80 days concerning about 100 variables stored at 1 sample/minute, so I think 80x100x60x24=11,5Million rows in table pointValues, instead the table is around 40Million points, corresponding to an H2 database of 3.5GB corresponding also to a mysql pointValues table using 3.5GB.
To delete data before last 80 days (originally 5 months) I used the datasource purge for each used datasource. The purge from admin settings seems not to work on a big H2 database or in any case it tooks a very long time. -
I'm searching for the reason why my H2 database is growing so fast,
I think that this "Logging TYpe = All Data" could be an error:
but, looking at the data, it seems that it takes a new value each minute
according to Data Source logging
so, it seems, it is not a problem, is it right?
-
If you are not using the NoSQL database, yes pointvalues are a contributing factor to the database expansion (and in some people's cases probably the only factor). But, the issue is here: https://github.com/infiniteautomation/ma-core-public/issues/1344
The likelihood is that sometime (probably sooner than later, maybe 3.6?) we will have a DB upgrade that does the necessary side work to upgrade an PageStore H2 database created in 196 to 197, as referenced in the issues referenced in that issue. In version 197 of H2 the problem of saving a LOB not reusing space has been resolved.
But, if something like your pointValues pointValueAnnotations or events tables are just very full, the next version of the database may not help in your case.
-
This post is deleted! -
I used this query to search for datapoints to reduce:
SELECT count(*) as totale, dataPointId, d.name FROM pointValues p, dataPoints d where p.dataPointID = d.id group by p.dataPointId , d.name order by totale desc;I will tune the results of this query,
I also founded 8500000 records in PointValueAnnotations all like the following row
TEXTPOINTVALUESHORT | TEXTPOINTVALUELONG | SOURCEMESSAGE --------------------+--------------------+---------------------- | | scripting.annotation|are these rows useful for something or I can delete it?
Who produce these rows?Thanks, Antonio
-
That is a consequence of a Scripting data source setting points that belong to other data sources, so that those values can be seen to have come from elsewhere after the fact. Yes you could delete those rows, you may possibly have to do a backup and a restore to reclaim the disk space.
-
Thanks, I will delete such rows but I think other similar rows will be created in the future, there's a way to avoid this?
Antonio -
Our solution was to code up the NoSQL module so that people wouldn't have trouble with their pointValues tables. That's the surefire way to avoid it.
-
At the end tuning datapoints purge I'm arrived to a reasonable size for H2 database, about 1.5GB, now I'm testing the mysql porting of this database. I've 2 troubles:
- backupMysqlWithOutDatabase:442) - java.io.IOException: Cannot run program "mysqldump"
the mysql setting on env.properties is the following:
# MySQL database settings. Your MySQL instance must already be running and configured before this can be used. db.type=mysql db.url=jdbc:mysql://127.0.0.1:3306/mango db.username=solergy db.password=secret db.mysqldump= db.mysql=what is required in "db.mysqldump" and "db.mysql" fileds?
- When using excel reports purge I've the following exception
what's the problem and how to solve it?
WARN 2018-11-06 10:23:35,344 (com.infiniteautomation.mango.web.mvc.rest.v1.exce lreports.ExcelReportRestController.purge:547) - java.lang.NullPointerException at java.io.File.<init>(Unknown Source) at com.infiniteautomation.mango.excelreports.ExcelReportsCommon.getRepor t(ExcelReportsCommon.java:172) at com.infiniteautomation.mango.excelreports.dao.ExcelReportDao.purgeRep ortsBefore(ExcelReportDao.java:190) at com.infiniteautomation.mango.web.mvc.rest.v1.excelreports.ExcelReport RestController.purge(ExcelReportRestController.java:543) at com.infiniteautomation.mango.web.mvc.rest.v1.excelreports.ExcelReport RestController.purgeUsingSettings(ExcelReportRestController.java:509) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvok e(InvocableHandlerMethod.java:221) at org.springframework.web.method.support.InvocableHandlerMethod.invokeF orRequest(InvocableHandlerMethod.java:137) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocabl eHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingH andlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:806) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingH andlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:729) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapt er.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(Dispatch erServlet.java:959) at org.springframework.web.servlet.DispatcherServlet.doService(Dispatche rServlet.java:893) at org.springframework.web.servlet.FrameworkServlet.processRequest(Frame workServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkSe rvlet.java:894) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkSer vlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808 ) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet Handler.java:1669) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.j ava:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet Handler.java:1652) at org.springframework.web.filter.CharacterEncodingFilter.doFilterIntern al(CharacterEncodingFilter.java:85) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR equestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet Handler.java:1652) at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterIntern al(ShallowEtagHeaderFilter.java:87) at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterIn ternal(MangoShallowEtagHeaderFilter.java:47) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR equestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet Handler.java:1652) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInter ceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInter ceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.do Filter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFi lter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticati onFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.servletapi.SecurityContextHolderAwar eRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter .doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.logout.LogoutFilter.d oFilter(LogoutFilter.java:121) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at com.serotonin.m2m2.web.mvc.spring.security.CsrfHeaderFilter.doFilterI nternal(CsrfHeaderFilter.java:41) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR equestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(Csr fFilter.java:124) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR equestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterIn ternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR equestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFi lter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManage rIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR equestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain. doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(Fi lterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChai nProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(D elegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(Delegat ingFilterProxy.java:262) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet Handler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java :585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav a:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl er.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl er.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: 515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle r.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle r.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont extHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper .java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.jav a:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java :540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo l.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool .java:555) at java.lang.Thread.run(Unknown Source)Thanks, Antonio
-
what is required in "db.mysqldump" and "db.mysql" fileds?
The path to invoke the mysqldump or mysql programs (or just the executables if they're on the Mango user's $Path). The error,
backupMysqlWithOutDatabase:442) - java.io.IOException: Cannot run program "mysqldump"
Indicates mysqldump is not on the path.
what's the problem and how to solve it?
That issue appears to have been solved in version 3.1.3 of the excel reports module. You will probably needs to work around it by manually deleting from the excelReports table and deleting from Mango/web/modules/excelReports/web/report-data/ as appropriate. It may only be one report with a null filename, you could perhaps investigate the excelReports SQL table to see.
