Corruptions on shard, Full Disk Space
-
I've found the offending file:
What can I do to this file?
When mango runs normally, it does not consume this must disk space, so I'm not sure how to treat this file.
-
If you have a core database backup zip in your Mango/backup directory then you can restore to that H2 backup. Doing a backup then a restore often shrinks an H2 database, sometimes significantly.
To delete all point values on a default MangoES you can run,
rm -r /opt/mango/databases/mangoTSDB
while Mango is off. -
Can I delete the mah2.h2.db and mah2.lock.db and mah2.trace.db files as well?
They're taking up all the space.
-
I was able to delete the folder contents and restore the latest H2 database.
I am seeing these errors in the log now:
WARN 2017-12-12T12:56:07,269 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full WARN 2017-12-12T12:56:27,268 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full WARN 2017-12-12T12:56:47,269 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full WARN 2017-12-12T12:57:07,268 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full WARN 2017-12-12T12:57:27,268 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full WARN 2017-12-12T12:57:36,559 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2017-12-12T12:57:36,560 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2017-12-12T12:57:36,561 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:727) - 2 BWB Task Failures, first is: Task Queue Full ERROR 2017-12-12T12:57:36,899 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:727) - 3 BWB Task Failures, first is: Task Queue Full WARN 2017-12-12T12:57:47,269 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full WARN 2017-12-12T12:58:07,268 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: Work item monitor because Task Queue Full
Do I just need to wait a bit longer for something to finish?
-
Nice!
I could say for sure from that information. I would take some thread dumps if it were me trying to look at that one. You could always restart if you thought it were in a funny state.
-
After a reboot, I mango is failing at the Initializing H2 connection manager stage:
INFO 2017-12-12T13:10:57,056 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:42) - Initializing H2 connection manager FATAL 2017-12-12T13:11:36,998 (com.serotonin.m2m2.db.DatabaseProxy.initialize:194) - Exception initializing database proxy: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValu$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); ]; nested exception is org.h2.jdbc$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:231) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:443) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:149) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:168) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.runScriptFile(DatabaseProxy.java:274) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:212) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106) [mango-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:656) [ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:262) [ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Main.main(Main.java:125) [ma-priv-3.2.2.jar:?] Caused by: org.h2.jdbc.JdbcSQLException: Table "SYSTEMSETTINGS" already exists; SQL statement: CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.ddl.CreateTable.update(CreateTable.java:111) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.CommandContainer.update(CommandContainer.java:78) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:185) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:159) ~[h2-1.4.181.jar:1.4.181] at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:435) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] ... 9 more ERROR 2017-12-12T13:11:37,013 (com.serotonin.m2m2.Main.main:129) - Error during initialization org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); ]; nested exception is org.h2.jdbc$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:231) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:443) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:149) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:168) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.runScriptFile(DatabaseProxy.java:274) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:212) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:656) ~[ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:262) ~[ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Main.main(Main.java:125) [ma-priv-3.2.2.jar:?] Caused by: org.h2.jdbc.JdbcSQLException: Table "SYSTEMSETTINGS" already exists; SQL statement: CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.ddl.CreateTable.update(CreateTable.java:111) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.CommandContainer.update(CommandContainer.java:78) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:185) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:159) ~[h2-1.4.181.jar:1.4.181] at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:435) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] ... 9 more INFO 2017-12-12T13:11:37,021 (com.serotonin.m2m2.Lifecycle.terminate:361) - Mango Lifecycle terminating... INFO 2017-12-12T13:11:38,247 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:115) - Terminating NoSQL Batch Write Manager. INFO 2017-12-12T13:11:38,315 (com.serotonin.m2m2.Lifecycle.terminate:480) - Mango Lifecycle terminated.
-
I tried to re-delete the database to try an earlier backup, and I am still getting this error:
FATAL 2017-12-12T13:57:13,192 (com.serotonin.m2m2.db.DatabaseProxy.initialize:194) - Exception initializing database proxy: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValu$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); ]; nested exception is org.h2.jdbc$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:231) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:443) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:149) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:168) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.runScriptFile(DatabaseProxy.java:274) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:212) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106) [mango-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:656) [ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:262) [ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Main.main(Main.java:125) [ma-priv-3.2.2.jar:?] Caused by: org.h2.jdbc.JdbcSQLException: Table "SYSTEMSETTINGS" already exists; SQL statement: CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.ddl.CreateTable.update(CreateTable.java:111) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.CommandContainer.update(CommandContainer.java:78) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:185) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:159) ~[h2-1.4.181.jar:1.4.181] at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:435) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] ... 9 more ERROR 2017-12-12T13:57:13,204 (com.serotonin.m2m2.Main.main:129) - Error during initialization org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); ]; nested exception is org.h2.jdbc$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.ddl.CreateTable.update(CreateTable.java:111) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.CommandContainer.update(CommandContainer.java:78) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:185) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:159) ~[h2-1.4.181.jar:1.4.181] at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:435) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] ... 9 more ERROR 2017-12-12T13:57:13,204 (com.serotonin.m2m2.Main.main:129) - Error during initialization org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); ]; nested exception is org.h2.jdbc$ CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:231) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:419) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:443) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:149) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.H2Proxy.runScript(H2Proxy.java:168) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.runScriptFile(DatabaseProxy.java:274) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:212) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:656) ~[ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:262) ~[ma-priv-3.2.2.jar:?] at com.serotonin.m2m2.Main.main(Main.java:125) [ma-priv-3.2.2.jar:?] Caused by: org.h2.jdbc.JdbcSQLException: Table "SYSTEMSETTINGS" already exists; SQL statement: CREATE TABLE systemSettings ( settingName varchar(64) NOT NULL, settingValue longtext, PRIMARY KEY (settingName) ); [42101-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.181.jar:1.4.181] at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.ddl.CreateTable.update(CreateTable.java:111) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.CommandContainer.update(CommandContainer.java:78) ~[h2-1.4.181.jar:1.4.181] at org.h2.command.Command.executeUpdate(Command.java:254) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:185) ~[h2-1.4.181.jar:1.4.181] at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:159) ~[h2-1.4.181.jar:1.4.181] at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:435) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) ~[spring-jdbc-4.3.7.RELEASE.jar:4.3.7.RELEASE] ... 9 more INFO 2017-12-12T13:57:13,206 (com.serotonin.m2m2.Lifecycle.terminate:361) - Mango Lifecycle terminating... INFO 2017-12-12T13:57:13,407 (com.infiniteautomation.nosql.MangoNoSqlProxy.shutdown:115) - Terminating NoSQL Batch Write Manager. INFO 2017-12-12T13:57:13,451 (com.serotonin.m2m2.Lifecycle.terminate:480) - Mango Lifecycle terminated.
What can I do now?
-
Huh. That's odd. One can delete an H2 database by deleting the Mango/databases/mah2.h2.db file, and it definitely will be deleted by that if Mango isn't running.
The check for a new database for Mango is the existence of the users table. So, you could definitely get this message if you had only dropped that table. or if your first initialization was unable to create the users table.
-
Ah understood, I deleted mah2.h2.db and mango was back to new again.
I attempted another database restore, with a version one prior to the last attempt with the same problem of Task Queue Full =(
-
Ahh success!!! The third backup seems to be have been good. Thanks for the help. I'll let you know if any further developments appear.
-
Glad to hear it! You're most welcome.