No space left on device // (org.tinytsdb.TinyTSDBException: java.io.IOException: No space left on device)
-
-
INFO 2017-05-30 16:47:35,858 (com.serotonin.m2m2.Main.main:82) - Starting Mango 2.7.12 with Schema Version: 12 INFO 2017-05-30 16:47:37,464 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sqlds', v1.6.2 by Infinite Automation Software INFO 2017-05-30 16:47:37,546 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'TCPIP', v1.1.5 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,558 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'asciiFile', v1.2.1 by Infinite Automation INFO 2017-05-30 16:47:37,570 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sqlConsole', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,583 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'galil', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,598 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstGlobalScripts', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,610 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mbus', v1.1.3 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,629 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'log4jDS', v1.1.2 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,640 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'persistent', v1.6.5 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,652 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'vmstat', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,663 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoApi', v1.1.7 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,674 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pointLinks', v1.5.5 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,687 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'log4JReset', v1.1.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,691 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstTheme', v1.1.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,698 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'jmxds', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:37,715 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'twilio', v1.1.2 by (unknown vendor) INFO 2017-05-30 16:47:37,731 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoESConfiguration', v1.1.7 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,746 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'meta', v2.2.4 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,759 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'excelReports', v1.0.6 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,782 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'snmp', v1.5.3 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,793 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'http', v1.5.2 by Infinite Automation Software INFO 2017-05-30 16:47:37,810 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'controlcore', v1.1.1 by Infinite Automation INFO 2017-05-30 16:47:37,821 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'graphicalViews', v1.6.7 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,869 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'scripting', v1.1.2 by Infinite Automation Software INFO 2017-05-30 16:47:37,882 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'watchlists', v1.7.4 by Infinite Automation Software INFO 2017-05-30 16:47:37,904 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataPointDetailsView', v1.0.5 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,917 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'modbus', v1.6.5 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:37,933 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dashboards', v3.2.2 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:38,175 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'onewire', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:38,196 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'virtualDS', v1.5.2 by Infinite Automation Software INFO 2017-05-30 16:47:38,208 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataFile', v1.1.1 by Infinite Automation INFO 2017-05-30 16:47:38,230 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'ssh', v1.1.1 by Infinite Automation INFO 2017-05-30 16:47:38,242 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dnp3', v1.6.2 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:38,254 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataImport', v1.4.1 by Infinite Automation Software INFO 2017-05-30 16:47:38,265 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pop3', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:38,276 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'jspViews', v2.1.1 by Infinite Automation Software INFO 2017-05-30 16:47:38,292 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'templateConfig', v1.0.8 by Infinite Automation Software INFO 2017-05-30 16:47:38,310 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'BACnet', v2.2.4 by Infinite Automation Software INFO 2017-05-30 16:47:38,325 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pid', v1.1.1 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:38,339 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'zwave', v1.1.2 by Infinite Automation INFO 2017-05-30 16:47:38,378 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'egauge', v1.1.1 by Infinite Automation INFO 2017-05-30 16:47:38,389 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'openv4j', v1.1.2 by aploese INFO 2017-05-30 16:47:38,393 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'scheduledEvents', v1.5.2 by (unknown vendor) INFO 2017-05-30 16:47:38,404 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'loggingConsole', v1.1.1 by Infinite Automation INFO 2017-05-30 16:47:38,415 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstGraphics', v1.1.1 by Infinite Automation Software INFO 2017-05-30 16:47:38,452 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoNoSqlDatabase', v1.2.3 by Infinite Automation Software INFO 2017-05-30 16:47:38,465 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'reports', v2.2.3 by Infinite Automation Systems Inc. INFO 2017-05-30 16:47:38,479 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'maintenanceEvents', v1.5.1 by Infinite Automation Software INFO 2017-05-30 16:47:38,491 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'serial', v1.2.1 by Infinite Automation INFO 2017-05-30 16:47:38,503 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'internal', v1.5.3 by Infinite Automation Software INFO 2017-05-30 16:47:41,325 (com.infiniteautomation.mangoApi.MangoApiReloadingProperties.checkForReload:146) - (mangoApiHeaders) Found updated file at /opt/mango/web/modules/mangoApi/classes/mangoApiHeaders.properties. Reloading properties INFO 2017-05-30 16:47:41,714 (com.serotonin.m2m2.Lifecycle.loadLic:464) - Checking license... INFO 2017-05-30 16:47:42,019 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:42) - Initializing H2 connection manager FATAL 2017-05-30 16:47:44,554 (com.serotonin.m2m2.db.DatabaseProxy.initialize:190) - Unable to connect to database of type H2 org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:244) at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:204) at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106) at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:602) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:280) at com.serotonin.m2m2.Main.main(Main.java:116) Caused by: org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) at org.h2.message.DbException.get(DbException.java:168) at org.h2.store.FileLock.getExceptionFatal(FileLock.java:456) at org.h2.store.FileLock.save(FileLock.java:225) at org.h2.store.FileLock.lockFile(FileLock.java:352) at org.h2.store.FileLock.lock(FileLock.java:135) at org.h2.engine.Database.open(Database.java:631) at org.h2.engine.Database.openDatabase(Database.java:266) at org.h2.engine.Database.<init>(Database.java:260) at org.h2.engine.Engine.openSession(Engine.java:60) at org.h2.engine.Engine.openSession(Engine.java:167) at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145) at org.h2.engine.Engine.createSession(Engine.java:128) at org.h2.engine.Engine.createSession(Engine.java:26) at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347) at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108) at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92) at org.h2.Driver.connect(Driver.java:72) at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190) at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353) at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385) at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227) at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ... 12 more Caused by: java.io.FileNotFoundException: /opt/mango/databases/mah2.lock.db (No space left on device) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:290) at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:233) at org.h2.store.FileLock.save(FileLock.java:213) ... 33 more ERROR 2017-05-30 16:47:44,562 (com.serotonin.m2m2.Main.main:120) - Error during initialization org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39) at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63) at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:244) at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:204) at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106) at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:602) at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:280) at com.serotonin.m2m2.Main.main(Main.java:116) Caused by: org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) at org.h2.message.DbException.get(DbException.java:168) at org.h2.store.FileLock.getExceptionFatal(FileLock.java:456) at org.h2.store.FileLock.save(FileLock.java:225) at org.h2.store.FileLock.lockFile(FileLock.java:352) at org.h2.store.FileLock.lock(FileLock.java:135) at org.h2.engine.Database.open(Database.java:631) at org.h2.engine.Database.openDatabase(Database.java:266) at org.h2.engine.Database.<init>(Database.java:260) at org.h2.engine.Engine.openSession(Engine.java:60) at org.h2.engine.Engine.openSession(Engine.java:167) at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145) at org.h2.engine.Engine.createSession(Engine.java:128) at org.h2.engine.Engine.createSession(Engine.java:26) at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347) at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108) at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92) at org.h2.Driver.connect(Driver.java:72) at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190) at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353) at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385) at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227) at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ... 12 more Caused by: java.io.FileNotFoundException: /opt/mango/databases/mah2.lock.db (No space left on device) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:290) at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:233) at org.h2.store.FileLock.save(FileLock.java:213) ... 33 more INFO 2017-05-30 16:47:44,565 (com.serotonin.m2m2.Lifecycle.terminate:344) - Mango Lifecycle terminating...
-
Sorry for the title. It was an older issue with this exception, but it seems not affect the current error. Here is still the "tsdb" exception:
ERROR 2017-05-13 11:58:46,511 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.execute:110) - Error saving point value. org.tinytsdb.TinyTSDBException: java.io.IOException: No space left on device at org.tinytsdb.impl.TinyTSDBImpl.write(TinyTSDBImpl.java:309) at com.infiniteautomation.nosql.MangoNoSqlPointValueDao.savePointValueSync(MangoNoSqlPointValueDao.java:65) at com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.execute(MangoNoSqlBatchWriteBehind.java:96) at com.serotonin.m2m2.rt.maint.BackgroundProcessing$1.run(BackgroundProcessing.java:54) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: No space left on device at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at org.tinytsdb.impl.Shard.ensureDirty(Shard.java:325) at org.tinytsdb.impl.Series.write(Series.java:160) at org.tinytsdb.impl.Series.write(Series.java:145) at org.tinytsdb.impl.TinyTSDBImpl.write(TinyTSDBImpl.java:301) ... 6 more
-
Hi Sasa,
Can you check
df -i
to see if you have inodes remaining?Here is a thread where someone ran out of inodes: https://forum.infiniteautomation.com/topic/2556/mango-not-starting-disk-is-full
The issue was traced to a cron script creating a lot of files in /var/spool/exim4/input and could be fixed by updating /etc/cron.d/system-info to redirect all output like
>/dev/null 2>&1
-
-
Yeah if it's the same problem you can just delete everything in /var/spool/exim4/input and it should free up a hundred thousand or more. You can just
find /var/spool/exim4/input -type f -delete
-
-
No problem and glad to hear you got it resolved!