Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website

No space left on device // (org.tinytsdb.TinyTSDBException: java.io.IOException: No space left on device)


  • Hi together,
    we had a deactivated point on the MangoES. After reboot the server did not start. The logs are attached. We have enough space on the disk.

    0_1496156952863_upload-29409c49-4fca-405b-940e-64e1d8b77bf2

    Some suggestions?

    Regards, Sasa

    [0_1496157253251_ma.log](Uploading 100%)


  • INFO  2017-05-30 16:47:35,858 (com.serotonin.m2m2.Main.main:82) - Starting Mango 2.7.12 with Schema Version: 12 
    INFO  2017-05-30 16:47:37,464 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sqlds', v1.6.2 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,546 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'TCPIP', v1.1.5 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,558 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'asciiFile', v1.2.1 by Infinite Automation 
    INFO  2017-05-30 16:47:37,570 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sqlConsole', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,583 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'galil', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,598 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstGlobalScripts', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,610 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mbus', v1.1.3 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,629 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'log4jDS', v1.1.2 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,640 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'persistent', v1.6.5 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,652 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'vmstat', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,663 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoApi', v1.1.7 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,674 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pointLinks', v1.5.5 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,687 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'log4JReset', v1.1.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,691 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstTheme', v1.1.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,698 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'jmxds', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,715 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'twilio', v1.1.2 by (unknown vendor) 
    INFO  2017-05-30 16:47:37,731 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoESConfiguration', v1.1.7 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,746 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'meta', v2.2.4 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,759 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'excelReports', v1.0.6 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,782 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'snmp', v1.5.3 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,793 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'http', v1.5.2 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,810 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'controlcore', v1.1.1 by Infinite Automation 
    INFO  2017-05-30 16:47:37,821 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'graphicalViews', v1.6.7 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,869 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'scripting', v1.1.2 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,882 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'watchlists', v1.7.4 by Infinite Automation Software 
    INFO  2017-05-30 16:47:37,904 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataPointDetailsView', v1.0.5 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,917 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'modbus', v1.6.5 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:37,933 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dashboards', v3.2.2 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:38,175 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'onewire', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,196 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'virtualDS', v1.5.2 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,208 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataFile', v1.1.1 by Infinite Automation 
    INFO  2017-05-30 16:47:38,230 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'ssh', v1.1.1 by Infinite Automation 
    INFO  2017-05-30 16:47:38,242 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dnp3', v1.6.2 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:38,254 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataImport', v1.4.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,265 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pop3', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,276 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'jspViews', v2.1.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,292 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'templateConfig', v1.0.8 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,310 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'BACnet', v2.2.4 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,325 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pid', v1.1.1 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:38,339 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'zwave', v1.1.2 by Infinite Automation 
    INFO  2017-05-30 16:47:38,378 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'egauge', v1.1.1 by Infinite Automation 
    INFO  2017-05-30 16:47:38,389 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'openv4j', v1.1.2 by aploese 
    INFO  2017-05-30 16:47:38,393 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'scheduledEvents', v1.5.2 by (unknown vendor) 
    INFO  2017-05-30 16:47:38,404 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'loggingConsole', v1.1.1 by Infinite Automation 
    INFO  2017-05-30 16:47:38,415 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstGraphics', v1.1.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,452 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoNoSqlDatabase', v1.2.3 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,465 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'reports', v2.2.3 by Infinite Automation Systems Inc. 
    INFO  2017-05-30 16:47:38,479 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'maintenanceEvents', v1.5.1 by Infinite Automation Software 
    INFO  2017-05-30 16:47:38,491 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'serial', v1.2.1 by Infinite Automation 
    INFO  2017-05-30 16:47:38,503 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'internal', v1.5.3 by Infinite Automation Software 
    INFO  2017-05-30 16:47:41,325 (com.infiniteautomation.mangoApi.MangoApiReloadingProperties.checkForReload:146) - (mangoApiHeaders) Found updated file at /opt/mango/web/modules/mangoApi/classes/mangoApiHeaders.properties. Reloading properties 
    INFO  2017-05-30 16:47:41,714 (com.serotonin.m2m2.Lifecycle.loadLic:464) - Checking license... 
    INFO  2017-05-30 16:47:42,019 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:42) - Initializing H2 connection manager 
    FATAL 2017-05-30 16:47:44,554 (com.serotonin.m2m2.db.DatabaseProxy.initialize:190) - Unable to connect to database of type H2 
    org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39)
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63)
    	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:244)
    	at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:204)
    	at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106)
    	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:602)
    	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:280)
    	at com.serotonin.m2m2.Main.main(Main.java:116)
    Caused by: org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
    	at org.h2.message.DbException.get(DbException.java:168)
    	at org.h2.store.FileLock.getExceptionFatal(FileLock.java:456)
    	at org.h2.store.FileLock.save(FileLock.java:225)
    	at org.h2.store.FileLock.lockFile(FileLock.java:352)
    	at org.h2.store.FileLock.lock(FileLock.java:135)
    	at org.h2.engine.Database.open(Database.java:631)
    	at org.h2.engine.Database.openDatabase(Database.java:266)
    	at org.h2.engine.Database.<init>(Database.java:260)
    	at org.h2.engine.Engine.openSession(Engine.java:60)
    	at org.h2.engine.Engine.openSession(Engine.java:167)
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)
    	at org.h2.engine.Engine.createSession(Engine.java:128)
    	at org.h2.engine.Engine.createSession(Engine.java:26)
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
    	at org.h2.Driver.connect(Driver.java:72)
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190)
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353)
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385)
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227)
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
    	... 12 more
    Caused by: java.io.FileNotFoundException: /opt/mango/databases/mah2.lock.db (No space left on device)
    	at java.io.FileOutputStream.open(Native Method)
    	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    	at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
    	at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:290)
    	at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:233)
    	at org.h2.store.FileLock.save(FileLock.java:213)
    	... 33 more
    ERROR 2017-05-30 16:47:44,562 (com.serotonin.m2m2.Main.main:120) - Error during initialization 
    org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39)
    	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63)
    	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:244)
    	at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:204)
    	at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106)
    	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:602)
    	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:280)
    	at com.serotonin.m2m2.Main.main(Main.java:116)
    Caused by: org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
    	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
    	at org.h2.message.DbException.get(DbException.java:168)
    	at org.h2.store.FileLock.getExceptionFatal(FileLock.java:456)
    	at org.h2.store.FileLock.save(FileLock.java:225)
    	at org.h2.store.FileLock.lockFile(FileLock.java:352)
    	at org.h2.store.FileLock.lock(FileLock.java:135)
    	at org.h2.engine.Database.open(Database.java:631)
    	at org.h2.engine.Database.openDatabase(Database.java:266)
    	at org.h2.engine.Database.<init>(Database.java:260)
    	at org.h2.engine.Engine.openSession(Engine.java:60)
    	at org.h2.engine.Engine.openSession(Engine.java:167)
    	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)
    	at org.h2.engine.Engine.createSession(Engine.java:128)
    	at org.h2.engine.Engine.createSession(Engine.java:26)
    	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
    	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
    	at org.h2.Driver.connect(Driver.java:72)
    	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190)
    	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353)
    	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385)
    	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227)
    	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
    	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
    	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
    	... 12 more
    Caused by: java.io.FileNotFoundException: /opt/mango/databases/mah2.lock.db (No space left on device)
    	at java.io.FileOutputStream.open(Native Method)
    	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    	at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
    	at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:290)
    	at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:233)
    	at org.h2.store.FileLock.save(FileLock.java:213)
    	... 33 more
    INFO  2017-05-30 16:47:44,565 (com.serotonin.m2m2.Lifecycle.terminate:344) - Mango Lifecycle terminating...
    

  • Sorry for the title. It was an older issue with this exception, but it seems not affect the current error. Here is still the "tsdb" exception:

    ERROR 2017-05-13 11:58:46,511 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.execute:110) - Error saving point value. 
    org.tinytsdb.TinyTSDBException: java.io.IOException: No space left on device
    	at org.tinytsdb.impl.TinyTSDBImpl.write(TinyTSDBImpl.java:309)
    	at com.infiniteautomation.nosql.MangoNoSqlPointValueDao.savePointValueSync(MangoNoSqlPointValueDao.java:65)
    	at com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.execute(MangoNoSqlBatchWriteBehind.java:96)
    	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$1.run(BackgroundProcessing.java:54)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.IOException: No space left on device
    	at java.io.UnixFileSystem.createFileExclusively(Native Method)
    	at java.io.File.createNewFile(File.java:1012)
    	at org.tinytsdb.impl.Shard.ensureDirty(Shard.java:325)
    	at org.tinytsdb.impl.Series.write(Series.java:160)
    	at org.tinytsdb.impl.Series.write(Series.java:145)
    	at org.tinytsdb.impl.TinyTSDBImpl.write(TinyTSDBImpl.java:301)
    	... 6 more
    

  • Hi Sasa,

    Can you check df -i to see if you have inodes remaining?

    Here is a thread where someone ran out of inodes: https://forum.infiniteautomation.com/topic/2556/mango-not-starting-disk-is-full

    The issue was traced to a cron script creating a lot of files in /var/spool/exim4/input and could be fixed by updating /etc/cron.d/system-info to redirect all output like >/dev/null 2>&1


  • Thank you Phil,

    here is the output and it is 100% used, right!

    0_1496159904254_upload-40b86bf8-a309-440c-93ed-b155b86f7183

    Should I remove all from /var/spool/exim4/input? What exact redirect do you recommend?


  • Yeah if it's the same problem you can just delete everything in /var/spool/exim4/input and it should free up a hundred thousand or more. You can just find /var/spool/exim4/input -type f -delete


  • @phildunlap

    That´s it - solved!

    Thx & Cheers
    Sasa


  • No problem and glad to hear you got it resolved!