• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    No space left on device // (org.tinytsdb.TinyTSDBException: java.io.IOException: No space left on device)

    MangoES Hardware
    2
    8
    3.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sasa
      last edited by sasa

      Hi together,
      we had a deactivated point on the MangoES. After reboot the server did not start. The logs are attached. We have enough space on the disk.

      0_1496156952863_upload-29409c49-4fca-405b-940e-64e1d8b77bf2

      Some suggestions?

      Regards, Sasa

      [0_1496157253251_ma.log](Uploading 100%)

      1 Reply Last reply Reply Quote 0
      • S
        sasa
        last edited by 

        INFO  2017-05-30 16:47:35,858 (com.serotonin.m2m2.Main.main:82) - Starting Mango 2.7.12 with Schema Version: 12 
INFO  2017-05-30 16:47:37,464 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sqlds', v1.6.2 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,546 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'TCPIP', v1.1.5 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,558 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'asciiFile', v1.2.1 by Infinite Automation 
INFO  2017-05-30 16:47:37,570 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sqlConsole', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,583 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'galil', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,598 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstGlobalScripts', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,610 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mbus', v1.1.3 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,629 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'log4jDS', v1.1.2 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,640 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'persistent', v1.6.5 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,652 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'vmstat', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,663 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoApi', v1.1.7 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,674 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pointLinks', v1.5.5 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,687 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'log4JReset', v1.1.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,691 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstTheme', v1.1.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,698 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'jmxds', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,715 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'twilio', v1.1.2 by (unknown vendor) 
INFO  2017-05-30 16:47:37,731 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoESConfiguration', v1.1.7 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,746 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'meta', v2.2.4 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,759 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'excelReports', v1.0.6 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,782 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'snmp', v1.5.3 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,793 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'http', v1.5.2 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,810 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'controlcore', v1.1.1 by Infinite Automation 
INFO  2017-05-30 16:47:37,821 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'graphicalViews', v1.6.7 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,869 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'scripting', v1.1.2 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,882 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'watchlists', v1.7.4 by Infinite Automation Software 
INFO  2017-05-30 16:47:37,904 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataPointDetailsView', v1.0.5 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,917 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'modbus', v1.6.5 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:37,933 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dashboards', v3.2.2 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:38,175 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'onewire', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,196 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'virtualDS', v1.5.2 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,208 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataFile', v1.1.1 by Infinite Automation 
INFO  2017-05-30 16:47:38,230 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'ssh', v1.1.1 by Infinite Automation 
INFO  2017-05-30 16:47:38,242 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dnp3', v1.6.2 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:38,254 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'dataImport', v1.4.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,265 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pop3', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,276 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'jspViews', v2.1.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,292 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'templateConfig', v1.0.8 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,310 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'BACnet', v2.2.4 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,325 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'pid', v1.1.1 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:38,339 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'zwave', v1.1.2 by Infinite Automation 
INFO  2017-05-30 16:47:38,378 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'egauge', v1.1.1 by Infinite Automation 
INFO  2017-05-30 16:47:38,389 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'openv4j', v1.1.2 by aploese 
INFO  2017-05-30 16:47:38,393 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'scheduledEvents', v1.5.2 by (unknown vendor) 
INFO  2017-05-30 16:47:38,404 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'loggingConsole', v1.1.1 by Infinite Automation 
INFO  2017-05-30 16:47:38,415 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'sstGraphics', v1.1.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,452 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'mangoNoSqlDatabase', v1.2.3 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,465 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'reports', v2.2.3 by Infinite Automation Systems Inc. 
INFO  2017-05-30 16:47:38,479 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'maintenanceEvents', v1.5.1 by Infinite Automation Software 
INFO  2017-05-30 16:47:38,491 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'serial', v1.2.1 by Infinite Automation 
INFO  2017-05-30 16:47:38,503 (com.serotonin.m2m2.Main.loadModules:417) - Loading module 'internal', v1.5.3 by Infinite Automation Software 
INFO  2017-05-30 16:47:41,325 (com.infiniteautomation.mangoApi.MangoApiReloadingProperties.checkForReload:146) - (mangoApiHeaders) Found updated file at /opt/mango/web/modules/mangoApi/classes/mangoApiHeaders.properties. Reloading properties 
INFO  2017-05-30 16:47:41,714 (com.serotonin.m2m2.Lifecycle.loadLic:464) - Checking license... 
INFO  2017-05-30 16:47:42,019 (com.serotonin.m2m2.db.H2Proxy.initializeImpl:42) - Initializing H2 connection manager 
FATAL 2017-05-30 16:47:44,554 (com.serotonin.m2m2.db.DatabaseProxy.initialize:190) - Unable to connect to database of type H2 
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39)
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63)
	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:244)
	at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:204)
	at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106)
	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:602)
	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:280)
	at com.serotonin.m2m2.Main.main(Main.java:116)
Caused by: org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:168)
	at org.h2.store.FileLock.getExceptionFatal(FileLock.java:456)
	at org.h2.store.FileLock.save(FileLock.java:225)
	at org.h2.store.FileLock.lockFile(FileLock.java:352)
	at org.h2.store.FileLock.lock(FileLock.java:135)
	at org.h2.engine.Database.open(Database.java:631)
	at org.h2.engine.Database.openDatabase(Database.java:266)
	at org.h2.engine.Database.<init>(Database.java:260)
	at org.h2.engine.Engine.openSession(Engine.java:60)
	at org.h2.engine.Engine.openSession(Engine.java:167)
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)
	at org.h2.engine.Engine.createSession(Engine.java:128)
	at org.h2.engine.Engine.createSession(Engine.java:26)
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190)
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353)
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385)
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227)
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	... 12 more
Caused by: java.io.FileNotFoundException: /opt/mango/databases/mah2.lock.db (No space left on device)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
	at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:290)
	at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:233)
	at org.h2.store.FileLock.save(FileLock.java:213)
	... 33 more
ERROR 2017-05-30 16:47:44,562 (com.serotonin.m2m2.Main.main:120) - Error during initialization 
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:39)
	at com.serotonin.db.spring.ExtendedJdbcTemplate.queryForObject(ExtendedJdbcTemplate.java:63)
	at com.serotonin.m2m2.db.H2Proxy.tableExists(H2Proxy.java:244)
	at com.serotonin.m2m2.db.DatabaseProxy.newDatabaseCheck(DatabaseProxy.java:204)
	at com.serotonin.m2m2.db.DatabaseProxy.initialize(DatabaseProxy.java:106)
	at com.serotonin.m2m2.Lifecycle.databaseInitialize(Lifecycle.java:602)
	at com.serotonin.m2m2.Lifecycle.initialize(Lifecycle.java:280)
	at com.serotonin.m2m2.Main.main(Main.java:116)
Caused by: org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /opt/mango/databases/mah2.lock.db" [8000-181]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:168)
	at org.h2.store.FileLock.getExceptionFatal(FileLock.java:456)
	at org.h2.store.FileLock.save(FileLock.java:225)
	at org.h2.store.FileLock.lockFile(FileLock.java:352)
	at org.h2.store.FileLock.lock(FileLock.java:135)
	at org.h2.engine.Database.open(Database.java:631)
	at org.h2.engine.Database.openDatabase(Database.java:266)
	at org.h2.engine.Database.<init>(Database.java:260)
	at org.h2.engine.Engine.openSession(Engine.java:60)
	at org.h2.engine.Engine.openSession(Engine.java:167)
	at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)
	at org.h2.engine.Engine.createSession(Engine.java:128)
	at org.h2.engine.Engine.createSession(Engine.java:26)
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190)
	at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:353)
	at org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:385)
	at org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:227)
	at org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	... 12 more
Caused by: java.io.FileNotFoundException: /opt/mango/databases/mah2.lock.db (No space left on device)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
	at org.h2.store.fs.FilePathDisk.newOutputStream(FilePathDisk.java:290)
	at org.h2.store.fs.FileUtils.newOutputStream(FileUtils.java:233)
	at org.h2.store.FileLock.save(FileLock.java:213)
	... 33 more
INFO  2017-05-30 16:47:44,565 (com.serotonin.m2m2.Lifecycle.terminate:344) - Mango Lifecycle terminating...
        1 Reply Last reply Reply Quote 0
        • S
          sasa
          last edited by

          Sorry for the title. It was an older issue with this exception, but it seems not affect the current error. Here is still the "tsdb" exception:

          ERROR 2017-05-13 11:58:46,511 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.execute:110) - Error saving point value. 
org.tinytsdb.TinyTSDBException: java.io.IOException: No space left on device
	at org.tinytsdb.impl.TinyTSDBImpl.write(TinyTSDBImpl.java:309)
	at com.infiniteautomation.nosql.MangoNoSqlPointValueDao.savePointValueSync(MangoNoSqlPointValueDao.java:65)
	at com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehind.execute(MangoNoSqlBatchWriteBehind.java:96)
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$1.run(BackgroundProcessing.java:54)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:1012)
	at org.tinytsdb.impl.Shard.ensureDirty(Shard.java:325)
	at org.tinytsdb.impl.Series.write(Series.java:160)
	at org.tinytsdb.impl.Series.write(Series.java:145)
	at org.tinytsdb.impl.TinyTSDBImpl.write(TinyTSDBImpl.java:301)
	... 6 more
          1 Reply Last reply Reply Quote 0
          • phildunlapP
            phildunlap
            last edited by phildunlap

            Hi Sasa,

            Can you check df -i to see if you have inodes remaining?

            Here is a thread where someone ran out of inodes: https://forum.infiniteautomation.com/topic/2556/mango-not-starting-disk-is-full

            The issue was traced to a cron script creating a lot of files in /var/spool/exim4/input and could be fixed by updating /etc/cron.d/system-info to redirect all output like >/dev/null 2>&1

            S 1 Reply Last reply Reply Quote 0
            • S
              sasa @phildunlap
              last edited by

              Thank you Phil,

              here is the output and it is 100% used, right!

              0_1496159904254_upload-40b86bf8-a309-440c-93ed-b155b86f7183

              Should I remove all from /var/spool/exim4/input? What exact redirect do you recommend?

              1 Reply Last reply Reply Quote 0
              • phildunlapP
                phildunlap
                last edited by

                Yeah if it's the same problem you can just delete everything in /var/spool/exim4/input and it should free up a hundred thousand or more. You can just find /var/spool/exim4/input -type f -delete

                S 1 Reply Last reply Reply Quote 0
                • S
                  sasa @phildunlap
                  last edited by

                  @phildunlap

                  That´s it - solved!

                  Thx & Cheers
                  Sasa

                  1 Reply Last reply Reply Quote 0
                  • phildunlapP
                    phildunlap
                    last edited by

                    No problem and glad to hear you got it resolved!

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post