• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    Easiest way to move data from one instance of Mango to another

    User help
    2
    15
    2.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      ddiezperez @phildunlap
      last edited by

      @phildunlap said in Easiest way to move data from one instance of Mango to another:

      Merge or Mergration ? You need to do a Mergration if you restored the configuration from the JSON import. Not Merge.

      I used the "Merge TSDB Data into Mango" tab. You are saying I should have used the "Migrate NoSQL Data Into Mango" - correct? Just want to be clear as I'm unsure of what the "Mergreation" tool is ..

      1 Reply Last reply Reply Quote 0
      • phildunlapP
        phildunlap
        last edited by phildunlap

        Oh, look at that, someone must have changed the tab names and I hadn't noticed. Yes, you want the "Migrate NoSQL Data Into Mango" tab. That should have inputs for an SQL database and a NoSQL database. Originally there was Migration (from SQL) and Merge (from a same-point-ids NoSQL) and so I called the 'both' type a Mergration. I guess that didn't stick!

        Edit: It looks like ten months ago Mergration became NoSQL Migration - sorry for my confusion.

        D 1 Reply Last reply Reply Quote 0
        • D
          ddiezperez @phildunlap
          last edited by ddiezperez

          Oh, look at that, someone must have changed the tab names and I hadn't noticed. Yes, you want the "Migrate NoSQL Data Into Mango" tab. That should have inputs for an SQL database and a NoSQL database. I guess someone else didn't like 'Mergration' either. Originally there was Migration (from SQL) and Merge (from a same-point-ids NoSQL) and so I called the 'both' type a Mergration. I guess that didn't stick!

          Ha gotcha - Thanks for the clarification.

          I just gave it a try and looks like having the Mango Internal data source in both servers is a problem when trying to migrate NoSQL data into Mango.

          1 Reply Last reply Reply Quote 0
          • phildunlapP
            phildunlap
            last edited by

            Are you getting an error? That should work fine, the only issues will arise if you have XIDs that exist in both SQL databases but are not the same data types.

            1 Reply Last reply Reply Quote 0
            • D
              ddiezperez
              last edited by

              Getting this ... even after renaming the XID of the Mango Internal data source

              org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement: select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]

              That shouldn't prevent the rest of the data migration to complete though - correct?

              1 Reply Last reply Reply Quote 0
              • phildunlapP
                phildunlap
                last edited by phildunlap

                Check to make sure your connection string is good. One interesting behavior of H2 is that if you point your connection string at a location that doesn't have a database and try to log into it with credentials, it will create a new, empty H2 database there. That would then not have a dataPoints table to query upon!

                Should be something like,

                jdbc:h2:/path/to/old-databases/mah2

                and probably the best connection string for H2 may also have the arguments added on the db.url normally in H2Proxy, which would be

                jdbc:h2:/path/to/old-databases/mah2;DB_CLOSE_ON_EXIT=FALSE;MV_STORE=FALSE;IGNORECASE=TRUE

                And you can optionally add ;IFEXISTS=TRUE to the end to ensure it won't just create a new, empty H2 database if the connection string isn't quite right.

                I will make the change in this issue to this type of migration as well: https://github.com/infiniteautomation/ma-core-public/issues/631

                1 Reply Last reply Reply Quote 0
                • D
                  ddiezperez
                  last edited by ddiezperez

                  Thanks for the help Phil - it is processing right now .. Keeping the fingers crossed.

                  If this works correctly then I'll go back and do a JSON config import of the rest of the data sources and points I care about and repeat the process.

                  This will be really helpful when integrating sites/controllers for clients on a dev/test Mango server first and then migrating the data and polling to the client's production environment.

                  I'll report back after it is done processing ..

                  UPDATE: status went to IDLE and it seems to be frozen at Run Time of 67 seconds. What can I check?0_1539109844073_Data_Migration.PNG

                  1 Reply Last reply Reply Quote 0
                  • phildunlapP
                    phildunlap
                    last edited by

                    Log file. I saw something like that recently where corruption exceptions were killing the merge task I tried to run. Was quite irritating. A shame we haven't released 3.5.1 of NoSQL yet, as I put retries into the Merge and the NoSQL Migration should there be corruption in a shard in the NoSQL. Maybe I can email you a pre-release of that....

                    1 Reply Last reply Reply Quote 0
                    • D
                      ddiezperez
                      last edited by

                      Connection string is pointing to the right directory and file, and I used the suggested arguments as well. It is pasted below:

                      jdbc:h2:file:/Mango/databases/_fromDot41/mah2;DB_CLOSE_ON_EXIT=FALSE;MV_STORE=FALSE;IGNORECASE=TRUE;IFEXISTS=TRUE

                      Latest ma.log file pasted below. Sorry I keep pasting the entire file within the post, if there's a better way to share it please let me know.

                      
INFO  2018-10-09T13:55:59,424 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:147) - Database at /Mango/databases/mangoTSDB_fromDot41 started 
INFO  2018-10-09T13:55:59,852 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:178) - Empty dir delete for database at /Mango/databases/mangoTSDB_fromDot41 took 399ms 
INFO  2018-10-09T13:56:06,508 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:40) - Starting Data Mergration, please wait... 
ERROR 2018-10-09T13:56:06,521 (com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId:88) - Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196] 
org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableOrView(Parser.java:5552) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableFilter(Parser.java:1266) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1946) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2095) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSub(Parser.java:1940) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectUnion(Parser.java:1755) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelect(Parser.java:1743) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parsePrepared(Parser.java:449) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:321) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:293) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.prepareCommand(Parser.java:258) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareLocal(Session.java:578) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareCommand(Session.java:519) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:73) ~[h2-1.4.196.jar:1.4.196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:78) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:52) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
ERROR 2018-10-09T13:56:06,522 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:102) - org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196] 
com.serotonin.ShouldNeverHappenException: org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:89) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:52) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableOrView(Parser.java:5552) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableFilter(Parser.java:1266) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1946) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2095) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSub(Parser.java:1940) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectUnion(Parser.java:1755) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelect(Parser.java:1743) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parsePrepared(Parser.java:449) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:321) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:293) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.prepareCommand(Parser.java:258) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareLocal(Session.java:578) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareCommand(Session.java:519) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:73) ~[h2-1.4.196.jar:1.4.196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:78) ~[?:?]
	... 7 more
WARN  2018-10-09T13:56:10,249 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source UT, Salt Lake Simple Agent KW Meter aborted 61 polls since it started. 
WARN  2018-10-09T13:57:24,948 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source TN, Nashville Simple Agent KW Meter aborted 62 polls since it started. 
INFO  2018-10-09T13:58:54,227 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:423) - Data source 'Mango Internal' stopped 
INFO  2018-10-09T13:58:54,317 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Mango Internal' initialized 
INFO  2018-10-09T13:58:54,318 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Mango Internal' took 79.100375ms to start 
INFO  2018-10-09T13:59:26,616 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:147) - Database at /Mango/databases/mangoTSDB_fromDot41 started 
INFO  2018-10-09T13:59:27,018 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:178) - Empty dir delete for database at /Mango/databases/mangoTSDB_fromDot41 took 375ms 
INFO  2018-10-09T13:59:32,963 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:40) - Starting Data Mergration, please wait... 
ERROR 2018-10-09T13:59:32,965 (com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId:88) - Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196] 
org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableOrView(Parser.java:5552) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableFilter(Parser.java:1266) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1946) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2095) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSub(Parser.java:1940) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectUnion(Parser.java:1755) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelect(Parser.java:1743) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parsePrepared(Parser.java:449) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:321) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:293) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.prepareCommand(Parser.java:258) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareLocal(Session.java:578) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareCommand(Session.java:519) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:73) ~[h2-1.4.196.jar:1.4.196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:78) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:52) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
ERROR 2018-10-09T13:59:32,965 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:102) - org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196] 
com.serotonin.ShouldNeverHappenException: org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:89) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:52) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableOrView(Parser.java:5552) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableFilter(Parser.java:1266) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1946) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2095) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSub(Parser.java:1940) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectUnion(Parser.java:1755) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelect(Parser.java:1743) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parsePrepared(Parser.java:449) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:321) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:293) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.prepareCommand(Parser.java:258) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareLocal(Session.java:578) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareCommand(Session.java:519) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:73) ~[h2-1.4.196.jar:1.4.196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:78) ~[?:?]
	... 7 more
INFO  2018-10-09T14:05:13,766 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:423) - Data source 'Mango Internal' stopped 
INFO  2018-10-09T14:05:13,777 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Mango Internal' initialized 
INFO  2018-10-09T14:05:13,778 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Mango Internal' took 9.333323ms to start 
INFO  2018-10-09T14:05:38,690 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:147) - Database at /Mango/databases/mangoTSDB_fromDot41 started 
INFO  2018-10-09T14:05:39,127 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:178) - Empty dir delete for database at /Mango/databases/mangoTSDB_fromDot41 took 411ms 
INFO  2018-10-09T14:05:45,359 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:40) - Starting Data Mergration, please wait... 
ERROR 2018-10-09T14:05:45,360 (com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId:88) - Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196] 
org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableOrView(Parser.java:5552) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableFilter(Parser.java:1266) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1946) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2095) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSub(Parser.java:1940) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectUnion(Parser.java:1755) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelect(Parser.java:1743) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parsePrepared(Parser.java:449) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:321) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:293) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.prepareCommand(Parser.java:258) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareLocal(Session.java:578) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareCommand(Session.java:519) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:73) ~[h2-1.4.196.jar:1.4.196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:78) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:52) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
ERROR 2018-10-09T14:05:45,360 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:102) - org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196] 
com.serotonin.ShouldNeverHappenException: org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:89) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:52) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.h2.jdbc.JdbcSQLException: Table "DATAPOINTS" not found; SQL statement:
select id from dataPoints where xid = 'internal_mango_num_active_user_sessions' [42102-196]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableOrView(Parser.java:5552) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.readTableFilter(Parser.java:1266) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1946) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2095) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectSub(Parser.java:1940) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelectUnion(Parser.java:1755) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parseSelect(Parser.java:1743) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parsePrepared(Parser.java:449) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:321) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.parse(Parser.java:293) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.command.Parser.prepareCommand(Parser.java:258) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareLocal(Session.java:578) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.engine.Session.prepareCommand(Session.java:519) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1204) ~[h2-1.4.196.jar:1.4.196]
	at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:73) ~[h2-1.4.196.jar:1.4.196]
	at com.infiniteautomation.nosql.maint.GenericSqlPointValueDao.getDataPointId(GenericSqlPointValueDao.java:78) ~[?:?]
	... 7 more
INFO  2018-10-09T14:16:06,885 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:147) - Database at /Mango/databases/mangoTSDB_fromDot41 started 
INFO  2018-10-09T14:16:07,308 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:178) - Empty dir delete for database at /Mango/databases/mangoTSDB_fromDot41 took 396ms 
INFO  2018-10-09T14:16:14,018 (com.infiniteautomation.nosql.maint.MangoNoSqlMergeWorkItem.execute:48) - Starting Data Merge, please wait... 
ERROR 2018-10-09T14:16:14,057 (com.infiniteautomation.nosql.maint.MangoNoSqlMergeWorkItem.execute:60) - Merge failed, you need 6248 available file descriptors but only have 3350 
INFO  2018-10-09T14:23:03,466 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:147) - Database at /Mango/databases/mangoTSDB_fromDot41 started 
INFO  2018-10-09T14:23:03,903 (com.infiniteautomation.tsdb.impl.IasTsdbImpl.<init>:178) - Empty dir delete for database at /Mango/databases/mangoTSDB_fromDot41 took 410ms 
INFO  2018-10-09T14:23:11,280 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:40) - Starting Data Mergration, please wait... 
WARN  2018-10-09T14:23:27,522 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source Mango Internal aborted 1 polls since it started. 
ERROR 2018-10-09T14:24:10,653 (com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute:102) - null 
java.nio.BufferOverflowException: null
	at com.infiniteautomation.tsdb.ByteArrayBuilder.ensureAvailable(ByteArrayBuilder.java:405) ~[?:?]
	at com.infiniteautomation.tsdb.ByteArrayBuilder.getString(ByteArrayBuilder.java:315) ~[?:?]
	at com.infiniteautomation.nosql.PointValueTimeSerializer.fromByteArray(PointValueTimeSerializer.java:81) ~[?:?]
	at com.infiniteautomation.nosql.PointValueTimeSerializer.fromByteArray(PointValueTimeSerializer.java:36) ~[?:?]
	at com.infiniteautomation.tsdb.impl.IasTsdbImpl$CallbackWrapper.sample(IasTsdbImpl.java:955) ~[?:?]
	at com.infiniteautomation.tsdb.impl.Shard.query(Shard.java:483) ~[?:?]
	at com.infiniteautomation.tsdb.impl.CompressibleShard.query(CompressibleShard.java:391) ~[?:?]
	at com.infiniteautomation.tsdb.impl.Series.query(Series.java:532) ~[?:?]
	at com.infiniteautomation.tsdb.impl.IasTsdbImpl.query(IasTsdbImpl.java:518) ~[?:?]
	at com.infiniteautomation.tsdb.impl.IasTsdbImpl.query(IasTsdbImpl.java:505) ~[?:?]
	at com.infiniteautomation.tsdb.impl.IasTsdbImpl.query(IasTsdbImpl.java:485) ~[?:?]
	at com.infiniteautomation.tsdb.impl.IasTsdbImpl.query(IasTsdbImpl.java:470) ~[?:?]
	at com.infiniteautomation.nosql.maint.MangoNoSqlMergrationWorkItem.execute(MangoNoSqlMergrationWorkItem.java:63) ~[?:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.5.0.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.5.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
WARN  2018-10-09T14:33:29,100 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source NC, Charlotte Simple Agent KW Meter aborted 6172 polls since it started. 
WARN  2018-10-09T14:37:19,099 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:157) - Data Source AL, Montgomery Simple Agent KW Meter aborted 4330 polls since it started.
                      1 Reply Last reply Reply Quote 0
                      • phildunlapP
                        phildunlap
                        last edited by phildunlap

                        We moved to email, but the next version of MangoNoSql, 3.5.1, will improve the handling of corruption in the database that is to be merged. This is what caused the buffer overflow exception at the end of the log, and the returning to IDLE state before finishing all the data points.

                        Sorry I keep pasting the entire file within the post, if there's a better way to share it please let me know.

                        Tis better to log the whole log than no log at all!

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post