Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
Migrate Data into Mango (NoSQL) does not appear to be doing anything
-
Having migrated to 3.2 and activated NoSQL, I am attempting to import historical data. However, I don't see the data showing up (e.g. Data Point chart) and the Migrate Data into Mango page increments the time but not "Total Processed" (I assume that's PointValues?)
Am I just being impatient, or should I see results after 14232 seconds?
-
Hi Aldo,
How'd it fare?
I would look a few places for more information. You could check your ma.log to see if any errors had been printed, and if it still purported to be running your can grab a few thread dumps from the /logging_console.shtm page and see if the thread performing the migration is stuck in the same place or in various places.
-
Now the "Total Processed" has grown:
But it still seems excruciatingly slow. Especially when this is an AWS instance that I've beefed up ti t2.2xlarge for the transition & import
However, I still don't see any actual data when I chart a Data Point "from inception"; the data starts at the time of the upgrade (when NoSQL was enabled)
-
And this is the only "interesting" message that appears in the tsdb logs (tsdb-series-385-shard-703-scan-report_2017-11-25_01-23-53.log):
-- Shard Scan Report Start: 11/25/2017 01:01:48.799 -- Callback was disordered by corruption: false ********** SERIES 519 ********** Path: /var/opt/mango/databases/mangoTSDB/9/519 Runtime: 00:00:00.00 Shard Repair: /var/opt/mango/databases/mangoTSDB/9/519/703.data.rev Corruption detected at position 4080 /var/opt/mango/databases/mangoTSDB/9/519/703.data.rev Cutting corrupt data at 4080, length 16 /var/opt/mango/databases/mangoTSDB/9/519/703.data.rev Corruption detected at position 8080 /var/opt/mango/databases/mangoTSDB/9/519/703.data.rev Cutting corrupt data at 8080, length 20However, ma.log has this repeating error message, which I thought was related to a separate data source, but might as well mention it here:
ERROR 2017-11-28T15:07:28,442 (org.springframework.web.socket.server.support.AbstractHandshakeHandler.handleInvalidUpgradeHeader:285) - Handshake failed due to invalid Upgrade header: null ERROR 2017-11-28T15:07:29,108 (org.springframework.web.socket.server.support.AbstractHandshakeHandler.handleInvalidUpgradeHeader:285) - Handshake failed due to invalid Upgrade header: null ERROR 2017-11-28T15:07:29,270 (org.springframework.web.socket.server.support.AbstractHandshakeHandler.handleInvalidUpgradeHeader:285) - Handshake failed due to invalid Upgrade header: null ERROR 2017-11-28T15:07:29,356 (org.springframework.web.socket.server.support.AbstractHandshakeHandler.handleInvalidUpgradeHeader:285) - Handshake failed due to invalid Upgrade header: null ERROR 2017-11-28T15:07:29,552 (org.springframework.web.socket.server.support.AbstractHandshakeHandler.handleInvalidUpgradeHeader:285) - Handshake failed due to invalid Upgrade header: null ERROR 2017-11-28T15:07:31,509 (org.springframework.web.socket.server.support.AbstractHandshakeHandler.handleInvalidUpgradeHeader:285) - Handshake failed due to invalid Upgrade header: null -
Interesting observation: had to restart mango, and therefore the migration process, and found this in the log:
INFO 2017-11-28T15:54:16,070 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... WARN 2017-11-28T15:54:16,082 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Migration, NaN% complete because Task Queue FullMango is configured at present for 25 high-priority threads, while I tried to launch with 20 high-priority threads, so according to my math, that should be sufficient.
-
Not including the detail of why?
I have fixed the threading of the migration work item for the next version. It's probably running in a single thread, but you may not actually see much performance gain in threading this migration, and at 25+ you may see degradation (well, had it actually launched all 25)
-
@phildunlap said in Migrate Data into Mango (NoSQL) does not appear to be doing anything:
Not including the detail of why?
Apologies, not quite following. Please clarify what you mean by that.
Meanwhile, the "Total Processed" shows 23517947 of 23517947, implying it should be completed; but the seconds keeps incrementing, and the Data Point Details is still missing the historical data.
-
Why you had to restart.
I bet that counter doesn't finish even though the migration is done. The timer not getting stopped is caused by the task rejection message you shared. Those tasks never report having finished (as they were never permitted to execute) but the invoker isn't aware. This should be resolved by preventing these tasks from having been rejected.
-
@phildunlap said in Migrate Data into Mango (NoSQL) does not appear to be doing anything:
Why you had to restart.
Ah, the server is on a trailer which had to be moved (and unplugged for the move)
So then what does the "23517947 of 23517947" mean, if the task was never permitted to execute?
-
1 of the 25 executed and probably finished everything.
-
But the data all starts at the time of the upgrade (i.e. installation of NoSQL). IOW, the historical data is not available, was not actually imported.
-
Give Mango a restart, I would expect it shows up.
-
No joy; data still starts at time of upgrade
-
Should I attempt to restart the migration?
-
You could update your NoSQL module before you try. It should permit multiple threads if you do. There should be relevant log messages but it finishing the migration task from each work item. I don't see a reason why it would say it was finished but your data isn't available.
-
Still getting confusing results.
It's still way slower than I would expect (been running over 24 hours; less than halfway through the data, even on the super-duper instance).
But the real confusion is when I look at a Data Point. At some point during the migration, I saw SOME historical data showing up (going back a few days prior to enabling NoSQL). But now when I check the same Data Point, the data starts at the time of upgrade.
Even more confusing, it seems to have introduced a few false zero values in the otherwise reasonable plot.
Could I send you a dump of the old data?
Aldo
-
Upon examining the server, I observed all data from the SQL database had been successfully migrated. It was speculated the data that was sought hadn't been in the SQL database, and this is why it didn't appear after the migration.
