Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
NoSQL migration crashes mango with no log message
-
I've upgraded an instance to 3.x, and migrating historical data into NoSQL, but it will run about 2 minutes then crash, with no log messages whatsoever:
less $MA_HOME/logs/ma.out INFO 2017-12-11T12:58:02,348 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... INFO 2017-12-11T12:58:02,384 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... INFO 2017-12-11T12:58:02,443 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... INFO 2017-12-11T12:58:28,664 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:51) - Running JSON File Import Job. INFO 2017-12-11T12:58:28,666 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:97) - Finished JSON File Import Job. INFO 2017-12-11T12:59:02,335 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 217370 point values. INFO 2017-12-11T12:59:02,336 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 217370 point values migrated to Time Series Datastore took 59.952s INFO 2017-12-11T12:59:02,339 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... INFO 2017-12-11T12:59:28,664 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:51) - Running JSON File Import Job. INFO 2017-12-11T12:59:28,665 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:97) - Finished JSON File Import Job. INFO 2017-12-11T13:00:00,596 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 1087343 point values. INFO 2017-12-11T13:00:00,597 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 1087343 point values migrated to Time Series Datastore took 118.154s INFO 2017-12-11T13:00:00,604 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... ma-start: no restart flag found, not restarting MA ma-start: MA doneless $MA_HOME/logs/ma.err /var/opt/mango/bin/ma-start.sh: line 147: 11343 Killed $EXECJAVA $JPDA $JAVAOPTS -server -cp "$MA_CP" "-Dma.home=$MA_HOME" "-Djava.library.path=$MA_HOME/overrides/lib:$MA_HOME/lib:/usr/lib/jni/:$PATH" com.serotonin.m2m2.MainWhat else can I try to get more diagnostics?
-
Do you have an hs_err file in your Mango/ directory? Usually HotSpot will write out (unless you're on a different JDK) an error file to the root Mango directory if it was killed by the OOM killer or something like that. That would be my first guess - the OOM killer got it. That should also appear in the dmesg output.
-
Those files do exist, but they are dated last week.
$ cat hs_err_pid1026.log # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 7158300672 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2640), pid=1026, tid=0x00007f92c42de700 # # JRE version: (8.0_144-b01) (build ) # Java VM: OpenJDK 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /var/opt/mango/core or core.1026 # --------------- T H R E A D --------------- Current thread (0x00007f92bc00e000): JavaThread "Unknown thread" [_thread_in_vm, id=1029, stack(0x00007f92c41de000,0x00007f92c42df000)] Stack: [0x00007f92c41de000,0x00007f92c42df000], sp=0x00007f92c42dd4a0, free space=1021k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xaa3d82] V [libjvm.so+0x4d85e7] V [libjvm.so+0x8e0be0] V [libjvm.so+0x8db29e] V [libjvm.so+0xaa01ec] V [libjvm.so+0x5dfde6] V [libjvm.so+0xa3c80a] V [libjvm.so+0x5e11ed] V [libjvm.so+0x5cb5df] V [libjvm.so+0xa6640a] V [libjvm.so+0xa666f5] V [libjvm.so+0x623f6f] V [libjvm.so+0xa49d5b] V [libjvm.so+0x6a3020] JNI_CreateJavaVM+0x60 C [libjli.so+0x2c53] C [libpthread.so.0+0x708a] start_thread+0xda --------------- P R O C E S S --------------- Java Threads: ( => current thread ) Other Threads: =>0x00007f92bc00e000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1029, stack(0x00007f92c41de000,0x00007f92c42df000)] VM state:not at safepoint (not fully initialized) VM Mutex/Monitor currently owned by a thread: None GC Heap History (0 events): No events Deoptimization events (0 events): No events Internal exceptions (0 events): No events Events (0 events): No events Dynamic libraries: 540000000-615550000 rw-p 00000000 00:00 0 5558f48ce000-5558f48cf000 r-xp 00000000 ca:01 407084 /usr/lib/jvm/java-8-openjdk/jre/bin/java 5558f4ace000-5558f4acf000 r--p 00000000 ca:01 407084 /usr/lib/jvm/java-8-openjdk/jre/bin/java 5558f4acf000-5558f4ad0000 rw-p 00001000 ca:01 407084 /usr/lib/jvm/java-8-openjdk/jre/bin/java 5558f55b7000-5558f55d8000 rw-p 00000000 00:00 0 [heap] 7f92abbff000-7f92ac2aa000 rw-p 00000000 00:00 0 7f92ac2aa000-7f92acfff000 ---p 00000000 00:00 0 7f92acfff000-7f92ad000000 rw-p 00000000 00:00 0 7f92ad000000-7f92ad270000 rwxp 00000000 00:00 0 7f92ad270000-7f92bc000000 ---p 00000000 00:00 0 7f92bc000000-7f92bc038000 rw-p 00000000 00:00 0 7f92bc038000-7f92c0000000 ---p 00000000 00:00 0 7f92c0da6000-7f92c0db0000 rw-p 00000000 00:00 0 7f92c0db0000-7f92c1166000 ---p 00000000 00:00 0 7f92c1166000-7f92c116e000 r-xp 00000000 ca:01 427757 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so 7f92c116e000-7f92c136d000 ---p 00008000 ca:01 427757 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so 7f92c136d000-7f92c136e000 r--p 00007000 ca:01 427757 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so 7f92c136e000-7f92c136f000 rw-p 00008000 ca:01 427757 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so 7f92c136f000-7f92c137a000 r-xp 00000000 ca:01 396550 /usr/lib/libnss_files-2.26.so 7f92c137a000-7f92c1579000 ---p 0000b000 ca:01 396550 /usr/lib/libnss_files-2.26.so 7f92c1579000-7f92c157a000 r--p 0000a000 ca:01 396550 /usr/lib/libnss_files-2.26.so 7f92c157a000-7f92c157b000 rw-p 0000b000 ca:01 396550 /usr/lib/libnss_files-2.26.so 7f92c157b000-7f92c1581000 rw-p 00000000 00:00 0 7f92c1581000-7f92c158c000 r-xp 00000000 ca:01 396574 /usr/lib/libnss_nis-2.26.so 7f92c158c000-7f92c178b000 ---p 0000b000 ca:01 396574 /usr/lib/libnss_nis-2.26.so 7f92c178b000-7f92c178c000 r--p 0000a000 ca:01 396574 /usr/lib/libnss_nis-2.26.so 7f92c178c000-7f92c178d000 rw-p 0000b000 ca:01 396574 /usr/lib/libnss_nis-2.26.so 7f92c178d000-7f92c17a1000 r-xp 00000000 ca:01 396559 /usr/lib/libnsl-2.26.so 7f92c17a1000-7f92c19a1000 ---p 00014000 ca:01 396559 /usr/lib/libnsl-2.26.so 7f92c19a1000-7f92c19a2000 r--p 00014000 ca:01 396559 /usr/lib/libnsl-2.26.so 7f92c19a2000-7f92c19a3000 rw-p 00015000 ca:01 396559 /usr/lib/libnsl-2.26.so 7f92c19a3000-7f92c19a5000 rw-p 00000000 00:00 0 7f92c19a5000-7f92c19ac000 r-xp 00000000 ca:01 396547 /usr/lib/libnss_compat-2.26.so 7f92c19ac000-7f92c1bab000 ---p 00007000 ca:01 396547 /usr/lib/libnss_compat-2.26.so 7f92c1bab000-7f92c1bac000 r--p 00006000 ca:01 396547 /usr/lib/libnss_compat-2.26.so 7f92c1bac000-7f92c1bad000 rw-p 00007000 ca:01 396547 /usr/lib/libnss_compat-2.26.so 7f92c1bad000-7f92c1bdb000 r-xp 00000000 ca:01 427720 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so 7f92c1bdb000-7f92c1dda000 ---p 0002e000 ca:01 427720 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so 7f92c1dda000-7f92c1ddb000 r--p 0002d000 ca:01 427720 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so 7f92c1ddb000-7f92c1ddd000 rw-p 0002e000 ca:01 427720 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so 7f92c1ddd000-7f92c1dea000 r-xp 00000000 ca:01 427706 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so 7f92c1dea000-7f92c1fe9000 ---p 0000d000 ca:01 427706 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so 7f92c1fe9000-7f92c1feb000 r--p 0000c000 ca:01 427706 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so 7f92c1feb000-7f92c1fec000 rw-p 0000e000 ca:01 427706 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so 7f92c1fec000-7f92c1ff3000 r-xp 00000000 ca:01 396587 /usr/lib/librt-2.26.so 7f92c1ff3000-7f92c21f2000 ---p 00007000 ca:01 396587 /usr/lib/librt-2.26.so 7f92c21f2000-7f92c21f3000 r--p 00006000 ca:01 396587 /usr/lib/librt-2.26.so 7f92c21f3000-7f92c21f4000 rw-p 00007000 ca:01 396587 /usr/lib/librt-2.26.so 7f92c21f4000-7f92c233f000 r-xp 00000000 ca:01 396580 /usr/lib/libm-2.26.so 7f92c233f000-7f92c253e000 ---p 0014b000 ca:01 396580 /usr/lib/libm-2.26.so 7f92c253e000-7f92c253f000 r--p 0014a000 ca:01 396580 /usr/lib/libm-2.26.so 7f92c253f000-7f92c2540000 rw-p 0014b000 ca:01 396580 /usr/lib/libm-2.26.so 7f92c2540000-7f92c31e3000 r-xp 00000000 ca:01 427760 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so 7f92c31e3000-7f92c33e3000 ---p 00ca3000 ca:01 427760 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so 7f92c33e3000-7f92c3477000 r--p 00ca3000 ca:01 427760 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so 7f92c3477000-7f92c349f000 rw-p 00d37000 ca:01 427760 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so 7f92c349f000-7f92c34d0000 rw-p 00000000 00:00 0 7f92c34d0000-7f92c367e000 r-xp 00000000 ca:01 396576 /usr/lib/libc-2.26.so 7f92c367e000-7f92c387d000 ---p 001ae000 ca:01 396576 /usr/lib/libc-2.26.so 7f92c387d000-7f92c3881000 r--p 001ad000 ca:01 396576 /usr/lib/libc-2.26.so 7f92c3881000-7f92c3883000 rw-p 001b1000 ca:01 396576 /usr/lib/libc-2.26.so 7f92c3883000-7f92c3887000 rw-p 00000000 00:00 0 7f92c3887000-7f92c388a000 r-xp 00000000 ca:01 396577 /usr/lib/libdl-2.26.so 7f92c388a000-7f92c3a89000 ---p 00003000 ca:01 396577 /usr/lib/libdl-2.26.so 7f92c3a89000-7f92c3a8a000 r--p 00002000 ca:01 396577 /usr/lib/libdl-2.26.so 7f92c3a8a000-7f92c3a8b000 rw-p 00003000 ca:01 396577 /usr/lib/libdl-2.26.so 7f92c3a8b000-7f92c3a99000 r-xp 00000000 ca:01 427763 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so 7f92c3a99000-7f92c3c98000 ---p 0000e000 ca:01 427763 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so 7f92c3c98000-7f92c3c99000 r--p 0000d000 ca:01 427763 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so 7f92c3c99000-7f92c3c9a000 rw-p 0000e000 ca:01 427763 /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so 7f92c3c9a000-7f92c3cb0000 r-xp 00000000 ca:01 399898 /usr/lib/libz.so.1.2.11 7f92c3cb0000-7f92c3eaf000 ---p 00016000 ca:01 399898 /usr/lib/libz.so.1.2.11 7f92c3eaf000-7f92c3eb0000 r--p 00015000 ca:01 399898 /usr/lib/libz.so.1.2.11 7f92c3eb0000-7f92c3eb1000 rw-p 00016000 ca:01 399898 /usr/lib/libz.so.1.2.11 7f92c3eb1000-7f92c3eca000 r-xp 00000000 ca:01 396603 /usr/lib/libpthread-2.26.so 7f92c3eca000-7f92c40c9000 ---p 00019000 ca:01 396603 /usr/lib/libpthread-2.26.so 7f92c40c9000-7f92c40ca000 r--p 00018000 ca:01 396603 /usr/lib/libpthread-2.26.so 7f92c40ca000-7f92c40cb000 rw-p 00019000 ca:01 396603 /usr/lib/libpthread-2.26.so 7f92c40cb000-7f92c40cf000 rw-p 00000000 00:00 0 7f92c40cf000-7f92c40f4000 r-xp 00000000 ca:01 396592 /usr/lib/ld-2.26.so 7f92c41de000-7f92c41e1000 ---p 00000000 00:00 0 7f92c41e1000-7f92c42e1000 rw-p 00000000 00:00 0 7f92c42e7000-7f92c42ef000 rw-s 00000000 00:26 21873 /tmp/hsperfdata_mango/1026 7f92c42ef000-7f92c42f0000 rw-p 00000000 00:00 0 7f92c42f0000-7f92c42f1000 r--p 00000000 00:00 0 7f92c42f1000-7f92c42f3000 rw-p 00000000 00:00 0 7f92c42f3000-7f92c42f4000 r--p 00024000 ca:01 396592 /usr/lib/ld-2.26.so 7f92c42f4000-7f92c42f5000 rw-p 00025000 ca:01 396592 /usr/lib/ld-2.26.so 7f92c42f5000-7f92c42f6000 rw-p 00000000 00:00 0 7ffd25f1a000-7ffd25f3b000 rw-p 00000000 00:00 0 [stack] 7ffd25f7b000-7ffd25f7d000 r--p 00000000 00:00 0 [vvar] 7ffd25f7d000-7ffd25f7f000 r-xp 00000000 00:00 0 [vdso] VM Arguments: jvm_args: -Xms10g -Xmx10g -Dma.home=/var/opt/mango -Djava.library.path=/var/opt/mango/overrides/lib:/var/opt/mango/lib:/usr/lib/jni/:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl java_command: com.serotonin.m2m2.Main java_class_path (initial): /var/opt/mango/overrides/classes:/var/opt/mango/classes:/var/opt/mango/overrides/properties:/var/opt/mango/overrides/lib/*:/var/opt/mango/lib/xmlbeans-2.6.0.jar:/var/opt/mango/lib/commons-dbcp-1.4.jar:/var/opt/mango/lib/commons-beanutils-1.9.3.jar:/var/opt/mango/lib/asm-commons-5.0.1.jar:/var/opt/mango/lib/spring-context-4.3.7.RELEASE.jar:/var/opt/mango/lib/taglibs-standard-impl-1.2.5.jar:/var/opt/mango/lib/spring-webmvc-4.3.7.RELEASE.jar:/var/opt/mango/lib/jsr311-api-1.1.1.jar:/var/opt/mango/lib/commons-codec-1.6.jar:/var/opt/mango/lib/jta-1.1.jar:/var/opt/mango/lib/ma-shared-1.2.3.jar:/var/opt/mango/lib/spring-test-4.0.5.RELEASE.jar:/var/opt/mango/lib/jscience-4.3.1.jar:/var/opt/mango/lib/http2-common-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-util-ajax-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-util-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-alpn-server-9.3.11.v20160721.jar:/var/opt/mango/lib/log4j-jcl-2.8.1.jar:/var/opt/mango/lib/dom4j-1.6.1.jar:/var/opt/mango/lib/websocket-server-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-servlets-9.3.11.v20160721.jar:/var/opt/mango/lib/swagger-core_2.10-1.3.7.jar:/var/opt/mango/lib/backport-util-concurrent-3.1.jar:/var/opt/mango/lib/jetty-http-9.3.11.v20160721.jar:/var/opt/mango/lib/jdom-1.1.3.jar:/var/opt/mango/lib/spring-tx-4.3.7.RELEASE.jar:/var/opt/mango/lib/jackson-annotations-2.8.6.jar:/var/opt/mango/lib/websocket-api-9.3.11.v20160721.jar:/var/opt/mango/lib/hibernate-core-3.3.0.GA.jar:/var/opt/mango/lib/annotations-2.0.1.jar:/var/opt/mango/lib/jackson-module-jsonSchema-2.1.0.jar:/var/opt/mango/lib/spring-beans-4.3.7.RELEASE.jar:/var/opt/mango/lib/jetty-xml-9.3.11.v20160721.jar:/var/opt/mango/lib/commons-collections-3.2.2.jar:/var/opt/mango/lib/jcommon-1.0.23.jar:/var/opt/mango/lib/paranamer-2.6.jar:/var/opt/mango/lib/java-semver-0.9.0.jar:/var/opt/mango/lib/json4s-core_2.10-3.2.9.jar:/var/opt/mango/lib/httpmime-4.5.3.jar:/var/opt/mango/lib/joda-convert-1.6.jar:/var/opt/mango/lib/a Launcher Type: SUN_STANDARD Environment Variables: PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl USERNAME=mango SHELL=/sbin/nologin Signal Handlers: SIGSEGV: [libjvm.so+0xaa46c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGBUS: [libjvm.so+0xaa46c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGFPE: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGPIPE: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGXFSZ: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGILL: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGUSR2: [libjvm.so+0x8dc280], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGQUIT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none --------------- S Y S T E M --------------- OS:NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch ID_LIKE=archlinux ANSI_COLOR="0;36" HOME_URL="https://www.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://bugs.archlinux.org/" uname:Linux 4.9.65-1-ec2-lts #1 SMP Wed Nov 29 09:39:59 PST 2017 x86_64 libc:glibc 2.26 NPTL 2.26 rlimit: STACK 8192k, CORE infinity, NPROC 7931, NOFILE 4096, AS infinity load average:0.00 0.01 0.00 /proc/meminfo: MemTotal: 2046976 kB MemFree: 1639152 kB MemAvailable: 1771804 kB Buffers: 46116 kB Cached: 190884 kB SwapCached: 0 kB Active: 174148 kB Inactive: 128432 kB Active(anon): 65776 kB Inactive(anon): 616 kB Active(file): 108372 kB Inactive(file): 127816 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 2097148 kB SwapFree: 2097148 kB Dirty: 404 kB Writeback: 0 kB AnonPages: 65652 kB Mapped: 93208 kB Shmem: 744 kB Slab: 73184 kB SReclaimable: 56428 kB SUnreclaim: 16756 kB KernelStack: 2636 kB PageTables: 13808 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 3120636 kB Committed_AS: 3805300 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 2048 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 12288 kB DirectMap2M: 2084864 kB CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2 /proc/cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz stepping : 2 microcode : 0x3b cpu MHz : 2400.085 cache size : 30720 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt bugs : bogomips : 4800.13 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: Memory: 4k page, physical 2046976k(1639152k free), swap 2097148k(2097148k free) vm_info: OpenJDK 64-Bit Server VM (25.144-b01) for linux-amd64 JRE (1.8.0_144-b01), built on Jul 28 2017 20:07:18 by "builduser" with gcc 7.1.1 20170630 time: Thu Dec 7 11:39:30 2017 elapsed time: 0 seconds (0d 0h 0m 0s)dmesg had no related information. :(
-
Were you redirecting stderr to ma.out? Perhaps there is an error escaping to stderr.
-
out.shis linked toext-enabledgenerating the 2 log files I posted above,$MA_HOME/logs/ma.outand$MA_HOME/logs/ma.errThat's what is so puzzling about all this, no error message output whatsoever.
-
I've noticed a few unexpected log messages, wonder if they give you any clues:
INFO 2017-12-12T12:11:37,696 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /var/opt/mango/web/modules/dashboardDesigner/web/dox/manifest.xml INFO 2017-12-12T12:11:37,505 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /var/opt/mango/web/modules/mangoUI/web/dox/manifest.xml WARN 2017-12-12T12:12:23,719 (org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandlers:114) - Neither 'urlMap' nor 'mappings' set on SimpleUrlHandlerMapping INFO 2017-12-12T12:12:25,371 (com.serotonin.m2m2.Lifecycle.configureDwr:1169) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.modbus.dwr.ModbusEditDwr(edited to include only interesting entries)
-
The good news: I finally migrated the data. I think.
The bad news: I still don't know why java crashes
In frustration, I finally ran the migration with 1 low-priority thread, and in about 15 minutes, it finally completed — sort of.
I tried updating java to 1.8.0_152; no effect
Added 4GB swap (in addition to the 1GB already configured): no effect
I executed the migration on a VirtualBox VM: no problem, success in about 3 minutesFinally, migrated on the server in question with 1 low-priority thread, It executed for 923 seconds — then stopped. I feared it had crashed again, but I noticed mango was still alive, so I checked a few things.
That's when it got interesting.
The migration page was seemingly stalled at 19648161 of 38879054
But in the log file, it claimed to have successfully completed:INFO 2017-12-15T16:03:36,024 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait... INFO 2017-12-15T16:18:59,072 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 19648161 point values. INFO 2017-12-15T16:18:59,072 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 19648161 point values migrated to Time Series Datastore took 923.048sEventually my eye caught that the 19648161 was (approximately) the same number I had seen in my successful simulation on the VirtualBox instance:
But a quick query on the original db shows:MariaDB [mango]> SELECT COUNT(*) FROM pointValues; +----------+ | COUNT(*) | +----------+ | 38879054 | +----------+Curiously, the “successful” migrations show about half the total. I also notice (especially on the actual server) that the total changes a few times during the migration.
So please help me understand: did the data migrate? Or only half? What am I not understanding?
The crashing seems obvious to be a java problem, but I can't glean any diagnostic information to pursue further. Note that this instance is running on an Arm architecture, so that might be a factor as well.
