NoSQL migration crashes mango with no log message

[AldoRamos]

I've upgraded an instance to 3.x, and migrating historical data into NoSQL, but it will run about 2 minutes then crash, with no log messages whatsoever:

less $MA_HOME/logs/ma.out
INFO  2017-12-11T12:58:02,348 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait...
INFO  2017-12-11T12:58:02,384 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait...
INFO  2017-12-11T12:58:02,443 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait...
INFO  2017-12-11T12:58:28,664 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:51) - Running JSON File Import Job.
INFO  2017-12-11T12:58:28,666 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:97) - Finished JSON File Import Job.
INFO  2017-12-11T12:59:02,335 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 217370 point values.
INFO  2017-12-11T12:59:02,336 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 217370 point values migrated to Time Series Datastore took 59.952s
INFO  2017-12-11T12:59:02,339 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait...
INFO  2017-12-11T12:59:28,664 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:51) - Running JSON File Import Job.
INFO  2017-12-11T12:59:28,665 (com.infiniteautomation.jsonfileimporter.JSONFileImportJob.run:97) - Finished JSON File Import Job.
INFO  2017-12-11T13:00:00,596 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 1087343 point values.
INFO  2017-12-11T13:00:00,597 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 1087343 point values migrated to Time Series Datastore took 118.154s
INFO  2017-12-11T13:00:00,604 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait...
ma-start: no restart flag found, not restarting MA
ma-start: MA done

less $MA_HOME/logs/ma.err
/var/opt/mango/bin/ma-start.sh: line 147: 11343 Killed                  $EXECJAVA $JPDA $JAVAOPTS -server -cp "$MA_CP" "-Dma.home=$MA_HOME" "-Djava.library.path=$MA_HOME/overrides/lib:$MA_HOME/lib:/usr/lib/jni/:$PATH" com.serotonin.m2m2.Main

What else can I try to get more diagnostics?

[phildunlap]

Do you have an hs_err file in your Mango/ directory? Usually HotSpot will write out (unless you're on a different JDK) an error file to the root Mango directory if it was killed by the OOM killer or something like that. That would be my first guess - the OOM killer got it. That should also appear in the dmesg output.

[AldoRamos]

Those files do exist, but they are dated last week.

$ cat hs_err_pid1026.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 7158300672 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=1026, tid=0x00007f92c42de700
#
# JRE version:  (8.0_144-b01) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /var/opt/mango/core or core.1026
#

---------------  T H R E A D  ---------------

Current thread (0x00007f92bc00e000):  JavaThread "Unknown thread" [_thread_in_vm, id=1029, stack(0x00007f92c41de000,0x00007f92c42df000)]

Stack: [0x00007f92c41de000,0x00007f92c42df000],  sp=0x00007f92c42dd4a0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xaa3d82]
V  [libjvm.so+0x4d85e7]
V  [libjvm.so+0x8e0be0]
V  [libjvm.so+0x8db29e]
V  [libjvm.so+0xaa01ec]
V  [libjvm.so+0x5dfde6]
V  [libjvm.so+0xa3c80a]
V  [libjvm.so+0x5e11ed]
V  [libjvm.so+0x5cb5df]
V  [libjvm.so+0xa6640a]
V  [libjvm.so+0xa666f5]
V  [libjvm.so+0x623f6f]
V  [libjvm.so+0xa49d5b]
V  [libjvm.so+0x6a3020]  JNI_CreateJavaVM+0x60
C  [libjli.so+0x2c53]
C  [libpthread.so.0+0x708a]  start_thread+0xda


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007f92bc00e000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1029, stack(0x00007f92c41de000,0x00007f92c42df000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
540000000-615550000 rw-p 00000000 00:00 0
5558f48ce000-5558f48cf000 r-xp 00000000 ca:01 407084                     /usr/lib/jvm/java-8-openjdk/jre/bin/java
5558f4ace000-5558f4acf000 r--p 00000000 ca:01 407084                     /usr/lib/jvm/java-8-openjdk/jre/bin/java
5558f4acf000-5558f4ad0000 rw-p 00001000 ca:01 407084                     /usr/lib/jvm/java-8-openjdk/jre/bin/java
5558f55b7000-5558f55d8000 rw-p 00000000 00:00 0                          [heap]
7f92abbff000-7f92ac2aa000 rw-p 00000000 00:00 0
7f92ac2aa000-7f92acfff000 ---p 00000000 00:00 0
7f92acfff000-7f92ad000000 rw-p 00000000 00:00 0
7f92ad000000-7f92ad270000 rwxp 00000000 00:00 0
7f92ad270000-7f92bc000000 ---p 00000000 00:00 0
7f92bc000000-7f92bc038000 rw-p 00000000 00:00 0
7f92bc038000-7f92c0000000 ---p 00000000 00:00 0
7f92c0da6000-7f92c0db0000 rw-p 00000000 00:00 0
7f92c0db0000-7f92c1166000 ---p 00000000 00:00 0
7f92c1166000-7f92c116e000 r-xp 00000000 ca:01 427757                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so
7f92c116e000-7f92c136d000 ---p 00008000 ca:01 427757                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so
7f92c136d000-7f92c136e000 r--p 00007000 ca:01 427757                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so
7f92c136e000-7f92c136f000 rw-p 00008000 ca:01 427757                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libzip.so
7f92c136f000-7f92c137a000 r-xp 00000000 ca:01 396550                     /usr/lib/libnss_files-2.26.so
7f92c137a000-7f92c1579000 ---p 0000b000 ca:01 396550                     /usr/lib/libnss_files-2.26.so
7f92c1579000-7f92c157a000 r--p 0000a000 ca:01 396550                     /usr/lib/libnss_files-2.26.so
7f92c157a000-7f92c157b000 rw-p 0000b000 ca:01 396550                     /usr/lib/libnss_files-2.26.so
7f92c157b000-7f92c1581000 rw-p 00000000 00:00 0
7f92c1581000-7f92c158c000 r-xp 00000000 ca:01 396574                     /usr/lib/libnss_nis-2.26.so
7f92c158c000-7f92c178b000 ---p 0000b000 ca:01 396574                     /usr/lib/libnss_nis-2.26.so
7f92c178b000-7f92c178c000 r--p 0000a000 ca:01 396574                     /usr/lib/libnss_nis-2.26.so
7f92c178c000-7f92c178d000 rw-p 0000b000 ca:01 396574                     /usr/lib/libnss_nis-2.26.so
7f92c178d000-7f92c17a1000 r-xp 00000000 ca:01 396559                     /usr/lib/libnsl-2.26.so
7f92c17a1000-7f92c19a1000 ---p 00014000 ca:01 396559                     /usr/lib/libnsl-2.26.so
7f92c19a1000-7f92c19a2000 r--p 00014000 ca:01 396559                     /usr/lib/libnsl-2.26.so
7f92c19a2000-7f92c19a3000 rw-p 00015000 ca:01 396559                     /usr/lib/libnsl-2.26.so
7f92c19a3000-7f92c19a5000 rw-p 00000000 00:00 0
7f92c19a5000-7f92c19ac000 r-xp 00000000 ca:01 396547                     /usr/lib/libnss_compat-2.26.so
7f92c19ac000-7f92c1bab000 ---p 00007000 ca:01 396547                     /usr/lib/libnss_compat-2.26.so
7f92c1bab000-7f92c1bac000 r--p 00006000 ca:01 396547                     /usr/lib/libnss_compat-2.26.so
7f92c1bac000-7f92c1bad000 rw-p 00007000 ca:01 396547                     /usr/lib/libnss_compat-2.26.so
7f92c1bad000-7f92c1bdb000 r-xp 00000000 ca:01 427720                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so
7f92c1bdb000-7f92c1dda000 ---p 0002e000 ca:01 427720                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so
7f92c1dda000-7f92c1ddb000 r--p 0002d000 ca:01 427720                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so
7f92c1ddb000-7f92c1ddd000 rw-p 0002e000 ca:01 427720                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libjava.so
7f92c1ddd000-7f92c1dea000 r-xp 00000000 ca:01 427706                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so
7f92c1dea000-7f92c1fe9000 ---p 0000d000 ca:01 427706                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so
7f92c1fe9000-7f92c1feb000 r--p 0000c000 ca:01 427706                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so
7f92c1feb000-7f92c1fec000 rw-p 0000e000 ca:01 427706                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/libverify.so
7f92c1fec000-7f92c1ff3000 r-xp 00000000 ca:01 396587                     /usr/lib/librt-2.26.so
7f92c1ff3000-7f92c21f2000 ---p 00007000 ca:01 396587                     /usr/lib/librt-2.26.so
7f92c21f2000-7f92c21f3000 r--p 00006000 ca:01 396587                     /usr/lib/librt-2.26.so
7f92c21f3000-7f92c21f4000 rw-p 00007000 ca:01 396587                     /usr/lib/librt-2.26.so
7f92c21f4000-7f92c233f000 r-xp 00000000 ca:01 396580                     /usr/lib/libm-2.26.so
7f92c233f000-7f92c253e000 ---p 0014b000 ca:01 396580                     /usr/lib/libm-2.26.so
7f92c253e000-7f92c253f000 r--p 0014a000 ca:01 396580                     /usr/lib/libm-2.26.so
7f92c253f000-7f92c2540000 rw-p 0014b000 ca:01 396580                     /usr/lib/libm-2.26.so
7f92c2540000-7f92c31e3000 r-xp 00000000 ca:01 427760                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so
7f92c31e3000-7f92c33e3000 ---p 00ca3000 ca:01 427760                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so
7f92c33e3000-7f92c3477000 r--p 00ca3000 ca:01 427760                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so
7f92c3477000-7f92c349f000 rw-p 00d37000 ca:01 427760                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/libjvm.so
7f92c349f000-7f92c34d0000 rw-p 00000000 00:00 0
7f92c34d0000-7f92c367e000 r-xp 00000000 ca:01 396576                     /usr/lib/libc-2.26.so
7f92c367e000-7f92c387d000 ---p 001ae000 ca:01 396576                     /usr/lib/libc-2.26.so
7f92c387d000-7f92c3881000 r--p 001ad000 ca:01 396576                     /usr/lib/libc-2.26.so
7f92c3881000-7f92c3883000 rw-p 001b1000 ca:01 396576                     /usr/lib/libc-2.26.so
7f92c3883000-7f92c3887000 rw-p 00000000 00:00 0
7f92c3887000-7f92c388a000 r-xp 00000000 ca:01 396577                     /usr/lib/libdl-2.26.so
7f92c388a000-7f92c3a89000 ---p 00003000 ca:01 396577                     /usr/lib/libdl-2.26.so
7f92c3a89000-7f92c3a8a000 r--p 00002000 ca:01 396577                     /usr/lib/libdl-2.26.so
7f92c3a8a000-7f92c3a8b000 rw-p 00003000 ca:01 396577                     /usr/lib/libdl-2.26.so
7f92c3a8b000-7f92c3a99000 r-xp 00000000 ca:01 427763                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so
7f92c3a99000-7f92c3c98000 ---p 0000e000 ca:01 427763                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so
7f92c3c98000-7f92c3c99000 r--p 0000d000 ca:01 427763                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so
7f92c3c99000-7f92c3c9a000 rw-p 0000e000 ca:01 427763                     /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/jli/libjli.so
7f92c3c9a000-7f92c3cb0000 r-xp 00000000 ca:01 399898                     /usr/lib/libz.so.1.2.11
7f92c3cb0000-7f92c3eaf000 ---p 00016000 ca:01 399898                     /usr/lib/libz.so.1.2.11
7f92c3eaf000-7f92c3eb0000 r--p 00015000 ca:01 399898                     /usr/lib/libz.so.1.2.11
7f92c3eb0000-7f92c3eb1000 rw-p 00016000 ca:01 399898                     /usr/lib/libz.so.1.2.11
7f92c3eb1000-7f92c3eca000 r-xp 00000000 ca:01 396603                     /usr/lib/libpthread-2.26.so
7f92c3eca000-7f92c40c9000 ---p 00019000 ca:01 396603                     /usr/lib/libpthread-2.26.so
7f92c40c9000-7f92c40ca000 r--p 00018000 ca:01 396603                     /usr/lib/libpthread-2.26.so
7f92c40ca000-7f92c40cb000 rw-p 00019000 ca:01 396603                     /usr/lib/libpthread-2.26.so
7f92c40cb000-7f92c40cf000 rw-p 00000000 00:00 0
7f92c40cf000-7f92c40f4000 r-xp 00000000 ca:01 396592                     /usr/lib/ld-2.26.so
7f92c41de000-7f92c41e1000 ---p 00000000 00:00 0
7f92c41e1000-7f92c42e1000 rw-p 00000000 00:00 0
7f92c42e7000-7f92c42ef000 rw-s 00000000 00:26 21873                      /tmp/hsperfdata_mango/1026
7f92c42ef000-7f92c42f0000 rw-p 00000000 00:00 0
7f92c42f0000-7f92c42f1000 r--p 00000000 00:00 0
7f92c42f1000-7f92c42f3000 rw-p 00000000 00:00 0
7f92c42f3000-7f92c42f4000 r--p 00024000 ca:01 396592                     /usr/lib/ld-2.26.so
7f92c42f4000-7f92c42f5000 rw-p 00025000 ca:01 396592                     /usr/lib/ld-2.26.so
7f92c42f5000-7f92c42f6000 rw-p 00000000 00:00 0
7ffd25f1a000-7ffd25f3b000 rw-p 00000000 00:00 0                          [stack]
7ffd25f7b000-7ffd25f7d000 r--p 00000000 00:00 0                          [vvar]
7ffd25f7d000-7ffd25f7f000 r-xp 00000000 00:00 0                          [vdso]

VM Arguments:
jvm_args: -Xms10g -Xmx10g -Dma.home=/var/opt/mango -Djava.library.path=/var/opt/mango/overrides/lib:/var/opt/mango/lib:/usr/lib/jni/:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
java_command: com.serotonin.m2m2.Main
java_class_path (initial): /var/opt/mango/overrides/classes:/var/opt/mango/classes:/var/opt/mango/overrides/properties:/var/opt/mango/overrides/lib/*:/var/opt/mango/lib/xmlbeans-2.6.0.jar:/var/opt/mango/lib/commons-dbcp-1.4.jar:/var/opt/mango/lib/commons-beanutils-1.9.3.jar:/var/opt/mango/lib/asm-commons-5.0.1.jar:/var/opt/mango/lib/spring-context-4.3.7.RELEASE.jar:/var/opt/mango/lib/taglibs-standard-impl-1.2.5.jar:/var/opt/mango/lib/spring-webmvc-4.3.7.RELEASE.jar:/var/opt/mango/lib/jsr311-api-1.1.1.jar:/var/opt/mango/lib/commons-codec-1.6.jar:/var/opt/mango/lib/jta-1.1.jar:/var/opt/mango/lib/ma-shared-1.2.3.jar:/var/opt/mango/lib/spring-test-4.0.5.RELEASE.jar:/var/opt/mango/lib/jscience-4.3.1.jar:/var/opt/mango/lib/http2-common-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-util-ajax-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-util-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-alpn-server-9.3.11.v20160721.jar:/var/opt/mango/lib/log4j-jcl-2.8.1.jar:/var/opt/mango/lib/dom4j-1.6.1.jar:/var/opt/mango/lib/websocket-server-9.3.11.v20160721.jar:/var/opt/mango/lib/jetty-servlets-9.3.11.v20160721.jar:/var/opt/mango/lib/swagger-core_2.10-1.3.7.jar:/var/opt/mango/lib/backport-util-concurrent-3.1.jar:/var/opt/mango/lib/jetty-http-9.3.11.v20160721.jar:/var/opt/mango/lib/jdom-1.1.3.jar:/var/opt/mango/lib/spring-tx-4.3.7.RELEASE.jar:/var/opt/mango/lib/jackson-annotations-2.8.6.jar:/var/opt/mango/lib/websocket-api-9.3.11.v20160721.jar:/var/opt/mango/lib/hibernate-core-3.3.0.GA.jar:/var/opt/mango/lib/annotations-2.0.1.jar:/var/opt/mango/lib/jackson-module-jsonSchema-2.1.0.jar:/var/opt/mango/lib/spring-beans-4.3.7.RELEASE.jar:/var/opt/mango/lib/jetty-xml-9.3.11.v20160721.jar:/var/opt/mango/lib/commons-collections-3.2.2.jar:/var/opt/mango/lib/jcommon-1.0.23.jar:/var/opt/mango/lib/paranamer-2.6.jar:/var/opt/mango/lib/java-semver-0.9.0.jar:/var/opt/mango/lib/json4s-core_2.10-3.2.9.jar:/var/opt/mango/lib/httpmime-4.5.3.jar:/var/opt/mango/lib/joda-convert-1.6.jar:/var/opt/mango/lib/a
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
USERNAME=mango
SHELL=/sbin/nologin

Signal Handlers:
SIGSEGV: [libjvm.so+0xaa46c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xaa46c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x8dc3d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x8dc280], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
ID_LIKE=archlinux
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"


uname:Linux 4.9.65-1-ec2-lts #1 SMP Wed Nov 29 09:39:59 PST 2017 x86_64
libc:glibc 2.26 NPTL 2.26
rlimit: STACK 8192k, CORE infinity, NPROC 7931, NOFILE 4096, AS infinity
load average:0.00 0.01 0.00

/proc/meminfo:
MemTotal:        2046976 kB
MemFree:         1639152 kB
MemAvailable:    1771804 kB
Buffers:           46116 kB
Cached:           190884 kB
SwapCached:            0 kB
Active:           174148 kB
Inactive:         128432 kB
Active(anon):      65776 kB
Inactive(anon):      616 kB
Active(file):     108372 kB
Inactive(file):   127816 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        2097148 kB
Dirty:               404 kB
Writeback:             0 kB
AnonPages:         65652 kB
Mapped:            93208 kB
Shmem:               744 kB
Slab:              73184 kB
SReclaimable:      56428 kB
SUnreclaim:        16756 kB
KernelStack:        2636 kB
PageTables:        13808 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3120636 kB
Committed_AS:    3805300 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       12288 kB
DirectMap2M:     2084864 kB


CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2

/proc/cpuinfo:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping        : 2
microcode       : 0x3b
cpu MHz         : 2400.085
cache size      : 30720 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs            :
bogomips        : 4800.13
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 2046976k(1639152k free), swap 2097148k(2097148k free)

vm_info: OpenJDK 64-Bit Server VM (25.144-b01) for linux-amd64 JRE (1.8.0_144-b01), built on Jul 28 2017 20:07:18 by "builduser" with gcc 7.1.1 20170630

time: Thu Dec  7 11:39:30 2017
elapsed time: 0 seconds (0d 0h 0m 0s)

dmesg had no related information. :(

[phildunlap]

Were you redirecting stderr to ma.out? Perhaps there is an error escaping to stderr.

[AldoRamos]

out.sh is linked to ext-enabled generating the 2 log files I posted above, $MA_HOME/logs/ma.out and $MA_HOME/logs/ma.err

That's what is so puzzling about all this, no error message output whatsoever.

[AldoRamos]

I've noticed a few unexpected log messages, wonder if they give you any clues:

INFO  2017-12-12T12:11:37,696 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /var/opt/mango/web/modules/dashboardDesigner/web/dox/manifest.xml
INFO  2017-12-12T12:11:37,505 (com.serotonin.m2m2.util.DocumentationManifest.parseManifestFile:60) - Documentation manifest file not found: /var/opt/mango/web/modules/mangoUI/web/dox/manifest.xml
WARN  2017-12-12T12:12:23,719 (org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandlers:114) - Neither 'urlMap' nor 'mappings' set on SimpleUrlHandlerMapping
INFO  2017-12-12T12:12:25,371 (com.serotonin.m2m2.Lifecycle.configureDwr:1169) - Duplicate definition of DWR class ignored: com.serotonin.m2m2.modbus.dwr.ModbusEditDwr

(edited to include only interesting entries)

[AldoRamos]

The good news: I finally migrated the data. I think.

The bad news: I still don't know why java crashes

In frustration, I finally ran the migration with 1 low-priority thread, and in about 15 minutes, it finally completed — sort of.

I tried updating java to 1.8.0_152; no effect
Added 4GB swap (in addition to the 1GB already configured): no effect
I executed the migration on a VirtualBox VM: no problem, success in about 3 minutes

Finally, migrated on the server in question with 1 low-priority thread, It executed for 923 seconds — then stopped. I feared it had crashed again, but I noticed mango was still alive, so I checked a few things.

That's when it got interesting.

The migration page was seemingly stalled at 19648161 of 38879054

But in the log file, it claimed to have successfully completed:

INFO  2017-12-15T16:03:36,024 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:70) - Starting Data Migration, please wait...
INFO  2017-12-15T16:18:59,072 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:151) - Migrated 19648161 point values.
INFO  2017-12-15T16:18:59,072 (com.infiniteautomation.nosql.maint.MangoNoSqlMigrationWorkItem.execute:155) - Finished Data Migration, 19648161 point values migrated to Time Series Datastore took 923.048s

Eventually my eye caught that the 19648161 was (approximately) the same number I had seen in my successful simulation on the VirtualBox instance:

But a quick query on the original db shows:

MariaDB [mango]> SELECT COUNT(*) FROM pointValues;
+----------+
| COUNT(*) |
+----------+
| 38879054 |
+----------+

Curiously, the “successful” migrations show about half the total. I also notice (especially on the actual server) that the total changes a few times during the migration.

So please help me understand: did the data migrate? Or only half? What am I not understanding?

The crashing seems obvious to be a java problem, but I can't glean any diagnostic information to pursue further. Note that this instance is running on an Arm architecture, so that might be a factor as well.