How To Restore a database backup ?

[phildunlap]

It is alright. I'm sure you can understand that to provide the best support I have to assess what information is probably relevant and available, and then get that information. I try to keep some eye to the long term for threads, and if our exchanges contain things like the text of the exact error the user encounters or simple descriptions of exact symptoms makes everything more searchable.

Best of luck on the new computer! I wonder, have you been setting a heap limit with an ext-enabled script?

[Phillip Weeks]

Ha ha you got it I was just messaging you back about the heap size and how I go about that . We have 8gb like this... What do you recommend and how to do it. Please and thanks. I'll start including some of the data from now on # top :)
I'm in a different time zone than our servers in case you were wondering.

This is from inside.. Our ES is much snappier than this system right now as its configured.

[Phillip Weeks]

Hard to see so

[phildunlap]

You can simply move a script for a correct size from Mango/bin/ext-avialable into Mango/bin/ext-enabled like memory-medium.sh, which sets a static heap of 5 Gb.

For 8 you could consider modifying it to 6 Gb, like,

#!/bin/bash
case "$1" in
    init)
        # Startup with Java Memory setup for Medium size installation
        # The heap is set to non-expanding for increased performance.  If memory use is a factor
        # set the Minimum heap size to a lower number and let the JVM adjust when necessary.
        JAVAOPTS="$JAVAOPTS -Xms6g -Xmx6g"
        ;;
esac

[Phillip Weeks]

That was easy .. Do I restart mango or the server to activate this configuration?

[phildunlap]

Yes, it won't take effect until the next startup, so that might mean restarting.

[Phillip Weeks]

OK thanks

[Phillip Weeks]

I restarted and JVM max memory is still about where it was at 1775mb before the change and the server says over 2 gb free does this look right for a 6gb heap?

[phildunlap]

Looks alright. If you do a ps $(pidof java) you can see if the command line options are set.

If you're getting the 1775Mb number from within Mango (i.e. internal metrics) know that Java is able to be more clever about its reporting, and even if it's allocated itself a minimum and maximum against the operating system it can expand and contract what's available for use in its own runtime.

[Phillip Weeks]

Its been a few days now since the server upgrade and here is what I have learned after tuning the system.. At first I modified the memory-medium.sh to 6gb heap as Phil suggested. This change did not take effect until Linux server was restarted because I restarted mango from inside mango no improvement or change in JVM. However once I rebooted the Linus server performance went up dramatically and UI became significantly faster, I then lowered it back to 5gb because JVM free memory was high. Unbenounced to me I had made an error in the sh file and JAVA consumed all the memory available so the heap was 8 and JVM was 3GB free memory!! At first I expected things to be great but soon experienced high degradation in performance and the system came to a crawl with constant Server Timeouts on every action. Linux could not keep up with limited memory I am guessing, So I shutdown and inspected the logs which had the script error on this sh config file. Fixed the error in memory-medium.sh file and uploaded the corrected file to the server, rebooted with 5gb heap. System came back up and leveled to about 15% above previous JVM levels but running perfectly now with few server timeouts and I set HI priority thread max to 2000 and 1000 for HI priority thread pool? Could anyone add significance to how and why mango responds to these changes?
This graph depicts how mango reconfigured memory through-out the changes made 2gb Heap to 5gb heap.

and this is current Mango stats after the process.

Linux Perspective:

[phildunlap]

The memory tuning works well because it creates a line where the operating system can reliably have a pool of memory, and Java knows that it has to do garbage collection when it reaches that line, so it ensure it does that before choking out any other processes on the computer, which will lead to disk swapping, which is about the slowest form of memory there is.

As to the thread settings, the lower thread pool setting for the high priority pool determines how many threads should always exist. This should be close to the number of how many threads can your system use. That's not so easy to figure out, but you can safely assume there are few reasonable condition where 1000 tasks need to be concurrently executing on 8 cores (unless you're doing immense amount of blocking IO with considerable latency.

The max setting of the high priority pool governs the size to which the pool can expand to before it rejects tasks because all the permissible threads have been allocated.

The medium and low pools are static pools, so they don't create or destroy thread objects while executing the tasks.

One reason having properly tuned pools can help is that it prevents memory allocation for thread objects, which would be very short-lived and create garbage that requires collection. For instance, if you only have a single high priority thread all the time but a much higher maximum, then it's very likely almost every task will expand the pool.

[Phillip Weeks]

Insightful information for sure. I hope that I will determine these levels over time and through experimentation. : Thanks

[Phillip Weeks]

Curiously I have been experimenting with the heap size reducing it from 6g to 5g and now to 4g because linux free memory is being cut to 41m after a few hours and used memory is almost 100% as below...

If I remove the memory-model config from ext-enabled, the system will default to a 2g heap and stabilize at about 3g free system memory in Linux indefinitely with no degradation and 800mb/1.7g free/max in mango. I want to get it to stabilize with 1g free linux and 4g max in mango. However all of the heap configurations I have tried so far start off great but reduce Linux memory to 50mb after running for a few hours and this makes the system sluggish. Yet the heap settings appear correctly applied as below. This is the last config for 4g heap and the system memory is almost 100% used.

Any ideas on why this is occurring? Thanks in advance.

[phildunlap]

I couldn't say for sure. I would advise sorting that top window by memory (shift+f, arrows to mem, press 's'). You could experiment with removing the Xms parameter.

Sometimes with larger heaps the garbage collector can be lazy, and then when it does run it has more work to do and a bigger pile of potential garbage to sift through, but I wouldn't expect that to cause what you're describing, as it also generally has more garbage it can collect, and it can meet its targets easily