8000 points to monit

bkm

Hi Guys,

I was using successfully mango M2M using Derby database on Windows 7 (x64), but recently I have a need to monit around 8 000 points in real time (on BACnet).

My problem is long delay in refreshing values on my graphical view. It take around 5 min, despiate of fact that I set 3min for saving database and 2 minutes in points specifications (INTERVAL) with 0.0 tolerance.

I am fighting with it quite long time.

I tried to increase cache dedicated to JAVA, my web browser, etc. Without success. Still I get information, that JAVA has Memory Cheap Overspace.
I have 32GB of RAM to use and I still have use of maksimum 4GB of memory.

I saw many topics related to switching Derby to MYSQL but I wasn't able to switch it to the MYSQL database. Even if I created empty MYSQL databse, Apache wasn't able to use MYSQL database.
I would appreciate if you help me go through whole process of running MYSQL on my operating system to be able to run Mango with following changes in file env.properities:

db.type=mysql
db.url=~/../../bin/mangoDB2
db.username=
db.password=
db.pool.maxActive=10
db.pool.maxIdle=10
db.update.log.dir=${catalina.base}/logs/

convert.db.type=mysql
convert.db.url=~/../../bin/mangoDB2
convert.db.username=
convert.db.password=

grove.url=http://mango.serotoninsoftware.com/servlet

If you have advices for me I would appreciate it!

hussam

your env.properities configuration is wrong,if you want to convert derby to mysql,configuration is like:

first creat one empty database in mysql,

then.
(new datase)
db.type=mysql
db.url=jdbc:mysql://localhost/samedb
db.username=
db.password=
db.pool.maxActive=10
db.pool.maxIdle=10
db.update.log.dir=${catalina.base}/logs/

(old database)
convert.db.type=derby
convert.db.url=~/../../bin/mangoDB2
convert.db.username=
convert.db.password=

grove.url=http://mango.serotoninsoftware.com/servlet

bkm

@hussam said:

your env.properities configuration is wrong,if you want to convert derby to mysql,configuration is like:

first creat one empty database in mysql,

then.
(new datase)
db.type=mysql
db.url=jdbc:mysql://localhost/samedb
db.username=
db.password=
db.pool.maxActive=10
db.pool.maxIdle=10
db.update.log.dir=${catalina.base}/logs/

(old database)
convert.db.type=derby
convert.db.url=~/../../bin/mangoDB2
convert.db.username=
convert.db.password=

grove.url=http://mango.serotoninsoftware.com/servlet

Thank you very much for your response!

And did you tried to increase values of parametres:
db.pool.maxActive=10
db.pool.maxIdle=10
without loosing stability?
If yes, did it cause improvement of Mango performance ?

phildunlap

I suspect the database conversion will help a lot, and thanks for answering, hussam.

In regards to the second part of the first post, you could investigate tuning the -X arguments passed to the JVM in your .bat or .sh start up script, such as your -Xms (Initial heap). I don't know that you're going to need to go into situations where Mango will require more than 4Gb with 8000pnts/2minutes, although this of course depends upon the configuration and use case.

bkm

Thank you for your answer - phildunlap. According your advice - I also improved Java parameters -> following this tutorial: http://www.wikihow.com/Increase-Java-Memory-in-Windows-7. There is also option to set JAVA heap space in TomCat service, but despite all this actions, I still had JAVA cheap space after some time (even If I set 4GB).

But I guess that should be set inside Mango to make it work, because It will be overwrite by constans inside Mango? Am I wrong?

@phildunlap said:

I suspect the database conversion will help a lot, and thanks for answering, hussam.

In regards to the second part of the first post, you could investigate tuning the -X arguments passed to the JVM in your .bat or .sh start up script, such as your -Xms (Initial heap). I don't know that you're going to need to go into situations where Mango will require more than 4Gb with 8000pnts/2minutes, although this of course depends upon the configuration and use case.

bkm

**phildunlap ** I have additional question :) .
I was able to run Mango on MySQL but time from hitting Apache *startup * till the moment Mango starts, increased from couple of seconds to the 4 hours (database has 4.5GB)....
Is it possible to fix it? I use Apache 7.0.

@phildunlap said:

I suspect the database conversion will help a lot, and thanks for answering, hussam.

In regards to the second part of the first post, you could investigate tuning the -X arguments passed to the JVM in your .bat or .sh start up script, such as your -Xms (Initial heap). I don't know that you're going to need to go into situations where Mango will require more than 4Gb with 8000pnts/2minutes, although this of course depends upon the configuration and use case.

phildunlap

A good thing to know is that you can put a file in your Mango root directory called "SAFE" and boot Mango with your datasources disabled. I suspect something in your configuration is causing an unnecessary delay, or your network is significantly slower than you're querying it. You could do some trial and error datasource/datapoint enabling.... or, have you investigated your mango/logs/ma.log? If so, swap debug-log4j.xml with log4j.xml in mango/classes for a boot, and let's have a look

JoelHaggar

Something to know is that in a very near release of the Mango Automation Enterprise license we will offer a NoSQL database option that gives impressive performance for very large systems.

bkm

Thank you very much for so prompt response! You have no idea how I appreciate it!

mango.log doesn't show anything special - without error/warnings but still freezed on startup level... ( maybe it is just me being impatient.... :oops: )
I decreased size of my database to the 1.2GB and problem is solved but question is how to avoid this kind of behavior - becouse in the future I will have - for sure huge database.

Thank you for advice with disabling all of the sources - I guess it could possibly help. I I will find solution I will share with you :-).

@phildunlap said:

A good thing to know is that you can put a file in your Mango root directory called "SAFE" and boot Mango with your datasources disabled. I suspect something in your configuration is causing an unnecessary delay, or your network is significantly slower than you're querying it. You could do some trial and error datasource/datapoint enabling.... or, have you investigated your mango/logs/ma.log? If so, swap debug-log4j.xml with log4j.xml in mango/classes for a boot, and let's have a look

bkm

I can't wait Joel :D !

@JoelHaggar said:

Something to know is that in a very near release of the Mango Automation Enterprise license we will offer a NoSQL database option that gives impressive performance for very large systems.