Apache Tomcat drops out often

[MattFox]

I keep seeing the mangoAPI server coming offline and online, to the point of interrupting user logins. What are your suggestions to stop this? I've got customers complaining about proxy errors (502) due to mango's webservice timing out.
Any suggestions would be great

Fox

[terrypacker]

What version of Mango? There are quite a few tuning options in the env.properties file perhaps you can post yours after scrubbing it for any sensitive info. This is all assuming your proxy is not the bottleneck.

In mango 3.7 there is. Quality of Service filter that could potentially be enabled and tuned for your needs. But before going into that you should figure out where the delay is, database query times, http request overload and simply server resources. I would do some testing to get a better idea first.

[MattFox]

Thanks for coming back to me at this ungodly hour Terry, sorry I'm stressing a little bit here due to being outside my knowledge base...
I'm still on 3.5.6, I don't want to do any upgrades due to the time of year and that our customers are more active during the spring/summer/autumn seasons and rely on our systems being up. Trying to maintain 99.9% uptime is a gripper.. That and I'm afraid of any breaking changes created could really throw a spanner in the development works...

I confirmed it's not the proxy as I loaded mango's pages directly and sure enough the pages dropped out.

#    Copyright (C) 2014 Infinite Automation Systems Inc. All rights reserved.
#    @author Matthew Lohbihler

###############################################################################
# TO OVERRIDE VALUES IN THIS FILE...
#
# Do not change the values in this file, because when you upgrade your core 
# your changes will be overwritten. Instead, create a new file called 
# <MA_HOME>/overrides/properties/env.properties and override properties 
# there. The overrides directory will never be overwritten by an upgrade, so 
# your customizations will be safe.
# 
###############################################################################

# The port at which Mango Automation will listen for browser connections
web.port=8080
# The host interface to which Mango Automation will bind and listen for new connections
#  0.0.0.0 is the special interface that will force a bind to all available interfaces
web.host=0.0.0.0

# Should Mango Automation open (if possible) a browser window when it starts up?
web.openBrowserOnStartup=true

# Web caching settings

# disable caching
web.cache.noStore=false
web.cache.noStore.rest=true
web.cache.noStore.resources=false

# set max age of cached files in seconds, only if noStore=false
# versioned resources are those with ?v=xxx on the query string
web.cache.maxAge=0
web.cache.maxAge.rest=0
web.cache.maxAge.resources=86400
web.cache.maxAge.versionedResources=31536000

#Upload file size limit (bytes) -1 means no limit
web.fileUpload.maxSize=50000000

# Default database settings, NOTE that on windows this must be an absolute path
#db.type=h2
#db.url=jdbc:h2:${ma.home}/databases/mah2
#db.location=${ma.home}/databases/mah2
#db.username=
#db.password=
#For web console
#db.web.start=false
#db.web.port=8091
#to compact the database size at shutdown (may take longer but will free up disk space)
#db.h2.shutdownCompact=false

#db.type=derby
#db.url=${ma.home}/databases/madb
#db.username=
#db.password=

#General Database Settings
db.pool.maxActive=100
db.pool.maxIdle=10
db.update.log.dir=${ma.home}/logs/

# setting to show query times in the logs as INFO
db.useMetrics=false
# if set, will only log slow queries, above this threshold in ms. Will be logged at WARN level instead of INFO
db.metricsThreshold=100

#--The following database properties are for RQL REST queries and can be changed during runtime and will be picked up at most in 5s--
#Force the use of indexes (Experimental and only on MySQL so far)
db.forceUseIndex=true
#Tell the jdbc driver to fetch this many rows at a time, useful over network connected dbs (Not MySQL)
# negative values will force use jdbc driver default
db.fetchSize=-1
#Tell the database to not return the entire result set (or fetch blocks) but to return row by row
# can slow down performance on network systems but reduce memory footprint for large queries
db.stream=false
#-- End auto-reloading Database Properties --

# MySQL database settings. Your MySQL instance must already be running and configured before this can be used.
#db.type=mysql
#db.url=jdbc:mysql://localhost/<your mysql schema name>
#db.username=<your mysql username>
#db.password=<your mysql password>
#db.mysqldump=<location/command for mysqldump executable for backups>
#db.mysql=<location/command for mysql executable for restore>

db.mysqldump=<location/command for mysqldump executable for backups>
db.mysql=<location/command for mysql executable for restore>


# MySQL database settings. Your MySQL instance must already be running and configured before this can be used.
db.type=mysql
db.url=jdbc:mysql://[DB_IP]/mangodb

db.username={USER}
db.password={PASS}
db.mysqldump=/usr/bin/mysqldump
db.mysql=/usr/bin/mysql

# Database settings for conversion. If the db.* settings point to a new database instance, and the convert type setting
# is set, Mango Automation will attempt to convert from the convert.db.* settings to the db.* settings
# Note that database conversions should not be performed in the same step as an upgrade. First upgrade, then convert.
#convert.db.type=h2
#convert.db.url=jdbc:h2:/mnt/disks/sdb/mango/databases/mah2
#convert.db.username=
#convert.db.password=
#convert.db.type=
#convert.db.url=${convert.db.url}
#convert.db.username=${convert.db.username}
#convert.db.password=${convert.db.password}

#Set the base path for where the NoSQL data will be stored
#db.nosql.location=${ma.home}/databases/
db.nosql.location=/mnt/disks/sdb/mango/databases/
#Set the folder name of the point value store
db.nosql.pointValueStoreName=mangoTSDB
#Set the number of files the database can have open at one time
db.nosql.maxOpenFiles=500
#Time after which a shard will be closed
db.nosql.shardStalePeriod=36000000
#Period to check for stale shards
db.nosql.flushInterval=300000
#Query Performance Tuning, File Access Type: Available[INPUT_STREAM,FILE_CHANNEL,RANDOM_ACCESS_FILE,MAPPED_BYTE_BUFFER]
db.nosql.shardStreamType=MAPPED_BYTE_BUFFER
#Setting to speed up NoSQL queries at the expense of a small increase in disk usage
db.nosql.reversible=true
#Setting this will convert your existing point value store [NONE, REVERSIBLE, UNREVERSIBLE]
db.nosql.convert=NONE
#Number of concurrent threads to use to convert the database
db.nosql.convertThreads=4
#Run the corruption scan if the db is marked dirty
db.nosql.runCorruptionOnStartupIfDirty=false

#Password encryption scheme [BCRYPT, SHA-1, NONE]
#Legacy is SHA-1, 2.8+ BCRYPT
#security.hashAlgorithm=BCRYPT

# The location of the Mango Automation store from which to get license files.
store.url=https://store.infiniteautomation.com

# SSL control

# *** NOTE ***
# You can generate a self-signed certificate for testing using the following command
# keytool -genkey -keyalg RSA -alias mango -keystore /location/to/keystore/file.jks -validity 365 -keysize 2048

# Enter keystore password: {type your keystore password <ENTER>}
# Re-enter new password: {type your keystore password <ENTER>}
# What is your first and last name?
#   [Unknown]: {the hostname mango is running on e.g. mymangotest.com OR localhost <ENTER>}
# What is the name of your organizational unit?
#   [Unknown]: {e.g. Mango testing <ENTER>}
# What is the name of your organization?
#   [Unknown]: {e.g. Infinite Automation Systems Inc. <ENTER>}
# What is the name of your City or Locality?
#   [Unknown]: {e.g. Erie <ENTER>}
# What is the name of your State or Province?
#   [Unknown]: {e.g. Colorado <ENTER>}
# What is the two-letter country code for this unit?
#   [Unknown]: {e.g. US <ENTER>}
# Is CN=localhost, OU=Development, O=Infinite Automation Systems Inc., L=Erie, ST=Colorado, C=US correct?
#   [no]:  {type yes <ENTER>}
# 
# Enter key password for <mango>
#         (RETURN if same as keystore password): {type your key password or just press <ENTER>}

# Note: Enabling SSL also turns on HSTS which may not be desirable, see below
ssl.on=false
ssl.port=8443
ssl.keystore.location=/location/to/keystore/file.jks
ssl.keystore.password=freetextpassword
# If they key password is commented out, it is assumed to be the same as the keystore password
#ssl.key.password=
#Time socket can be idle before being closed (ms)
ssl.socketIdleTimeout=70000

#Enable ALPN (Application-Layer Protocol Negotiation) for HTTP/2
# on current browsers HTTP/2 is only available for TLS/SSL connections.
# Note that with this setting you must also have the ALPN script extension enabled for Mango to start.
# (Adds -javaagent:${MA_HOME}/boot/jetty-alpn-agent-x.x.x.jar to the java options)
ssl.alpn.on=true
#Show debug output for alpn connections in log
ssl.alpn.debug=false

# Configure HSTS (HTTP Strict Transport Security)
# Enabled by default when ssl.on=true
# Sets the Strict-Transport-Security header, web browsers will always connect using HTTPS when they
# see this header and they will cache the result for max-age seconds
ssl.hsts.enabled=true
ssl.hsts.maxAge=31536000
ssl.hsts.includeSubDomains=false

# System time zone. Leave blank to use default VM time zone.
timezone=

#Rest API Configuration
rest.enabled=true

#Enable to make JSON More readable
rest.indentJSON=true
#Cross Origin Request Handling
rest.cors.enabled=true
rest.cors.allowedOrigins=*
rest.cors.allowedMethods=PUT,POST,GET,OPTIONS,DELETE
rest.cors.allowedHeaders=content-type,x-requested-with,authorization
rest.cors.exposedHeaders=
rest.cors.allowCredentials=true
rest.cors.maxAge=3600

#For rest API Documentation at /swagger-ui.html
swagger.enabled=true
#path to api-docs for swagger tools
springfox.documentation.swagger.v2.path=/swagger/v2/api-docs
#By default the api-docs endpoint used by the swagger-ui is protected
#  if it is allowed to be public then one can generate tokens in for Users
#  to supply to the swagger-ui and access the api endpoints
#  To use, enter: Bearer <space> <token value> into the Authorize value input in the swagger ui
swagger.apidocs.protected=true
#Regex Pattern to scan for REST API endpoints for Swagger to document/display
swagger.mangoApiVersion=v[12]


#Distributor Settings
distributor=IA

#Jetty Thread Pool Tuning
# Time a thread must be idle before killing to keep pool size at minimum
web.threads.msIdleTimeout=10000
# Number of threads to keep around to handle incoming connections
web.threads.minimum=10
# Number of threads allowed to be created to handle incoming requests as needed
web.threads.maximum=250
# Number of Requests To Queue if all threads are busy
web.requests.queueSize=200
# Ping timeout for response from browser
web.websocket.pingTimeoutMs=20000
#Time socket can be idle before being closed (ms)
web.socketIdleTimeout=30000

#Jetty JSP Configuration
# See here for options: http://www.eclipse.org/jetty/documentation/9.2.10.v20150310/configuring-jsp.html
web.jsp.development=true
web.jsp.genStringsAsCharArray=true
web.jsp.trimSpaces=false
web.jsp.classdebuginfo=false
web.jsp.supressSmap=true
web.jsp.compilerClassName=org.apache.jasper.compiler.JDTCompiler
web.jsp.compiler=modern
web.jsp.fork=false
web.jsp.keepgenerated=true

#iFrame Header Control iFrame Header Control 'X-Frame-Options' (case sensitive options)
# SAMEORIGIN - Only allow Mango to embed i-frames when the requesting page was loaded from the Mango domain
# DENY - Do not allow at all
# ANY - Do not even use the header at all 
# One specific domain name can be supplied so that the header becomes: ALLOW-FROM http://foo.bar.com 
web.security.iFrameAccess=SAMEORIGIN

#Regex used to match serial ports so they show up in the menu
serial.port.linux.regex=((cu|ttyS|ttyUSB|ttyACM|ttyAMA|rfcomm|ttyO|COM)[0-9]{1,3}|rs(232|485)-[0-9])
serial.port.linux.path=/dev/
serial.port.windows.regex=
serial.port.windows.path=
serial.port.osx.path=/dev/
serial.port.osx.regex=(cu|tty)..*

#Start data sources in parallel threads
runtime.datasource.startupThreads=8
#Log startup times for runtime manager
runtime.datasource.logStartupMetrics=true
#Log number of aborted polls for a polling data source this often at a minimum (only logged after next aborted poll past this time)
runtime.datasource.pollAbortedLogFrequency=3600000

#Report Javascript Execution Times at INFO Level logging
# add this to log4j.xml   <category name="org.perf4j.TimingLogger"><level value="info"/></category>
runtime.javascript.metrics=false

#Default task queue size for the Real Time Timer, should multiple tasks of the same type be queued up?
# Tasks are rejected from a full queue, a size of 0 means reject multiple instances of the same task
runtime.realTimeTimer.defaultTaskQueueSize=0
#When a task queue is full should the waiting tasks be discarded and replaced with the most recent
runtime.realTimeTimer.flushTaskQueueOnReject=false
#Delay (in ms) to wait to rate limit task rejection log messages so they don't fill up logs and use too much cpu doing it
runtime.taskRejectionLogPeriod=10000
#Maximum counts to wait to terminate the thread pool's tasks that are running or queued to run
# each count is 1 second.  So the default of 60 = 1 minute.  Note that the medium and low
# timeout happens first and then the remaining time is spent waiting of the high priority tasks.
# So by setting both to the same value will result in waiting only as long as that value.
runtime.shutdown.medLowTimeout=60
runtime.shutdown.highTimeout=60

# Set the location of the file stores
# If not set, the location is MA_HOME/filestore
filestore.location=

# Limits the rate at which an unauthenticated IP address can access the REST API Defaults to an initial 10 request burst then 2 requests per 1 second thereafter
rateLimit.rest.anonymous.enabled=true 
rateLimit.rest.anonymous.burstQuantity=10 
rateLimit.rest.anonymous.quanitity=2 
rateLimit.rest.anonymous.period=1 
rateLimit.rest.anonymous.periodUnit=SECONDS

# Limits the rate at which an authenticated user can access the REST API Disabled by default
rateLimit.rest.user.enabled=false 
rateLimit.rest.user.burstQuantity=20 
rateLimit.rest.user.quanitity=10 
rateLimit.rest.user.period=1 
rateLimit.rest.user.periodUnit=SECONDS

# Limits the rate at which authentication attempts can occur by an IP address Defaults to an initial 5 attempt burst then 1 attempt per 1 minute thereafter
rateLimit.authentication.ip.enabled=true 
rateLimit.authentication.ip.burstQuantity=5 
rateLimit.authentication.ip.quanitity=1 
rateLimit.authentication.ip.period=1 
rateLimit.authentication.ip.periodUnit=MINUTES

# Limits the rate at which authentication attempts can occur against a username Defaults to an initial 5 attempt burst then 1 attempt per 1 minute thereafter
rateLimit.authentication.user.enabled=true 
rateLimit.authentication.user.burstQuantity=5 
rateLimit.authentication.user.quanitity=1 
rateLimit.authentication.user.period=1
rateLimit.authentication.user.periodUnit=MINUTES

Follow symbolic links when serving files from Jetty
web.security.followSymlinks=true

# Content Security Policy settings, please see https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
# The reasons for the default policy are outlined below
# style-src 'unsafe-inline' - inline styles are used by AngularJS Material for the dynamic theming
# script-src 'unsafe-eval' - needed by Fabric.js used in amCharts for drawing on charts, also gives AngularJS a 30% performance boost
# connect-src ws: wss: - necessary as 'self' does not permit connections to websockets on the same origin, this should be configured to restrict it to your server's actual hostname
# img-src data: - allows for small base64 encoded images to be embedded inline into the html
# img-src/script-src https://www.google-analytics.com - allows for enabling Google analytics (not enabled by default, must be manually enabled by admin via UI Settings page)
# img-src/script-src https://maps.google.com https://maps.googleapis.com https://maps.gstatic.com - allows for using the Google maps component
# style-src/font-src https://fonts.googleapis.com https://fonts.gstatic.com - allows for using Google fonts in dashboards
web.security.contentSecurityPolicy.enabled=false
web.security.contentSecurityPolicy.reportOnly=false
web.security.contentSecurityPolicy.defaultSrc='self'
web.security.contentSecurityPolicy.scriptSrc='self' 'unsafe-eval' https://maps.google.com https://maps.googleapis.com https://www.google-analytics.com
web.security.contentSecurityPolicy.styleSrc='self' 'unsafe-inline' https://fonts.googleapis.com
web.security.contentSecurityPolicy.connectSrc='self' ws: wss:
web.security.contentSecurityPolicy.imgSrc='self' data: https://maps.google.com https://maps.gstatic.com https://www.google-analytics.com
web.security.contentSecurityPolicy.fontSrc='self' https://fonts.gstatic.com
web.security.contentSecurityPolicy.mediaSrc=
web.security.contentSecurityPolicy.objectSrc=
web.security.contentSecurityPolicy.frameSrc=
web.security.contentSecurityPolicy.workerSrc=
web.security.contentSecurityPolicy.manifestSrc=
web.security.contentSecurityPolicy.other=

# script-src 'unsafe-inline' - inline scripts are used extensively throughout the Mango legacy UI
# script-src 'unsafe-eval' - The Dojo JS library uses eval()
# style-src 'unsafe-inline' - inline styles are used throughout the Mango legacy UI
# connect-src ws: wss: - necessary as 'self' does not permit connections to websockets on the same origin, this should be configured to restrict it to your server's actual hostname
# img-src data: - allows for small base64 encoded images to be embedded inline into the html
# img-src/script-src https://www.google-analytics.com - allows for enabling Google analytics
web.security.contentSecurityPolicy.legacyUi.enabled=false
web.security.contentSecurityPolicy.legacyUi.reportOnly=false
web.security.contentSecurityPolicy.legacyUi.defaultSrc='self'
web.security.contentSecurityPolicy.legacyUi.scriptSrc='self' 'unsafe-inline' 'unsafe-eval' https://www.google-analytics.com
web.security.contentSecurityPolicy.legacyUi.styleSrc='self' 'unsafe-inline'
web.security.contentSecurityPolicy.legacyUi.connectSrc='self' ws: wss:
web.security.contentSecurityPolicy.legacyUi.imgSrc='self' data: https://www.google-analytics.com
web.security.contentSecurityPolicy.legacyUi.fontSrc=
web.security.contentSecurityPolicy.legacyUi.mediaSrc=
web.security.contentSecurityPolicy.legacyUi.objectSrc=
web.security.contentSecurityPolicy.legacyUi.frameSrc=
web.security.contentSecurityPolicy.legacyUi.workerSrc=
web.security.contentSecurityPolicy.legacyUi.manifestSrc=
web.security.contentSecurityPolicy.legacyUi.other=

# Set the location of the modules data directory, (relative to $MA_HOME if not absolute) If not set, the location is $MA_HOME/data
moduleData.location=data

# HTTP session (authentication) cookie name and domain name settings.
#
# Use the Mango GUID as the session cookie name
sessionCookie.useGuid=true
# name takes precedence over useGuid if set
sessionCookie.name=
# Set the domain name that the cookie is valid for, can be used to make the session login valid for subdomains too. If left blank the session cookie can only be used for the domain that you login at.
sessionCookie.domain=

Let me know if there's anything I can do here to increase performance

Fox

[terrypacker]

You can try logging slow queries to see if that is the problem, but it will require a restart.

db.useMetrics=true

Depending on how many points will be used concurrently you could increase this to allow more files to remain open longer:

db.nosql.maxOpenFiles=500

What are the specific endpoints that are slow if there are any specific ones?

[MattFox]

I know the system drops out when we are updating the values via the mango API every 5 minutes.
I'm looking at

/v1/point-values
Update one or many data point's current value

But am unsure if this will still apply an update event to affected datapoints, that and I'm not sure of the correct format.
thought it might be something like

{
xid:"aasdf",
timestamp:12314556677,
value:10,
type:"NUMERIC"
}

I'll try the max open files later tonight if you feel that could be a bottle neck if I have several 100 different points coming in with a 100 values at a time.

[terrypacker]

One thing I noted is that you are not using ssl which is likely due to being behind a proxy with ssl. However I believe this will force disable HTTP 2.0 which if enabled would improve the user experience via the multiplexing feature.

[terrypacker]

It could also be the updates firing events if there are a lot of listeners. There is a new endpoint in the later versions of Mango that let you choose to fire events or not but you may require these events to fire if you are using them in the context of meta points.

[terrypacker]

One more thought is to take some thread dumps via the UI and see what threads are using excessive cpu time, this won’t show much if it’s just a lot of short requests.

[MattFox]

Good idea with the thread dumps, if I can get through to have them exported during an import session I will post them here.

Thanks for your input here. Perhaps some feedback regarding my server configuration may help here so I can document best practices for running and getting the best out of the mango system.
Seeing how if SSL with http2 is the way to go perhaps I need to rejig how my server is configured.
Let's see how I go

Fox

[MattFox]

I've still got the API dropping in and out, can't always log in. Can I get someone to remote in and take a look?

Fox