We did get the upgrade to work today so we will let that sit as is for a while before I start changing things again. We do get a few of these now:
High priority task: com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$PointDataCleaner was rejected because it is already running.
Can't load system settings modele after upgrade. I get a system exception when trying to view the tab, everything else appears to be working. Tried two different computers and different browsers, cleared cache, restarted manngo. Can't get it to come back, any ideas?
org.apache.jasper.JasperException: javax.servlet.ServletException: javax.servlet.jsp.JspException: javax.servlet.jsp.JspException: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:455)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:405)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:349)
at org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:107)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:595)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:191)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:72)
at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168)
at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1244)
at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1027)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:971)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at com.serotonin.m2m2.web.filter.WebContextFilter.doFilter(WebContextFilter.java:43)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at com.serotonin.m2m2.web.filter.LoggedInFilter.doFilter(LoggedInFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at com.serotonin.m2m2.web.filter.UrlSecurityFilter.doFilter(UrlSecurityFilter.java:206)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at com.serotonin.web.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:37)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:87)
at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterInternal(MangoShallowEtagHeaderFilter.java:47)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.serotonin.m2m2.web.mvc.spring.security.CsrfHeaderFilter.doFilterInternal(CsrfHeaderFilter.java:41)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.servlet.ServletException: javax.servlet.jsp.JspException: javax.servlet.jsp.JspException: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
at org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:901)
at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:834)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp._jspService(systemSettings_jsp.java:144)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
... 97 more
Caused by: javax.servlet.jsp.JspException: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp$Helper.invoke(systemSettings_jsp.java:9501)
at org.apache.jsp.tag.web.labelledSection_tag.doTag(labelledSection_tag.java:153)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp._jspx_meth_tag_005flabelledSection_005f12(systemSettings_jsp.java:3018)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp.access$37(systemSettings_jsp.java:3000)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp$Helper.invoke0(systemSettings_jsp.java:7327)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp$Helper.invoke(systemSettings_jsp.java:9442)
at org.apache.jsp.tag.web.page_tag.doTag(page_tag.java:354)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp._jspService(systemSettings_jsp.java:131)
... 100 more
Caused by: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358)
at org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:107)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:595)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.include(Dispatcher.java:115)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:934)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp$Helper.invoke13(systemSettings_jsp.java:8984)
at org.apache.jsp.WEB_002dINF.jsp.systemSettings_jsp$Helper.invoke(systemSettings_jsp.java:9481)
... 107 more
Caused by: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:131)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:62)
at org.apache.tomcat.SimpleInstanceManager.newInstance(SimpleInstanceManager.java:51)
at org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:172)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:405)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:349)
... 123 more
Caused by: java.lang.ClassNotFoundException: org.apache.jasper.runtime.JspSourceImports
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:125)
at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:62)
... 139 more
REQUEST URL
https://mango.iders.ca:8443/exception/error.jsp
REQUEST PARAMETERS
REQUEST HEADERS
Cookie=XSRF-TOKEN=35b4558d-8274-42da-90aa-66e789747f30; MANGO80=4ut6dlen5ow11xnbe9p673362
Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests=1
Connection=keep-alive
User-Agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Referer=https://mango.iders.ca:8443/events.shtm?level=none
Host=mango.iders.ca:8443
Accept-Encoding=gzip, deflate, sdch, br
Accept-Language=en-US,en;q=0.8
REQUEST ATTRIBUTES
javax.servlet.forward.context_path=
org.springframework.web.context.request.async.WebAsyncManager.WEB_ASYNC_MANAGER=org.springframework.web.context.request.async.WebAsyncManager@3cf2f214
javax.servlet.error.status_code=500
javax.servlet.forward.servlet_path=/system_settings.shtm
org.springframework.web.servlet.DispatcherServlet.CONTEXT=WebApplicationContext for namespace 'springDispatcher-servlet': startup date [Thu Jan 05 12:22:31 CST 2017]; parent: Root WebApplicationContext
javax.servlet.jsp.jstl.fmt.localizationContext.request=org.springframework.web.servlet.support.JstlUtils$SpringLocalizationContext@462587a2
org.springframework.web.servlet.HandlerMapping.introspectTypeLevelMapping=false
org.eclipse.jetty.server.error_page=/exception/error.jsp
__spring_security_session_mgmt_filter_applied=true
urlSecurity=true
modulePath=/
org.springframework.web.servlet.DispatcherServlet.LOCALE_RESOLVER=org.springframework.web.servlet.i18n.SessionLocaleResolver@10dae506
org.springframework.web.servlet.HandlerMapping.bestMatchingPattern=/system_settings.shtm
org.springframework.web.servlet.DispatcherServlet.OUTPUT_FLASH_MAP=FlashMap [attributes={}, targetRequestPath=null, targetRequestParams={}]
javax.servlet.error.exception_type=class org.apache.jasper.JasperException
org.springframework.web.servlet.DispatcherServlet.FLASH_MAP_MANAGER=org.springframework.web.servlet.support.SessionFlashMapManager@1c619d23
lang=en
javax.servlet.forward.request_uri=/system_settings.shtm
javax.servlet.request.key_size=128
availableLanguages=[com.serotonin.db.pair.StringStringPair@27b8eff4]
javax.servlet.http.HttpServletResponse=org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse@af490ab
__spring_security_filterSecurityInterceptor_filterApplied=true
_csrf=org.springframework.security.web.csrf.DefaultCsrfToken@5c8d7918
javax.servlet.error.servlet_name=springDispatcher
javax.servlet.error.message=org.apache.jasper.JasperException: javax.servlet.ServletException: javax.servlet.jsp.JspException: javax.servlet.jsp.JspException: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
javax.servlet.request.cipher_suite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
javax.servlet.request.ssl_session_id=586e92C58810C38aB4D1528fEf344b1aCc483bDa8f874bC15751028f0112Cc6e
org.springframework.web.servlet.DispatcherServlet.THEME_SOURCE=WebApplicationContext for namespace 'springDispatcher-servlet': startup date [Thu Jan 05 12:22:31 CST 2017]; parent: Root WebApplicationContext
instanceDescription=Iders HVAC Automation
NEW_ID=-1
org.springframework.web.servlet.HandlerMapping.pathWithinHandlerMapping=/system_settings.shtm
org.springframework.security.web.csrf.CsrfToken=org.springframework.security.web.csrf.DefaultCsrfToken@5c8d7918
javax.servlet.error.request_uri=/system_settings.shtm
org.springframework.web.servlet.DispatcherServlet.THEME_RESOLVER=org.springframework.web.servlet.theme.FixedThemeResolver@5a285ca8
javax.servlet.jsp.jstl.fmt.locale.request=en_US
javax.servlet.error.exception=org.apache.jasper.JasperException: javax.servlet.ServletException: javax.servlet.jsp.JspException: javax.servlet.jsp.JspException: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceImports
SESSION ATTRIBUTES
org.springframework.security.web.csrf.HttpSessionCsrfTokenRepository.CSRF_TOKEN=org.springframework.security.web.csrf.DefaultCsrfToken@5c8d7918
sessionUser=User [id=2, username=Brett, password={SHA-1}51rl7x8yy4n1+z1k+IXMuYKy1Fw=, email=btrombo@iders.ca, phone=, disabled=false, homeUrl=, lastLogin=1483633169866, receiveAlarmEmails=-3, receiveOwnAuditEvents=false, timezone=, permissions=Brett,user,superadmin]
org.eclipse.jetty.security.sessionKnownOnlytoAuthenticated=true
LONG_POLL_DATA_TIMEOUT=1483641541699
SPRING_SECURITY_CONTEXT=org.springframework.security.core.context.SecurityContextImpl@14b15681: Authentication: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@14b15681: Principal: User [id=2, username=Brett, password={SHA-1}51rl7x8yy4n1+z1k+IXMuYKy1Fw=, email=btrombo@iders.ca, phone=, disabled=false, homeUrl=, lastLogin=1483633169866, receiveAlarmEmails=-3, receiveOwnAuditEvents=false, timezone=, permissions=Brett,user,superadmin]; Credentials: [PROTECTED]; Authenticated: true; Details: null; Granted Authorities: ROLE_Brett, ROLE_user, ROLE_superadmin, ROLE_ADMIN
LONG_POLL_DATA=[com.serotonin.m2m2.web.dwr.longPoll.LongPollData@77c452cf, com.serotonin.m2m2.web.dwr.longPoll.LongPollData@1db6c302]
Basically 0 swap over the timeframe.
Virtual Memory Used
Idle Memory
Cache
A restart fixed the issue, it looks like the issue takes some time to come back or maybe some task we do causes the issue to come back but here is a CPU non kernal over 4 weeks. The drops are all manual restarts. We will try to upgrade again today, the last try failed so we rolled back.
Well this problem has come back, all the changes we did before are still in place so I'm guessing those changes just helped the root cause be less of an issue or another issue has come up maybe just all the restarts during the last debugging helped. All cores are pinned this time.
I changed to MAPPED_BYTE_BUFFER and also tried the 200 and 50 settings. The 50 setting has no change that I could notice but changing it to 200 had even higher CPU usage. I put it at 500 and looks reasonable now. The core is no longer pinned and looks like the major issue has been resolved.
I adjusted the core pool to your suggestion except high priority max I changed to 10000. My system is 95% BACnet thus a while ago we needed to increase it because of issues and we have not touched it since. Those pictures were using htop a standard top is attached.
Here is the current NoSQL settings:
There are no scripts enabling or disabling data sources or data points. I know that we do use similar to return p.past(MINUTE).sum; but I will have to look a little closer at what and when that is happening.
I will try out some of your suggestions today and see if I can notice anything. Currently mango is running on a vmware server (ESXI) with about 7 other servers. The hardware of the server is an 8x2ghz cpu and mango has access to all 8 cores and 8GB of memory. The disks in the server I believe are Western digital red drives in a RAID configuration. The other 7 servers don’t use much of any of the resources (cpu time, disk access) most of the time. It is currently not an option to move mango to a dedicated server. It may be possible in the future but at this time I see no evidence of the other machines interfering with mango.
Hey thanks for the reply, I downloaded 5 dumps and emailed them in. I do have internal metrics for almost all the parameters available.
Here are two internal metric screenshots, I look at these often and they are stable as in I don't notice much change to either of these.
Idle Memory:
Virtual Memory:
Gone through all these options, but still a pinned core. I tried to clean a lot of stuff up in the system(unused sources and points), I purged a bunch of old event logs and such and the UI is much more responsive in combination with all the meta point changes. The memory usage does still creep up over time as well. Any more ideas?
I'm not sure if this has any help in trying to debug this but I noticed that I would get many alarms(timeouts, aborted polls) during the autobackup of the H2 Database Backup Settings.
So this took a while but I did both of these and still a CPU core pinned. I'm going to double check the meta data points tomorrow as that was a lot to go through and might have missed some. Although I did get at least 95%+ of them there is no improvement, I'm not sure if one could cause this issue or if the CPU usage should have been progressively less as I changed them.
Those errors are no longer logging.
I would say yes I have a lot of these(75ish without counting) and a bunch of them do use points from the same data source and set to update on context change.
Yes 1.2.3, db.nosql.shardStreamType=INPUT_STREAM
I found this on the thread monitoring, I'm assuming the high CPU time would relate to the two cores that are pinned. Not sure if this can help you diagnose the issues. I also emailed you the log files from the BACnet issue.
Just want to clarify that by process event handlers you are talking about an event detector that runs a local process or shell command? I have many event detectors that send emails if a data point is above or below a threshold but nothing that runs OS level commands.
I received your email.
Hey Phildunlap,
The way we have it arranged is each physical device is a datasource. I tried this command and didn't really change anything in terms of performance. Same resource usage and everything was working.
Looks like there is a lot of theses in the logs.
2016-10-06 15:40:48,337 (com.serotonin.ma.bacnet.BACnetDataSourceRT.fireMessageExceptionEvent:651) -
java.lang.IllegalStateException: Cannot send future request in the transport thread. Use a callback call instead, or make this
call in a new thread.
at com.serotonin.bacnet4j.transport.DefaultTransport.send(DefaultTransport.java:239)
at com.serotonin.bacnet4j.LocalDevice.send(LocalDevice.java:389)
at com.serotonin.ma.bacnet.BACnetDataSourceRT.sendCovSubscriptionImpl(BACnetDataSourceRT.java:630)
at com.serotonin.ma.bacnet.BACnetDataSourceRT.covNotificationReceived(BACnetDataSourceRT.java:548)
at com.serotonin.bacnet4j.event.DeviceEventHandler.fireCovNotification(DeviceEventHandler.java:134)
at com.serotonin.bacnet4j.service.unconfirmed.UnconfirmedCovNotificationRequest.handle(Uncon 2016-10-06 15:40:48,337
(com.serotonin.ma.bacnet.BACnetDataSourceRT.fireMessageExceptionEvent:651) -
java.lang.IllegalStateException: Cannot send future request in the transport thread. Use a callback call instead, or make this call in a new thread.
at com.serotonin.bacnet4j.transport.DefaultTransport.send(DefaultTransport.java:239)
at com.serotonin.bacnet4j.LocalDevice.send(LocalDevice.java:389)
at com.serotonin.ma.bacnet.BACnetDataSourceRT.sendCovSubscriptionImpl(BACnetDataSourceRT.java:630)
at com.serotonin.ma.bacnet.BACnetDataSourceRT.covNotificationReceived(BACnetDataSourceRT.java:548)
at com.serotonin.bacnet4j.event.DeviceEventHandler.fireCovNotification(DeviceEventHandler.java:134)
at com.serotonin.bacnet4j.service.unconfirmed.UnconfirmedCovNotificationRequest.handle(UnconfirmedCovNotificationRequest.java:69)
at com.serotonin.bacnet4j.transport.DefaultTransport.receiveAPDU(DefaultTransport.java:539)
at com.serotonin.bacnet4j.transport.DefaultTransport.receiveImpl(DefaultTransport.java:463)
at com.serotonin.bacnet4j.transport.DefaultTransport.run(DefaultTransport.java:404)
at java.lang.Thread.run(Thread.java:745)
INFO
On a clean reboot two cores are pinned to 100% while the remaining 6 are low as expected. Ram usage is 50% and Mango still functions but after days to weeks I need to restart the system as datasources stop updating(At this point almost all the ram is used). I have been trying to figure out where the issues arise. I have 6 GB allocated to Java and 8GB total for the system. I have all the newest Mango updates. Restarted the server recently. I included some screenshot metrics page and htop of the system. This is a very BACnet heavy system. I do get the timeout alerts that others have seen from the recent update also. I'm looking for some advice on how to resolve this issue.
Twilio is a publisher not a datasource so you will find it on the publisher module.
I'm not sure if this exists or not but I would like to be able to implement the Timesync function in BACnet. It is basically a simple Broadcast that sets the time on all devices in the network. Mango being the highest level of an automation system is the ideal place to implement this. Anyone done this or have a way to do it? My guess is that this would have to be an update to the BACnet module to allow this type of message to be sent.
Thanks for the video, had a report working in under a minute. Is there anyway to run these reports besides a set schedule or on demand? Say from a script or event detector? The use case here is a failure or unexpected data from a device, it would then email a report will all the details of that specific device.