@phildunlap

Thanks for getting back to me.

It sounds like you have Mango running, so perhaps on the /sqlConsole.shtm page (or the /ui/administration/sql-console page) you can run.

drop index if exists "DATA4";

I just gave this a try but it didn't seem like DATA4 existed. If it did, deleting it didn't resolve the issue.

Do you have any existing core database backups in the Mango/backup directory? You could move your existing Mango/databases/ directory while Mango is off, then try restoring an older backup via the system settings page. If that works you could move in your old mangoTSDB directory to keep your data.

Unfortunately I didn't pick up this issue for a while so our latest backup is from 03/12/18. I will try using the recovery tool and hopefully that resolves unless you have a different idea?

Bryce

Hi Phil,

Sorry for the late reply, I got pulled onto another project and am only just getting back to this issue.

I have tried running the SQL database backup built in to Mango (Administration>System Settings> SQL database backup > BACKUP NOW) and get the following error:

WARN 2019-01-17T16:43:36,408 (com.serotonin.m2m2.rt.maint.work.DatabaseBackupWorkItem.execute:203) - org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [SCRIPT DROP TO 'c:\Users\admin\Downloads\free-m2m2-core-3.4.4\backup\core-database-H2-Jan-17-2019_164317.zip' COMPRESSION ZIP; ]; SQL state [90030]; error code [90030]; File corrupted while reading record: "index not found 228". Possible solution: use the recovery tool; SQL statement:SCRIPT DROP TO 'c:\Users\admin\Downloads\free-m2m2-core-3.4.4\backup\core-database-H2-Jan-17-2019_164317.zip' COMPRESSION ZIP; [90030-196]; nested exception is org.h2.jdbc.JdbcSQLException: File corrupted while reading record: "index not found 228". Possible solution: use the recovery tool; SQL statement:SCRIPT DROP TO 'c:\Users\admin\Downloads\free-m2m2-core-3.4.4\backup\core-database-H2-Jan-17-2019_164317.zip' COMPRESSION ZIP; [90030-196]

What is the recovery tool mentioned?

Bryce

Hello,

I've been trying to set up an excel report on our Mango system but am mostly getting this error:

ERROR 2018-10-29T12:26:28,740 (com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute:705) - PreparedStatementCallback; uncategorized SQLException; SQL state [HYC00]; error code [50100]; Feature not supported: "Index on BLOB or CLOB column: DATA4 CLOB"; SQL statement:
(select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, p
org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException; SQL state [HYC00]; error code [50100]; Feature not supported: "Index on BLOB or CLOB column: DATA4 CLOB"; SQL statement:
(select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  [50100-196]; nested exception is org.h2.jdbc.JdbcSQLException: Feature not supported: "Index on BLOB or CLOB column: DATA4 CLOB"; SQL statement:
(select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  [50100-196]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89) ~[spring-jdbc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1402) ~[spring-jdbc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:620) ~[spring-jdbc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
        at com.serotonin.m2m2.db.dao.PointValueDaoSQL.wideBookendQuery(PointValueDaoSQL.java:1056) ~[mango-3.5.1.jar:?]
        at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:586) ~[?:?]
        at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$WorkItemRunnable.run(BackgroundProcessingImpl.java:676) ~[mango-3.5.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_181]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: org.h2.jdbc.JdbcSQLException: Feature not supported: "Index on BLOB or CLOB column: DATA4 CLOB"; SQL statement:
(select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  UNION (select pv.dataPointId, pv.dataType, pv.pointValue, pva.textPointValueShort, pva.textPointValueLong, pv.ts, pva.sourceMessage from pointValues pv   left join pointValueAnnotations pva on pv.id = pva.pointValueId where pv.dataPointId=? AND pv.ts <= ? ORDER BY ts DESC LIMIT 1)  [50100-196]
        at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.message.DbException.get(DbException.java:179) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.message.DbException.get(DbException.java:155) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.message.DbException.getUnsupportedException(DbException.java:216) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.index.BaseIndex.checkIndexColumnTypes(BaseIndex.java:79) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.index.TreeIndex.<init>(TreeIndex.java:37) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.table.RegularTable.addIndex(RegularTable.java:254) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.result.ResultTempTable.createIndex(ResultTempTable.java:115) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.result.ResultTempTable.<init>(ResultTempTable.java:70) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.result.LocalResult.addRow(LocalResult.java:305) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Select.queryFlat(Select.java:523) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Select.queryWithoutCache(Select.java:625) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Query.query(Query.java:371) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Query.query(Query.java:333) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.SelectUnion.queryWithoutCache(SelectUnion.java:229) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Query.query(Query.java:347) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.dml.Query.query(Query.java:333) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.CommandContainer.query(CommandContainer.java:113) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.command.Command.executeQuery(Command.java:201) ~[h2-1.4.196.jar:1.4.196]
        at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:200) ~[h2-1.4.196.jar:1.4.196]
        at com.serotonin.m2m2.db.dao.PointValueDaoSQL$BookendSinglePointValuesPreparedStatementCreator.doInPreparedStatement(PointValueDaoSQL.java:918) ~[mango-3.5.1.jar:?]
        at com.serotonin.m2m2.db.dao.PointValueDaoSQL$BookendSinglePointValuesPreparedStatementCreator.doInPreparedStatement(PointValueDaoSQL.java:862) ~[mango-3.5.1.jar:?]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:605) ~[spring-jdbc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
        ... 6 more

Previously I was able to get around this with 2 of the 8 data points I want to bring into the Excel Report (so we were only populating a fraction of the report) but now no amount of exclusion gets rid of the error. I've checked the H2 error 50100 and this corresponds to an unsupported SQL feature.

I've tried to diagnose the difference between those two points and the other six but can't figure it out.

Look forward to hearing back.

Bryce

Thanks Phil,

We performed a clean install and this resolved the issue of our Mango instance crashing and not restoring itself. Performance was still poor so we also created meta points that rolled-up our data and then purged the old data source back to only a couple of days and this has significantly improved performance.

Bryce

The browser tab crashes? What's crashing when you say that? What browser are you using?

Sorry, mango hangs and we have to stop and start it. I'm using Chrome.

If you went through the modules page to do the upgrade this should have happened for you (it would delete the old lib files, move in the new). But, I have had reports of clean installations helping people, so it's something to try. I can assure you that charting does in fact work, generally, It could also be good to know what version of Java you are running, as updating to the latest JDK 8 is also a good course of action.

Will give it a shot, so I just copy the database and over rides folders in to the new installation home (but hold on the over-rides until after I've checked whether charting works or not). I thought the same thing about JDK so I have tried updating with no change. We're at 8u181 at the moment.

Hi Phil,

Hmm. Over what kind of time period, and with what data density? Have you tried short time ranges?

Even just opening the data point details page with one data point and no charting crashes, so I don't think the size matters.

You could open your browser's developer tools and check the network tab for the actual request for the data. That would tell us if the request isn't returning (what I expecT) or if there's some other issue going on in the front end.

I have checked this and three elements time out almost in sync in the network tab ( I have exported one of these as a .har which I am happy to share with you if it is of interest), and in the console there are lots of events similar to the following:

mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230 TypeError: Cannot read property 'push' of undefined
    at t.$applyAsync (mangoUi~ngMango.js?v=13b0f75f5f39034b19d2:84)
    at c.$eval (mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230)
    at mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230
    at _ (mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230)
    at c.$eval (mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230)
    at c.$apply (mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230)
    at mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230
    at p (mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230)
    at mangoUi~ngMango~ngMangoServices.js?v=f6d868035ceca7ed0360:230

and I've saved the log which if you would like a closer look at I'm also happy to share.

Note that for these I logged into the computer remotely and used localhost:8080 to ensure that there weren't any issues with headers getting tampered by the network.

The second error in the first post is weird to me. Do you have a web.xml file in your Mango/overrides/web/ directory? Have you modified the web.xml file? For some reason, your requests are not properly being identified as coming from your logged in user. I think that's the source of the issues, but I can only imagine it happening if there were some issue like overridden classes or duplicate jars in the Mango/lib/ directory, or maybe some intervening server messing with the headers.

Our overrides folder is fairly meager, with only 2 files. One is env.properties which I've changed the timezone in, and the other is web\images\logo.png.

Can you try starting up a newly downloaded Mango after copying your Mango/databases directory into the new Mango? Does that alleviate any of what you're seeing?

Recently I performed an update from 3.3.4 (I think) to 3.4.4 which sounds very similar to the process you're describing and the issue happened both before and after the upgrade. I will test this soon.

Are you trying to chart anywhere other than in a report instance? The errors you posted are from using the legacy reports page, but it sounded to me like your charts were not working on the /ui/data-point-details page. Can you clarify which charts you're talking about (since obviously you can generate old JFree charts!)

Currently anywhere other than a old style report chart crashes the system, so I'm using them to display data as an interim measure.

WARN  2018-09-21T14:25:26,655 (com.serotonin.m2m2.web.mvc.spring.exception.ExceptionUtils.logWebException:56) -
REQUEST URL
http://hostname-removed.com:8080/reportChart.shtm

REQUEST PARAMETERS
   instanceId=193


REQUEST HEADERS
   Cookie=MANGO8080=something; XSRF-TOKEN=something-else
   Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
   Upgrade-Insecure-Requests=1
   Connection=keep-alive
   User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
   Referer=http://hostname-removed.com:8080/reports.shtm
   Host=hostname-removed.com:8080
   Accept-Encoding=gzip, deflate
   Accept-Language=en-US,en;q=0.9


REQUEST ATTRIBUTES
   org.springframework.web.context.request.async.WebAsyncManager.WEB_ASYNC_MANAGER=org.springframework.web.context.request.async.WebAsyncManager@2a8e4a94
   javax.servlet.error.status_code=500
   org.springframework.web.servlet.DispatcherServlet.CONTEXT=WebApplicationContext for namespace 'springDispatcher-servlet': startup date [Fri Sep 21 18:38:52 PONT 2018]; parent: Root WebApplicationContext
   org.springframework.web.servlet.HandlerMapping.introspectTypeLevelMapping=false
   __spring_security_session_mgmt_filter_applied=true
   org.springframework.web.servlet.DispatcherServlet.LOCALE_RESOLVER=org.springframework.web.servlet.i18n.SessionLocaleResolver@7a7a9f08
   org.eclipse.jetty.server.newSessionId=1b0n6x32vb9rn1e21pfril6sqx
   org.springframework.web.servlet.HandlerMapping.bestMatchingPattern=/reportChart.shtm
   org.springframework.web.servlet.DispatcherServlet.OUTPUT_FLASH_MAP=FlashMap [attributes={}, targetRequestPath=null, targetRequestParams={}]
   javax.servlet.error.exception_type=class org.springframework.web.util.NestedServletException
   org.springframework.web.servlet.DispatcherServlet.FLASH_MAP_MANAGER=org.springframework.web.servlet.support.SessionFlashMapManager@526fa0ca
   lang=en
   lastUpgrade=1536312911
   availableLanguages=[]
   javax.servlet.http.HttpServletResponse=org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse@73b6fa4c
   _csrf=org.springframework.security.web.csrf.DefaultCsrfToken@2b34c444
   __spring_security_filterSecurityInterceptor_filterApplied=true
   javax.servlet.error.servlet_name=springDispatcher
   javax.servlet.error.message=org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.RuntimeException: User does not exist
   org.springframework.web.servlet.DispatcherServlet.THEME_SOURCE=WebApplicationContext for namespace 'springDispatcher-servlet': startup date [Fri Sep 21 18:38:52 PONT 2018]; parent: Root WebApplicationContext
   instanceDescription=Balance Group
   NEW_ID=-1
   org.springframework.web.servlet.HandlerMapping.pathWithinHandlerMapping=/reportChart.shtm
   org.springframework.security.web.csrf.CsrfToken=org.springframework.security.web.csrf.DefaultCsrfToken@2b34c444
   javax.servlet.error.request_uri=/reportChart.shtm
   org.springframework.web.servlet.DispatcherServlet.THEME_RESOLVER=org.springframework.web.servlet.theme.FixedThemeResolver@5989cf5
   javax.servlet.error.exception=org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.RuntimeException: User does not exist

java.lang.RuntimeException: User does not exist
        at com.serotonin.m2m2.reports.web.ReportCommon.ensureReportInstancePermission(ReportCommon.java:38) ~[reports-3.4.2.jar:?]
        at com.serotonin.m2m2.reports.web.ReportChartHandler.handleRequest(ReportChartHandler.java:32) ~[reports-3.4.2.jar:?]
        at com.serotonin.m2m2.web.mvc.UrlHandlerController.handleRequest(UrlHandlerController.java:36) ~[mango-3.4.5.jar:?]
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) ~[spring-webmvc-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at com.serotonin.m2m2.web.filter.WebContextFilter.doFilter(WebContextFilter.java:43) ~[mango-3.4.5.jar:?]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at com.serotonin.m2m2.web.filter.UrlSecurityFilter.doFilter(UrlSecurityFilter.java:191) ~[mango-3.4.5.jar:?]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at com.serotonin.web.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:37) ~[mango-3.4.5.jar:?]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterInternal(MangoShallowEtagHeaderFilter.java:90) ~[mango-3.4.5.jar:?]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at com.serotonin.m2m2.web.mvc.spring.security.PermissionExceptionFilter.doFilter(PermissionExceptionFilter.java:32) ~[mango-3.4.5.jar:?]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) ~[spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) ~[jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.Server.handle(Server.java:539) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
WARN  2018-09-21T14:30:45,958 (com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketHandler$PointValueWebSocketListener.pointUpdated:241) - Error sending websocket message
com.serotonin.m2m2.web.mvc.websocket.WebSocketSendException: org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSED], expecting [OPEN or CONNECTED]
        at com.serotonin.m2m2.web.mvc.websocket.MangoWebSocketHandler.sendStringMessageAsync(MangoWebSocketHandler.java:135) ~[mango-3.4.5.jar:?]
        at com.serotonin.m2m2.web.mvc.websocket.MangoWebSocketHandler.sendMessage(MangoWebSocketHandler.java:94) ~[mango-3.4.5.jar:?]
        at com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketHandler.sendMessage(PointValueWebSocketHandler.java:143) ~[mango-api-3.4.6.jar:?]
        at com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketHandler$PointValueWebSocketListener.sendNotification(PointValueWebSocketHandler.java:190) ~[mango-api-3.4.6.jar:?]
        at com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketHandler$PointValueWebSocketListener.pointUpdated(PointValueWebSocketHandler.java:238) [mango-api-3.4.6.jar:?]
        at com.serotonin.m2m2.rt.dataImage.DataPointEventMulticaster.pointUpdated(DataPointEventMulticaster.java:151) [mango-3.4.5.jar:?]
        at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:741) [mango-3.4.5.jar:?]
        at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) [mango-3.4.5.jar:?]
        at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.4.5.jar:?]
        at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.4.5.jar:?]
        at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:336) [mango-3.4.5.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSED], expecting [OPEN or CONNECTED]
        at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:308) ~[websocket-common-9.3.23.v20180228.jar:9.3.23.v20180228]
        at com.serotonin.m2m2.web.mvc.websocket.MangoWebSocketHandler.sendStringMessageAsync(MangoWebSocketHandler.java:133) ~[mango-3.4.5.jar:?]
        ... 13 more

Below is the report chart of Free JVM, as you suggested it does reach zero, how should I combat this?

Bryce

Hi Phil,

Sorry about the formatting.

We have not been able to get any charting. We have set roll-up to 10 mins and simplify with a 500 tolerance.

We have 8GB of memory on the system and have used the medium memory extension (I changed it from 5G to 4G).

I have just recreated the issue but nothing appeared in the command line for me to provide. I checked task manager while it was frozen and the memory used by java was ~3.2GB with ~75% of total memory being used. Disk speed was sitting at 80-95%.

A similar issue happened again while trying to get the free java memory plot and this was in the command line:

ERROR 2018-09-21T14:24:52,809 (com.serotonin.m2m2.web.mvc.websocket.MangoPingPongTracker.sendPing:53) - Error sending websocket ping
java.io.IOException: Unable to obtain RemoteEndpoint in session 76ce53dc
        at org.springframework.web.socket.adapter.jetty.JettyWebSocketSession.getRemoteEndpoint(JettyWebSocketSession.java:296) ~[spring-websocket-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.socket.adapter.jetty.JettyWebSocketSession.sendPingMessage(JettyWebSocketSession.java:283) ~[spring-websocket-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:107) ~[spring-websocket-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at com.serotonin.m2m2.web.mvc.websocket.MangoPingPongTracker.sendPing(MangoPingPongTracker.java:50) [mango-3.4.5.jar:?]
        at com.serotonin.m2m2.web.mvc.websocket.MangoPingPongTracker.scheduleTimeout(MangoPingPongTracker.java:82) [mango-3.4.5.jar:?]
        at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:59) [mango-3.4.5.jar:?]
        at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.4.5.jar:?]
        at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.4.5.jar:?]
        at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:336) [mango-3.4.5.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]
        at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:308) ~[websocket-common-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.springframework.web.socket.adapter.jetty.JettyWebSocketSession.getRemoteEndpoint(JettyWebSocketSession.java:293) ~[spring-websocket-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        ... 11 more
ERROR 2018-09-21T14:25:26,561 (com.serotonin.m2m2.web.dwr.util.ExceptionDetectionFilter.doFilter:38) - DWR invocation exception
com.serotonin.m2m2.vo.permission.PermissionException: Not authenticated
        at com.serotonin.m2m2.vo.permission.Permissions.ensureValidUser(Permissions.java:59) ~[mango-3.4.5.jar:?]
        at com.serotonin.m2m2.vo.permission.Permissions.hasAdmin(Permissions.java:77) ~[mango-3.4.5.jar:?]
        at com.serotonin.m2m2.reports.web.ReportsDwr.getReportInstances(ReportsDwr.java:409) ~[reports-3.4.2.jar:?]
        at com.serotonin.m2m2.reports.web.ReportsDwr.getReportInstances(ReportsDwr.java:398) ~[reports-3.4.2.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_181]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_181]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_181]
        at org.directwebremoting.impl.ExecuteAjaxFilter.doFilter(ExecuteAjaxFilter.java:34) ~[dwr-2.0.11.jar:?]
        at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428) ~[dwr-2.0.11.jar:?]
        at com.serotonin.m2m2.web.dwr.util.TranslationsFilter.doFilter(TranslationsFilter.java:37) ~[mango-3.4.5.jar:?]
        at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428) ~[dwr-2.0.11.jar:?]
        at com.serotonin.m2m2.web.dwr.util.ExceptionDetectionFilter.doFilter(ExceptionDetectionFilter.java:26) [mango-3.4.5.jar:?]
        at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428) [dwr-2.0.11.jar:?]
        at com.serotonin.m2m2.web.dwr.util.DwrPermissionFilter.doFilter(DwrPermissionFilter.java:46) [mango-3.4.5.jar:?]
        at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428) [dwr-2.0.11.jar:?]
        at org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:431) [dwr-2.0.11.jar:?]
        at org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:283) [dwr-2.0.11.jar:?]
        at org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:52) [dwr-2.0.11.jar:?]
        at org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101) [dwr-2.0.11.jar:?]
        at org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146) [dwr-2.0.11.jar:?]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterInternal(MangoShallowEtagHeaderFilter.java:90) [mango-3.4.5.jar:?]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at com.serotonin.m2m2.web.mvc.spring.security.PermissionExceptionFilter.doFilter(PermissionExceptionFilter.java:32) [mango-3.4.5.jar:?]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.Server.handle(Server.java:539) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]

Bryce

Page 1
HI,
Need Help!
We are having problems trying to chat with any of our data points, even using the Mango examples – “Line chart” - selecting a single Data point.
We lose connectivity, system stops and need to restart Mango.
I have copy and paste the log file below.
Thanks in advance.
B

INFO  2018-09-19T14:09:45,931 (com.serotonin.m2m2.Main.main:97) - Starting Mango 3.4.5 with Schema Version: 25 
...
INFO 2018-09-19T14:30:11,526 (org.springframework.web.servlet.handler.AbstractUrlHandlerMapping.registerHandler:362) - Mapped URL path [/globalScripts.shtm] onto handler of type [class com.serotonin.m2m2.web.mvc.UrlHandlerController]
WARN 2018-09-19T14:31:08,480 (com.serotonin.m2m2.web.mvc.spring.security.MangoAccessDeniedHandler.handle:53) - Denying access to Mango resource /sdk to IP 172.16.0.164
    org.springframework.security.web.csrf.MissingCsrfTokenException: Could not verify the provided CSRF token because your session was not found.
    at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:114) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.17.RELEASE.jar:4.3.17.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.Server.handle(Server.java:539) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.23.v20180228.jar:9.3.23.v20180228]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]

@phildunlap

Thank you, that has it fixed. I appreciate your quick response.

Bryce