Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.
Meta Data point with Excel Report
-
The following will actually crash my mango ES to the point where I have to manually restart it in terminal.
Make new Meta Data source. Make new object inside meta data source with following script:
return String(point.getDataPointWrapper().name);
"point" is of course set to another object, in order to get the name of said object into the excel report.
Then, add this object and with any rollup or no rollup, any time period, it will crash my mango. However, manually set this name as a new point in a virtual point, and it will work 100%.
Yes the object is alphanumeric.
I wasted a ton of time trying to fix this and just resolved to working around it with more work in the future.
-
Hi thewizardguy,
I was not able to replicate from this description. I created the point as described, added it to a very simple Excel Report and successfully generated reports with the name of the source point as the value of the alphanumeric meta point. Can you post a JSON you expect to cause it?
If it's crashing, there will be some kind of output. You can launch Mango on the command line with either
ma-start.bat
or./ma.sh start
and if there is an error going to stderr it will show up. Have you checked your /opt/mango/logs/ma.log file for relevant information? Otherwise it is a Java error and you will have an hs_err file in your Mango/ directory. -
@phildunlap First, thank you for responding so quickly!
I have to amend my previous statement also -- I tried it with a regular virtual data source and it worked ONCE, but now it won't work in the same way. When I say "crash" I mean it becomes unresponsive on the web interface, the top command shows java using about 50% total CPU, but the mango ES itself is responsive in terminal.
Here is some of the java output in ma.log:
WARN 2018-01-10T12:44:44,139 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T12:44:44,141 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T12:44:44,142 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full WARN 2018-01-10T12:44:44,143 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T12:44:44,155 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T12:44:51,559 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T12:44:58,851 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full ERROR 2018-01-10T12:44:58,852 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T12:44:58,852 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T12:44:58,854 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full WARN 2018-01-10T12:44:58,856 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T12:45:06,324 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 27 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T12:45:13,619 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T12:45:13,621 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T12:45:13,622 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T12:45:13,641 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T12:45:13,641 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T12:45:20,969 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 12 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T12:45:28,273 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T12:45:28,274 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T12:45:28,275 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
-
@phildunlap I started again with monitoring the ma.sh process from terminal and this is the "fresh" info right from when I tried the report:
This report was run just with the virtual data source:
WARN 2018-01-10T13:07:06,567 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full ERROR 2018-01-10T13:07:07,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:07:44,407 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:07:44,417 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:07:44,422 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:07:51,690 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full ERROR 2018-01-10T13:07:51,691 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:07:52,028 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:07:58,985 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:07:58,991 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:07:58,993 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:08:06,343 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:136) - Data Source MangoES System aborted 1 polls since it started. WARN 2018-01-10T13:08:06,364 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full ERROR 2018-01-10T13:08:06,367 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 11 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:08:06,411 (com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketPublisher.pointUpdated:179) - RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED] org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED] at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:308) ~[websocket-common-9.3.11.v20160721.jar:9.3.11.v20160721] at com.serotonin.m2m2.web.mvc.websocket.MangoWebSocketPublisher.sendMessage(MangoWebSocketPublisher.java:140) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketPublisher.pointUpdated(PointValueWebSocketPublisher.java:176) [mango-api-3.2.4.jar:?] at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:674) [mango-3.2.2.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessing$RejectableWorkItemRunnable.run(BackgroundProcessing.java:556) [mango-3.2.2.jar:?] at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.2.2.jar:?] at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.2.2.jar:?] at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:307) [mango-3.2.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] WARN 2018-01-10T13:08:17,147 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:08:17,151 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:08:17,161 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:08:17,161 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:08:17,167 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:08:26,969 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:08:27,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:08:38,925 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:08:38,927 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:08:38,928 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:08:38,929 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:08:38,936 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full WARN 2018-01-10T13:08:49,173 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full ERROR 2018-01-10T13:08:49,173 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:08:49,174 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full WARN 2018-01-10T13:08:49,175 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full ERROR 2018-01-10T13:08:49,177 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:09:00,568 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:09:00,570 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:09:00,571 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:09:00,572 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 9 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:09:02,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
-
Hmm. There is not a "Timer already cancelled" showing up anywhere in the logs, is there?
A high CPU utilization could suggest that you're close to the line on memory and so it's doing a lot of garbage collection. Do you by chance have a large events, userEvents, or audit table? I would encourage you to purge your events through the purge settings portion of the system settings ("Purge all events") and run
DELETE FROM audit;
in the SQL console. I would guess that it becomes responsive after a while, no? Does the report finish running? Have you tried a very small time range for the report?If the virtual point has the same behavior, I'm going to reason the meta script is incidental.
-
Also it may be wise to check if you're fully up to date. I don't think it's the case in your instance, but there is an issue fixed recently regarded the rejected BWB tasks.
-
@phildunlap This happens after the report "finishes"
ERROR 2018-01-10T13:09:45,101 (com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run:603) - Error in work item java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) ~[?:1.8.0_131] at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[?:1.8.0_131] at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[?:1.8.0_131] at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[?:1.8.0_131] at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88) ~[poi-ooxml-3.14.jar:3.14] at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590) ~[xmlbeans-2.6.0.jar:?] at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544) ~[xmlbeans-2.6.0.jar:?] at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223) ~[xmlbeans-2.6.0.jar:?] at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:3363) ~[poi-ooxml-3.14.jar:3.14] at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:3321) ~[poi-ooxml-3.14.jar:3.14] at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:464) ~[poi-ooxml-3.14.jar:3.14] at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:469) ~[poi-ooxml-3.14.jar:3.14] at org.apache.poi.POIXMLDocument.write(POIXMLDocument.java:211) ~[poi-ooxml-3.14.jar:3.14] at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:565) ~[excel-reports-3.2.1.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run(BackgroundProcessing.java:600) [mango-3.2.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] ERROR 2018-01-10T13:11:25,646 (com.infiniteautomation.mango.rest.v1.excelreports.ExcelReportRestController.getReportFile:287) - null java.lang.NullPointerException: null at java.io.File.<init>(File.java:360) ~[?:1.8.0_131] at com.infiniteautomation.mango.excelreports.ExcelReportsCommon.getReport(ExcelReportsCommon.java:174) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.rest.v1.excelreports.ExcelReportRestController.getReportFile(ExcelReportRestController.java:273) [excel-reports-3.2.1.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterInternal(MangoShallowEtagHeaderFilter.java:80) [mango-3.2.2.jar:?] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.authentication.switchuser.SwitchUserFilter.doFilter(SwitchUserFilter.java:198) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at com.serotonin.m2m2.web.mvc.spring.security.PermissionExceptionFilter.doFilter(PermissionExceptionFilter.java:32) [mango-3.2.2.jar:?] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:461) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.Server.handle(Server.java:524) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
-
Likely the out of memory error caused the null pointer exception. The solution there is going to be to shorten the time range, simplify the report, free memory up elsewhere (hence my suggestions for purging the events, userEvents and audit tables), or perhaps you can eek out a few more MB in the /opt/mango/bin/ext-enabled/memory-small.sh script.
-
@phildunlap I am completely up to date as far as i know, check for upgrades returns nothing new, Version is 3.2.2+2017009170034. I clicked "Purge all events," 23699 events deleted. The report only runs with "None" in rollup at the moment, and I've tried minutes, hours, weeks, days etc, DELETE FROM audit returned an error, I also tried "DELETE * FROM audit;" and it returned a similar error, im guessing that wasn't typed in or at least it doesnt show properly on the post.
The report finishes running, but I can never download it, and it runs for between 7-15 minutes.
Again, thank you for responding so quickly !
-
I try :D
I'm guessing the error with running the SQL had to do with pressing the 'Submit query' button instead of the 'Submit update' button. The likelihood is that the actual report file is not generated, as it ran out of memory while trying to generate that file, so it wouldn't be available for download. Had it been generated, I think you could find the xlsx file in /opt/mango/web/modules/excelReports/web/report-data
I would expect the memory footprint to decrease with the use of rollups, and larger rollups should do more to decrease the overhead.
-
@phildunlap I cannot use rollups for some reason, for when I do the job fails immediately (but didn't used to?) with this error:
ERROR 2018-01-10T13:06:15,836 (com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute:579) - Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell. com.serotonin.ShouldNeverHappenException: Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell. at com.infiniteautomation.mango.excelreports.rt.ExcelUtility.setCellValue(ExcelUtility.java:54) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportDataPointReference.insert(ExcelReportDataPointReference.java:102) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:81) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:28) ~[excel-reports-3.2.1.jar:?] at com.serotonin.m2m2.view.quantize2.ValueChangeCounterQuantizer.closePeriod(ValueChangeCounterQuantizer.java:41) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.view.quantize2.AbstractDataQuantizer.nextPeriod(AbstractDataQuantizer.java:69) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.view.quantize2.AbstractDataQuantizer.done(AbstractDataQuantizer.java:60) ~[mango-3.2.2.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportRollupDataPointReference.finish(ExcelReportRollupDataPointReference.java:76) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:537) [excel-reports-3.2.1.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run(BackgroundProcessing.java:600) [mango-3.2.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
The file in question shows 0 bytes like you said:
-rw-r--r-- 1 mango mango 0 Jan 10 13:08 Jan_10__2018_1_07_00_PM_alpha-test.xlsx
-
@phildunlap Ok, when I switched from "Past" to "Previous" it allowed rollups, but it gives the same Task Queue Full errors. The report is "Previous 1 Days" and Rollup is "Last 8 Hours," the report has like 8 named ranges available and I only use one for TIME and ALPHA, same errors. When I'm referencing Numeric points I have FAR more greedy time periods and rollups with no issues, the job is done in SECONDS. If it's running out of memory just trying to get 3 values for the last day... I daresay it isn't the reports fault =P
-
@phildunlap For posterity, this is the report with Past attempting to run, then the one with Previous, and it's still running:
ERROR 2018-01-10T13:43:50,046 (com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute:579) - Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell. com.serotonin.ShouldNeverHappenException: Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell. at com.infiniteautomation.mango.excelreports.rt.ExcelUtility.setCellValue(ExcelUtility.java:54) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportDataPointReference.insert(ExcelReportDataPointReference.java:102) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:81) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:28) ~[excel-reports-3.2.1.jar:?] at com.serotonin.m2m2.view.quantize2.ValueChangeCounterQuantizer.closePeriod(ValueChangeCounterQuantizer.java:41) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.view.quantize2.AbstractDataQuantizer.done(AbstractDataQuantizer.java:61) ~[mango-3.2.2.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportRollupDataPointReference.finish(ExcelReportRollupDataPointReference.java:76) ~[excel-reports-3.2.1.jar:?] at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:537) [excel-reports-3.2.1.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run(BackgroundProcessing.java:600) [mango-3.2.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] WARN 2018-01-10T13:45:13,809 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:45:13,811 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:45:13,811 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:45:13,856 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:45:22,589 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 12 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:45:22,592 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full WARN 2018-01-10T13:45:29,867 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:45:29,868 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:45:29,869 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:45:29,870 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:45:36,946 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:45:37,028 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:45:44,099 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:45:44,100 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:45:44,108 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:45:44,108 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:45:52,055 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:45:52,070 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:45:59,154 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:45:59,156 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:45:59,159 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:45:59,181 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:46:06,341 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:46:15,288 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full ERROR 2018-01-10T13:46:15,288 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:46:15,289 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:46:15,291 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:46:15,314 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:46:20,543 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 14 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:46:29,712 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:46:29,714 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:46:29,715 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full WARN 2018-01-10T13:46:29,716 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T13:46:29,718 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:46:37,268 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:46:44,436 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:46:44,437 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:46:44,438 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:46:44,469 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 6 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:46:52,092 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T13:46:52,092 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 17 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:47:00,146 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:47:00,150 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:47:00,151 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:47:00,153 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:08,045 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 9 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:08,051 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:47:15,816 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:47:15,817 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:47:15,879 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:47:15,885 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:23,243 (com.serotonin.bacnet4j.util.PropertyUtils.requestPropertiesFromDevice:209) - Timeout while finding device 1100 com.serotonin.bacnet4j.exception.BACnetTimeoutException: No response from instanceId 1100 at com.serotonin.bacnet4j.util.RemoteDeviceFinder$DeviceFutureImpl.get(RemoteDeviceFinder.java:176) ~[bacnet4j-4.1.2.jar:?] at com.serotonin.bacnet4j.LocalDevice$2.get(LocalDevice.java:714) ~[bacnet4j-4.1.2.jar:?] at com.serotonin.bacnet4j.util.PropertyUtils.requestPropertiesFromDevice(PropertyUtils.java:205) ~[bacnet4j-4.1.2.jar:?] at com.serotonin.bacnet4j.util.PropertyUtils.lambda$readProperties$0(PropertyUtils.java:166) ~[bacnet4j-4.1.2.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] ERROR 2018-01-10T13:47:23,250 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:23,263 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:23,264 (com.serotonin.m2m2.util.timeout.TimeoutTask.run:61) - Uncaught Task Exception org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED] at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:308) ~[websocket-common-9.3.11.v20160721.jar:9.3.11.v20160721] at com.serotonin.m2m2.web.mvc.websocket.MangoPingPongTracker.scheduleTimeout(MangoPingPongTracker.java:55) ~[mango-3.2.2.jar:?] at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:59) [mango-3.2.2.jar:?] at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.2.2.jar:?] at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.2.2.jar:?] at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:307) [mango-3.2.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] WARN 2018-01-10T13:47:31,948 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:47:31,949 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:47:31,950 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:47:31,954 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:32,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 6 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:40,003 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 13 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:47:48,616 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:47:48,618 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:47:48,619 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:47:48,626 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:48,685 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:47:56,570 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 13 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:04,536 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full ERROR 2018-01-10T13:48:04,542 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:04,542 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:48:04,543 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full WARN 2018-01-10T13:48:04,545 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T13:48:12,652 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 19 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:48:12,686 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:20,008 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:48:20,009 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:48:20,011 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:48:20,019 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 13 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:48:27,601 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:48:27,626 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:35,440 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:48:35,442 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:48:35,443 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:48:35,449 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 11 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:42,932 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T13:48:42,946 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 14 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:50,881 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:48:50,882 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:48:50,883 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:48:50,885 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 9 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:48:58,519 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full ERROR 2018-01-10T13:48:58,521 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full WARN 2018-01-10T13:49:03,694 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full WARN 2018-01-10T13:49:03,696 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full WARN 2018-01-10T13:49:03,698 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full ERROR 2018-01-10T13:49:03,698 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full ERROR 2018-01-10T13:49:08,778 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
-
@phildunlap I tried with "Submit Update" button and it "updated" 1600 entries (assuming it means deleted), so that worked, thanks!
-
I will send you an email about the "Cannot insert alphanumeric" error, as that is almost certainly a bug.
There isn't a very significant difference in handling the alphanumeric points for a report and handling the numeric points, so my guess is that the memory is being consumed elsewhere and the observation of one working quickly and one failing may also be incidental. But, I will consider that angle.
-
It was discovered upon investigation that the Excel report with a chart and the various dynamic elements had a much larger memory impact than a simple data report. The complex report would drain the memory until the garbage collector started thrashing, which caused the slowdown and 'crashing'. Resolution was found in modifying the Mango/bin/ext-enabled script to supply a few hundred more MB to Java, which led the complex Excel report to complete very quickly and without garbage collector thrashing. There was also a brief discussion of why this memory was previously reserved for the operating system.
-
TL;DR If you make an excel template, limit your named ranges to as small as possible
Alrighty, so after a few more days of troubleshooting, @phildunlap took a look at my excel template. He determined that the named range I designated in Excel was too long (only by a million cells!) for EACH named range, of which there are about 50. So, the theory was that the POI Apache plugin that allows Mango to interact with Excel was loading these empty cells into memory somehow, and was sporadically causing a memory leak or memory usage at least until the point where it would cripple the amount of available memory to the JVM to VERY VERY low, and render the garbage collection useless. Thats a mouthful.
And so, the problem only reared its ugly head once in awhile, it was hard to track down, but I've been operating for about a week now with no "crippling" memory issue, and I've launched thousands of reports without issue. For anyone who wants to know, the final template filled in is 82kb, has charts and 5 sheets of data, the time range spans 14 months and there are multiple different time ranges used and approximately 10-15 data points queried.