• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    Meta Data point with Excel Report

    Scripting general Discussion
    2
    17
    2.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      thewizardguy @phildunlap
      last edited by phildunlap

      @phildunlap First, thank you for responding so quickly!

      I have to amend my previous statement also -- I tried it with a regular virtual data source and it worked ONCE, but now it won't work in the same way. When I say "crash" I mean it becomes unresponsive on the web interface, the top command shows java using about 50% total CPU, but the mango ES itself is responsive in terminal.

      Here is some of the java output in ma.log:

      WARN  2018-01-10T12:44:44,139 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T12:44:44,141 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T12:44:44,142 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
WARN  2018-01-10T12:44:44,143 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T12:44:44,155 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T12:44:51,559 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T12:44:58,851 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-10T12:44:58,852 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 2 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T12:44:58,852 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T12:44:58,854 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
WARN  2018-01-10T12:44:58,856 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T12:45:06,324 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 27 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T12:45:13,619 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T12:45:13,621 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T12:45:13,622 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T12:45:13,641 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T12:45:13,641 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T12:45:20,969 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 12 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T12:45:28,273 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T12:45:28,274 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T12:45:28,275 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
      1 Reply Last reply Reply Quote 0
      • T
        thewizardguy
        last edited by phildunlap

        @phildunlap I started again with monitoring the ma.sh process from terminal and this is the "fresh" info right from when I tried the report:

        This report was run just with the virtual data source:

        WARN  2018-01-10T13:07:06,567 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-10T13:07:07,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:07:44,407 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:07:44,417 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:07:44,422 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:07:51,690 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
ERROR 2018-01-10T13:07:51,691 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:07:52,028 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:07:58,985 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:07:58,991 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:07:58,993 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:08:06,343 (com.serotonin.m2m2.rt.dataSource.PollingDataSource.incrementUnsuccessfulPolls:136) - Data Source MangoES System aborted 1 polls since it started.
WARN  2018-01-10T13:08:06,364 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
ERROR 2018-01-10T13:08:06,367 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 11 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:08:06,411 (com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketPublisher.pointUpdated:179) - RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]
org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]
	at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:308) ~[websocket-common-9.3.11.v20160721.jar:9.3.11.v20160721]
	at com.serotonin.m2m2.web.mvc.websocket.MangoWebSocketPublisher.sendMessage(MangoWebSocketPublisher.java:140) ~[mango-3.2.2.jar:?]
	at com.serotonin.m2m2.web.mvc.rest.v1.publisher.pointValue.PointValueWebSocketPublisher.pointUpdated(PointValueWebSocketPublisher.java:176) [mango-api-3.2.4.jar:?]
	at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:674) [mango-3.2.2.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$RejectableWorkItemRunnable.run(BackgroundProcessing.java:556) [mango-3.2.2.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.2.2.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.2.2.jar:?]
	at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:307) [mango-3.2.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
WARN  2018-01-10T13:08:17,147 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:08:17,151 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:08:17,161 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:08:17,161 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:08:17,167 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:08:26,969 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:08:27,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:08:38,925 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:08:38,927 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:08:38,928 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:08:38,929 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:08:38,936 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
WARN  2018-01-10T13:08:49,173 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-10T13:08:49,173 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:08:49,174 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
WARN  2018-01-10T13:08:49,175 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
ERROR 2018-01-10T13:08:49,177 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:09:00,568 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:09:00,570 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:09:00,571 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:09:00,572 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 9 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:09:02,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
        1 Reply Last reply Reply Quote 0
        • phildunlapP
          phildunlap
          last edited by

          Hmm. There is not a "Timer already cancelled" showing up anywhere in the logs, is there?

          A high CPU utilization could suggest that you're close to the line on memory and so it's doing a lot of garbage collection. Do you by chance have a large events, userEvents, or audit table? I would encourage you to purge your events through the purge settings portion of the system settings ("Purge all events") and run DELETE FROM audit; in the SQL console. I would guess that it becomes responsive after a while, no? Does the report finish running? Have you tried a very small time range for the report?

          If the virtual point has the same behavior, I'm going to reason the meta script is incidental.

          T 2 Replies Last reply Reply Quote 0
          • phildunlapP
            phildunlap
            last edited by

            Also it may be wise to check if you're fully up to date. I don't think it's the case in your instance, but there is an issue fixed recently regarded the rejected BWB tasks.

            1 Reply Last reply Reply Quote 0
            • T
              thewizardguy @phildunlap
              last edited by phildunlap

              @phildunlap This happens after the report "finishes"

              ERROR 2018-01-10T13:09:45,101 (com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run:603) - Error in work item
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236) ~[?:1.8.0_131]
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[?:1.8.0_131]
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[?:1.8.0_131]
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[?:1.8.0_131]
	at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88) ~[poi-ooxml-3.14.jar:3.14]
	at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590) ~[xmlbeans-2.6.0.jar:?]
	at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544) ~[xmlbeans-2.6.0.jar:?]
	at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223) ~[xmlbeans-2.6.0.jar:?]
	at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:3363) ~[poi-ooxml-3.14.jar:3.14]
	at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:3321) ~[poi-ooxml-3.14.jar:3.14]
	at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:464) ~[poi-ooxml-3.14.jar:3.14]
	at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:469) ~[poi-ooxml-3.14.jar:3.14]
	at org.apache.poi.POIXMLDocument.write(POIXMLDocument.java:211) ~[poi-ooxml-3.14.jar:3.14]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:565) ~[excel-reports-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run(BackgroundProcessing.java:600) [mango-3.2.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
ERROR 2018-01-10T13:11:25,646 (com.infiniteautomation.mango.rest.v1.excelreports.ExcelReportRestController.getReportFile:287) - null
java.lang.NullPointerException: null
	at java.io.File.<init>(File.java:360) ~[?:1.8.0_131]
	at com.infiniteautomation.mango.excelreports.ExcelReportsCommon.getReport(ExcelReportsCommon.java:174) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.rest.v1.excelreports.ExcelReportRestController.getReportFile(ExcelReportRestController.java:273) [excel-reports-3.2.1.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at com.serotonin.m2m2.web.filter.MangoShallowEtagHeaderFilter.doFilterInternal(MangoShallowEtagHeaderFilter.java:80) [mango-3.2.2.jar:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.authentication.switchuser.SwitchUserFilter.doFilter(SwitchUserFilter.java:198) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at com.serotonin.m2m2.web.mvc.spring.security.PermissionExceptionFilter.doFilter(PermissionExceptionFilter.java:32) [mango-3.2.2.jar:?]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:461) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.Server.handle(Server.java:524) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
              1 Reply Last reply Reply Quote 0
              • phildunlapP
                phildunlap
                last edited by

                Likely the out of memory error caused the null pointer exception. The solution there is going to be to shorten the time range, simplify the report, free memory up elsewhere (hence my suggestions for purging the events, userEvents and audit tables), or perhaps you can eek out a few more MB in the /opt/mango/bin/ext-enabled/memory-small.sh script.

                1 Reply Last reply Reply Quote 0
                • T
                  thewizardguy @phildunlap
                  last edited by

                  @phildunlap I am completely up to date as far as i know, check for upgrades returns nothing new, Version is 3.2.2+2017009170034. I clicked "Purge all events," 23699 events deleted. The report only runs with "None" in rollup at the moment, and I've tried minutes, hours, weeks, days etc, DELETE FROM audit returned an error, I also tried "DELETE * FROM audit;" and it returned a similar error, im guessing that wasn't typed in or at least it doesnt show properly on the post.

                  The report finishes running, but I can never download it, and it runs for between 7-15 minutes.

                  Again, thank you for responding so quickly !

                  1 Reply Last reply Reply Quote 0
                  • phildunlapP
                    phildunlap
                    last edited by

                    I try :D

                    I'm guessing the error with running the SQL had to do with pressing the 'Submit query' button instead of the 'Submit update' button. The likelihood is that the actual report file is not generated, as it ran out of memory while trying to generate that file, so it wouldn't be available for download. Had it been generated, I think you could find the xlsx file in /opt/mango/web/modules/excelReports/web/report-data

                    I would expect the memory footprint to decrease with the use of rollups, and larger rollups should do more to decrease the overhead.

                    T 4 Replies Last reply Reply Quote 0
                    • T
                      thewizardguy @phildunlap
                      last edited by thewizardguy

                      @phildunlap I cannot use rollups for some reason, for when I do the job fails immediately (but didn't used to?) with this error:

                      ERROR 2018-01-10T13:06:15,836 (com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute:579) - Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell.
com.serotonin.ShouldNeverHappenException: Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell.
	at com.infiniteautomation.mango.excelreports.rt.ExcelUtility.setCellValue(ExcelUtility.java:54) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportDataPointReference.insert(ExcelReportDataPointReference.java:102) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:81) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:28) ~[excel-reports-3.2.1.jar:?]
	at com.serotonin.m2m2.view.quantize2.ValueChangeCounterQuantizer.closePeriod(ValueChangeCounterQuantizer.java:41) ~[mango-3.2.2.jar:?]
	at com.serotonin.m2m2.view.quantize2.AbstractDataQuantizer.nextPeriod(AbstractDataQuantizer.java:69) ~[mango-3.2.2.jar:?]
	at com.serotonin.m2m2.view.quantize2.AbstractDataQuantizer.done(AbstractDataQuantizer.java:60) ~[mango-3.2.2.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportRollupDataPointReference.finish(ExcelReportRollupDataPointReference.java:76) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:537) [excel-reports-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run(BackgroundProcessing.java:600) [mango-3.2.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

                      The file in question shows 0 bytes like you said:

                      -rw-r--r-- 1 mango mango 0 Jan 10 13:08 Jan_10__2018_1_07_00_PM_alpha-test.xlsx

                      1 Reply Last reply Reply Quote 0
                      • T
                        thewizardguy @phildunlap
                        last edited by

                        @phildunlap Ok, when I switched from "Past" to "Previous" it allowed rollups, but it gives the same Task Queue Full errors. The report is "Previous 1 Days" and Rollup is "Last 8 Hours," the report has like 8 named ranges available and I only use one for TIME and ALPHA, same errors. When I'm referencing Numeric points I have FAR more greedy time periods and rollups with no issues, the job is done in SECONDS. If it's running out of memory just trying to get 3 values for the last day... I daresay it isn't the reports fault =P

                        1 Reply Last reply Reply Quote 0
                        • T
                          thewizardguy @phildunlap
                          last edited by thewizardguy

                          @phildunlap For posterity, this is the report with Past attempting to run, then the one with Previous, and it's still running:

                          ERROR 2018-01-10T13:43:50,046 (com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute:579) - Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell.
com.serotonin.ShouldNeverHappenException: Unable to write a value of type: com.serotonin.m2m2.rt.dataImage.types.AlphanumericValue into a cell.
	at com.infiniteautomation.mango.excelreports.rt.ExcelUtility.setCellValue(ExcelUtility.java:54) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportDataPointReference.insert(ExcelReportDataPointReference.java:102) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:81) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportNonNumericRollupDataPointReference.quantizedStatistics(ExcelReportNonNumericRollupDataPointReference.java:28) ~[excel-reports-3.2.1.jar:?]
	at com.serotonin.m2m2.view.quantize2.ValueChangeCounterQuantizer.closePeriod(ValueChangeCounterQuantizer.java:41) ~[mango-3.2.2.jar:?]
	at com.serotonin.m2m2.view.quantize2.AbstractDataQuantizer.done(AbstractDataQuantizer.java:61) ~[mango-3.2.2.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportRollupDataPointReference.finish(ExcelReportRollupDataPointReference.java:76) ~[excel-reports-3.2.1.jar:?]
	at com.infiniteautomation.mango.excelreports.rt.ExcelReportWorkItem.execute(ExcelReportWorkItem.java:537) [excel-reports-3.2.1.jar:?]
	at com.serotonin.m2m2.rt.maint.BackgroundProcessing$WorkItemRunnable.run(BackgroundProcessing.java:600) [mango-3.2.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
WARN  2018-01-10T13:45:13,809 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:45:13,811 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:45:13,811 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:45:13,856 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:45:22,589 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 12 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:45:22,592 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
WARN  2018-01-10T13:45:29,867 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:45:29,868 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:45:29,869 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:45:29,870 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:45:36,946 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:45:37,028 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:45:44,099 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:45:44,100 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:45:44,108 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:45:44,108 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:45:52,055 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:45:52,070 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:45:59,154 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:45:59,156 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:45:59,159 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:45:59,181 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:46:06,341 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:46:15,288 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-10T13:46:15,288 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:46:15,289 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:46:15,291 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:46:15,314 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:46:20,543 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 14 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:46:29,712 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:46:29,714 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:46:29,715 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
WARN  2018-01-10T13:46:29,716 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T13:46:29,718 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:46:37,268 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:46:44,436 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:46:44,437 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:46:44,438 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:46:44,469 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 6 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:46:52,092 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T13:46:52,092 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 17 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:47:00,146 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:47:00,150 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:47:00,151 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:47:00,153 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:08,045 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 9 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:08,051 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:47:15,816 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:47:15,817 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:47:15,879 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:47:15,885 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:23,243 (com.serotonin.bacnet4j.util.PropertyUtils.requestPropertiesFromDevice:209) - Timeout while finding device 1100
com.serotonin.bacnet4j.exception.BACnetTimeoutException: No response from instanceId 1100
	at com.serotonin.bacnet4j.util.RemoteDeviceFinder$DeviceFutureImpl.get(RemoteDeviceFinder.java:176) ~[bacnet4j-4.1.2.jar:?]
	at com.serotonin.bacnet4j.LocalDevice$2.get(LocalDevice.java:714) ~[bacnet4j-4.1.2.jar:?]
	at com.serotonin.bacnet4j.util.PropertyUtils.requestPropertiesFromDevice(PropertyUtils.java:205) ~[bacnet4j-4.1.2.jar:?]
	at com.serotonin.bacnet4j.util.PropertyUtils.lambda$readProperties$0(PropertyUtils.java:166) ~[bacnet4j-4.1.2.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
ERROR 2018-01-10T13:47:23,250 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:23,263 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:23,264 (com.serotonin.m2m2.util.timeout.TimeoutTask.run:61) - Uncaught Task Exception
org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]
	at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:308) ~[websocket-common-9.3.11.v20160721.jar:9.3.11.v20160721]
	at com.serotonin.m2m2.web.mvc.websocket.MangoPingPongTracker.scheduleTimeout(MangoPingPongTracker.java:55) ~[mango-3.2.2.jar:?]
	at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:59) [mango-3.2.2.jar:?]
	at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.2.2.jar:?]
	at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.2.2.jar:?]
	at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:307) [mango-3.2.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
WARN  2018-01-10T13:47:31,948 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:47:31,949 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:47:31,950 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:47:31,954 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 5 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:32,029 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 6 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:40,003 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 13 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:47:48,616 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:47:48,618 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:47:48,619 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:47:48,626 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 10 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:48,685 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 3 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:47:56,570 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 13 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:04,536 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
ERROR 2018-01-10T13:48:04,542 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:04,542 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:48:04,543 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
WARN  2018-01-10T13:48:04,545 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T13:48:12,652 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 19 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:48:12,686 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 1 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:20,008 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:48:20,009 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:48:20,011 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:48:20,019 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 13 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:48:27,601 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 4 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:48:27,626 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:35,440 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:48:35,442 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:48:35,443 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:48:35,449 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 11 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:42,932 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T13:48:42,946 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 14 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:50,881 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:48:50,882 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:48:50,883 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:48:50,885 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 9 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:48:58,519 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Batch Writing Status Provider because Task Queue Full
ERROR 2018-01-10T13:48:58,521 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 15 BWB Task Failures, first is: Task Queue Full
WARN  2018-01-10T13:49:03,694 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Point Value Mover because Task Queue Full
WARN  2018-01-10T13:49:03,696 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: NoSQL Stale Point Data Cleaner because Task Queue Full
WARN  2018-01-10T13:49:03,698 (com.serotonin.m2m2.util.timeout.TaskRejectionHandler.rejectedTask:75) - Rejected task: User event cache cleaner because Task Queue Full
ERROR 2018-01-10T13:49:03,698 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 8 BWB Task Failures, first is: Task Queue Full
ERROR 2018-01-10T13:49:08,778 (com.infiniteautomation.nosql.MangoNoSqlBatchWriteBehindManager$StatusProvider.scheduleTimeout:729) - 7 BWB Task Failures, first is: Task Queue Full
                          1 Reply Last reply Reply Quote 0
                          • T
                            thewizardguy @phildunlap
                            last edited by

                            @phildunlap I tried with "Submit Update" button and it "updated" 1600 entries (assuming it means deleted), so that worked, thanks!

                            1 Reply Last reply Reply Quote 0
                            • phildunlapP
                              phildunlap
                              last edited by

                              I will send you an email about the "Cannot insert alphanumeric" error, as that is almost certainly a bug.

                              There isn't a very significant difference in handling the alphanumeric points for a report and handling the numeric points, so my guess is that the memory is being consumed elsewhere and the observation of one working quickly and one failing may also be incidental. But, I will consider that angle.

                              1 Reply Last reply Reply Quote 0
                              • phildunlapP
                                phildunlap
                                last edited by

                                It was discovered upon investigation that the Excel report with a chart and the various dynamic elements had a much larger memory impact than a simple data report. The complex report would drain the memory until the garbage collector started thrashing, which caused the slowdown and 'crashing'. Resolution was found in modifying the Mango/bin/ext-enabled script to supply a few hundred more MB to Java, which led the complex Excel report to complete very quickly and without garbage collector thrashing. There was also a brief discussion of why this memory was previously reserved for the operating system.

                                T 1 Reply Last reply Reply Quote 0
                                • T
                                  thewizardguy @phildunlap
                                  last edited by thewizardguy

                                  TL;DR If you make an excel template, limit your named ranges to as small as possible

                                  Alrighty, so after a few more days of troubleshooting, @phildunlap took a look at my excel template. He determined that the named range I designated in Excel was too long (only by a million cells!) for EACH named range, of which there are about 50. So, the theory was that the POI Apache plugin that allows Mango to interact with Excel was loading these empty cells into memory somehow, and was sporadically causing a memory leak or memory usage at least until the point where it would cripple the amount of available memory to the JVM to VERY VERY low, and render the garbage collection useless. Thats a mouthful.

                                  And so, the problem only reared its ugly head once in awhile, it was hard to track down, but I've been operating for about a week now with no "crippling" memory issue, and I've launched thousands of reports without issue. For anyone who wants to know, the final template filled in is 82kb, has charts and 5 sheets of data, the time range spans 14 months and there are multiple different time ranges used and approximately 10-15 data points queried.

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post