Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website

persistent tcp publishers stop transfer


  • Hi,

    I have a few publishers via TCP sync history and real time data to one server (different ports and crons).
    Both systems same core 3.7.4 and modules, openjdk-8.0.232.09.

    0_1578661334224_132edf1b-09ce-4699-8ee1-3ff589c04334-image.png

    Since the last 24 hours the history-sync stop and after restart the publisher it is working a few hours and stop again:

    0_1578661461044_636395f0-848e-4d90-8290-7f5447aa6851-image.png

    The real time data is syncing, client and server are working normal.

    Any help?

    Runtime status:

    0_1578665872072_c63921ad-3110-41b2-8bae-e2c4edcd0f39-image.png

    Log file from one publisher gateway:

    WARN 2020-01-10 14:15:53,000 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    WARN 2020-01-10 14:15:54,000 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    WARN 2020-01-10 14:15:55,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    WARN 2020-01-10 14:15:56,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    WARN 2020-01-10 14:15:57,002 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    WARN 2020-01-10 14:15:58,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    WARN 2020-01-10 14:15:59,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
    

    Ma log file gateway:

    ERROR 2020-01-10T14:01:14,266 (com.serotonin.m2m2.db.dao.AbstractBasicDao$1.extractData:1063) - Internal Server Error � org.eclipse.jetty.io.EofException: Closed 
    com.infiniteautomation.mango.rest.v2.exception.GenericRestException: Internal Server Error � org.eclipse.jetty.io.EofException: Closed
    	at com.infiniteautomation.mango.rest.v2.model.StreamedVOQueryWithTotal$StreamedVOArray.lambda$writeArrayValues$0(StreamedVOQueryWithTotal.java:70) ~[?:?]
    	at com.serotonin.m2m2.db.dao.AbstractBasicDao$1.extractData(AbstractBasicDao.java:1058) ~[mango-3.7.4.jar:?]
    	at com.serotonin.m2m2.db.dao.AbstractBasicDao$1.extractData(AbstractBasicDao.java:1050) ~[mango-3.7.4.jar:?]
    	at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:678) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:616) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at com.serotonin.db.DaoUtils.query(DaoUtils.java:258) ~[mango-3.7.4.jar:?]
    	at com.serotonin.m2m2.db.dao.AbstractBasicDao.customizedQuery(AbstractBasicDao.java:1050) ~[mango-3.7.4.jar:?]
    	at com.serotonin.m2m2.db.dao.AbstractBasicDao.customizedQuery(AbstractBasicDao.java:1024) ~[mango-3.7.4.jar:?]
    	at com.infiniteautomation.mango.rest.v2.model.StreamedVOQueryWithTotal$StreamedVOArray.writeArrayValues(StreamedVOQueryWithTotal.java:65) ~[?:?]
    	at com.serotonin.m2m2.web.mvc.rest.v1.mapping.JSONStreamedArraySerializer.serialize(JSONStreamedArraySerializer.java:24) ~[?:?]
    	at com.serotonin.m2m2.web.mvc.rest.v1.mapping.JSONStreamedArraySerializer.serialize(JSONStreamedArraySerializer.java:17) ~[?:?]
    	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:103) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:180) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:122) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]
    	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at com.infiniteautomation.mango.webapp.filters.MangoCacheControlHeaderFilter.doFilter(MangoCacheControlHeaderFilter.java:155) ~[mango-3.7.4.jar:?]
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.authentication.switchuser.SwitchUserFilter.doFilter(SwitchUserFilter.java:200) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at com.serotonin.m2m2.web.mvc.spring.security.RateLimitingFilter.doFilterInternal(RateLimitingFilter.java:91) ~[mango-3.7.4.jar:?]
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at com.serotonin.m2m2.web.mvc.spring.security.PermissionExceptionFilter.doFilter(PermissionExceptionFilter.java:34) ~[mango-3.7.4.jar:?]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:124) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE]
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) ~[jetty-security-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.Server.handle(Server.java:500) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
    Caused by: org.eclipse.jetty.io.EofException: Closed
    	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:575) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118]
    	at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:639) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:639) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
    	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085) ~[jackson-core-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097) ~[jackson-core-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2658) ~[jackson-databind-2.9.10.jar:2.9.10]
    	at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381) ~[jackson-core-2.9.10.jar:2.9.10]
    	at com.infiniteautomation.mango.rest.v2.model.StreamedVOQueryWithTotal$StreamedVOArray.lambda$writeArrayValues$0(StreamedVOQueryWithTotal.java:68) ~[?:?]
    	... 119 more
    

  • Ok, I invest some time, deactivate all percistent TCP publishers (ptp), start again at 10:00 with only 3 and found this after one hour:

    0_1578736891653_893676a0-d5cc-4b9b-bc06-a5c2a4c6c30e-image.png

    One ptp sync all the data, finished and then it is repeating that message in the log-file and stop the history-sync. The real time sync is working.
    The next ptp stop at 11:00, the last stop at 11:30, same message in the log-file.

    Another thing I found:

    One ptp is starting and sync more than 1 Million records in 3m20s, all ok:

    0_1578737707293_f082d7f7-5ffe-43cf-ada0-4612d4403226-image.png

    The next sync, same ptp, 8650 records and 4m15s?:

    0_1578737826092_137c8d6a-441e-47d2-b7f4-282d868c470f-image.png

    Thanks for the help!


  • Ok, I found a solution!

    The old thread pool settings:

    0_1578997281515_ad7a49b3-2b49-4a5b-bdcf-9577fd404f36-image.png

    Settings now:

    0_1578997216570_5c7fd5ef-9f5b-478a-8084-3f4391eea992-image.png

    The setup with more than 120 BACnet-Devices needs a lot of threads, so the ptp-sync hasn't enough and failed.

    Now it needs more CPU (20-45%) but it is working, that is fine.