persistent tcp publishers stop transfer
-
Hi,
I have a few publishers via TCP sync history and real time data to one server (different ports and crons).
Both systems same core 3.7.4 and modules, openjdk-8.0.232.09.Since the last 24 hours the history-sync stop and after restart the publisher it is working a few hours and stop again:
The real time data is syncing, client and server are working normal.
Any help?
Runtime status:
Log file from one publisher gateway:
WARN 2020-01-10 14:15:53,000 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running WARN 2020-01-10 14:15:54,000 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running WARN 2020-01-10 14:15:55,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running WARN 2020-01-10 14:15:56,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running WARN 2020-01-10 14:15:57,002 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running WARN 2020-01-10 14:15:58,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running WARN 2020-01-10 14:15:59,001 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.startSync:912) - A data synchronization run was not started because a previous one is still running
Ma log file gateway:
ERROR 2020-01-10T14:01:14,266 (com.serotonin.m2m2.db.dao.AbstractBasicDao$1.extractData:1063) - Internal Server Error � org.eclipse.jetty.io.EofException: Closed com.infiniteautomation.mango.rest.v2.exception.GenericRestException: Internal Server Error � org.eclipse.jetty.io.EofException: Closed at com.infiniteautomation.mango.rest.v2.model.StreamedVOQueryWithTotal$StreamedVOArray.lambda$writeArrayValues$0(StreamedVOQueryWithTotal.java:70) ~[?:?] at com.serotonin.m2m2.db.dao.AbstractBasicDao$1.extractData(AbstractBasicDao.java:1058) ~[mango-3.7.4.jar:?] at com.serotonin.m2m2.db.dao.AbstractBasicDao$1.extractData(AbstractBasicDao.java:1050) ~[mango-3.7.4.jar:?] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:678) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:616) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711) ~[spring-jdbc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at com.serotonin.db.DaoUtils.query(DaoUtils.java:258) ~[mango-3.7.4.jar:?] at com.serotonin.m2m2.db.dao.AbstractBasicDao.customizedQuery(AbstractBasicDao.java:1050) ~[mango-3.7.4.jar:?] at com.serotonin.m2m2.db.dao.AbstractBasicDao.customizedQuery(AbstractBasicDao.java:1024) ~[mango-3.7.4.jar:?] at com.infiniteautomation.mango.rest.v2.model.StreamedVOQueryWithTotal$StreamedVOArray.writeArrayValues(StreamedVOQueryWithTotal.java:65) ~[?:?] at com.serotonin.m2m2.web.mvc.rest.v1.mapping.JSONStreamedArraySerializer.serialize(JSONStreamedArraySerializer.java:24) ~[?:?] at com.serotonin.m2m2.web.mvc.rest.v1.mapping.JSONStreamedArraySerializer.serialize(JSONStreamedArraySerializer.java:17) ~[?:?] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913) ~[jackson-databind-2.9.10.jar:2.9.10] at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:103) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:180) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:122) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.7.RELEASE.jar:5.1.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at com.infiniteautomation.mango.webapp.filters.MangoCacheControlHeaderFilter.doFilter(MangoCacheControlHeaderFilter.java:155) ~[mango-3.7.4.jar:?] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.authentication.switchuser.SwitchUserFilter.doFilter(SwitchUserFilter.java:200) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at com.serotonin.m2m2.web.mvc.spring.security.RateLimitingFilter.doFilterInternal(RateLimitingFilter.java:91) ~[mango-3.7.4.jar:?] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at com.serotonin.m2m2.web.mvc.spring.security.PermissionExceptionFilter.doFilter(PermissionExceptionFilter.java:34) ~[mango-3.7.4.jar:?] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:124) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) ~[spring-web-5.1.7.RELEASE.jar:5.1.7.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) ~[jetty-security-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) ~[jetty-servlet-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.Server.handle(Server.java:500) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.23.v20191118.jar:9.4.23.v20191118] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] Caused by: org.eclipse.jetty.io.EofException: Closed at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:575) ~[jetty-server-9.4.23.v20191118.jar:9.4.23.v20191118] at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:639) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:639) ~[spring-security-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085) ~[jackson-core-2.9.10.jar:2.9.10] at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097) ~[jackson-core-2.9.10.jar:2.9.10] at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2658) ~[jackson-databind-2.9.10.jar:2.9.10] at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381) ~[jackson-core-2.9.10.jar:2.9.10] at com.infiniteautomation.mango.rest.v2.model.StreamedVOQueryWithTotal$StreamedVOArray.lambda$writeArrayValues$0(StreamedVOQueryWithTotal.java:68) ~[?:?] ... 119 more
-
Ok, I invest some time, deactivate all percistent TCP publishers (ptp), start again at 10:00 with only 3 and found this after one hour:
One ptp sync all the data, finished and then it is repeating that message in the log-file and stop the history-sync. The real time sync is working.
The next ptp stop at 11:00, the last stop at 11:30, same message in the log-file.Another thing I found:
One ptp is starting and sync more than 1 Million records in 3m20s, all ok:
The next sync, same ptp, 8650 records and 4m15s?:
Thanks for the help!
-
Ok, I found a solution!
The old thread pool settings:
Settings now:
The setup with more than 120 BACnet-Devices needs a lot of threads, so the ptp-sync hasn't enough and failed.
Now it needs more CPU (20-45%) but it is working, that is fine.