Mango Persistent TCP publisher constantly resets connection

[MaP]

After setting up our local mango instance to sync with an enterprise mango instance using mango persistent tcp publisher the connection seems to be successful but will continually reset and throw an error. We have tried changing the ports/passowrd/using crc etc and adding ACCEPT rules to the IP tables of the server holding the enterprise mango instance but can not get a stable connection.

Any hints on what causes the connection reset would be appreciated.
![0_1477379541984_upload-f374193f-07ca-494a-8c58-5da83f2d742f]
(https://i.imgur.com/Pg4BXD3.png)

[JoelHaggar]

How many data points are you publishing? I would suggest increasing your Socket Timeout. It needs to be long enough to allow all your points to reconnect. You can also provide a screen shot of your publisher settings and we might be able to suggest other setting changes.

[phildunlap]

You can also enable logging on the data source and publisher at varying levels, so perhaps that will add some information.

[terrypacker]

I have seen this happen when one of the data points on the receiving end is configured with the wrong data type. The receiver will create new data points for new incoming points but will also match points based on XID so if you have been making changes on the publishing system this can happen. This will showup in the log output that Phillip suggested turning on.

[mihairosu]

@phildunlap

Just a friendly warning with regards to Phil's suggestions to turn on Mango Persistent TCP Publisher logs. On our MangoES they do not appear to rotate, so eventually they will eat up the entire hard disk space.

In fact, I'm surprised more people haven't posted on this issue.

You can disregard the warning if it has been fixed already =)

[MaP]

Hi Guys,

Thanks for the speedy replies - much appreciated. I have disabled the publisher while i try to solve the issue as i don't want to clog up the events.

I have attached some screenshots.

[phildunlap]

@mihairosu Right you are. I have made an issue for this; https://github.com/infiniteautomation/ma-core-public/issues/855

@MaP Can you give us a description of what happens when you enable it? If you press the "Get Status" button every few seconds after enabling it, do you see it finish connecting the points? Judging from the image in your first post, where the connection is resetting every three seconds, I am guessing Terry's suggestion is pretty good: it could be an issue with the point configurations.

Consider changing the XID prefix and seeing if you get new points created on the receiver. If so, it could be a problem with the configuration of a point, as Terry suggested.

If the points do finish connecting and begin to publish or sync, then the connection resets, I would consider trying to turn down the point values to send per packet.

[MaP]

No points have been created by the receiver when you press get status it says 0 points, connected for 284ms, 1 packet received, 0 timeouts if you press get status a second time it will say no connection. It toggles between these two states..

No points have been created by the receiver at all yet just lots of connection reset errors on the sending side and the data points tab on the receiver side are blank.

[JoelHaggar]

Is it possible you have two different versions of the module? Regardless from your screen shot I can see you have an older version so I would recommend doing an update.

[MaP]

I have upgraded the local mango to the latest persistent module but we still see the same connection reset behavior.

[MaP]

I enabled the Debug log for a while and received the following logs (only put about a third here):

INFO 2016-10-27 15:00:50,050 (com.serotonin.m2m2.persistent.pub.PersistentSenderRT.initialize:228) - Initializing
INFO 2016-10-27 15:00:50,101 (com.serotonin.m2m2.persistent.pub.PersistentSenderRT.initialize:242) - Initialized
WARN 2016-10-27 15:00:51,127 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl:182) - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at com.serotonin.m2m2.persistent.common.Packet.readPacket(Packet.java:93)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.getPointReceipt(PersistentSendThread.java:617)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.openConnection(PersistentSendThread.java:599)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl(PersistentSendThread.java:179)
at com.serotonin.m2m2.rt.publish.SendThread.run(SendThread.java:56)
WARN 2016-10-27 15:00:54,074 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl:182) - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at com.serotonin.m2m2.persistent.common.Packet.readPacket(Packet.java:93)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.getPointReceipt(PersistentSendThread.java:617)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.openConnection(PersistentSendThread.java:599)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl(PersistentSendThread.java:179)
at com.serotonin.m2m2.rt.publish.SendThread.run(SendThread.java:56)
WARN 2016-10-27 15:00:57,354 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl:182) - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at com.serotonin.m2m2.persistent.common.Packet.readPacket(Packet.java:93)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.getPointReceipt(PersistentSendThread.java:617)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.openConnection(PersistentSendThread.java:599)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl(PersistentSendThread.java:179)
at com.serotonin.m2m2.rt.publish.SendThread.run(SendThread.java:56)
WARN 2016-10-27 15:01:00,324 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl:182) - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at com.serotonin.m2m2.persistent.common.Packet.readPacket(Packet.java:93)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.getPointReceipt(PersistentSendThread.java:617)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.openConnection(PersistentSendThread.java:599)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl(PersistentSendThread.java:179)
at com.serotonin.m2m2.rt.publish.SendThread.run(SendThread.java:56)
WARN 2016-10-27 15:01:03,604 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl:182) - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at com.serotonin.m2m2.persistent.common.Packet.readPacket(Packet.java:93)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.getPointReceipt(PersistentSendThread.java:617)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.openConnection(PersistentSendThread.java:599)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl(PersistentSendThread.java:179)
at com.serotonin.m2m2.rt.publish.SendThread.run(SendThread.java:56)
WARN 2016-10-27 15:01:06,564 (com.serotonin.m2m2.persistent.pub.PersistentSendThread.runImpl:182) - Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at com.serotonin.m2m2.persistent.common.Packet.readPacket(Packet.java:93)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.getPointReceipt(PersistentSendThread.java:617)
at com.serotonin.m2m2.persistent.pub.PersistentSendThread.openConnection(PersistentSendThread.java:599)

Any assistance would be appreciated.

[phildunlap]

Is this to imply there was no debug output on the receiver's end? You can set the level even lower, at Trace. When I asked about the status button, I was wondering about from the Publisher's side as well.

I would try to assess whether it was a configuration issue or a network issue. To do that I would export your Publisher's configuration, remove all the items in the "points" list, change the host to localhost and import it on your receiver's Mango. Create some virtual points, and see if you can publish them to the localhost receiver. If so, and both Mangos are running on the same versions of the module, you can be sure it's a network problem.

Another thought is to remove the data points from the publisher, and see if the connection is stable.

[MaP]

I have some screenshots of the receiver side Trace Logs here:

Constraint A foreign key constrant fails - DataSourceId References DataSources_Old

Any ideas how to solve this?

[MaP]

This post is deleted!

[MaP]

Also encountering the below error while trying to construct a new publisher using only mango data source points like CPU temp.

The published data points section just hangs with a loading image after seeing this error

[phildunlap]

dataSources_old? You were playing around in your database schema! I assume you created a table dataSources_new like dataSources, then renamed dataSources to dataSources_old and moved dataSources_new to dataSources. Is there a reason you would try this?

If there isn't data that matters, I would fix it by exporting your configuration, going into the mysql command line, creating a database "mango2" and redirecting the connection string in your env.properties to this new database, and then importing your configuration again after Mango boots.

If there is data that matters, you can update the foreign key constraints to reference the right table (grumble grumble) by running some SQL:

ALTER TABLE dataPoints DROP FOREIGN KEY dataPointsFk1;
ALTER TABLE dataPoints ADD CONSTRAINT dataPointsFk1 FOREIGN KEY (dataSourceId) REFERENCES dataSources(id);

[MaP]

Hi Phil,

I just imported another sites data sources config into the mango and did a find replace to ensure unique XIDs. I was pretty sure this was a brand-new mango but I cant be sure that it hasnt been toyed with.

Is there a way to do a factory reset and clear all previous data/config from a mango?

[MaP]

Hi Phil,

I entered the SQL queries listed in your last post and received the following errors:

[MaP]

[phildunlap]

That's okay, the foreign key was the bigger problem. I'm going to remove the index recommendation from that post. But, it also looks like you're using the H2 web console now, but from your SQL error I think you're using MySQL? If so, running those commands in the H2 console won't help, you would have to run them in your MySQL database.

Using H2, you can cause a new Mango instance by renaming your Mango/databases directory while Mango is off. if using MySQL, you should create another database via CREATE DATABASE newMangoDatabaseName and then modify the connection string to point to this database. You would also need to rename the Mango/databases directory to set the old data aside, if you are using the NoSQL module.