Modbus/Serial Driver Changes on 3.6 under Linux?

Turbo

Greetings, all:

I've found a new interesting behavior with some of our sites after updating to 3.6 under Linux.

Brief summary, looking for help:
(1) Connected to solar inverters, 9600, via optically-isolated RS-485 adapter. This unit works fine (Generally) with all my other modbus devices I've tested it on. This particular inverter is doing something, that sometimes breaks 3.6. It worked fine with 3.5.

(2) When using the Modbus Tools, Mango never sees that data come back from the inverter. (No Response from Slave XX is the error message)

(3) When using the Enable/Disable data source option, Mango never sees the data come back from the inverter. (No Response from Slave XX is the error message)

(4) When just polling the device, Mango DOES see the inverter responses, but only AFTER the system is restarted or the computer is restarted.

(5) If you disable the data source, and restart the data source, it doesn't see the respones.

(6) If you restart mango entirely, it DOES see the responses from the inverter.

This behavior is (apparently) new in 3.6, as I had another site that was working fine with 3.5 and is now showing this behavior with 3.6.

My hypothesis is that these inverters are maybe responding too quickly (or something), and that's tripping up the RX/TX control circuit in the 485 adapters. I've tried messing with the flow control, and the behavior described above seems dependent on the resetting of Mango entirely.

I'm using Linux, and the JSSC drivers as we've discussed here before.

I've got (2) identical serial converters on most of my sites- FTDI chip based devices. They work fine running my other (2) devices that are standard in my install, an electrical meter and a (self-designed) Modbus-talking RS-485 input module.

These inverters always seem to talk properly using Modscan32 or similar tools, but fail with Mango as described above.

If I get a chance to visit one of these sites, I'll take a scope out and see what's happening on the bus, to test my RX/TX switching hypothesis.

Any thoughts?

Cheers,
-Greg

phildunlap

Hi Turbo,

Hmm. The packaged version of Modbus4J did go from 3.0.4 to 3.0.5 so there were a few changes. But, that it works sometimes and after restarts and whatnot is confusing detail.

When does it not work? After some amount of time, some percentage of restarts, etc
Does this apply even in cases like (6) where the restart seems to make the responses appear?

If it is a serial port issue, I think it could be interesting to see from the command line what the setting of the serial port is when in each state to see if the port is being configured differently: stty -F /dev/ttyUSB0 -a

I found this issue while investigating this question, and it would be a difference between 3.5 and 3.6, so perhaps it is the issue at hand: https://github.com/infiniteautomation/ma-core-public/issues/1497

If so, thanks for bringing it to our attention! Perhaps you could use stty command to manually tinker with the flow control settings when you are observing the issue.

Turbo

Heya:

I appreciate you looking into this. I also noticed a new update to Mango's Modbus library. Does this address this issue?

Do you want me to still look into stty output for my configurations?

Cheers,
-Greg

Turbo

Ok:

Since I'm logged in anyways, here we go:

stty -F /dev/ttyUSB1 -a
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z;
rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 0; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke -flusho -extproc

After disable inside Mango:
stty -F /dev/ttyUSB1 -a
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z;
rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 0; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke -flusho -extproc

After re-enable inside Mango:
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z;
rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 0; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke -flusho -extproc

cwangv

@turbo
Greg
Did you see a lot of events being generated for this modbus device? Such as lot of 'RTN' events popping up?

Turbo

Nope.

Right now, the device is working fine. If I turn it OFF and ON again within Mango, it just shows the data as uncertain.

I did disable the events for this, though, as I generally don't need my database flooded with lost poll type numbers.

That all being said, it's working Fine Now, so long as I don't use the Enable/Disable Data Source buttons.

phildunlap

All those stty outputs are the same. Were those correlated with the data source not working?

Turbo

Yeah.

I was just about to go look into it again, and one of my computers in the field fell over again.

Ugh:

Anyways: Was the big you found earlier up in this post fixed?

phildunlap

I described it in the git issue, but it was that flow control was being improperly set. That doesn't seem to me that it would be the issue if toggling the data source resolves it, as it would be passing the incorrect flow control every time if that were the issue. It was fixed, but there hasn't been a core release in that time, so it's not released.

If it's not the serial port configuration, it could be the adapter or the noise on the wire. Or, it could be the modbus data source is somehow stuck, but I wouldn't expect that if you are able to disable it. Certainly it would have to finish the poll it was attempting.

Perhaps you can take a thread dump when you observe this occurring?

You could perhaps apply a band-aid of an event handler to restart the data source, if it's giving grief.

Turbo

Toggling the data source does not resolve it.

Restarting the Mango instance does.

When Mango starts, it works, fine, for apparently long periods of time. If you toggle it off and on again, it does not come back online. Only the ? shows up on data, and the Modbus tools fail to work.

If you restart all of Mango (or restart the computer entirely) it comes up on the first startup.

Turbo

Okay.

Just was verifying this: What log files can I send you to verify the behavior I'm seeing and help you out?

(1) On startup, the Data Sources come up Just Fine, and we're all solid.
The system has been running here since Tuesday of last week with data coming in fine, an no problems reported.
(2) If I toggle any single Data source, it causes all the data sources on the same port to fail. The polls are still going out, but the returns are not getting seen by Mango. I have Modbus/Serial data sources defined one data source per inverter, meaning in this case there are (5) Data sources all using /dev/ttyUSB1. They are all the same serial settings, with the only difference being the modbus address used.
(3) The only way to get that device back up again is to restart mango, at which point everything is Fine.

It looks as if disabling a Single Data Source disables the entire port-- In looking at this now, I'm seeing the same behavior I previously described on BOTH ports (USB1 and USB0).

Previously, in Mango, if you disabled, say, Data Source A on /dev/ttyUSB0, it just turned off Data Source A (stopped polling). With 3.6, it appears that disabling Data Source A disables all data sources using that particular port. Is this the way this behavior is supposed to be?

phildunlap

I sent you an email. I was not able to replicate an issue with two data sources sharing a modbus serial connection and disabling them. No it is not the expected behavior.