Loss of Communication to AB PLC
-
Yesterday the Mango install we have talking with an AB PLC stopped communicating around 3:00 PM. There has been no changes to the PLC since install.
The following is put into the log when the datasource is enabled:
INFO 2016-09-13 09:34:01,025 (com.serotonin.m2m2.rt.RuntimeManager.initializeDataSource:302) - Data source 'PLC 2' initialized INFO 2016-09-13 09:34:01,025 (com.serotonin.m2m2.rt.RuntimeManager.initializeDataSource:307) - Data source 'PLC 2' took 897.459516ms to start WARN 2016-09-13 09:34:01,527 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:01.526 aborted because Task Currently Running WARN 2016-09-13 09:34:02,033 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:02.026 aborted because Task Currently Running WARN 2016-09-13 09:34:02,534 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:02.526 aborted because Task Currently Running WARN 2016-09-13 09:34:03,034 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:03.026 aborted because Task Currently Running WARN 2016-09-13 09:34:03,530 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:03.526 aborted because Task Currently Running WARN 2016-09-13 09:34:04,029 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:04.026 aborted because Task Currently Running WARN 2016-09-13 09:34:04,529 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:04.526 aborted because Task Currently Running WARN 2016-09-13 09:34:05,027 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:05.026 aborted because Task Currently Running WARN 2016-09-13 09:34:05,527 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:05.526 aborted because Task Currently Running WARN 2016-09-13 09:34:06,028 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:06.026 aborted because Task Currently Running WARN 2016-09-13 09:34:07,029 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:07.026 aborted because Task Currently Running WARN 2016-09-13 09:34:07,526 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:07.526 aborted because Task Currently Running WARN 2016-09-13 09:34:08,026 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:08.026 aborted because Task Currently Running WARN 2016-09-13 09:34:08,526 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:08.526 aborted because Task Currently Running WARN 2016-09-13 09:34:09,026 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:09.026 aborted because Task Currently Running WARN 2016-09-13 09:34:09,527 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:09.526 aborted because Task Currently Running WARN 2016-09-13 09:34:10,026 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:10.026 aborted because Task Currently Running WARN 2016-09-13 09:34:10,526 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:10.526 aborted because Task Currently Running WARN 2016-09-13 09:34:11,026 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:11.026 aborted because Task Currently Running WARN 2016-09-13 09:34:11,526 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 09:34:11.526 aborted because Task Currently Running INFO 2016-09-13 09:34:11,656 (com.serotonin.m2m2.rt.RuntimeManager.stopDataSource:375) - Data source 'PLC 2' stopped
I purged the /work/jsp folder and updated to the latest and greatest Mango.
Any ideas?
-
Can you ping the PLC from the mango machine?
-
Yep, full communication to the PLC and associated HMI devices.
No errors shown in the ethernet card.
-
Hmm, OK, it seems like its having trouble connecting or the reads are timing out. Is it the Ethernet/IP driver or the PCCC driver?
-
Ethernet IP
If had a runtime of approx 37 days until yesterday.
No matter what I do I cannot kickstart the datasource.
-
Can you copy log4j-debug.xml from your classes directory into overrides/classes and rename it to log4j.xml, also add the following line below the BACnet4J entry.
<category name="com.deltamation"><level value="debug"/></category>
Restart Mango and post up the log again. Thanks.
-
If you have the Log4JReset module installed, you can reload your log4j settings without restarting Mango from the System Settings page, in the Log4J Reset section.
-
This is the init portion.
INFO 2016-09-13 13:40:12,478 (com.deltamation.mango.abeip.ABEipDataSourceRT.initialize:57) - abeip initialize() DEBUG 2016-09-13 13:40:12,482 (com.deltamation.mango.abeip.plc.compatibility.BaseClient.printPacket:141) - Sending connection request: 0x65 0x0 0x4 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x0 0x0 0x0 DEBUG 2016-09-13 13:40:12,485 (com.deltamation.mango.abeip.plc.compatibility.BaseClient.printPacket:141) - Recieved connection response: 0x65 0x0 0x4 0x0 0x25 0x0 0x43 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x0 0x0 0x0 DEBUG 2016-09-13 13:40:12,485 (com.deltamation.mango.abeip.plc.allenbradley.eipclass3.EipClass3Client.requestSession:69) - Connected with sessionID: 4390949 DEBUG 2016-09-13 13:40:12,486 (com.deltamation.mango.abeip.plc.compatibility.BaseClient.printPacket:141) - Sending Forward Open: 0x6f 0x0 0x42 0x0 0x25 0x0 0x43 0x0 0x0 0x0 0x0 0x0 0xd 0x3 0x1 0x7 0xa 0xe 0x1 0xe 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x0 0x0 0x0 0xb2 0x0 0x32 0x0 0x54 0x2 0x20 0x6 0x24 0x1 0xa 0xff 0xc2 0xe7 0x7f 0x31 0x65 0xf3 0x51 0x47 0x1 0x0 0x8b 0xb9 0xb9 0xf3 0x2 0x19 0x2 0x0 0x0 0x0 0x80 0x84 0x1e 0x0 0xf4 0x43 0x80 0x84 0x1e 0x0 0xf4 0x43 0xa3 0x4 0x1 0x0 0x20 0x2 0x24 0x1 0x2c 0x1 DEBUG 2016-09-13 13:40:12,490 (com.deltamation.mango.abeip.plc.compatibility.BaseClient.printPacket:141) - Recieved Fwd Open Response: 0x6f 0x0 0x2e 0x0 0x25 0x0 0x43 0x0 0x0 0x0 0x0 0x0 0xd 0x3 0x1 0x7 0xa 0xe 0x1 0xe 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x0 0x0 0x0 0xb2 0x0 0x1e 0x0 0xd4 0x0 0x0 0x0 0x93 0x60 0xaa 0xff 0x65 0xf3 0x51 0x47 0x1 0x0 0x8b 0xb9 0xb9 0xf3 0x2 0x19 0x80 0x84 0x1e 0x0 0x80 0x84 0x1e 0x0 0x0 0x0 INFO 2016-09-13 13:40:13,312 (com.serotonin.m2m2.rt.RuntimeManager.initializeDataSource:302) - Data source 'PLC 2' initialized INFO 2016-09-13 13:40:13,313 (com.serotonin.m2m2.rt.RuntimeManager.initializeDataSource:307) - Data source 'PLC 2' took 834.922472ms to start
-
Initial Warning
WARN 2016-09-13 13:40:18,313 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 13:40:18.313 aborted because Task Currently Running DEBUG 2016-09-13 13:40:18,319 (com.serotonin.m2m2.rt.EventManager.raiseEvent:190) - Event raised: type=DataSoureEventType(dataSourceId=7, eventTypeId=6), message='PLC 2': Data source with xid: Zirkle_PLC_2 and name: PLC 2, aborted 1 polls. DEBUG 2016-09-13 13:40:18,433 (com.serotonin.m2m2.rt.EventManager.raiseEvent:190) - Event raised: type=DataSoureEventType(dataSourceId=7, eventTypeId=4), message='PLC 2': Read failed WARN 2016-09-13 13:40:18,434 (com.deltamation.mango.abeip.ABEipDataSourceRT.doPoll:282) - Error reading data point. java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.deltamation.mango.abeip.plc.allenbradley.eipclass3.EipClass3Client.processServices(EipClass3Client.java:779) at com.deltamation.mango.abeip.ABEipDataSourceRT.syncProcessServices(ABEipDataSourceRT.java:425) at com.deltamation.mango.abeip.ABEipDataSourceRT.doPoll(ABEipDataSourceRT.java:276) at com.serotonin.m2m2.rt.dataSource.PollingDataSource.doPollNoSync(PollingDataSource.java:182) at com.serotonin.m2m2.rt.dataSource.PollingDataSource.scheduleTimeout(PollingDataSource.java:148) at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:69) at com.serotonin.timer.TimerTask.runTask(TimerTask.java:148) at com.serotonin.timer.OrderedTimerTaskWorker.run(OrderedTimerTaskWorker.java:29) at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTask.run(OrderedThreadPoolExecutor.java:278) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) DEBUG 2016-09-13 13:40:19,747 (com.serotonin.m2m2.persistent.pub.TimedOutputStream$CheckTask.run:131) - 1 (PersistentSenderRT.SendThread): not in a write operation
-
Final
DEBUG 2016-09-13 13:40:27,247 (com.serotonin.m2m2.persistent.pub.TimedOutputStream$CheckTask.run:131) - 1 (PersistentSenderRT.SendThread): not in a write operation WARN 2016-09-13 13:40:28,313 (com.serotonin.m2m2.rt.dataSource.PollingDataSource$2.execute:226) - PLC 2: poll scheduled at 2016/09/13 13:40:28.313 aborted because Task Currently Running WARN 2016-09-13 13:40:28,424 (com.deltamation.mango.abeip.ABEipDataSourceRT.doPoll:282) - Error reading data point. java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.deltamation.mango.abeip.plc.allenbradley.eipclass3.EipClass3Client.processServices(EipClass3Client.java:779) at com.deltamation.mango.abeip.ABEipDataSourceRT.syncProcessServices(ABEipDataSourceRT.java:425) at com.deltamation.mango.abeip.ABEipDataSourceRT.doPoll(ABEipDataSourceRT.java:276) at com.serotonin.m2m2.rt.dataSource.PollingDataSource.doPollNoSync(PollingDataSource.java:182) at com.serotonin.m2m2.rt.dataSource.PollingDataSource.scheduleTimeout(PollingDataSource.java:148) at com.serotonin.m2m2.util.timeout.TimeoutTask.run(TimeoutTask.java:69) at com.serotonin.timer.TimerTask.runTask(TimerTask.java:148) at com.serotonin.timer.OrderedTimerTaskWorker.run(OrderedTimerTaskWorker.java:29) at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTask.run(OrderedThreadPoolExecutor.java:278) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) INFO 2016-09-13 13:40:28,471 (com.deltamation.mango.abeip.ABEipDataSourceRT.terminate:116) - abeip terminate() INFO 2016-09-13 13:40:28,473 (com.serotonin.m2m2.rt.RuntimeManager.stopDataSource:375) - Data source 'PLC 2' stopped
-
I'm recreating the old datasource one point at a time, and so far the new datasource is functioning without issue.
I do not have all the points loaded just yet, but so far it is working.
If I attempt to enable the old datasource, it still produces the same errors.
-
Hey Brad, from the logs you posted its definitely timing out on the read, i.e. its sending a command and not receiving a response. I know this will happen with AB PLCs when the session is invalid or something like that.
I would try disabling points on your existing datasource until you can get it to work. I suspect its something to do with the payload sizes, i.e. the driver is requesting too much data at once or something. Its odd that it used to work and just stopped though.
Unfortunately the AB datasources are provided by a 3rd party vendor - Deltamation, so we don't have access to the source code to debug or fix the issue. I will forward the issue onto them.