Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website

Serial Port Problem, Listener task queue full, data will be lost!


  • Hi guys, any idea what this warning issued is referencing?
    Hoping it's the USB port serial adaptor and not the mangoES having a tizzy:

    WARN  2019-03-25T00:04:20,388 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:150) - Serial Port Problem, Listener task queue full, data will be lost!
    java.lang.IllegalStateException: Queue full
            at java.util.AbstractQueue.add(AbstractQueue.java:98) ~[?:1.8.0_172]
            at java.util.concurrent.ArrayBlockingQueue.add(ArrayBlockingQueue.java:312) ~[?:1.8.0_172]
            at com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent(JsscSerialPortInputStream.java:147) ~[mango-3.5.6.jar:?]
            at jssc.SerialPort$LinuxEventThread.run(SerialPort.java:1299) ~[jssc-2.8.0.jar:?]
    
    

    Thanks Fox


  • Hi Fox,

    Hmm. The code in question is here: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/infiniteautomation/mango/io/serial/JsscSerialPortInputStream.java#L164 and the tasks it's running are here: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/infiniteautomation/mango/io/serial/SerialPortProxyEventTask.java

    It looks like if the SerialPortProxyEventTask were executed, it would definitely remove itself from the queue of listener tasks. This makes me wonder if a thread pool were full, but upon checking the JsscSerialPortInputStream it seems like the event tasks are executed in their own threads and not in the Mango thread pools. For more information, collect a thread dump from the system status page under administration so we can see where these threads have gotten in their execution and perhaps why they are not being removed from the existing tasks.

    It does seem like the JsscSerialPortInputStream doesn't do exactly what some of its comments purport it to do: blocking if the task list is full until the task can be added. I will look into that more, awaiting some thread dumps. Thanks for bringing this to our attention!


  • No thanks for the rapid response Phil, you want the dump here or shall I email it to you?


  • Either would likely be fine, but may as well email.


  • Sent


  • Thanks for that Fox, I think it revealed what the underlying issue is. I think if you restart the serial data sources, this problem may go away for a while. Will try to have a fix out this week.


  • Yeah, I wonder if this ties in with the issue I was having last year with them dropping out and stopping data from coming in...


  • I don't think so. I think that had to do with the virtual serial server socket being relatively unused at that point. I am guessing your serial data is still working?

    This is more like the serial data source is being told 20 times to read the port (the max queue size) and then every time it's registered as a listener > 20 it's hitting that stack trace in the OP, is my thought. Being told to process the serial even 20 times is not a problem, since the first time it would drain the input stream and that is synchronized in the data source.

    So, serial data is still flowing, correct?


  • No I had to turn it off and on like you suggested before it would accept data. It's almost as if the buffers were being filled and discarded. I think I might have to send you my software to help with testing it. Testing with socat would work too though I think...


  • But, the stack trace in the OP is from a physical serial port I would think, not the virtual serial server socket, by virtue of being in the JSSC's event thread.


  • Right gotcha. I'll watch this space and we can see how you get on. Thanks again my man.


  • You could enable debug logging for the com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream and it should output every character received byte by byte. It may also reveal over 20 sequential invocations of the serialEvent method, which may add information.

    <AsyncLogger includeLocation="true" name="com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream" level="debug"/>
    

    in Mango/classes/log4j2.xml near the bottom. Reload via /system_settings.shtm page Log4J Reset section


  • @phildunlap said in Serial Port Problem, Listener task queue full, data will be lost!:

    <AsyncLogger includeLocation="true" name="com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream" level="debug"/>

    If it's beneficial to you then sure. I'll get it implemented now for you.


  • Well, the byte level output may be more on the overwhelming side, but if you get the stack trace in the OP I would maybe check out the logging to see if "Serial Receive Event fired." shows up a bunch just before it. You do have a physical serial port in use here, yeah?


  • Ah yes I do, only one though. The rest work through the virtual socket.


  • Hrmm it appears it was turned off... Neither I or any colleagues disabled it...
    I wonder if the error caused that. I only toggled my data related ports, not this keepalive port...


  • DEBUG 2019-03-26T10:52:40,708 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.:62) - Creating Jssc Serial Port Input Stream for: /dev/ttyUSB11 
    INFO 2019-03-26T10:52:40,714 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Connection' initialized 
    INFO 2019-03-26T10:52:40,716 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Connection' took 24.55144ms to start 
    INFO 2019-03-26T10:54:30,269 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Message' initialized 
    INFO 2019-03-26T10:54:30,270 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Message' took 80.276995ms to start 
    INFO 2019-03-26T10:57:34,036 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:423) - Data source 'Keepalive Connection' stopped 
    DEBUG 2019-03-26T10:58:39,274 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.:62) - Creating Jssc Serial Port Input Stream for: /dev/ttyUSB11 
    INFO 2019-03-26T10:58:39,280 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Connection' initialized 
    INFO 2019-03-26T10:58:39,281 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Connection' took 13.000694ms to start 
    INFO 2019-03-26T10:59:36,361 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:423) - Data source 'Keepalive Connection' stopped 
    DEBUG 2019-03-26T10:59:36,929 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.:62) - Creating Jssc Serial Port Input Stream for: /dev/ttyUSB11 
    INFO 2019-03-26T10:59:36,934 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Connection' initialized 
    INFO 2019-03-26T10:59:36,936 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Connection' took 12.735079ms to start 
    
    DEBUG 2019-03-26T11:14:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:14:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
    ERROR 2019-03-26T11:14:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
    DEBUG 2019-03-26T11:16:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:16:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
    ERROR 2019-03-26T11:16:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
    DEBUG 2019-03-26T11:17:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:17:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
    ERROR 2019-03-26T11:17:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
    DEBUG 2019-03-26T11:18:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:18:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
    ERROR 2019-03-26T11:18:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
    DEBUG 2019-03-26T11:21:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:21:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
    ERROR 2019-03-26T11:21:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
    DEBUG 2019-03-26T11:24:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:24:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: fe 
    DEBUG 2019-03-26T11:25:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
    DEBUG 2019-03-26T11:25:00,005 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: ff 
    

    I'll have to monitor...