• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    Serial Port Problem, Listener task queue full, data will be lost!

    User help
    2
    17
    2.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MattFoxM
      MattFox
      last edited by

      Hi guys, any idea what this warning issued is referencing?
      Hoping it's the USB port serial adaptor and not the mangoES having a tizzy:

      WARN  2019-03-25T00:04:20,388 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:150) - Serial Port Problem, Listener task queue full, data will be lost!
java.lang.IllegalStateException: Queue full
        at java.util.AbstractQueue.add(AbstractQueue.java:98) ~[?:1.8.0_172]
        at java.util.concurrent.ArrayBlockingQueue.add(ArrayBlockingQueue.java:312) ~[?:1.8.0_172]
        at com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent(JsscSerialPortInputStream.java:147) ~[mango-3.5.6.jar:?]
        at jssc.SerialPort$LinuxEventThread.run(SerialPort.java:1299) ~[jssc-2.8.0.jar:?]

      Thanks Fox

      Do not follow where the path may lead; go instead where there is no path.
      And leave a trail - Muriel Strode

      1 Reply Last reply Reply Quote 0
      • phildunlapP
        phildunlap
        last edited by phildunlap

        Hi Fox,

        Hmm. The code in question is here: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/infiniteautomation/mango/io/serial/JsscSerialPortInputStream.java#L164 and the tasks it's running are here: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/infiniteautomation/mango/io/serial/SerialPortProxyEventTask.java

        It looks like if the SerialPortProxyEventTask were executed, it would definitely remove itself from the queue of listener tasks. This makes me wonder if a thread pool were full, but upon checking the JsscSerialPortInputStream it seems like the event tasks are executed in their own threads and not in the Mango thread pools. For more information, collect a thread dump from the system status page under administration so we can see where these threads have gotten in their execution and perhaps why they are not being removed from the existing tasks.

        It does seem like the JsscSerialPortInputStream doesn't do exactly what some of its comments purport it to do: blocking if the task list is full until the task can be added. I will look into that more, awaiting some thread dumps. Thanks for bringing this to our attention!

        1 Reply Last reply Reply Quote 0
        • MattFoxM
          MattFox
          last edited by

          No thanks for the rapid response Phil, you want the dump here or shall I email it to you?

          Do not follow where the path may lead; go instead where there is no path.
          And leave a trail - Muriel Strode

          1 Reply Last reply Reply Quote 0
          • phildunlapP
            phildunlap
            last edited by

            Either would likely be fine, but may as well email.

            1 Reply Last reply Reply Quote 0
            • MattFoxM
              MattFox
              last edited by

              Sent

              Do not follow where the path may lead; go instead where there is no path.
              And leave a trail - Muriel Strode

              1 Reply Last reply Reply Quote 0
              • phildunlapP
                phildunlap
                last edited by

                Thanks for that Fox, I think it revealed what the underlying issue is. I think if you restart the serial data sources, this problem may go away for a while. Will try to have a fix out this week.

                1 Reply Last reply Reply Quote 0
                • MattFoxM
                  MattFox
                  last edited by

                  Yeah, I wonder if this ties in with the issue I was having last year with them dropping out and stopping data from coming in...

                  Do not follow where the path may lead; go instead where there is no path.
                  And leave a trail - Muriel Strode

                  1 Reply Last reply Reply Quote 0
                  • phildunlapP
                    phildunlap
                    last edited by

                    I don't think so. I think that had to do with the virtual serial server socket being relatively unused at that point. I am guessing your serial data is still working?

                    This is more like the serial data source is being told 20 times to read the port (the max queue size) and then every time it's registered as a listener > 20 it's hitting that stack trace in the OP, is my thought. Being told to process the serial even 20 times is not a problem, since the first time it would drain the input stream and that is synchronized in the data source.

                    So, serial data is still flowing, correct?

                    1 Reply Last reply Reply Quote 0
                    • MattFoxM
                      MattFox
                      last edited by

                      No I had to turn it off and on like you suggested before it would accept data. It's almost as if the buffers were being filled and discarded. I think I might have to send you my software to help with testing it. Testing with socat would work too though I think...

                      Do not follow where the path may lead; go instead where there is no path.
                      And leave a trail - Muriel Strode

                      1 Reply Last reply Reply Quote 0
                      • phildunlapP
                        phildunlap
                        last edited by

                        But, the stack trace in the OP is from a physical serial port I would think, not the virtual serial server socket, by virtue of being in the JSSC's event thread.

                        1 Reply Last reply Reply Quote 0
                        • MattFoxM
                          MattFox
                          last edited by

                          Right gotcha. I'll watch this space and we can see how you get on. Thanks again my man.

                          Do not follow where the path may lead; go instead where there is no path.
                          And leave a trail - Muriel Strode

                          1 Reply Last reply Reply Quote 0
                          • phildunlapP
                            phildunlap
                            last edited by

                            You could enable debug logging for the com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream and it should output every character received byte by byte. It may also reveal over 20 sequential invocations of the serialEvent method, which may add information.

                            <AsyncLogger includeLocation="true" name="com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream" level="debug"/>

                            in Mango/classes/log4j2.xml near the bottom. Reload via /system_settings.shtm page Log4J Reset section

                            1 Reply Last reply Reply Quote 0
                            • MattFoxM
                              MattFox
                              last edited by

                              @phildunlap said in Serial Port Problem, Listener task queue full, data will be lost!:

                              <AsyncLogger includeLocation="true" name="com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream" level="debug"/>

                              If it's beneficial to you then sure. I'll get it implemented now for you.

                              Do not follow where the path may lead; go instead where there is no path.
                              And leave a trail - Muriel Strode

                              1 Reply Last reply Reply Quote 0
                              • phildunlapP
                                phildunlap
                                last edited by phildunlap

                                Well, the byte level output may be more on the overwhelming side, but if you get the stack trace in the OP I would maybe check out the logging to see if "Serial Receive Event fired." shows up a bunch just before it. You do have a physical serial port in use here, yeah?

                                1 Reply Last reply Reply Quote 0
                                • MattFoxM
                                  MattFox
                                  last edited by

                                  Ah yes I do, only one though. The rest work through the virtual socket.

                                  Do not follow where the path may lead; go instead where there is no path.
                                  And leave a trail - Muriel Strode

                                  1 Reply Last reply Reply Quote 0
                                  • MattFoxM
                                    MattFox
                                    last edited by

                                    Hrmm it appears it was turned off... Neither I or any colleagues disabled it...
                                    I wonder if the error caused that. I only toggled my data related ports, not this keepalive port...

                                    Do not follow where the path may lead; go instead where there is no path.
                                    And leave a trail - Muriel Strode

                                    1 Reply Last reply Reply Quote 0
                                    • MattFoxM
                                      MattFox
                                      last edited by MattFox 

                                      DEBUG 2019-03-26T10:52:40,708 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.:62) - Creating Jssc Serial Port Input Stream for: /dev/ttyUSB11 
INFO 2019-03-26T10:52:40,714 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Connection' initialized 
INFO 2019-03-26T10:52:40,716 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Connection' took 24.55144ms to start 
INFO 2019-03-26T10:54:30,269 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Message' initialized 
INFO 2019-03-26T10:54:30,270 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Message' took 80.276995ms to start 
INFO 2019-03-26T10:57:34,036 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:423) - Data source 'Keepalive Connection' stopped 
DEBUG 2019-03-26T10:58:39,274 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.:62) - Creating Jssc Serial Port Input Stream for: /dev/ttyUSB11 
INFO 2019-03-26T10:58:39,280 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Connection' initialized 
INFO 2019-03-26T10:58:39,281 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Connection' took 13.000694ms to start 
INFO 2019-03-26T10:59:36,361 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSourceShutdown:423) - Data source 'Keepalive Connection' stopped 
DEBUG 2019-03-26T10:59:36,929 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.:62) - Creating Jssc Serial Port Input Stream for: /dev/ttyUSB11 
INFO 2019-03-26T10:59:36,934 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:380) - Data source 'Keepalive Connection' initialized 
INFO 2019-03-26T10:59:36,936 (com.serotonin.m2m2.rt.RuntimeManagerImpl.initializeDataSourceStartup:385) - Data source 'Keepalive Connection' took 12.735079ms to start 

DEBUG 2019-03-26T11:14:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:14:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
ERROR 2019-03-26T11:14:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
DEBUG 2019-03-26T11:16:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:16:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
ERROR 2019-03-26T11:16:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
DEBUG 2019-03-26T11:17:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:17:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
ERROR 2019-03-26T11:17:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
DEBUG 2019-03-26T11:18:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:18:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
ERROR 2019-03-26T11:18:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
DEBUG 2019-03-26T11:21:00,003 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:21:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: 
ERROR 2019-03-26T11:21:00,005 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:476) - null java.lang.NullPointerException: null 
DEBUG 2019-03-26T11:24:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:24:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: fe 
DEBUG 2019-03-26T11:25:00,004 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:121) - Serial Receive Event fired. 
DEBUG 2019-03-26T11:25:00,005 (com.infiniteautomation.mango.io.serial.JsscSerialPortInputStream.serialEvent:130) - Recieved: ff

                                      I'll have to monitor...

                                      Do not follow where the path may lead; go instead where there is no path.
                                      And leave a trail - Muriel Strode

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post