Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website

Serial Data Point Value Not Displaying as Hex


  • Hi, All,

    We had a TCP/IP data source that was polling HEX values, and when saving the complete packet to the data point, the value was displayed and saved as HEX.

    We now have created a Serial Data source with Hex config and the data points are displaying and saving differently.

    I did see this post https://forum.mango-os.com/topic/2788/help-with-a-serial-data-point which is basically the same thing I'm running into. Is there any way to view the HEX, or be able to parse it on the fly?

    The I/O log is capturing the values in Hex:
    2ce553d3-597e-4725-944d-d8e3af5f2228-image.png

    Looks like Mango is trying to convert the hex resulting in garbled values:
    c920921c-6c80-4e5a-8703-d6544a1abcdf-image.png

    I'm trying a very basic data point setup so I can get the entire packet and parse it later to get the specific data needed.
    35e5859a-132a-4ffc-a25c-6c6271ee959b-image.png

    Any help to figure out either of the following would be appreciated:

    • Get the point value to display in Hex
    • Parse the hex on the fly to get the data I need

    Thanks,
    Chad


  • @cmusselm the logging mechanism displays the entire message and uses the same code to convert the input from a byte array into a Hex String as the logic that saves the matched String to the point. My guess is that since it is displaying in the log correctly that you are not matching the entire message to save into the point?

    You can get some detailed logging by enabling debug logging for that data source, this might point you in the right direction. Add this to the log4j2.xml configuration and reload it:

    <AsyncLogger name="com.infiniteautomation.serial.rt.SerialDataSourceRT" includeLocation="true" level="debug"/>
    

  • Thanks, Terry, we'll try that out and see.


  • Terry,

    We added the extra logging and found the following in the log:

    DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:419) - Matching will use String: 74657374696e67 
    DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
    DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
    DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: (.*) and extracted value 74657374696e67 
    DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(testing@2021/09/27 21:58:50.158) 
    

    It appears the data point is trying to convert the extracted value/hex to ascii and save it as the data point value. We need to store the hex value so it can be parsed and converted to decimal format. What is the best way for us to do that?

    Is there a way for us to access the extracted value directly?

    Thanks,
    Chad


  • @cmusselm I think I see the confusion. With "Configuration in Hex" set, after the message is received it is converted to ASCII characters that represent the HEX values of the bytes. This ASCII string is matched using the regex while still in the ASCII representation of 2 byte HEX characters. However after the logging is done the value is converted from the 2 byte HEX ASCII representation back into the bytes that were sent over the wire.

    That is why when you view the values in the UI they are "garbled", they are basically bytes that are not valid UTF-8 characters but are wrapped in a Java UTF-8 String object.

    So to make use of the value in the data point you would need to use a meta point to translate the bytes that are stored in the Alphanumeric point into whatever those bytes represent.

    So to recap:

    1. Message is received as byte array
    byte[] b = readMessage();
    
    1. Message is converted to HEX ASCII String:
    String message = dumpHex(b, 0, b.length);
    
    public static String dumpHex(byte[] b, int pos, int len) {
        StringBuilder sb = new StringBuilder();
        for (int i = pos; i < len; i++) {
            sb.append(StringUtils.leftPad(Integer.toHexString(b* & 0xff), 2, '0'));
        }
        return sb.toString();
    }
    
    1. Message is matched and value is extracted from String
    String value = matchAndExtractValue(message);
    
    1. Alphanumeric value is saved into data point after converting back to bytes
    byte[] data = convertHexStringToBytes(value);
    String valueThatIsSaved = new String(data, StandardCharsets.UTF_8)
    
       /**
         * Convert a string of ASCII HEX to bytes
         * @param stringValue
         * @return
         */
        public static byte[] convertHexStringToBytes(String stringValue) throws ConvertHexException {
            int len = stringValue.length();
            if((len&1) == 1)
                throw new ConvertHexException("Odd value lengths not permitted");
            if(!Pattern.matches(HEX_REGEX, stringValue))
                throw new ConvertHexException("Non-hex character detected.");
            byte[] data = new byte[len / 2];
            for (int i = 0; i < len; i += 2) {
                data[i / 2] = (byte) ((Character.digit(stringValue.charAt(i), 16) << 4)
                        + Character.digit(stringValue.charAt(i+1), 16));
            }
    
            return data;
        }
    
    

    This final step is arguably odd since it is converting something that may not be filled with valid UTF-8 bytes into a UTF-8 String. However if you need access to the raw bytes you should be able to extract them out of the String wrapper in a script. I would start by writing a meta script that prints the bytes of the String stored in the value of the point.


  • Terry,

    We've been working on this and have it sort of working, but still running into an issue. Hopefully you'll be able to provide some insight and get us over the hump.

    I'm using Packet Sender software to test sending varius HEX values to the server and being read by the serial data source. As you suggested a Meta data source was created to convert the Mango stored value back to HEX.

    This is close to working, but the issue is that whenever I send a hex value greater than the 128th ASCII value, Mango saves it as � Is this due to Mango trying to convert it to ASCII, or could there be something on the server side that we need to modify to allow Mango to save it properly?

    Here is a walkthrough of our process working with HEX values that convert to regular ASCII:

    • Pass "this is a test" in hex to the server (hex: 74 68 69 73 20 69 73 20 61 20 74 65 73 74)
    • Data point is saved as "this is a test"
      1ebeff07-15d4-42b8-9b18-72d8661c31ee-image.png
    • Meta DP converts it properly back to the HEX value
    • Meta DP Code:
    var str = varPacket.value;
    print("Str: "+str);
    var bytes = []; // char codes
    
    for (var i = 0; i < str.length; ++i) {
      var code = str.charCodeAt(i);
      bytes = bytes.concat([code]);
    }
    print('bytes', bytes.join(', '));
    
    function toHexString(bytes) {
      return bytes.map(function(byte) {
        return ('0'+(byte & 0xFF).toString(16)).slice(-2);
      }).join('')
    }
    print ("\nTEST: "+toHexString(bytes));
    
    • Output of script validation:
      b3ce524d-bea4-4709-94da-2ebdcd3e759e-image.png

    You can see all looks good.

    If I pass another HEX value to the DS, but use values past the 128 characters of ASCII, then things don't work.

    • Pass a9 a9 a9 a9 a9 a9 to Mango DS (© character)
    • The DP value is incorrectly saved as������
      8313d65c-69e9-4ce6-a2ad-15a259a10793-image.png
    • The Meta DS cannot validate since the � character converts to 65533 / fd in HEX
      70206811-f369-41d3-b6b2-efe36b193a00-image.png

    I hope we're doing something wrong on our side and it's a simple setting that can fix this. Is Mango truly trying to convert the hex to ASCII and not converting it properly since there are a lot more hex values than ASCII?

    Here's a quick snippet of the serial DS log, which sees the proper hex values:

    2021/10/18-11:40:34,891 I 7468697320697320612074657374
    2021/10/18-11:45:13,536 I a9a9a9a9a9a9
    

    Also, the ma.log file shows the values being read properly, but saved incorrectly.

    DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
    DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
    DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: .* and extracted value 7468697320697320612074657374 
    DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(this is a test@2021/10/18 11:40:35.406) 
    
    
    DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
    DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
    DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: .* and extracted value a9a9a9a9a9a9 
    DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(??????@2021/10/18 11:45:14.052) 
    

    Look forward to hearing any suggestions.

    Thanks,
    Chad


  • One other note on this.

    I found the following info on the 65533 value/� symbol:

    U+FFFD (decimal 65533) is the "replacement character". When a decoder encounters an invalid sequence of bytes, it may (depending on its configuration) substitute � for the corrupt sequence and continue.

    One common reason for a "corrupt" sequence is that the wrong decoder has been applied.