Serial Data Point Value Not Displaying as Hex

cmusselm

Hi, All,

We had a TCP/IP data source that was polling HEX values, and when saving the complete packet to the data point, the value was displayed and saved as HEX.

We now have created a Serial Data source with Hex config and the data points are displaying and saving differently.

I did see this post https://forum.mango-os.com/topic/2788/help-with-a-serial-data-point which is basically the same thing I'm running into. Is there any way to view the HEX, or be able to parse it on the fly?

The I/O log is capturing the values in Hex:

Looks like Mango is trying to convert the hex resulting in garbled values:

I'm trying a very basic data point setup so I can get the entire packet and parse it later to get the specific data needed.

Any help to figure out either of the following would be appreciated:

Get the point value to display in Hex
Parse the hex on the fly to get the data I need

Thanks,
Chad

terrypacker

@cmusselm the logging mechanism displays the entire message and uses the same code to convert the input from a byte array into a Hex String as the logic that saves the matched String to the point. My guess is that since it is displaying in the log correctly that you are not matching the entire message to save into the point?

You can get some detailed logging by enabling debug logging for that data source, this might point you in the right direction. Add this to the log4j2.xml configuration and reload it:

<AsyncLogger name="com.infiniteautomation.serial.rt.SerialDataSourceRT" includeLocation="true" level="debug"/>

cmusselm

Thanks, Terry, we'll try that out and see.

cmusselm

Terry,

We added the extra logging and found the following in the log:

DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:419) - Matching will use String: 74657374696e67 
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: (.*) and extracted value 74657374696e67 
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(testing@2021/09/27 21:58:50.158)

It appears the data point is trying to convert the extracted value/hex to ascii and save it as the data point value. We need to store the hex value so it can be parsed and converted to decimal format. What is the best way for us to do that?

Is there a way for us to access the extracted value directly?

Thanks,
Chad

terrypacker

@cmusselm I think I see the confusion. With "Configuration in Hex" set, after the message is received it is converted to ASCII characters that represent the HEX values of the bytes. This ASCII string is matched using the regex while still in the ASCII representation of 2 byte HEX characters. However after the logging is done the value is converted from the 2 byte HEX ASCII representation back into the bytes that were sent over the wire.

That is why when you view the values in the UI they are "garbled", they are basically bytes that are not valid UTF-8 characters but are wrapped in a Java UTF-8 String object.

So to make use of the value in the data point you would need to use a meta point to translate the bytes that are stored in the Alphanumeric point into whatever those bytes represent.

So to recap:

Message is received as byte array

byte[] b = readMessage();

Message is converted to HEX ASCII String:

String message = dumpHex(b, 0, b.length);

public static String dumpHex(byte[] b, int pos, int len) {
    StringBuilder sb = new StringBuilder();
    for (int i = pos; i < len; i++) {
        sb.append(StringUtils.leftPad(Integer.toHexString(b* & 0xff), 2, '0'));
    }
    return sb.toString();
}

Message is matched and value is extracted from String

String value = matchAndExtractValue(message);

Alphanumeric value is saved into data point after converting back to bytes

byte[] data = convertHexStringToBytes(value);
String valueThatIsSaved = new String(data, StandardCharsets.UTF_8)

   /**
     * Convert a string of ASCII HEX to bytes
     * @param stringValue
     * @return
     */
    public static byte[] convertHexStringToBytes(String stringValue) throws ConvertHexException {
        int len = stringValue.length();
        if((len&1) == 1)
            throw new ConvertHexException("Odd value lengths not permitted");
        if(!Pattern.matches(HEX_REGEX, stringValue))
            throw new ConvertHexException("Non-hex character detected.");
        byte[] data = new byte[len / 2];
        for (int i = 0; i < len; i += 2) {
            data[i / 2] = (byte) ((Character.digit(stringValue.charAt(i), 16) << 4)
                    + Character.digit(stringValue.charAt(i+1), 16));
        }

        return data;
    }

This final step is arguably odd since it is converting something that may not be filled with valid UTF-8 bytes into a UTF-8 String. However if you need access to the raw bytes you should be able to extract them out of the String wrapper in a script. I would start by writing a meta script that prints the bytes of the String stored in the value of the point.

cmusselm

Terry,

We've been working on this and have it sort of working, but still running into an issue. Hopefully you'll be able to provide some insight and get us over the hump.

I'm using Packet Sender software to test sending varius HEX values to the server and being read by the serial data source. As you suggested a Meta data source was created to convert the Mango stored value back to HEX.

This is close to working, but the issue is that whenever I send a hex value greater than the 128th ASCII value, Mango saves it as � Is this due to Mango trying to convert it to ASCII, or could there be something on the server side that we need to modify to allow Mango to save it properly?

Here is a walkthrough of our process working with HEX values that convert to regular ASCII:

Pass "this is a test" in hex to the server (hex: 74 68 69 73 20 69 73 20 61 20 74 65 73 74)
Data point is saved as "this is a test"
Meta DP converts it properly back to the HEX value
Meta DP Code:

var str = varPacket.value;
print("Str: "+str);
var bytes = []; // char codes

for (var i = 0; i < str.length; ++i) {
  var code = str.charCodeAt(i);
  bytes = bytes.concat([code]);
}
print('bytes', bytes.join(', '));

function toHexString(bytes) {
  return bytes.map(function(byte) {
    return ('0'+(byte & 0xFF).toString(16)).slice(-2);
  }).join('')
}
print ("\nTEST: "+toHexString(bytes));

Output of script validation:

You can see all looks good.

If I pass another HEX value to the DS, but use values past the 128 characters of ASCII, then things don't work.

The DP value is incorrectly saved as��
The Meta DS cannot validate since the � character converts to 65533 / fd in HEX

I hope we're doing something wrong on our side and it's a simple setting that can fix this. Is Mango truly trying to convert the hex to ASCII and not converting it properly since there are a lot more hex values than ASCII?

Here's a quick snippet of the serial DS log, which sees the proper hex values:

2021/10/18-11:40:34,891 I 7468697320697320612074657374
2021/10/18-11:45:13,536 I a9a9a9a9a9a9

Also, the ma.log file shows the values being read properly, but saved incorrectly.

DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: .* and extracted value 7468697320697320612074657374 
DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(this is a test@2021/10/18 11:40:35.406) 


DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: .* and extracted value a9a9a9a9a9a9 
DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(??????@2021/10/18 11:45:14.052)

Look forward to hearing any suggestions.

Thanks,
Chad

cmusselm

One other note on this.

I found the following info on the 65533 value/� symbol:

U+FFFD (decimal 65533) is the "replacement character". When a decoder encounters an invalid sequence of bytes, it may (depending on its configuration) substitute � for the corrupt sequence and continue.

One common reason for a "corrupt" sequence is that the wrong decoder has been applied.