One other note on this.

I found the following info on the 65533 value/� symbol:

U+FFFD (decimal 65533) is the "replacement character". When a decoder encounters an invalid sequence of bytes, it may (depending on its configuration) substitute � for the corrupt sequence and continue.

One common reason for a "corrupt" sequence is that the wrong decoder has been applied.

Terry,

We've been working on this and have it sort of working, but still running into an issue. Hopefully you'll be able to provide some insight and get us over the hump.

I'm using Packet Sender software to test sending varius HEX values to the server and being read by the serial data source. As you suggested a Meta data source was created to convert the Mango stored value back to HEX.

This is close to working, but the issue is that whenever I send a hex value greater than the 128th ASCII value, Mango saves it as � Is this due to Mango trying to convert it to ASCII, or could there be something on the server side that we need to modify to allow Mango to save it properly?

Here is a walkthrough of our process working with HEX values that convert to regular ASCII:

• Pass "this is a test" in hex to the server (hex: 74 68 69 73 20 69 73 20 61 20 74 65 73 74)
• Data point is saved as "this is a test"
  
• Meta DP converts it properly back to the HEX value
• Meta DP Code:

var str = varPacket.value;
print("Str: "+str);
var bytes = []; // char codes

for (var i = 0; i < str.length; ++i) {
  var code = str.charCodeAt(i);
  bytes = bytes.concat([code]);
}
print('bytes', bytes.join(', '));

function toHexString(bytes) {
  return bytes.map(function(byte) {
    return ('0'+(byte & 0xFF).toString(16)).slice(-2);
  }).join('')
}
print ("\nTEST: "+toHexString(bytes));

• Output of script validation:
  

You can see all looks good.

If I pass another HEX value to the DS, but use values past the 128 characters of ASCII, then things don't work.

• Pass a9 a9 a9 a9 a9 a9 to Mango DS (© character)
• The DP value is incorrectly saved as������
  
• The Meta DS cannot validate since the � character converts to 65533 / fd in HEX
  

I hope we're doing something wrong on our side and it's a simple setting that can fix this. Is Mango truly trying to convert the hex to ASCII and not converting it properly since there are a lot more hex values than ASCII?

Here's a quick snippet of the serial DS log, which sees the proper hex values:

2021/10/18-11:40:34,891 I 7468697320697320612074657374
2021/10/18-11:45:13,536 I a9a9a9a9a9a9

Also, the ma.log file shows the values being read properly, but saved incorrectly.

DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: .* and extracted value 7468697320697320612074657374 
DEBUG 2021-10-18T15:40:35,406 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(this is a test@2021/10/18 11:40:35.406) 


DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: .* and extracted value a9a9a9a9a9a9 
DEBUG 2021-10-18T15:45:14,052 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(??????@2021/10/18 11:45:14.052) 

Look forward to hearing any suggestions.

Thanks,
Chad

Terry,

We added the extra logging and found the following in the log:

DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.serialEvent:419) - Matching will use String: 74657374696e67 
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:640) - Message matched regex:  
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:653) - Point Identified:  
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.matchPointValue:659) - Point Value matched regex: (.*) and extracted value 74657374696e67 
DEBUG 2021-09-28T01:58:50,158 (com.infiniteautomation.serial.rt.SerialDataSourceRT.updatePointValue:545) - Saving value: PointValueTime(testing@2021/09/27 21:58:50.158) 

It appears the data point is trying to convert the extracted value/hex to ascii and save it as the data point value. We need to store the hex value so it can be parsed and converted to decimal format. What is the best way for us to do that?

Is there a way for us to access the extracted value directly?

Thanks,
Chad

Thanks, Terry, we'll try that out and see.

Hi, All,

We had a TCP/IP data source that was polling HEX values, and when saving the complete packet to the data point, the value was displayed and saved as HEX.

We now have created a Serial Data source with Hex config and the data points are displaying and saving differently.

I did see this post https://forum.mango-os.com/topic/2788/help-with-a-serial-data-point which is basically the same thing I'm running into. Is there any way to view the HEX, or be able to parse it on the fly?

The I/O log is capturing the values in Hex:

Looks like Mango is trying to convert the hex resulting in garbled values:

I'm trying a very basic data point setup so I can get the entire packet and parse it later to get the specific data needed.

Any help to figure out either of the following would be appreciated:

• Get the point value to display in Hex
• Parse the hex on the fly to get the data I need

Thanks,
Chad

It's been quite a while, but figured I would post a message to let everyone know our fix.

Turns out we were running into a Jetty bug which the support team identified. The latest to v3 core (3.7.8) was applied, and so far after 3+ days we have not seen the issue anymore.

Thanks,
Chad

Hi, Terry,

Is the comment you made here: https://forum.mango-os.com/topic/4691/virtual-serial-client/24 still valid? This seems like the exact situation we have.

This device is a black box, and cannot connect to our IP:port, we have to open a socket to it and just listen to capture packets that it sends out. Based on your note, it sounds like using a Virtual Serial Client with a Serial data source won't work.

If that's the case, is the only real solution to create some middleware/applet that connects to the host and reads the data, then pass it along to Mango in a way it can read?

We thought that if the device passes data every 6 seconds, and we use a TCP/IP Data Source to poll it every 3 seconds, we'd be guaranteed to get all of the packets, but that actually made the results worse, and we got fewer packets than when we had it polling every 6.3 seconds. (Don't know why adding the 0.3 seconds makes a difference, but it definitely does.)

Thanks,
Chad

Thanks, Terry, for the suggestions. I'll check these out and see if we can get anything going.

Thanks,
Chad

Hi, All,

We're testing new Radio/Analog hardware that pushes a value every x seconds. We have set up a TCP/IP data source to poll the IP:port every x seconds, and we get most of the packets, but miss quite a few as well.

When working with the vendor, they performed a couple tests and were able to get 100% of the packets. The packets are being sent by a 4-20mA unit. They used Node-Red to view and capture packets, as well as TCP/IP viewer. Both got all the packets.

I was wondering if there is some sort of listener data source that we could use to keep an open connection to the port:ip and record all packet values as they are updated.

Thanks,
Chad

Wanted to check in and see if this issue was able to be re-created, and if there are any updates.

Thanks,
Chad