How to use the new Serial Data Source in Mango Automation 2.1

phildunlap

I'm not surprised. Unconsumed characters are lingering in the serial stream, and it isn't configured to discard anything. This is causing it to gunk up, but it's getting one value so we can be sure now that Mango is getting the data we're expecting...

Which means the \r terminator character is likely incorrect on one of our ends... Ah, it appears we needed to unescape our terminator field in the code... Give this version a shot:

Edit: Switched versions on ya... give this one a shot

Attachment: download link

Marc

Hi, Philips. Is there anything we should take notice of when using serial-1.1.2 ?
When we tried to add a serial data source, there was a verification failure on the "message terminator" field. We inputted \r , yet it acted like there was no input value.

@phildunlap said:

I'm not surprised. Unconsumed characters are lingering in the serial stream, and it isn't configured to discard anything. This is causing it to gunk up, but it's getting one value so we can be sure now that Mango is getting the data we're expecting...

Which means the \r terminator character is likely incorrect on one of our ends... Ah, it appears we needed to unescape our terminator field in the code... Give this version a shot:

Edit: Switched versions on ya... give this one a shot

Attachment: download link

phildunlap

My bad... fixed that, hopefully this one works for ya.

Attachment: download link

Marc

It worked. Thanks, Philips.
Now we have to work on the issue of multiple response messages from the same serial port.
Let's say there're two sets of messages:

If we send the command "~10012A410000FDA6\r" , the response will be "~10012A00xxxxxx\r" .
If we send the command "~10012AE100003D9E\r" , the response will be "~10012A00xxxxxx\r" .

We are trying meta data points to make automatic and periodic sending.
One sending point can be used for two commands, but both responses will appear in the same receiving point.
How can we differentiate them, or how can we connect one response to its command?
@phildunlap said:

My bad... fixed that, hopefully this one works for ya.

phildunlap

When setting a particular point, the message sent is pointIndentifier + [setValue] + messageTerminator

When differentiating points, you need a more subtle capture group in the datasource regex for the pointIdentifier to match the point's identifiers individually.

Sometimes the best tactic for a protocol is to set all points individually, but sometimes the better tactic is to have a single command point to send all commands through and use other points to receive values.

Marc

Thanks, Philps. One sending point or multiple sending points, these ways are both good.

Since response messages have the same format, how can we tell which is the response to command 1, which is to command 2?

@Marc said:

It worked. Thanks, Philips.
Now we have to work on the issue of multiple response messages from the same serial port.
Let's say there're two sets of messages:

If we send the command "~10012A410000FDA6\r" , the response will be "~10012A00xxxxxx\r" .

If we send the command "~10012AE100003D9E\r" , the response will be "~10012A00xxxxxx\r" .

We are trying meta data points to make automatic and periodic sending.
One sending point can be used for two commands, but both responses will appear in the same receiving point.
How can we differentiate them, or how can we connect one response to its command?
@phildunlap said:
My bad... fixed that, hopefully this one works for ya.

phildunlap

So, in theory, if you sent both messages, the protocol is ambiguous as to which response refers to which question?

The only way I can see that working is using two meta points to buffer the commands and responses for sorting by a script to the appropriate endpoints. Then the serial datasource may have very simple points that catch everything, the meta points addend whatever new message is coming or going, and the script processes the message queues into its own points. You could also write a module for this protocol that hides/accelerates the analysis portion.

It may help to know the serial data source is open source: https://github.com/infiniteautomation/ma-modules-public/tree/master/Serial Data Source

Marc

Managed that using scripting data source and meta data source.
Thanks again for all the help, Phil.

phildunlap

Glad it worked :D

Thanks for bringing the escape character issue to our attention!

Marc

Hi, Philips. Didn't expect to come back so soon myself :)

We start working on a new device with mango 2.4, and I have to use \r\r\n as the message terminator.
Turns out serial data source doesn't accept strings more than one character as the input.
I've checked with the source codes from where you gave, and found that message terminator was of a java.lang.String type.
So is there any chance we get serial data source into accepting characters, other than one character, for the message terminator?

phildunlap

Hi Marc,

It looks like that restriction was improper. I removed it and tried to read around to validate it, but my serial cables have seemingly grown legs and I can't test as thoroughly as I'd like. Let me know!

Attachment: download link

Marc

Thanks, Philips. I've tested this version.
First of all, I was wrong about the protocol. It's a "20H 20H 0DH" at the end of the message, not a "0DH 0DH 0AH" as I thought before.
So I tried "\s\s\r" at first, and something about escape was wrong. On saving data source properties, this version lost the first two \ charactors and turned "\s\s\r" into "ss\r". You may want to look into that.

"[space][space]\r" seems not so well-formed to me, yet it got a correct command message to send.
Then again, the protocol, which is definitely not my favourite, has this response like "0DH 41H xxxxxxx 20H 20H 0DH xxxxxxx 20H 20H 0DH xxxxxxx 20H 20H 0DH".

"20H 20H 0DH" is the end of response, but it also shows up in the middle, twice. Is it still OK to use it as the message terminator?
@phildunlap said:

Hi Marc,

It looks like that restriction was improper. I removed it and tried to read around to validate it, but my serial cables have seemingly grown legs and I can't test as thoroughly as I'd like. Let me know!

phildunlap

The difference from what you'd expect in the resulting terminator using the \s is because Java escape characters are different than regular expression escape characters (spaces are okay with Java). This is unclear, I should probably add that to the documentation.

Unfortunately no, the serial data source wasn't written for protocols that are length dependent or have ambiguous terminators. I think the existing solution is to use the same trick and buffer the serial data into a meta point, then shuffle it to the right point from a script.

Marc

Thanks, Philips.
Learned more about the response. It's like "0DH [X1] [X2] xxxxxxx 20H 20H 0DH 4CH 32H xxxxxxx 20H 20H 0DH 4CH 33H xxxxxxx 20H 20H 0DH", in which [X1] may be letter 'A','B' or 'C', and [X2] is a single digit.

Tryed to split it into 3 parts using 3 data points, each of which ended with "20H 20H 0DH". A sample of the first part is in the screenshot attached.
But "patternMismatch" alarms showed up.
Screenshots of Serial Data Source Configuration, alarm, and event in Mango database are attached.

As may not be seen directly in the screeshot, there are two spaces in both message terminator and message regex. Whole strings are written below:
Message Terminator: [space][space]\r
Message Regex: (?s)([A-CL][0-9].*)[space][space]\r

Embbeded flag expression (?s) is used because there are several single '0DH' in the "xxxxxxxx" part of the response. So the ".*" can match them.

I took out the piece of "messageMatcher" java code from SerialDataSourceRT.java, and tested it with sample message and regex above. It was a match. And I got to find the problem elsewhere.

Now it seems to me there is only one space left in the message and the regex, seen from the highlighted parts of the screenshots. Not sure that is a display issue or really a clue.
What do you see, Philips?

@phildunlap said:

The difference from what you'd expect in the resulting terminator using the \s is because Java escape characters are different than regular expression escape characters (spaces are okay with Java). This is unclear, I should probably add that to the documentation.

Unfortunately no, the serial data source wasn't written for protocols that are length dependent or have ambiguous terminators. I think the existing solution is to use the same trick and buffer the serial data into a meta point, then shuffle it to the right point from a script.

Attachment: download link

phildunlap

Hi Marc,

I modified the serial data source to have a second protocol parsing method: zealous regex. I have updated the help documents, but the short of it is that if the datasource is configured not to use the terminator, it will attempt to match every point's regex with the message as received thusfar and give a value to the first match, popping that match from the input.

Attachment: download link