How to use the new Serial Data Source in Mango Automation 2.1

terrypacker

Both the 'Message Terminator' and 'Message Regex' are used to define how the data source will parse the incoming serial data.

The 'Message Terminator' is the last character of all messages, otherwise known as the termination character of the protocol.

The 'Message Regex' is a Regular Expression that helps extract the desired information from a message. Please look here http://www.regular-expressions.info for information on Regular Expressions and here http://regexpal.com for a nice tool to help you test your regex.

For example if you wanted to collect the value 3 from the following serial message:

0FF3;

you would set

'Message Terminator' to a semicolon ;

and

'Message Regex' to 0FF([0-9]);

mebius

Terry, thanks for your reply. I got it.

then I set the data source as attached pictures ds.PNG, but

a point identifier and value regex are required when I created a data point in the data source(shown as details.PNG). what's the meaning of these two items?
In my case, a response will be sent to the serial port only after the device received a command. How should I config that in Mango?

thanks.

Attachment: download link

terrypacker

No Problem.

The point identifier is a a unique part of a message that will link that message type to that point. The Value Regex is the regex used that will strip out the point value from the captured Message Regex.
In order to send a message out, create a data source and add one point then write the value of the message to the point from the Point Details page.

When writing a point value the message created and sent out is as follows:

'Point Identifier' + 'Point Value' + 'Message Terminator'

So using our previous example, if we want to send the message 0FF?; and we have set:

'Point Identifier' as 0FF
'Message Terminator' as ;

We just have to write the value ? to the point

When reading a point value the message is first processed by the 'Message Regex' and then each point uses its 'Value Regex' and 'Value Index' to pull the point value out. This means that if there are 2 capture groups in the 'Message Regex' there will be 2 points one with a 'Value Index' of 0 and the other with a 'Value Index' of 1 to indicate the capture group that they will apply their Regex to. Capture groups are anything surrounded by ( and ).

I would be happy provide you with an example for your Protocol if you would like to post the message you are trying to use?

mebius

I need to send $016 to a com_port, then the device will response !FFFF00 to the port.

Command message format is $AA6
Response message format is !(data_field1)(data_field2)00
Message terminator is (cr)

Following your instruction, I am confused.

It seems that can't create a settable point under Serial Data Source to send message to a com_port.
What is the right input in Mango for the message terminator (cr) ?

thanks.

terrypacker

After seeing your example I can understand how this is confusing you, this is a bit tricky for your application.

It was a bit misleading telling you to create a 'settable' point for the Serial Data Source. All points created are always settable for this data source. See the attached image on where you would write the value out, remember that the Data Source AND Data Point must both be active before you can write a value to it.

I can understand your confusion for how to set the 'Message Terminator' to a Carriage Return, we are using Apache's String Escape Utils to handle those situations so you would need to input the character sequence of: \n

Another Problem is going to be setting the point value since the 'Point Identifier' is not contained within both the transmitted and recieved messages. This module wasn't designed for such a protocol but we can make it work by doing the following:

Using the message you gave me and assuming the data is being transmitted as ASCII chars here are the settings that would allow you to log different responses to different points by filtering on data_field1.

Data Source Settings
'Message Terminator' - \n (the actual characters of backslash and n that will be escaped into a CR inside of Mango)
'Message Regex' - !(.){2}(.){2}(0)0
'Point Identifier Index' - 0 (To use data_field1 for the point identifier)

**Point Settings - You will need 2 points, one to write and another to read **

Receiving Point
'Data Type' - Alphanumeric
'Point Identifier' - FF
'Value Index' - 1 (To use data_field2 as the value that will have the 'Value Regex' applied to it in mango)
'Value Regex' - .* (To capture all information from data_field2 and store it in mango)

Sending Point
'Data Type' - Alphanumeric
'Point Identifier' - $
'Value Index' - (This won't matter because you will never receive a message for this point because the point identifier will not be received assuming no response messages have a $ in it.
'Value Regex' - (Doesn't matter, same as above)

Then to send the message you would write the characters: 016 to Sending Point and expect to see the value of FF from data_field2 as the value of recieving point. There are many ways of configuring the receiving point, my example just assumes all you want to save is data_field2.

Attachment: download link

phildunlap

Hi mebius,

A lot of this may be restating some of what Terry has already said here, but I just wrote up some documentation for this data source and posted it to our wiki, including a use case on a client's sprinkler system.
Serial Data Source
Serial Point Properties
Sprinkler Use Case

I encourage you to read the use case!

mebius

Thanks Terry and Phildunlap.

I will read the wiki carefully, that's a great supporting resource which I don't know before.

I will come back if the problem can't be solved. :)

mebius

We tried according to Terry's guide and wiki but failed to receive response successfully.

Good to say, Sending the command through mango point is work, but failed to receive response message successfully through another point.

But the response message can be watched clearly by using a serial-port monitoring tool after sending the command message in mango.

But no any response message is displayed in mango, even the receiving point's value regex is set to display all it can receive.

phildunlap

In your case, checking whether any messages can be received would probably mean setting your message regex to (!)(.)\n with point identifier 1 and having an enabled alphanumeric point with point identifier "!", values regex (!)(.)\n and value index 2.

When this data source was first developed it was printing all the serial traffic to the mango console, which made debugging the regex setup much easier. I need to look a little closer when I'm at a computer, and not just waiting for coffee, but I remember having message terminators in the value regex for the point was important.

Edit: So, both the message regex and the value regex should end in your message terminator. We've said your terminator for (cr) is \n, but it depends on your system. If your serial stream watching utility doesn't display special characters in human readable ways, your message terminator is probably one of these three: "\n", "\r" or "\r\n" which will only cause you problems when trying to read a value (as the regex could fail to match the whole message, and since it's currently implemented as a "matches" and not a "find" it will need to match the entire serial message received).

In the vast majority of default implementations, (.*) doesn't match \r or \n. Maybe the best catch-all you could use would be (!)(**) value index 2. I chose vertical bar because nobody uses vertical bar for anything, but if it is in your serial stream you can select a different character for your catch-all debugging point. You can check your message terminator's form this way, by examining it in a scripting or meta data source.

mebius

hi Terry, I set the data source and data point as the first sentence in your latest reply. you can check that from attached screenshot pictures.

and I tried several another replacement, ex. use \r to replace \n.

It still can't work. But I can captured the command sent from mango and response from the device by using a capturing tool.

Any other clue to help me? :)

BTW, mango can't accept \r\n as message terminator.

If in your convenient, you can access my mango instance directly. if you want to do that, please contact me by skype hxg2008. thanks.

Attachment: download link

phildunlap

Hi mebius,

I recompiled the module with debug logging statements.

Backup your serial data source module ([Mango Home]/web/modules/serial) and your log4j configuration files ([Mango Home]/classes debug-log4j.xml and log4j.xml).
Place the attached zip into your modules folder
Place the attached log4j into your classes folder, overwrite the version already in the folder (you should have it backed up).

Now if you start Mango with a serial source, it should log all traffic in both directions on your serial data source. Remember, all regular expressions need to match a whole message (terminator and all)!

Attachment: download link

mebius

Bingo!

I replaced log4j and loaded the new serial module, it works this time.

please see attached screenshot.

Thank you. Phildunlap .

Attachment: download link

phildunlap

Does that mean if you put your old log4j file back, it stops working? That seems unlikely to me, but would be most disturbing.

mebius

It seems that a new log4j.xml was created automatically after debug-log4j.xml was put into the classes folder.
When the old log4j file was put back to replace the new log4j.xml, it stops working.
when put the new log4j.xml back, it works again.

phildunlap

That is not what happened when I tested it on my machine. On my machine it is definitely easier to configure, but the data source still works without the debug-style log4j

mebius

It's same on my machine.

Just now I removed the debug-log4j.xml which you gave to me from the folder and left the new created log4j.xml still there, it still works.

I think this problem is solved, isn't it?

Marc

..

Marc

Hello, Terry and Phil, this is Marc.
Same problem happened while I was trying it on another COM port: the receiving data point showed nothing.
Since we have upgraded to Mango core 2.2, the logging mod cannot be used. I've tried changing the log4j.xml alone, and still no luck.

The command message is ~10012A410000FDA6\r . And response message is a long one with more than 100 characters, which starts with ~10012A00 ,and ends with a \r.

Data Source Settings
'Message Terminator' - \r
'Message Regex' - (.*)
'Point Identifier Index' - 0

Receiving Point
'Data Type' - Alphanumeric
'Point Identifier' - .
'Value Index' - 0
'Value Regex' - (.*)

Sending Point
'Data Type' - Alphanumeric
'Point Identifier' - ~10012A410000FDA (Just to make it easy when I set the point value manually)
'Value Index' - 0
'Value Regex' - (.*)

With the configuration above, every time I set the value of the sending point to 6, a command message is sent to the COM port, and a response message is returned.
Both messages can be seen from a serial monitoring tool. But the receiving point in Mango gets no value.
I've changed all the regex to (.*) , so it should show anything received from the COM port.
The wiki pages do not seem available now, so I'm not sure if there's anything wrong about that configuration. Can you guys give me some tips?

@phildunlap said:

Hi mebius,

I recompiled the module with debug logging statements.

Backup your serial data source module ([Mango Home]/web/modules/serial) and your log4j configuration files ([Mango Home]/classes debug-log4j.xml and log4j.xml).

Place the attached zip into your modules folder

Place the attached log4j into your classes folder, overwrite the version already in the folder (you should have it backed up).

Now if you start Mango with a serial source, it should log all traffic in both directions on your serial data source. Remember, all regular expressions need to match a whole message (terminator and all)!

terrypacker

Can you ensure that the message being received by Mango is terminated by \r

Mango will not start parsing the message until it receives the terminator character.

phildunlap

It looks to me like the configuration is a bit off...

On the data source,

Point identifier index 0 means regular expression match index 0, which will be your entire match (specified match groups start at 1). So, if your response is 100 characters that whole 100 characters would need to be the Point Identifier of the receiving point. So, for testing, let's capture an easy point identifier,
Data Source Settings:
'Message Terminator' - \r
'Message Regex' - (~).*\r
'Point Identifier Index' - 1
The Point Identifier must match the capture through exact comparison, not regex. So,
Receiving Point
'Data Type' - Alphanumeric
'Point Identifier' - ~
'Value Index' - 1
'Value Regex' - ~(.*)\r