Thanks Terry, I'm tracking the issue and hoping it can be done sooner.

Hi, guys. I've tested this new version in Mango 2.7, but not sure I was using it by the right way.
The module seems to process only one character after the escape one. So I can get HEX value 00 by typing \0 as read command, and HEX value 01 by typing \1 .
When it comes to complicated HEX value like FD, \253 cannot work because it is treated as three characters, as in HEX value 02 35 33 .
The job is to send and receive strings in HEX format, and the commands may be like 01 00 00 FD AE CA 5C. Using escape characters is one way to input these strings on Mango pages, maybe there are other ways, please let me know.

Quick question: how to input HEX value like FD or CA?
I was writing this in a hurry, any mistakes, please correct me.
Thank you guys.

Thanks to your reply, Terry.
I'm tracking your issue at github and have seen your fix about write command. I can't wait to see all codes fixed and try out the new version of TCP/IP module.

This TCP/IP data source is really handy.
Yet it seems to send and receive data as strings as always, despite of escape characters like '\' .
Is there a way to send and receive HEX data using this data source? @administrators

Solved.

There is a SQL grammer error in Line 18 of the "createTables-MYSQL.sql" file, which is in the "m2m2-reports-2.0.7.zip" module.
There is a dot character where there should be a comma in SQL.

There are some problems that stop a new mango 2.4.2 installation from starting on a Windows 7 machine.

There are some problems that stop a new mango 2.4.2 installation from starting on a Windows 7 machine.

Hello, Joel. This is Marc working with Mebius.
We have acknowladged all alarms and then deleted all from EVENTS table using the SQL statement.
Unluckily it didn't work much. Maybe our trouble comes elsewhere.

We did find something. As you and your colleague Phil know, we are using combinations of scripting data source, serial test data source and meta data source to send and receive messages through serial ports.
There are more than 20 combinations of these data sources in our mango system now.

1. When these 20+ combinations are disabled, other 100+ data sources and publishers can work at a normal speed.
2. When these 20+ combinations are enabled, with periods of 20 seconds on scripting data sources, the whole mango system slow down. All 120+ data sources updates slowly. And all publishers become slower and slower until they stop to publish any data.
3. After we change the periods to 5 mins on scripting data sources, the publishers can work normally, but data sources are still slow. We got stuck when opening the "datasources.shtm" page, several minutes after enabling them.

We need a way that provides both shorter periods and normal speed updates.
Also we see a very high memory usage of mango, up to 1.4G. Is that because the total number of data sources, or something else?

@JoelHaggar said:

I see you have a lot of unacknowledged events. I would recommend purging your events table and see if that speeds things up. I seem to be sharing this thread a lot lately:
http://forum.infiniteautomation.com/forum/posts/list/1796.page

If you can use the SQL statement to delete everything in the events table. Go through your settings and turn things to Do Not Log that you don't need to minimize the number of events that are being created.

Let me know if that helps,
Joel.

Thanks, Philips.
Learned more about the response. It's like "0DH [X1] [X2] xxxxxxx 20H 20H 0DH 4CH 32H xxxxxxx 20H 20H 0DH 4CH 33H xxxxxxx 20H 20H 0DH", in which [X1] may be letter 'A','B' or 'C', and [X2] is a single digit.

Tryed to split it into 3 parts using 3 data points, each of which ended with "20H 20H 0DH". A sample of the first part is in the screenshot attached.
But "patternMismatch" alarms showed up.
Screenshots of Serial Data Source Configuration, alarm, and event in Mango database are attached.

As may not be seen directly in the screeshot, there are two spaces in both message terminator and message regex. Whole strings are written below:
Message Terminator: [space][space]\r
Message Regex: (?s)([A-CL][0-9].*)[space][space]\r

Embbeded flag expression (?s) is used because there are several single '0DH' in the "xxxxxxxx" part of the response. So the ".*" can match them.

I took out the piece of "messageMatcher" java code from SerialDataSourceRT.java, and tested it with sample message and regex above. It was a match. And I got to find the problem elsewhere.

Now it seems to me there is only one space left in the message and the regex, seen from the highlighted parts of the screenshots. Not sure that is a display issue or really a clue.
What do you see, Philips?

@phildunlap said:

The difference from what you'd expect in the resulting terminator using the \s is because Java escape characters are different than regular expression escape characters (spaces are okay with Java). This is unclear, I should probably add that to the documentation.

Unfortunately no, the serial data source wasn't written for protocols that are length dependent or have ambiguous terminators. I think the existing solution is to use the same trick and buffer the serial data into a meta point, then shuffle it to the right point from a script.

Attachment: download link

Thanks, Philips. I've tested this version.
First of all, I was wrong about the protocol. It's a "20H 20H 0DH" at the end of the message, not a "0DH 0DH 0AH" as I thought before.
So I tried "\s\s\r" at first, and something about escape was wrong. On saving data source properties, this version lost the first two \ charactors and turned "\s\s\r" into "ss\r". You may want to look into that.

"[space][space]\r" seems not so well-formed to me, yet it got a correct command message to send.
Then again, the protocol, which is definitely not my favourite, has this response like "0DH 41H xxxxxxx 20H 20H 0DH xxxxxxx 20H 20H 0DH xxxxxxx 20H 20H 0DH".

"20H 20H 0DH" is the end of response, but it also shows up in the middle, twice. Is it still OK to use it as the message terminator?
@phildunlap said:

Hi Marc,

It looks like that restriction was improper. I removed it and tried to read around to validate it, but my serial cables have seemingly grown legs and I can't test as thoroughly as I'd like. Let me know!

Hi, Philips. Didn't expect to come back so soon myself :)

We start working on a new device with mango 2.4, and I have to use \r\r\n as the message terminator.
Turns out serial data source doesn't accept strings more than one character as the input.
I've checked with the source codes from where you gave, and found that message terminator was of a java.lang.String type.
So is there any chance we get serial data source into accepting characters, other than one character, for the message terminator?

Managed that using scripting data source and meta data source.
Thanks again for all the help, Phil.

Thanks, Philps. One sending point or multiple sending points, these ways are both good.

Since response messages have the same format, how can we tell which is the response to command 1, which is to command 2?

@Marc said:

It worked. Thanks, Philips.
Now we have to work on the issue of multiple response messages from the same serial port.
Let's say there're two sets of messages:

1. If we send the command "~10012A410000FDA6\r" , the response will be "~10012A00xxxxxx\r" .
2. If we send the command "~10012AE100003D9E\r" , the response will be "~10012A00xxxxxx\r" .

We are trying meta data points to make automatic and periodic sending.
One sending point can be used for two commands, but both responses will appear in the same receiving point.
How can we differentiate them, or how can we connect one response to its command?
@phildunlap said:
My bad... fixed that, hopefully this one works for ya.

It worked. Thanks, Philips.
Now we have to work on the issue of multiple response messages from the same serial port.
Let's say there're two sets of messages:

1. If we send the command "~10012A410000FDA6\r" , the response will be "~10012A00xxxxxx\r" .
2. If we send the command "~10012AE100003D9E\r" , the response will be "~10012A00xxxxxx\r" .

We are trying meta data points to make automatic and periodic sending.
One sending point can be used for two commands, but both responses will appear in the same receiving point.
How can we differentiate them, or how can we connect one response to its command?
@phildunlap said:

My bad... fixed that, hopefully this one works for ya.

Hi, Philips. Is there anything we should take notice of when using serial-1.1.2 ?
When we tried to add a serial data source, there was a verification failure on the "message terminator" field. We inputted \r , yet it acted like there was no input value.

@phildunlap said:

I'm not surprised. Unconsumed characters are lingering in the serial stream, and it isn't configured to discard anything. This is causing it to gunk up, but it's getting one value so we can be sure now that Mango is getting the data we're expecting...

Which means the \r terminator character is likely incorrect on one of our ends... Ah, it appears we needed to unescape our terminator field in the code... Give this version a shot:

Edit: Switched versions on ya... give this one a shot

Attachment: download link

The tests on the working machine got nothing as before.
But with the same configuration running on the testing notebook, it succeeded at the first time, as the receiving point had a value of "1001". Unfortunately, on every try after that we got only an event instead.

In the test, we changed the message terminator to '2'. That made the sending message become "~10012A410000FDA62".
And the response changed to "~10012A050000FDA6", which was actually a message telling us the sending message was malformatted. But it was good for the test.

At the first time, we got a match, "1001", from the response, which became the point value. We could not tell if there was an event at that time.
After that, we got no point values, but events saying "Recieved message: (~).*2 does not match pattern: A050000FDA6 ~10012".
It seems the places of the received message and the pattern were switched in the event message.

The serial port monitoring logs, screenshots of data source/points configuration and events are archived and attached. Please let us know if there's anything wrong. Thanks.
@Marc said:

Hello, Philips.
There are two machines: a working machine with mango core 2.2.2, serial 1.1.0, and with commercial mango license; the other is a notebook for testing, with the same version of mango but a free license.
In the lib folder, no rxtx jar file, but a jssc-2.8.0.jar. I assume that's what it should be.

The test results were confusing, but some things came out, I'll describe that in the next post.

@phildunlap said:
Okay, let's be sure of things like what version you're using. On the modules page, what's your core version? What's the version of the serial module? In your mango/lib folder, do you have A. RXTX jar file B. JSSC jar file?

As far as further tests, let's change your message terminator in all relevant spots to '2', point identifier to '1' and see if it is getting values then.

Attachment: download link

Hello, Philips.
There are two machines: a working machine with mango core 2.2.2, serial 1.1.0, and with commercial mango license; the other is a notebook for testing, with the same version of mango but a free license.
In the lib folder, no rxtx jar file, but a jssc-2.8.0.jar. I assume that's what it should be.

The test results were confusing, but some things came out, I'll describe that in the next post.

@phildunlap said:

Okay, let's be sure of things like what version you're using. On the modules page, what's your core version? What's the version of the serial module? In your mango/lib folder, do you have A. RXTX jar file B. JSSC jar file?

As far as further tests, let's change your message terminator in all relevant spots to '2', point identifier to '1' and see if it is getting values then.

Hi,Philips. We followed your instruction but Mango showed no value of the receiving point.
The attachment is the log from COM monitoring tool, as same as last time with my configuration.
Both sending and receiving messages can be seen. The "java.exe" means those are from and to the Mango thread. The suffix "." is the printable form of "\r", as the hexadecimal form of the message shows that's a "0D" charactor.

I've also set ```
<category name="com.infiniteautomation"><level value="all"/></category>

 
Could you guys have a look and find what's wrong there?
@phildunlap said:
> It looks to me like the configuration is a bit off...
>
>On the data source,
>1. Point identifier index 0 means regular expression match index 0, which will be your entire match (specified match groups start at 1). So, if your response is 100 characters that whole 100 characters would need to be the Point Identifier of the receiving point. So, for testing, let's capture an easy point identifier,
>Data Source Settings:
>'Message Terminator' - \r
>'Message Regex' - (~).*\r
>'Point Identifier Index' - 1
>
>2. The Point Identifier must match the capture through exact comparison, not regex. So,
>Receiving Point
>'Data Type' - Alphanumeric
>'Point Identifier' - ~
>'Value Index' - 1
>'Value Regex' - ~(.*)\r 

 Attachment: [download link](http://archive.infiniteautomation.com/forum/posts/downloadAttach/276.page)

I think that's a yes, Terry.
I should have posted the log of COM monitoring tool. It shows clearly that mango receives the message which terminated by \r.
Since Philips've given a set of new configurations, it's better that I offer you new logs. Will do that as soon as I get those logs.

@terrypacker said:

Can you ensure that the message being received by Mango is terminated by \r

Mango will not start parsing the message until it receives the terminator character.

Hello, Terry and Phil, this is Marc.
Same problem happened while I was trying it on another COM port: the receiving data point showed nothing.
Since we have upgraded to Mango core 2.2, the logging mod cannot be used. I've tried changing the log4j.xml alone, and still no luck.

The command message is ~10012A410000FDA6\r . And response message is a long one with more than 100 characters, which starts with ~10012A00 ,and ends with a \r.

Data Source Settings
'Message Terminator' - \r
'Message Regex' - (.*)
'Point Identifier Index' - 0

Receiving Point
'Data Type' - Alphanumeric
'Point Identifier' - .
'Value Index' - 0
'Value Regex' - (.*)

Sending Point
'Data Type' - Alphanumeric
'Point Identifier' - ~10012A410000FDA (Just to make it easy when I set the point value manually)
'Value Index' - 0
'Value Regex' - (.*)

With the configuration above, every time I set the value of the sending point to 6, a command message is sent to the COM port, and a response message is returned.
Both messages can be seen from a serial monitoring tool. But the receiving point in Mango gets no value.
I've changed all the regex to (.*) , so it should show anything received from the COM port.
The wiki pages do not seem available now, so I'm not sure if there's anything wrong about that configuration. Can you guys give me some tips?

@phildunlap said:

Hi mebius,

I recompiled the module with debug logging statements.

1. Backup your serial data source module ([Mango Home]/web/modules/serial) and your log4j configuration files ([Mango Home]/classes debug-log4j.xml and log4j.xml).
2. Place the attached zip into your modules folder
3. Place the attached log4j into your classes folder, overwrite the version already in the folder (you should have it backed up).

Now if you start Mango with a serial source, it should log all traffic in both directions on your serial data source. Remember, all regular expressions need to match a whole message (terminator and all)!