Virtual Serial Client

[jcaballeroa]

I am trying to integrate the information of a new peripheral into Mango. The way of working of this peripheral is as follows: You are listening on port 9000, when an external software (in my case it would be Mango) establishes a socket on that port, it starts to send you information periodically (without receiving any command for it).

In order to integrate it I have tried 2 different ways:
1.- Creating a Virtual Serial Port, configured as a Serial Client, pointing to the IP and the port of the new peripheral. Next I have created a Serial Data Source that uses the Virtual Serial Port created, and a data Point to collect the information. The result is that the socket between Mango and the new peripheral is perfectly established, and I see that the new peripheral sends the information through said socket (with Wireshark I monitor the exchange of frames). But it seems that Mango does not read this information, in my log I / O the information that arrives from the peripheral does not appear and that I see in wireshark.

2.- Directly creating a TcpIP Data Source pointing to the IP and the port of the new peripheral, and a data point to collect the information. In this case I see that the communication socket is also established without problems, but the only way for Mango to collect the information that arrives from the socket is to mark the data point as "Queryable". In that case he periodically sends the information in the "Read Command" field, and in that case it seems that if he hears the information back from the socket and updates the Data Point, but I don't need to send anything, just listen.

Is it possible to do this?

[MattFox]

@jcaballeroa I am an avid user of the virtual serial socket for control. Make sure your regular expressions are set correctly in order to capture the data you desire. Also ensure your outgoing isn't a wildcard as that will just absorb everything going out and in.
If needs be, show me the data and I can help you configure your set up to perform as required

Fox

[jcaballeroa]

@MattFox thanks for your answer. I don't think the problem is in regular expressions, or not at least for now. The problem is that the virtual port does not seem to capture the traffic that reaches the communications port.

For example, the log output of the serial data source does not show traffic on the port, however with wireshark I see that if information arrives.

2020/02/06-08:25:50,120 Data source started
2020/02/06-08:41:41,999 Data source stopped
2020/02/06-08:46:53,731 Data source started
2020/02/06-08:47:02,234 Data source stopped
2020/02/06-08:47:02,751 Data source started
2020/02/06-10:42:06,322 Data source stopped
2020/02/06-10:42:59,319 Data source started
2020/02/06-10:43:07,054 Data source stopped
2020/02/06-10:43:07,605 Data source started
2020/02/06-10:45:29,504 Data source stopped

It's as if mango wasn't listening on that ip: port.

And for the TcpIp data source option, the log is the same way, except if I activate the "Queryable" option, then if the frames arrive after the Read Command request, but if I deactivate the "Queryable" option it does not reach me nothing either, and in my case it should be disabled. In fact with the "Queryable" option active, and when I get the information I can see that my regular expressions are fine, since the information is parsed correctly.

2020/02/06-08:42:17,933 Data source started
2020/02/06-08:42:45,936 O 31
2020/02/06-08:42:54,586 I 0233203920302036302036363832204d20312030203020302030203020302030203020302030203020302030203020353420342e35203633202b52412032302e322039312041424344030d0a0233203920302036302036363832204d20312030203020302030203020302030203020302030203020302030203020353420342e35203633202b52412032302e322039312041424344030d0a0233203920302036302036363832204d20312030203020302030203020302030203020302030203020302030203020353420342e35203633202b52412032302e322039312041424344030d0a0233203920302036302036363832204d20312030203020302030203020302030203020302030203020302030203020353420342e35203633202b52412032302e322039312041424344030d0a0233203920302036302036363832204d20312030203020302030203020302030203020302030203020302030203020353420342e35203633202b52412032302e322039312041424344030d0a

[MattFox]

Whats the timeout on your virtual port?
Why on earth is there a lot of toggling on your serial data source?
Edit: if you want query able calls, use a scripted datasource issuing the commands out of your serial port out a point from it

[jcaballeroa]

The timeout of my virtual port = 1000ms (1s), I think is a great value. Should I reduce it?

The openings and closures of the virtual port, have been produced manually, while testing to see if data arrived, do not give more importance. In a stable situation, it opens and stays open.

I don't want "query able calls". What I need is Mango initiate the connection to an IP: external port (that does it well), and once the socket is established, without sending any command, keep listening for data reception (it is in this part where I have problems, what is received by the socket is not received in mango, or so it seems because it is not even logged in the log files).

[MattFox]

Set the socket timeout to 0 first. Judging by what you are doing, start of by making a Honeypot point just for everything to fall into. so .* for the data source, nothing in the point identifier the same as above for the regex.
You say it's not the regex but if it weren't you'd be logging info. Also I feel I'm missing something here. I'll ask another question soon.

Fox

[jcaballeroa]

In case it provides more information. This is my current setting, after changing the timeout to 0, as you recommend.

Virtual Serial Port Configuration:

Data Source Configuration (In this case Mango not permitted me a 0 value for timeout, must be greater than 0):

Data Point Configuration:

The frame that the peripheral send to Mango is: 0238203920302036302036363832204D20312030203020302030203020302030203020302030203020302030203020353420342E35203633202B52412032302E322039312041424344030D0A

And if i make a test String, the value is correct:

In wireshark I see the plot that arrives at the port every 500ms, however the value of the Data Point or the log is not updated with the information that arrives at the port.

[terrypacker]

@jcaballeroa I think your issue is the decoding of the message. You message appears to be in Hexadecimal format but your data source is not configured for that. Mango is reading the input bytes as UTF_8 so the first 2 bytes will not translate to 02 in your regex matching.

[MattFox]

Thanks Terry, I thought we were missing something here

[jcaballeroa]

@terrypacker Thanks Terry for your answer.

425/5000
I just tried changing the data source and checking the "Configuration is in hex" checkbox, and of course changing the regex value to (. *) And the result is the same, I don't have information either in the logger or the data point .

I have also tried a program to inject frames (dockligth scripting), with values in ASCII, and sending the frame "Hello World", and I don't have any kind of result either.

Any other idea?

[terrypacker]

What are the contents of your IO log, I see that you have it enabled.

[jcaballeroa]

That is the problem, that the log is also empty. Traces only appear when I activate and deactivate the data source. The feeling is that Mango is not listening at the port (I don't know if it really will be that way).

2020/02/07-20:21:10,068 Data source started
2020/02/07-20:22:23,056 Data source stopped
2020/02/07-20:22:23,063 Data source started
2020/02/07-20:23:01,703 Data source stopped
2020/02/07-20:23:03,734 Data source started
2020/02/07-20:23:24,410 Data source stopped
2020/02/07-23:05:11,358 Data source started
2020/02/07-23:07:08,568 Data source stopped
2020/02/07-23:07:08,571 Data source started
2020/02/07-23:07:42,151 Data source stopped
2020/02/07-23:07:42,154 Data source started

[MattFox]

FYI:
Datasource Regex: (.*), group 1

Datapoint:
Point Identifier leave empty, value group 1, regex (.*)

We want a honeypot that takes anything that comes in, you may have to have your 02 as \u0002 or something to that effect for it to register. Until we can see how the serial port parses it then no point assuming 02 in front will work.

[jcaballeroa]

Thanks @MattFox. I think it's a great idea to first get all the information from the port, and then filter.

I just tested your directions:

Datasource Regex: (.*), group 1

&
Datapoint:
Point Identifier leave empty, value group 1, regex (.*)

But this doesn't produce any results in the Test String, something it doesn't like.

I have also tried the following pattern in the "Point Identifier" of the "Data Point": ([0-9a-zA-Z]*), and neither produces any result.

Is it possible that it is necessary to indicate a fixed pattern in the "Point Identifier" of the "Data Point" to allow all the information to arrive?

[MattFox]

@jcaballeroa said in Virtual Serial Client:

Is it possible that it is necessary to indicate a fixed pattern in the "Point Identifier" of the "Data Point" to allow all the information to arrive?

No, that's why I want you to use a wildcard. I'm a bit taken aback that there's still nothing in the logfile. Having that regex format is basically an 'open door' to allow everything to come through. I'd half expected we'd have something to work from...

PS: ([0-9a-zA-Z]*) should be (([0-9]|[a-z]|[A-Z]) * ) but * just repeats whatever was previous (0 or more) - hence why I used ' . '

EDIT: Do you think it's entirely possible to change your output from being hex to something else? I think if this is still unsuccessful you'll just have to go with the TCP/IP datasource...

[jcaballeroa]

@mattfox said in Virtual Serial Client:

No, that's why I want you to use a wildcard. I'm a bit taken aback that there's still nothing in the logfile. Having that regex format is basically an 'open door' to allow everything to come through. I'd half expected we'd have something to work from...

Of course, that is what worries me, that nothing is seen in the logs. That is why I think there is some kind of problem with listening to the port, that the incoming information is not heard. It is possible that regex is also not good, but that would be another problem in my opinion besides reading the port, but without having incoming information it is very difficult to verify it.

I just tested the following parameters:
Data source

• Message regex = empty
• Point identifier group = empty (Mango change it to 0)

Data Point:

• Point identifier = empty
• Value group = empty (Mango change it to 0)
• Regex = (.*)

This combination seems to have liked it, and at least produces a match in the Test String. But I still have no data in the log, nor in the data point.

[MattFox]

I've got another idea, maybe a last ditch effort, but it would enable us to see everything...
Are you familiar with socat? Let's use that to test the RAW data coming in for you...

once installed fire it up:

socat /dev/null,ignoreeof TCP:localhost:9000,fork

This will spit everything into a black hole, but it may give us the info we need to parse everything...

[jcaballeroa]

The problem is that the integration I have to do it in a Windows environment, and as I think the socat tool does not exist in Windows.

Anyway tomorrow I will try in one of the Linux environments I have (now it is too late here), with the recommendation that you are making me, it will not be that the problem is because of Windows, and we are going crazy.

[MattFox]

Yes, I'm trying to go with being as generic and simple as possible, once we know what the RAW data is, we can definitely hit this sucker on the head.

Chat tomorrow!

Fox

[jcaballeroa]

Hi @mattfox ,

Sorry, finally yesterday I couldn' t do tests. I've taken it again today.

I just mounted my mango environment on my Centos7, to verify first that there was no problem with Windows when doing these tests. And after replicating the Virtual Serial Port, Data Source and Data Point configuration, the result remains the same.

As you recommended, I used the socat application to record the raw data that arrives from the peripheral. I have made a couple of changes regarding the instruction you sent me, the IP of the peripheral (192.168.1.155), and to display the information on the screen (/dev/stdin)

Below is a screenshot of the result. As you can see, socat does record the information that arrives from the peripheral, but the mango logs still do not.