Virtual Serial Client

[jcaballeroa]

The timeout of my virtual port = 1000ms (1s), I think is a great value. Should I reduce it?

The openings and closures of the virtual port, have been produced manually, while testing to see if data arrived, do not give more importance. In a stable situation, it opens and stays open.

I don't want "query able calls". What I need is Mango initiate the connection to an IP: external port (that does it well), and once the socket is established, without sending any command, keep listening for data reception (it is in this part where I have problems, what is received by the socket is not received in mango, or so it seems because it is not even logged in the log files).

[MattFox]

Set the socket timeout to 0 first. Judging by what you are doing, start of by making a Honeypot point just for everything to fall into. so .* for the data source, nothing in the point identifier the same as above for the regex.
You say it's not the regex but if it weren't you'd be logging info. Also I feel I'm missing something here. I'll ask another question soon.

Fox

[jcaballeroa]

In case it provides more information. This is my current setting, after changing the timeout to 0, as you recommend.

Virtual Serial Port Configuration:

Data Source Configuration (In this case Mango not permitted me a 0 value for timeout, must be greater than 0):

Data Point Configuration:

The frame that the peripheral send to Mango is: 0238203920302036302036363832204D20312030203020302030203020302030203020302030203020302030203020353420342E35203633202B52412032302E322039312041424344030D0A

And if i make a test String, the value is correct:

In wireshark I see the plot that arrives at the port every 500ms, however the value of the Data Point or the log is not updated with the information that arrives at the port.

[terrypacker]

@jcaballeroa I think your issue is the decoding of the message. You message appears to be in Hexadecimal format but your data source is not configured for that. Mango is reading the input bytes as UTF_8 so the first 2 bytes will not translate to 02 in your regex matching.

[MattFox]

Thanks Terry, I thought we were missing something here

[jcaballeroa]

@terrypacker Thanks Terry for your answer.

425/5000
I just tried changing the data source and checking the "Configuration is in hex" checkbox, and of course changing the regex value to (. *) And the result is the same, I don't have information either in the logger or the data point .

I have also tried a program to inject frames (dockligth scripting), with values in ASCII, and sending the frame "Hello World", and I don't have any kind of result either.

Any other idea?

[terrypacker]

What are the contents of your IO log, I see that you have it enabled.

[jcaballeroa]

That is the problem, that the log is also empty. Traces only appear when I activate and deactivate the data source. The feeling is that Mango is not listening at the port (I don't know if it really will be that way).

2020/02/07-20:21:10,068 Data source started
2020/02/07-20:22:23,056 Data source stopped
2020/02/07-20:22:23,063 Data source started
2020/02/07-20:23:01,703 Data source stopped
2020/02/07-20:23:03,734 Data source started
2020/02/07-20:23:24,410 Data source stopped
2020/02/07-23:05:11,358 Data source started
2020/02/07-23:07:08,568 Data source stopped
2020/02/07-23:07:08,571 Data source started
2020/02/07-23:07:42,151 Data source stopped
2020/02/07-23:07:42,154 Data source started

[MattFox]

FYI:
Datasource Regex: (.*), group 1

Datapoint:
Point Identifier leave empty, value group 1, regex (.*)

We want a honeypot that takes anything that comes in, you may have to have your 02 as \u0002 or something to that effect for it to register. Until we can see how the serial port parses it then no point assuming 02 in front will work.

[jcaballeroa]

Thanks @MattFox. I think it's a great idea to first get all the information from the port, and then filter.

I just tested your directions:

Datasource Regex: (.*), group 1

&
Datapoint:
Point Identifier leave empty, value group 1, regex (.*)

But this doesn't produce any results in the Test String, something it doesn't like.

I have also tried the following pattern in the "Point Identifier" of the "Data Point": ([0-9a-zA-Z]*), and neither produces any result.

Is it possible that it is necessary to indicate a fixed pattern in the "Point Identifier" of the "Data Point" to allow all the information to arrive?

[MattFox]

@jcaballeroa said in Virtual Serial Client:

Is it possible that it is necessary to indicate a fixed pattern in the "Point Identifier" of the "Data Point" to allow all the information to arrive?

No, that's why I want you to use a wildcard. I'm a bit taken aback that there's still nothing in the logfile. Having that regex format is basically an 'open door' to allow everything to come through. I'd half expected we'd have something to work from...

PS: ([0-9a-zA-Z]*) should be (([0-9]|[a-z]|[A-Z]) * ) but * just repeats whatever was previous (0 or more) - hence why I used ' . '

EDIT: Do you think it's entirely possible to change your output from being hex to something else? I think if this is still unsuccessful you'll just have to go with the TCP/IP datasource...

[jcaballeroa]

@mattfox said in Virtual Serial Client:

No, that's why I want you to use a wildcard. I'm a bit taken aback that there's still nothing in the logfile. Having that regex format is basically an 'open door' to allow everything to come through. I'd half expected we'd have something to work from...

Of course, that is what worries me, that nothing is seen in the logs. That is why I think there is some kind of problem with listening to the port, that the incoming information is not heard. It is possible that regex is also not good, but that would be another problem in my opinion besides reading the port, but without having incoming information it is very difficult to verify it.

I just tested the following parameters:
Data source

• Message regex = empty
• Point identifier group = empty (Mango change it to 0)

Data Point:

• Point identifier = empty
• Value group = empty (Mango change it to 0)
• Regex = (.*)

This combination seems to have liked it, and at least produces a match in the Test String. But I still have no data in the log, nor in the data point.

[MattFox]

I've got another idea, maybe a last ditch effort, but it would enable us to see everything...
Are you familiar with socat? Let's use that to test the RAW data coming in for you...

once installed fire it up:

socat /dev/null,ignoreeof TCP:localhost:9000,fork

This will spit everything into a black hole, but it may give us the info we need to parse everything...

[jcaballeroa]

The problem is that the integration I have to do it in a Windows environment, and as I think the socat tool does not exist in Windows.

Anyway tomorrow I will try in one of the Linux environments I have (now it is too late here), with the recommendation that you are making me, it will not be that the problem is because of Windows, and we are going crazy.

[MattFox]

Yes, I'm trying to go with being as generic and simple as possible, once we know what the RAW data is, we can definitely hit this sucker on the head.

Chat tomorrow!

Fox

[jcaballeroa]

Hi @mattfox ,

Sorry, finally yesterday I couldn' t do tests. I've taken it again today.

I just mounted my mango environment on my Centos7, to verify first that there was no problem with Windows when doing these tests. And after replicating the Virtual Serial Port, Data Source and Data Point configuration, the result remains the same.

As you recommended, I used the socat application to record the raw data that arrives from the peripheral. I have made a couple of changes regarding the instruction you sent me, the IP of the peripheral (192.168.1.155), and to display the information on the screen (/dev/stdin)

Below is a screenshot of the result. As you can see, socat does record the information that arrives from the peripheral, but the mango logs still do not.

[MattFox]

Agreed, something is definitely amiss here.
Have you tried sending all of this with the wildcard regex and the hex box unchecked?
I'm out of thoughts. With the knowledge I have I've tested all that comes to my mind.
@terrypacker You're up!
I'm gonna see what you find because I'm at a loss here...

[terrypacker]

@jcaballeroa go ahead and post the JSON export for this data source and its point(s). Also include the virtual serial port export JSON. I will setup a local test and see if I can replicate your problem.

Also:

1. What version of Mango are you using?
2. What is the version of the data source module?

[jcaballeroa]

@MattFox

Have you tried sending all of this with the wildcard regex and the hex box unchecked?

Yes, but the result is the same. No Data in log and no data in Data Point.

@terrypacker

What version of Mango are you using?
What is the version of the data source module?

I have tried in 2 versions of Mango: 3.5.6 and 3.7.4. Now I am using version 3.7.4

The version of Data Source Module: 3.7.0

Then the JSON with the requested information:

{
"virtualSerialPorts":[
{
"xid":"VSP_Client",
"address":"localhost",
"port":9000,
"portName":"1",
"type":"SERIAL_SOCKET_BRIDGE",
"timeout":0
}
],
"dataPoints":[
{
"purgePeriod":1,
"setExtremeLowLimit":-179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"intervalLoggingType":"INSTANT",
"dataSourceXid":"DS_e4aa6461-15c2-4f52-8fc8-5deac4ecc66c",
"discardHighLimit":179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"deviceName":"hdfhdh",
"enabled":true,
"rollup":"NONE",
"purgeOverride":false,
"chartRenderer":{
"limit":10,
"type":"TABLE"
},
"xid":"DP_97638d3f-6fb2-42dd-bad5-367219b78e98",
"eventDetectors":[
],
"textRenderer":{
"useUnitAsSuffix":false,
"type":"PLAIN",
"suffix":""
},
"loggingType":"ON_CHANGE",
"overrideIntervalLoggingSamples":false,
"defaultCacheSize":1,
"readPermission":"",
"setPermission":"",
"intervalLoggingPeriodType":"MINUTES",
"intervalLoggingSampleWindowSize":0,
"tolerance":0,
"purgeType":"YEARS",
"plotType":"STEP",
"intervalLoggingPeriod":1,
"discardExtremeValues":false,
"tags":{
},
"unit":"",
"pointLocator":{
"valueIndex":0,
"dataType":"ALPHANUMERIC",
"valueRegex":"(.*)",
"pointIdentifier":""
},
"discardLowLimit":-179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"setExtremeHighLimit":179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"name":"pyupyp",
"chartColour":"",
"preventSetExtremeValues":false,
"simplifyType":"NONE"
}
],
"dataSources":[
{
"purgePeriod":1,
"stopBits":"STOP_BITS_1",
"parity":"NONE",
"dataBits":"DATA_BITS_8",
"type":"SERIAL",
"flowControlIn":"NONE",
"enabled":true,
"logIO":true,
"purgeOverride":false,
"xid":"DS_e4aa6461-15c2-4f52-8fc8-5deac4ecc66c",
"messageRegex":"",
"hex":false,
"maxMessageSize":1024,
"editPermission":"",
"purgeType":"YEARS",
"commPortId":"1",
"maxHistoricalIOLogs":1,
"baudRate":9600,
"retries":1,
"pointIdentifierIndex":0,
"ioLogFileSizeMBytes":1,
"messageTerminator":"",
"alarmLevels":{
"POINT_READ_PATTERN_MISMATCH_EVENT":"URGENT",
"POINT_WRITE_EXCEPTION":"URGENT",
"DATA_SOURCE_EXCEPTION":"URGENT",
"POINT_READ_EXCEPTION":"URGENT"
},
"flowControlOut":"NONE",
"useTerminator":false,
"name":"hdfhdh"
}
]
}

[terrypacker]

Ok I think I've got it figured out. You are trying to do something that isn't supported by that combination of data source and virtual serial port type.

The Serial Data Source is event driven and will only work with the Serial Server type of virtual port as that is the only one that generates events. The Serial Client must be controlled by a polling data source to read and write out of the socket each poll. We probably could/should implement an event driven approach for this other type of port but I'll leave that for now.

Stepping back from the problem it seems the reason the queryable setting on the TcpIp data source works is because after it sends a command it reads the response out of the socket. Probably not ideal as it will be sending something out of the port each poll.

Is there no way to use a Virtual Server type serial port and have the device initiate the connection to a port on the Mango machine?