Virtual Serial Client

[terrypacker]

What are the contents of your IO log, I see that you have it enabled.

[jcaballeroa]

That is the problem, that the log is also empty. Traces only appear when I activate and deactivate the data source. The feeling is that Mango is not listening at the port (I don't know if it really will be that way).

2020/02/07-20:21:10,068 Data source started
2020/02/07-20:22:23,056 Data source stopped
2020/02/07-20:22:23,063 Data source started
2020/02/07-20:23:01,703 Data source stopped
2020/02/07-20:23:03,734 Data source started
2020/02/07-20:23:24,410 Data source stopped
2020/02/07-23:05:11,358 Data source started
2020/02/07-23:07:08,568 Data source stopped
2020/02/07-23:07:08,571 Data source started
2020/02/07-23:07:42,151 Data source stopped
2020/02/07-23:07:42,154 Data source started

[MattFox]

FYI:
Datasource Regex: (.*), group 1

Datapoint:
Point Identifier leave empty, value group 1, regex (.*)

We want a honeypot that takes anything that comes in, you may have to have your 02 as \u0002 or something to that effect for it to register. Until we can see how the serial port parses it then no point assuming 02 in front will work.

[jcaballeroa]

Thanks @MattFox. I think it's a great idea to first get all the information from the port, and then filter.

I just tested your directions:

Datasource Regex: (.*), group 1

&
Datapoint:
Point Identifier leave empty, value group 1, regex (.*)

But this doesn't produce any results in the Test String, something it doesn't like.

I have also tried the following pattern in the "Point Identifier" of the "Data Point": ([0-9a-zA-Z]*), and neither produces any result.

Is it possible that it is necessary to indicate a fixed pattern in the "Point Identifier" of the "Data Point" to allow all the information to arrive?

[MattFox]

@jcaballeroa said in Virtual Serial Client:

Is it possible that it is necessary to indicate a fixed pattern in the "Point Identifier" of the "Data Point" to allow all the information to arrive?

No, that's why I want you to use a wildcard. I'm a bit taken aback that there's still nothing in the logfile. Having that regex format is basically an 'open door' to allow everything to come through. I'd half expected we'd have something to work from...

PS: ([0-9a-zA-Z]*) should be (([0-9]|[a-z]|[A-Z]) * ) but * just repeats whatever was previous (0 or more) - hence why I used ' . '

EDIT: Do you think it's entirely possible to change your output from being hex to something else? I think if this is still unsuccessful you'll just have to go with the TCP/IP datasource...

[jcaballeroa]

@mattfox said in Virtual Serial Client:

No, that's why I want you to use a wildcard. I'm a bit taken aback that there's still nothing in the logfile. Having that regex format is basically an 'open door' to allow everything to come through. I'd half expected we'd have something to work from...

Of course, that is what worries me, that nothing is seen in the logs. That is why I think there is some kind of problem with listening to the port, that the incoming information is not heard. It is possible that regex is also not good, but that would be another problem in my opinion besides reading the port, but without having incoming information it is very difficult to verify it.

I just tested the following parameters:
Data source

• Message regex = empty
• Point identifier group = empty (Mango change it to 0)

Data Point:

• Point identifier = empty
• Value group = empty (Mango change it to 0)
• Regex = (.*)

This combination seems to have liked it, and at least produces a match in the Test String. But I still have no data in the log, nor in the data point.

[MattFox]

I've got another idea, maybe a last ditch effort, but it would enable us to see everything...
Are you familiar with socat? Let's use that to test the RAW data coming in for you...

once installed fire it up:

socat /dev/null,ignoreeof TCP:localhost:9000,fork

This will spit everything into a black hole, but it may give us the info we need to parse everything...

[jcaballeroa]

The problem is that the integration I have to do it in a Windows environment, and as I think the socat tool does not exist in Windows.

Anyway tomorrow I will try in one of the Linux environments I have (now it is too late here), with the recommendation that you are making me, it will not be that the problem is because of Windows, and we are going crazy.

[MattFox]

Yes, I'm trying to go with being as generic and simple as possible, once we know what the RAW data is, we can definitely hit this sucker on the head.

Chat tomorrow!

Fox

[jcaballeroa]

Hi @mattfox ,

Sorry, finally yesterday I couldn' t do tests. I've taken it again today.

I just mounted my mango environment on my Centos7, to verify first that there was no problem with Windows when doing these tests. And after replicating the Virtual Serial Port, Data Source and Data Point configuration, the result remains the same.

As you recommended, I used the socat application to record the raw data that arrives from the peripheral. I have made a couple of changes regarding the instruction you sent me, the IP of the peripheral (192.168.1.155), and to display the information on the screen (/dev/stdin)

Below is a screenshot of the result. As you can see, socat does record the information that arrives from the peripheral, but the mango logs still do not.

[MattFox]

Agreed, something is definitely amiss here.
Have you tried sending all of this with the wildcard regex and the hex box unchecked?
I'm out of thoughts. With the knowledge I have I've tested all that comes to my mind.
@terrypacker You're up!
I'm gonna see what you find because I'm at a loss here...

[terrypacker]

@jcaballeroa go ahead and post the JSON export for this data source and its point(s). Also include the virtual serial port export JSON. I will setup a local test and see if I can replicate your problem.

Also:

1. What version of Mango are you using?
2. What is the version of the data source module?

[jcaballeroa]

@MattFox

Have you tried sending all of this with the wildcard regex and the hex box unchecked?

Yes, but the result is the same. No Data in log and no data in Data Point.

@terrypacker

What version of Mango are you using?
What is the version of the data source module?

I have tried in 2 versions of Mango: 3.5.6 and 3.7.4. Now I am using version 3.7.4

The version of Data Source Module: 3.7.0

Then the JSON with the requested information:

{
"virtualSerialPorts":[
{
"xid":"VSP_Client",
"address":"localhost",
"port":9000,
"portName":"1",
"type":"SERIAL_SOCKET_BRIDGE",
"timeout":0
}
],
"dataPoints":[
{
"purgePeriod":1,
"setExtremeLowLimit":-179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"intervalLoggingType":"INSTANT",
"dataSourceXid":"DS_e4aa6461-15c2-4f52-8fc8-5deac4ecc66c",
"discardHighLimit":179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"deviceName":"hdfhdh",
"enabled":true,
"rollup":"NONE",
"purgeOverride":false,
"chartRenderer":{
"limit":10,
"type":"TABLE"
},
"xid":"DP_97638d3f-6fb2-42dd-bad5-367219b78e98",
"eventDetectors":[
],
"textRenderer":{
"useUnitAsSuffix":false,
"type":"PLAIN",
"suffix":""
},
"loggingType":"ON_CHANGE",
"overrideIntervalLoggingSamples":false,
"defaultCacheSize":1,
"readPermission":"",
"setPermission":"",
"intervalLoggingPeriodType":"MINUTES",
"intervalLoggingSampleWindowSize":0,
"tolerance":0,
"purgeType":"YEARS",
"plotType":"STEP",
"intervalLoggingPeriod":1,
"discardExtremeValues":false,
"tags":{
},
"unit":"",
"pointLocator":{
"valueIndex":0,
"dataType":"ALPHANUMERIC",
"valueRegex":"(.*)",
"pointIdentifier":""
},
"discardLowLimit":-179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"setExtremeHighLimit":179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368,
"name":"pyupyp",
"chartColour":"",
"preventSetExtremeValues":false,
"simplifyType":"NONE"
}
],
"dataSources":[
{
"purgePeriod":1,
"stopBits":"STOP_BITS_1",
"parity":"NONE",
"dataBits":"DATA_BITS_8",
"type":"SERIAL",
"flowControlIn":"NONE",
"enabled":true,
"logIO":true,
"purgeOverride":false,
"xid":"DS_e4aa6461-15c2-4f52-8fc8-5deac4ecc66c",
"messageRegex":"",
"hex":false,
"maxMessageSize":1024,
"editPermission":"",
"purgeType":"YEARS",
"commPortId":"1",
"maxHistoricalIOLogs":1,
"baudRate":9600,
"retries":1,
"pointIdentifierIndex":0,
"ioLogFileSizeMBytes":1,
"messageTerminator":"",
"alarmLevels":{
"POINT_READ_PATTERN_MISMATCH_EVENT":"URGENT",
"POINT_WRITE_EXCEPTION":"URGENT",
"DATA_SOURCE_EXCEPTION":"URGENT",
"POINT_READ_EXCEPTION":"URGENT"
},
"flowControlOut":"NONE",
"useTerminator":false,
"name":"hdfhdh"
}
]
}

[terrypacker]

Ok I think I've got it figured out. You are trying to do something that isn't supported by that combination of data source and virtual serial port type.

The Serial Data Source is event driven and will only work with the Serial Server type of virtual port as that is the only one that generates events. The Serial Client must be controlled by a polling data source to read and write out of the socket each poll. We probably could/should implement an event driven approach for this other type of port but I'll leave that for now.

Stepping back from the problem it seems the reason the queryable setting on the TcpIp data source works is because after it sends a command it reads the response out of the socket. Probably not ideal as it will be sending something out of the port each poll.

Is there no way to use a Virtual Server type serial port and have the device initiate the connection to a port on the Mango machine?

[jcaballeroa]

@terrypacker

Thank you very much for your answer.

The problem is that there is no way to make the peripheral establish the socket towards Mango. For me it is a black box, which I cannot touch, it is a commercial device that I have acquired. And his way of working is that, he expects the remote device (in this case Mango) to establish a client socket by opening the connection and only then begins to send information through this socket (without the need for any command to be sent for it) . We can discuss whether this method is good or bad, but it is the way it works, and it is not possible to change it unfortunately.

[MattFox]

Use socat then.
Make it act like a relay!
One socat service linking your server to a mango serial server!
If you can't use linux it can be done with a node js app or a c# app.

[jcaballeroa]

Yes, it is a very good idea. I had already thought of something similar, in my case I had thought to use Phyton. So that my python app would open a socket against the peripheral and listen to everything it receives, and then send it to Mango through a Virtual Serial Port configured as Server Port.

Thanks!!

[MattFox]

Anytime, that's what we're (the forum) here for.