How to use Publishers - HTTP Sender?

[BG]

Hello,
I am hoping some one can help me.

I would like to push several data points out to another server.

It looks like HTTP sender should do what I need, however I do not know how to test it.

How do I set up a receiving entity?

How is the data packaged when it is sent, i.e. is it an csv file or does it need a database connection.

Also how are the static headers and static parameters used?

I have tried to set up an HTTP sender with a url that goes to a shared folder and I tried to enter the user name and password for the folder into the static entries in various formats but I get "Error: invalid response code 401" each time I try.

Thank you in advance for any advice.

Cheers

Brian

[mlavelle42]

I'm interested in this too. I could not get HTTP Sender to operate as I thought it would.

[phildunlap]

Hi all!

Receiving HTTP from the sender requires an HTTP server on the other end which will parse the incoming HTTP packets. Mango provides one such option, the HTTP Receiver, which will handle HTTP packets sent to http[s]://[ip:port]/httpds where this URL points to a Mango with a configured HTTP receiver data source.

You can, of course, create your own HTTP server to receive and parse the messages. A glance at the actual protocol layer can be achieved with some netcat usage.

So, I created a datasource on server 1, and published the data point "virtual - source" to server2:5566. On server 2, I am listening with nc -l 5566. The results are thus:

POST / HTTP/1.1
User-Agent: Mango M2M2 HTTP Sender publisher
Content-Length: 822
Content-Type: application/x-www-form-urlencoded
Host: 192.168.1.20:5566
Connection: Keep-Alive
Expect: 100-continue
Accept-Encoding: gzip,deflate

virtual+-+source=17.129741762312932%4020160322152926&virtual+-+source=4.424986565823971%4020160322152931&virtual+-+source=47.95866368432249%4020160322152936&virtual+-+source=33.30561421067151%4020160322152941&virtual+-+source=53.62508055424671%4020160322152946&virtual+-+source=96.34891172700515%4020160322152951&virtual+-+source=77.90802062514759%4020160322152956&virtual+-+source=61.14097255107299%4020160322153001&virtual+-+source=52.075519464097944%4020160322153006&virtual+-+source=71.37153382113706%4020160322153011&virtual+-+source=100.0%4020160322153016&virtual+-+source=78.40540813062631%4020160322153021&virtual+-+source=34.218161343751845%4020160322153026&virtual+-+source=77.59530286191702%4020160322153031&virtual+-+source=68.57529645445203%4020160322153036&virtual+-+source=40.70898195645236%4020160322153041

We can see I had a little data built up in my buffer, and so it was all published together in the body of the message. I encourage you to read the contextual help dialogues for the publisher and data source. The publisher is hoping to get HTTP status 200 messages back from the target system.

For static parameters (url encoded values) and static headers, we can see their effect in this sample GET form of the publisher:

GET /?paramkey=paramvalue&virtual+-+source=0.0%4020160322154046 HTTP/1.1
User-Agent: Mango M2M2 HTTP Sender publisher
key: value
Host: 192.168.1.20:5566
Connection: Keep-Alive
Accept-Encoding: gzip,deflate

You can see that the target URL contains the parameters (after the ? in the URL, which is just / since I am publishing to ip:port as opposed to ip:port/resource) paramkey=paramvalue and that the HTTP header "key: value" has been added to the message.

[BG]

Hello Phil,
Thank you very much for the information.

Cheers

Brian

[sasa]

@phildunlap

Hi Phil,

what do you prefer in this case. We have to push values to a webservice (HTTP: PUSH) The excepted form is:

{
"identifier": "1",
"timestamp": "1475157787",
"value" : "10",
"type" : "27",
"status" : "1",
"smgwid" : "1",
"obis" : "1-0:1.8.0",
"validation": "xyz"
}

I hope it matches this topic. Some suggstions?
Regards
Sasa

[phildunlap]

Hi Sasa,

I had to do a little research as I'm not overly familiar with HTTP: PUSH. This is a part of the HTTP/2 protocol, correct?

If so, I suspect the easiest path may be writing a small application to sit beside Mango, receive HTTP publisher messages and forward them out on the HTTP/2 connection for servers which have registered with it for push notifications. Writing an HTTP/2 module in Mango is probably not an option at the moment, as we're using Jetty 9.2 which doesn't have HTTP/2 support.

If not, and any style of pushing will do, you could have the client system register with a websocket for notifications, then rephrase the messages into your desired format. The documentation to assist in this is sparse, but you can find an example websocket registration and exchange using a browser's developer tools on one of the Custom Dashboards pages that has a live value. It'll be in the network tab of type "websocket"

[sasa]

@phildunlap

Hi Phil,

I had an error in the post. It is "post" not "push". Sorry for that!
Mango has to send the JSON data to an external service (REST).

Is there something ready in Mango?

Regards
Sasa

[sasa]

@phildunlap

Hi Phil,

I had an error in the post. It is "post" not "push". Sorry for that!
Mango has to send the JSON data to an external service (REST).

Is there something ready in Mango?

Regards
Sasa

[phildunlap]

Hi Sasa,

There is no way to configure the HTTP publisher to send arbitrary JSON in its body.

Depending on the requirements of your situation, you may be able to achieve this with the TCP/IP data source. If the responses to the posts aren't especially important, and if getting data synchronized isn't especially important, meaning live data is the focus, I think TCP/IP and a script could be an adequate solution. I don't know how many values/second you could get, though... hopefully several hundred or thousand.

You'd want a TCP/IP data source, pointed at the IP or host and port that you are wishing to POST data to. Create a single data point on this data source, let's call it "dataOut". Properties;

Datatype: Alphanumeric
Queryable: Unchecked
Read Command:
Value Regex: .*      //Also, you could consider using something that catches \n
Value Index: 0
Settable: Checked
Write Command: %VALUE%

Now you'll want a scripting data source, running on a cron, with one or more points to be published and also dataOut in context. I've presumed the service we're publishing to accepts batches, but the logic of it all should be similar regardless.

var pointsToPublish = [p1, p2, p3]
var startTime = new Date().getTime() - 1000; //This is related to the cron of the script

function buildPost(jsonBody) {
	//Adjust your Host for sure, but you may need additional headers
	return "POST /dataEndpoint HTTP/1.1\nUser-Agent: Sasa custom publisher\nContent-Length: " + jsonBody.length +
	"\nContent-Type: application/json\nHost: localhost:1234\n\n" + jsonBody;
}

for(var p = 0; p < pointsToPublish.length; p+=1) {
	var values = pointsToPublish[p].getValuesSince(startTime);
	var queue = []
	for(var k = 0; k < values.length; k+=1) {
		queue.push({
			"identifier": "1",
			"timestamp": String(values[k].time),
			"value" : String(values[k].value),
			"type" : "27",
			"status" : "1",
			"smgwid" : "1",
			"obis" : "1-0:1.8.0",
			"validation": "xyz"
			});
	}
	
	dataOut.set(buildPost(JSON.stringify(queue)));
}

This script would be running on a cron like 0/1 * * * * ? but this may cause problems. The Timeout for the TCP/IP data source will be limitting for each dataOut.set() call, so you'll want the cron to perhaps be 0/5, then the startTime to be now-5000 such that you can send 15 messages with a 300 ms timeout with some extra wiggle room. Just watch for aborted polls of the Scripting data source, and adjust accordingly.