HTTP Json Receiver Timestamp

[cbyrne]

The timestamp parameter of my JSON payload doesn't seem to be used at the moment, but rather the system time.

I can't see anything in the logs about failed datetime parsing, but it might be failing silently.

Anyone know what the required format is?

[MattFox]

From what I've read in the 3.7 docs:

The JSON variant of the HTTP data point contains two additional fields: the device identifier key and the device identifier itself. These are formed from the use case where a device publishes its name in the JSON object, i.e. "id":"eGauge42", where "id" is the key and "eGauge42" is the identifier. The other properties are inherited from the HTTP Receiver data point properties. If the device identifier key is left blank, the receiver will behave as though every JSON object corresponds to the point.

If you're using receive type 2, then you can also specify a Jackson JSON path to a number of milliseconds epoch timestamp. The device identifier key, parameter name, and timestamp key are all Jackson JSON paths within the received Json object.

And to expand regarding your timestamp question:

Timestamps can be represented in multiple ways. If a timestamp is detected (i.e. is not empty), Mango will attempt to parse it first in the format "yyyyMMddHHmmss", then "yyyy-MM-dd'T'HH:mm:ss'Z'", and finally as a UTC millisecond count since midnight on Jan 1, 1970. (See the "Date/time formats" documentation for more information.)

__device – the device ID of the sender
__time – the time override
__point – the prefix which provides an with which an alternate method of providing a point name. The characters following the prefix must match the characters that follow a '__value' parameter prefix. The characters that follow the prefix are used for matching only, and are otherwise insignificant.
__value – the prefix which provides the corresponding value for a key given with a '__point' prefix.

Basically use a __time parameter and give an ISO timestamp or an epoch timestamp. Alternatively, use the JSON path notation and tell it what your timestamp is in epoch milliseconds.

Hope that shines some light on the situation. if you can, fire your request into https://webhook.site and perhaps copy the request here if you still encounter issues.

Best of luck!

Fox

[Holzatelier]

I have the same issue. My Json data as it is recieved in the "TEST RECIEVER" window:

{
  "analyser": [
    {
      "parameter": "ACI",
      "value": 0.5678807,
      "timestamp": 1628632800000
    },
    {
      "parameter": "ACV",
      "value": 229.658
    }
  ]
}

This is my point setup:

The problem is that there are no errors but the point is not updated at all. If I remove the 'Timestamp identifier' or even point it to some non existing JSON entry like /analyser/0/xyz the point updates just fine using system time as the timestamp.

This is using Version 4.0.0

[Jared Wiltshire]

I was actually looking at this code yesterday. There are 3 formats available but unfortunately someone forgot to add the inputs to the UI.

If you export the data source using the JSON configuration export you will see
"dateFormat":"DATE_FORMAT_BASIC",

There are 3 options -

• DATE_FORMAT_BASIC - yyyyMMddHHmmss
• DATE_FORMAT_TZ - ISO8601 format, e.g. 2021-08-11T15:58:35.204Z
• DATE_FORMAT_UTC - Epoch time stamp (numeric value, ms since 1970)

So you should export your DS, edit the above field then import.

I will file a ticket to get the UI fixed.

[Holzatelier]

@jared-wiltshire Thank you for your prompt response.
Unfortunately I'm still unable to get the timestamp logged with the point value.

I changed my data source as suggested:

{
  "dataSources":[
    {
      "xid":"DS_580c90e4-2b83-47aa-a4e7-3062c1a748f0",
      "name":"JSON-test",
      "enabled":true,
      "type":"HTTP_JSON_RECEIVER",
      "alarmLevels":{
        "SET_POINT_FAILURE":"URGENT",
        "JSON_READ_FAILURE":"URGENT"
      },
      "purgeType":"YEARS",
      "setType":"PUBLISHER",
      "dateFormat":"DATE_FORMAT_UTC",
      "receiveType":"JSON",
      "deviceIdWhiteList":[
        "*"
      ],
      "ipWhiteList":[
        "*.*.*.*"
      ],
      "setPointUrl":"",
      "data":null,
      "editPermission":[
        [
          "superadmin"
        ]
      ],
      "purgeOverride":false,
      "purgePeriod":1,
      "readPermission":[
      ]
    }
  ]
}

My data point is setup like this:


Let's say I want to send a value of 0.123456 with a timestamp of 12 Aug 2021 11:00:00 (ie epoch in ms: 1628766000000).

This is what the HTTP JSON Reciever revieves:

Zeit: Donnerstag, August 12. 2021, 2:28:54 pm

{
  "analyser": [
    {
      "parameter": "ACI",
      "value": 0.123456,
      "timestamp": 1628766000000
    }
  ]
}

But the point is not updated:

However if I send a timestamp of 0:

Zeit: Donnerstag, August 12. 2021, 2:37:55 pm

{
  "analyser": [
    {
      "parameter": "ACI",
      "value": 0.123456,
      "timestamp": 0
    }
  ]
}

The data point is updated using system time:

So it seems that if a JSON message with a valid timestamp is recieved the point is not updated but if the timestamp is not valid or missing then the point is updated using current system time. There are no error messages in the browser or in the ma.log. Neither are there any events logged.

Looks like this has come up before: https://forum.mango-os.com/topic/4976/event-detector-created-event-with-wrong-timestamp?_=1628768475714

[terrypacker]

@holzatelier I've been testing this locally and found that it may be possible that the address of the sending system is using an IPv6 address which is being blocked by your whitelist.

During testing I found my local system was in fact doing this. I also found a few bugs around IPv6 in the data source which will likely prevent you from being able to use it this way.

I suggest you confirm that the sending system is using IPv6 and if so try and force IPv4. Once I did this using your configuration I was able to receive values with the correct timestamps.

[Holzatelier]

@terrypacker Thanks again for looking at this.

I have disabled ipv6 on all adaptors of the sending system:

user@imac21:~$ sudo nano /etc/sysctl.conf
[sudo] password for user:        
user@imac21:~$ sudo sysctl -p
net.ipv6.conf.all.disable_ipv6 = 1
user@imac21:~$ ifconfig
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.126  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 10:dd:b1:af:71:03  txqueuelen 1000  (Ethernet)
        RX packets 1778257  bytes 2274299048 (2.1 GiB)
        RX errors 0  dropped 1399  overruns 0  frame 0
        TX packets 878148  bytes 93002850 (88.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 826  bytes 41215 (40.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 826  bytes 41215 (40.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.129  netmask 255.255.255.0  broadcast 192.168.1.255
        ether ec:35:86:38:91:5e  txqueuelen 1000  (Ethernet)
        RX packets 8  bytes 1257 (1.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 189
        TX packets 126  bytes 16513 (16.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 19  

However this has not changed the behavior regarding the timestamp on my system.
IMHO this looks like an issue with the parsing of the timestamp from the JSON message since the data is recieved correctly yet the point is not updated when a valid timestamp is present in the 'timestamp' tag. If the timestamp is 0 or not present then the point is updated using current system time. It also seems odd to me that there would be no parsing error if the JSON pointer for the timestamp is pointing to a non existing tag.

[Holzatelier]

@holzatelier Just a little update on this.
It seems the data points are indeed updated on my system when timestamps are passed along with the JSON data. What I didn't realize is that if the timestamp predates the last entry in the point value history then the browser window needs to be refreshed to show the added historic data value.

So this works for me now besides the missing JSON parsing Error.

Thank you again for your attention.