Event Detector - Created Event with Wrong Timestamp

[cmusselm]

Hello,

We have a High limit event detector set up, and when we tested it (Put the limit below the current data point value) it fired off fine and generated the event as expected.

Last night the data point value went over the threshold for the first time. The event was created, but the timestamp was way off. For some reason the time stamp was future dated by 4 hours.

We use US/Eastern as our Mango timezone, so we're current 4 hours difference from GMT. It seems like the system thought the time was GMT and tried to do an adjustment to EDT making it off by 4 hours.

This never happened when we tested, and I tried some quick tests this morning to re-create it, but cannot. This event triggers texts to go out to a team of people, so because of the time difference we were getting texted every 30 seconds for the same thing.

Here's a quick screenshot of the list of events. As you can see the duration is -14.2K seconds, which means the 3:36am time stamp is 4 hours in the future. This actually triggered at 11:36am EDT, not 3:36am.

One thing to note is, as I said above, we do use US/Eastern as the Mango timezone in the env.properties file, but the server itself is on UTC/GMT. In testing event detectors this didn't seem to matter, but not certain.

Thanks,
Chad

[cmusselm]

Testing text notifications yesterday and the future dating hit me two more times.

This seems like a bug in the system since it doesn't happen consistently and nothing was changed with the event detector. Can this be looked into?

I did go into the purge settings and updated the Discard point values future dates to more than 2 hours instead of 24. This should help, but may not really resolve the problem.

What is the systems behavior if a datapoint future dated beyond the 2 hours is attempted to be inserted relative to the event detector and handler? Will it fail and do nothing, so the event handler will miss the event, or will the event detector not flag the data point in alert status and catch it the next go-around? We need to be sure that it will be caught so users are notified.

It seems that when a data point value is future dated, no other values can be written to it until that future date/time has passed. Is this be design, or is that incorrect? This is a big problem for our event handlers since they build the text message that will be sent to users and put that text message into a Scripting Data Point that interfaced with our Twilio publisher.

If our new message can't be written to the data point and the date/time stamp is off nothing will go out to the users when we have a threshold being passed.

[terrypacker]

@cmusselm I think you are seeing the result of the event detector using the time of the point value that caused the event detector to go active. We don't use "wall clock" time when the detector raises the event. Some data sources will acquire the time from the values it retrieves if it comes with a timestamp and others will use Mango time to assign the the value.

First:

What is the systems behavior if a datapoint future dated beyond the 2 hours is attempted to be inserted relative to the event detector and handler?

It is discarded and nothing happens.

It seems that when a data point value is future dated, no other values can be written to it until that future date/time has passed. Is this be design, or is that incorrect?

This is not true, the values are being backdated and won't show up in the UI unless you chart the time period. If you are using MangoNoSQL the backdates are added when possible so you may see some queuing and a delay before they show up.

[cmusselm]

Terry, thanks for the information it's very helpful. What you say makes some sense to me, but I still have several questions since the behavior I'm seeing doesn't necessarily jive with your explanation.

Timestamp for the event detector
For this data point we are using a JSON Retriever which in addition to the data point value, is getting the date/time stamp in MM/dd/yyyy HH:mm:ss format, and relative to my account’s timezone, is 4 hours in the future (GMT). That helps explain why the timestamp was future dated.

What doesn’t make sense to me is that if it really does this, then why isn’t every event created and every data point’s time stamp future dated? This only happened 3 times in ~20 on event creation from the detector. Something seems off with that.

Also, if I’m getting the date/time stamp from the JSON feed, should the data point details show the time as the JSON feed’s time? That isn’t what I’m seeing, the data point time is in my account’s time zone.

See the screenshot below. 8:32am is my local time. Based on what you’re saying, shouldn’t this display 12:32pm since that’s the date/time sent by the JSON feed?

JSON Feed:

Inserting Data when Future Dated
What you say about future dating and being able to insert data point values between the “now” and future date did not work for me when this happened. A scripting data source had a data point that was future dated, and I could not insert any value into it. I even tried using the set button on the data point details screen. That acted like it worked, but nothing showed up. Even now when I look at the data point details, the values I tried to put in don’t show up.

In the screenshot below there should be values between the 4:21pm and 8:49pm timestamp. The data point is set to log when the point timestamp changes, so even if we log the same value it should still log and show up shouldn’t it?

I will admit, it was pretty frantic when trying to get this to work and dealing with the issue, so this may not have been a great test of that functionality.

To help with all of this though, and reduce/hopefully eliminate the chance of it happening again, I'll not have the JSON Retriever get the date/time stamp from the JSON feed and let Mango generate this on its own.

Thanks,
Chad

[terrypacker]

@cmusselm

What doesn’t make sense to me is that if it really does this, then why isn’t every event created and every data point’s time stamp future dated? This only happened 3 times in ~20 on event creation from the detector. Something seems off with that.

I can't speak to this but I would assume that the time you are receiving is different to what you think (or out of order)

Also, if I’m getting the date/time stamp from the JSON feed, should the data point details show the time as the JSON feed’s time? That isn’t what I’m seeing, the data point time is in my account’s time zone.

The Mango UI will render the time in the timezone setup for the system or logged in user if it is set.

In the screenshot below there should be values between the 4:21pm and 8:49pm timestamp. The data point is set to log when the point timestamp changes, so even if we log the same value it should still log and show up shouldn’t it?

I should have been more specific. Depending on the logging type of the point "backdated" values are handled differently. I tend to use Log All data points which will log everything. These types will ignore backdates:

• On Timestamp Change
• On Change
• On Change Interval
• Interval

[cmusselm]

Thanks for the reply and clarifications.

We're diving into the source system's data to see if there was a blip or something that may have contributed to this.

Knowing the logging type affects the "backdating" helps a lot!

Thanks for all the help,
Chad

[cmusselm]

We set up a test JSON feed and messed around with the date/time stamp a bit. Here's what we found. Not sure how Mango is designed to work, but this is how it's working for us in reality.

• We set up the data point to log on an interval of 5 seconds, with the instant value and the data source polls every 5 seconds
• Mango doesn't actually use the date/time string as date and time stamp for the value being recorded.
• No matter the date/time in the JSON feed, Mango records the value's timestamp as the current Mango/system time
• The feed's date/time must be the same as or later than what was in the feed during the previous poll for Mango to record it, otherwise its ignored
• This leads me to believe the date/time stamp in the JSON feed is used more as an ordinal so Mango inserts values in the correct order, not actually using the date/time that the point value was recorded

[terrypacker]

@cmusselm the JSON Retriever data source will definitely use the time supplied if it is parsed from the message correctly. If the parsing fails you should see a "Parse Exception" event raised. In that situation it will use Mango's poll time as the timestamp for the value. I would triple check your settings on the data source.

[cmusselm]

The behavior I'm seeing doesn't match what you're telling me for our retriever.

In the data point the Validate Time JSON Pointer returns the correct value:

We also are not getting any events/exceptions for this data source or data point.

In the data point details on the left pane it shows the timestamp being passed by the JSON feed, but in the point value history the timestamp is the current Mango stamp, not the feed's.

I'm not really expecting a resolution to this right now, just posting the info so you are aware of what we're seeing. Maybe there's something else that is being done wrong?

Thanks,
Chad

[cmusselm]

One more bit of information I discovered a couple days ago.

For a different reason, we had to reboot the production instance. After the reboot, we were getting warning in the MA.log file about datapoints being too far into the future. Of course it was the data points we were retrieving via the JSON retriever.

As a quick fix I updated the data points, removing the Time JSON Pointer so it used the Mango timestamp.

Something seems off with how that played out. Why didn't the system start to flag the datapoints with that warning right away after I updated the system to not allow data points that far into the future? Does that setting not take effect until reboot? If so, some warning about that on-screen would be nice.

Also, after the reboot it was complaining about every poll having a future date/time stamp. If that's the case, then why didn't the point details screenshot shown above use that future time stamp when the values were being recorded. Seems that some of the date/time functions aren't behaving properly.