Data Point Timing

[BG]

Hello,
First of all I am using
Mango Core: 3.5.6
Mango API: 3.5.2
Mango UI: 3.5.5
Platform: Centos 7.4.1708
Java Version: 1.8.0_161
Browser: Chrome 72.0.3626.121 (Official Build) (64-bit)

I am having a little bit of trouble getting some data points to behave properly.

I have a rain bucket logger that updates every 10 minutes. From the channel "P01 - Rain Gauge" I have created 3 more channels that check the past 3 hours, 24 hours and 72 hours. Our server is set to GMT and does not adjust for daylight savings time.

The 3 / 24/ 72 hour channels each use a simple script:
return p1.past(HOUR, 3).sum;
return p1.past(HOUR, 24).sum;
return p1.past(HOUR, 72).sum

where p1 is the variable set for the original channel

Here is a screen shot of the watchlist:

In this screen shot the original channel "P01 - Rain Gauge" is correct for the latest 10 minute interval.

The first problem I discovered is that the 3 / 24 / 72 hour channels were not picking up the latest reading. It seems that the update for these channel (triggered by the update of the original) happens just before the data is entered into the database. This is effectively making those readings 10 minutes behind.

I have tried putting in an "Execution Delay" in the update settings under the script section of the meta point but this has had no affect on the timings.

So I changed the 72 hour channel to use a cron pattern and update itself every 10 minutes and 30 seconds. The crom pattern was entered as: 30 0/10 * * * ?

Now the value is correct as it has picked up the latest reading from the original channel. But the time has picked up local time and not the server time.

So 2 issues I have encountered.

1. The timing of updating the extrapolated data lags by probably milliseconds
2. The appearance of local time when the server and loggers are set to GMT.

Any suggestions on how to solve these issues would be greatly appreciated.

p.s. the values for the 3 / 24 / and 72 hour readings should be the same in the screen shot as I have only been working on this for a couple of hours.

Cheers

Brian

p.p.s. I have just checked the user time zone settings as well to see what affect they have on this issue. My user account is set to "Use server time zone" but if I change it to Etc/GMT which is the same time zone as the server is set to, the times on the watchlist reduce by an hour. This would make the 72 hour display correct but the other 3 channels show as an hour behind. So this is not a solution either.

[phildunlap]

Hi Brian,

The first problem I discovered is that the 3 / 24 / 72 hour channels were not picking up the latest reading. It seems that the update for these channel (triggered by the update of the original) happens just before the data is entered into the database. This is effectively making those readings 10 minutes behind.

You could probably avoid that by adding the cache control argument into the call to past, like,

return p1.past(HOUR, 3, true).sum; //include cached values

The timing of updating the extrapolated data lags by probably milliseconds

Not sure exactly what you means, but, a meta point script's timestamp upon return will either be,

1. The value of the TIMESTAMP variable as epoch milliseconds (not assigned by default).
2. The time the meta point was expected to get to run, whether by cron or context event.

The appearance of local time when the server and loggers are set to GMT.

Are you saying data is logged with the wrong timestamp or are you saying it's displaying in your browser in another timezone? You could ensure your user in the instance does not have a timezone defined (edit: okay I see you did this!). Otherwise can you check pages in the new UI and see if it's related to that.

[BG]

Hello Phil,
Thank you for your reply.
I have tested the channels with the cache value used and it is working the way I need it to now.

In regards to the local time appearing when I use the cron pattern to update the data point, It seems to happen in the new ui pages as well. Sort of.

On the new watchlist UI, the graph shows the server times ok but the downloaded csv still shows local time for some reason. So it looks like the data was actually logged with local time. I have purged all the data from that data point and will look at it with the cron pattern again using the new UI pages only and let you know what I find.

What I meant by the milliseconds lag is that I suspected that the data from the original meta data point did not get stored in the database before the extrapolated data used it for updates. So I thought it must take several milliseconds (or seconds even) for the original meta data point to get the new data from memory and store it into the database. This is confirmed by using the cached values as you suggested.

So my issue is solved by using the cached values as well. But I still think there may be an issue if anyone needs to update using a cron pattern.

Cheers

Brian

[BG]

Hello Phil,
Just to let you know, I purged the data for the 72 hour data point and reset it to use the cron pattern again. As you can see from the screen shot below, it has started to use the local time again. This is from the new UI pages.

The graph also shows the local time for that data point.

And yet if I download the data from this page and choose the UTC time zone before I download the csv, all the data timestamps are using the correct server time not the local time.

So I can assume that the data has been logged correct just not displayed correctly when using a cron pattern to update the data point.

Cheers

Brian

[Jared Wiltshire]

@BG

1. What timezone is the server's operating system set to?
2. What is the timezone property set to in env.properties (in your overrides)?
3. What is the timezone setting on your user set to in the new UI? Go to /ui/administration/users and choose your the user you are logged in as
4. Please include a screen shot of the watchlist download options so I can see what it displays, like this -
   
5. Do you have any overriden files or an AngularJS user module?
6. What version of Mango and the UI module are you using?

[BG]

Hello Jared,

1. The Linux web server is set to UTC +0000 and will not adjust for daylight savings time. All of our loggers sync their time with the web server.

2. From the env.properties file (both the default and the file in overrides) (no quotes in file):
   "# System time zone. Leave blank to use default VM time zone.
   timezone= "

3. All of our users as set to "Use Server time zone"

4. Here is the screen shot, but I notice since you are asking these questions that the Server and User time zones shown do not match what I have checked. Seems there may be an interpretation error here.
   

5. We don't have any special programming for overrides or Angular. We only have the env.properties in the overrides folder.

6. Mango Core: 3.5.6
   Mango API: 3.5.2
   Mango UI: 3.5.5
   Platform: Centos 7.4.1708

Cheers

Brian

[Jared Wiltshire]

@BG it seems that your JVM is somehow using Europe/London as its timezone, this is not the same as UTC. Currently London is UTC+1 due to summer DST. Double check your OS is setup with UTC timezone and not GMT or Europe/London. If that fails, try setting timezone=UTC in your env.properties file.

[BG]

Hello Jarrod,
I have seen in some Java forums about the issue you mentioned. That would explain the differences that I have seen.

I have taken your advice and set the env.properties file so that timezone=UTC and that seems to have solved the problem.

My downloads are now giving correct times. And the cron patterns are now working and not changing the times!

Thank you all for your help in sorting out a confusion for us.

Cheers

Brian