Inconsistent values between MQTT and meta points

[iperry]

Heya,

Mango version: 3.5.6

I am trying to resolve an issue with differences between values recorded by a MQTT data point and its associated meta data point.

Data flow:

• Device transmits messages on an hourly basis, which contains sensor values for each 10 minute interval during that previous hour (thus 6 packets of values)
• MQTT service receives these messages, parses them, and posts the values to the topics. The date/time for each packet is set to the appropriate 10 minute interval it occurred on. All 6 values are published to the topic in < 1 sec
• (Mango) MQTT client receives these changes to the topic. The meta point is changed based on the context change

The values recorded by the MQTT data point are:

The value recorded by the meta data point are:

The pattern is interesting: the first and last values in the hour are consistent. The middle 4 values are the same as the last value.

The meta point currently calls a global function to process the raw value. However, for this value (temperature), no manipulation is occurring. The same value should be returned by the meta point.

The logging properties of the meta point is when timestamp changes.

My belief is there is a timing issue as maybe the rapid changes to the topic are occurring too quickly. I have been using this process for other projects successfully but the updates to the topics have been > 30 secs.

Are there considerations I need to make if changes to a data point occur at millisecond frequency? Any insights would be great.

Thanks
Ian

[MattFox]

@iperry said in Inconsistent values between MQTT and meta points:

Are there considerations I need to make if changes to a data point occur at millisecond frequency?

Just looking at what you're saying here just so I don't get my proverbial wires crossed...

1. Your point values update every ten minutes.
2. The updates within the MQTT space occur within a span of a few milliseconds.
3. You suspect that the points aren't updating fast enough prior to the context in your meta point calc firing.

If 3 is correct, my idea is running the meta script every ten minutes, and setting itself with the value and timestamp. myPoint.set(calcTemperatureWithOffset('L1-CS01-01',p2576.value,0),p2576.timestamp ); That way there is no issue with trying to run your system in a reactive nature. just ensure meta point has the 'Is Settable' checkbox ticked.

[iperry]

Hi @MattFox

1. Which point values are you referring to? The meta point values? They should be updated when the MQTT point changes, which in this case is 6 time in < 1 sec.
2. Correct, the updates to the MQTT topics (and thus the MQTT points) occur within a span of a few milliseconds.
3. Not quite (I think). To me it seems that the meta data point is not updating fast enough in response to its context point (the MQTT point). The meta point should be updating 6 times < 1 sec like the MQTT point.

I noticed something in your response. Is there a difference between?
return calcTemperature...
versus
myPoint.set(calcTemperature...)

To me its odd/interesting that the timestamp of the meta point values are consistent with the MQTT point but the value isn't. To me this would seem the meta point is changing sorta right? heh

Is there maybe a synchronization issue with using the global function?

Thanks
Ian

[MattFox]

@iperry said in Inconsistent values between MQTT and meta points:

updated when the MQTT point changes, which in this case is 6 time in < 1 sec.

Ah so this function is practically being called each time each of the points are updated, not just a single point in particular... There could be some sort of deadlock if each metapoint doesn't fire in it's own thread quick enough or something silly like that. But I doubt that's the issue here.

return ... just will set the metapoint at the time it is returned, which in your case is during the context update of a given point. My suggestion was merely to see if this improves capturing all changed values rather than trying to smack them all simultaneously in the same meta data source. Only playing with ideas here. Phil will likely have something more to add.
My idea of using set() just meant that you weren't pressed to have the meta point firing right when every point updates nearly simultaneously. It could fire each of them in their own cron thread every ten minutes, setting the meta point with the correct value and timestamp. Create an additional point and run a historic on it for me then let it run for a few hours to see if it gives you the desired solution.

Fox

[phildunlap]

@mattfox said in Inconsistent values between MQTT and meta points:

Phil will likely have something more to add.

And so I do!

Hi Ian,

This indeed looks like a timing issue. In the runtime of a meta point, the .value is the latest cached value. When doing a history generation it is the value the point had at or before the timestamp being executed on, but in the present it is the DataPointRT.value

To solve this, I would perhaps use p2576.pointValueAt(CONTEXT.getTimestamp(), true).doubleValue instead of p2576.value in the call to the global function. If you have a lot of database batching going on (not the default), then you may need to set the default cache size for the source point to 6, such that a whole run of the data is temporarily in the data point's cache for fast access.

The .doubleValue is because the pointValueAt function would return a PointValueTime, and I presume your function wanted a number.

[MattFox]

Thanks Phil, so from what you're saying here even my suggestion wouldn't change the outcome due to how the system caches data?
Will be good to know with an upcoming project I'm doing will have to resolve as much as 16 points at once through the same algorithm.

[phildunlap]

Correct, your one code snippet that I see is still using .value which is going to refer to the latest cached value when the script runs, unless doing history generation as previously explained.

[iperry]

Hi @phildunlap

The solution you proposed works... most of the time. After I implemented the solution, the data was changing as I expected over the hour time period:

However, as I continued to monitor the data, I was seeing gaps:

The gaps appear to match errors that I was seeing in the events table:

Global function call in meta point:
return calcTemperatureWithOffset('L1-CS02-02', p3452.pointValueAt(CONTEXT.getTimestamp(), true).doubleValue, 0);

It seems like the context is not available when (sometimes) the function is called? It is odd/interesting that the correct value is returned most of the time.

Any ideas?

Thanks
Ian

[phildunlap]

There is a small window of time where the value has been sent to be written to the database, but it has not been written. If the thread that is propelling the meta point gets ahead of that other thread, the value will not be available. It could perhaps be solved by a faster machine, but the solution in the current version of Mango is to increase the data point's default cache size. This will retain the values in the cache and then they should be available. You should set the cache to the size of the number of values the point will receive at once, which sounds like 6 in this case.

We did develop a fix, but didn't pull it in yet (to keep values in the cache until the database has written, which is not a significant memory add since the database controller has the references to the values). We didn't bring it in yet because interval samples would be retained in that fix as it is, which would potentially be a significant memory add. For now, the default cache size would resolve what you're observing.

[iperry]

Hi Phil,

I increased the default cache size to 6 as you suggested and the data points appear to be all created. Thanks! I will continue to monitor the data for any issues.

@phildunlap said in Inconsistent values between MQTT and meta points:

It could perhaps be solved by a faster machine

This Mango instance is currently running on a 2 core Linode VM so I don't believe that the machine is really a factor here. Perhaps the amount of memory could be increased but the VM looks to be running ok. I will likely review the amount of memory needed once I implement the default cache change on all the necessary points.

As an aside: as above, I am currently passing only the value I need to the global function, instead of the entire point (or context). Is there any downside to passing the whole object to the function and let it get the data it needs? Is there a preferred pattern to use?

i.e. return calcTemperatureWithOffset('L1-CS02-02', p3452, 0);

I am thinking ideally that since I have to update the function call to all the necessary points, passing the object would be better. Then any future changes would be limited to only the global function.

Thanks
Ian

[phildunlap]

As you are likely the maintainer of the system, you should do what makes sense to you.

Global scripts are simply stitched into scripts to provide functions or global variables or what have you. Refactoring what gets passed where and how in a script is an implementation decision that I don't see a clear advisement on. It certainly can simplify things if lots of scripts would be doing the same thing to make that a uniform function.

The only thing I can think to add to that is that currently script engines which are in something's runtime do not suddenly execute a global scripts when it is saved to pick up the changes. You would have to restart any meta data sources or scripting sources, for instance.