New Feature Request: Expansion of "Discard Extreme Values" feature with a "Discard if Delta > X" argument.

[Turbo]

Greetings, all:

I'm dealing with some Angry Devices that behave in annoying ways. An example of this behavior is a power register- It merrily reports 45.5 45.7, 48, 90.2, 46.1, 44.2... Or something similar. That 90.2 is NOT a real reading. Big Extremes I can toss out with the "Discard Extreme Values"- This is very useful when some of these machines end up writing Full Scale INT to their registers at the end of the day. However, for these random within-range random excursions, there's no reasonable way to filter this out.

It's not possible for some of these data points to change from 45.7 to 90.2 (for example) in the space of a single poll cycle, so that data is clearly errant. But it's still in the reasonable range of data points values that would be expecting for that data point, so Mango merrily logs it.

Such a setting maybe to "Discard values where delta is greater than X" for each point. Then, for that device, I could stick like "10" in there that would prevent those errant writes from getting into the database.

I see this on some temperature channels as well, where suddenly the temperature will go from 45C to -0.1C in a single 5-second poll. Unless Elsa released her Freeze Powers on that particular sensor, that's not a real measurement. It's an annoying problem right now, since these things sort of pollute the database with hard to post-process trash.

Cheers,
-Greg Linder

[cwangv]

@turbo
i second your suggestion.

[MattFox]

The extreme values are excellent, however:

Why have a delta reading for this? You need to know this if something is malfunctioning at least. Alternatively, why not just use a metapoint to parse your raw data?? I do that because it means you still are able to record your raw info so if there were issues you've got raw proof that something was just firing rubbish as opposed to clean information.

Fox

[cwangv]

@mattfox
Don't the meta data points count against the licensed points allowance? If they do, we basically pay twice for the same point. Am I wrong on this?

[MattFox]

@cwangv Sorry, I was unaware that you are using a gateway/commercial/free license.
You are correct.
I thought this was running on a local mangoES install or of the like... apologies.

EDIT: i've just have another idea... use an event that fires on update and deletes the latest value if it's greater than n size of your previous value... it'd be no different than infinite coding the exact same behaviour in...

Fox

[Turbo]

The problem here is that--- Well, something is ALWAYS broken, and always will be. I've got so far about two-dozen different sites, about half of them are running different types of machine, each with Serial Numbers started with 100 or less, which are constantly breaking in random / different ways.

Here's how this works:
I notice something like the watt/kw/whatever problem I described originally- Clearly, this is Broken, and isn't working right. Great. Now I know. It happens just often enough to Annoy Me, but not often enough that the people who make the equipment care. So, the People who Make The Equipment tell me things like "Well, you know you can only poll our equipment every 10 minutes on obverse full moon, or else it isn't going to work". So, fine, okay, I start polling every 10 minutes on obverse full moon cycles, and some other Dumb Thing pops up with that, so round around it goes again. Eventually I inevitably get the "we haven't seen this before", and then it ends, and it just stays Broken until I roll out metapoints or some other random workaround to get it working.

No matter what, it seems to work for about 143 hours, and then they remote-update the firmware, and some other Damn Thing breaks, since everyone designs everything nowadays to basically require field firmware upgrades every 48 hours, which of course they DO, and then the wheels fall off some different point. "Oh yeah, we forgot to tell you that the current register is now expressed in mA instead of Amps, and the volts have been converted to ergs-per-unit area, because someone thought that was a good idea-- Also, if you don't read it every 10 seconds, the Inverter shuts down the Modbus and won't respond to polls again unless you power cycle the inverter and poll it within 30 seconds. That last thing ACTUALLY IS TRUE.. That was a Huge Pain to work out.

While in principle I completely agree- I do need to know if Something is Broken. But I'm talking largely to Solar Inverters, in quantities of dozens at a time. The sales life of these machines is maybe 2-4 year, and then the manufacturers really don't seem to care about fixing Broken. So it falls on Me and Mango to report data the best I can, so that we can tell our customers when their machines are broken, and if so offer help in fixing them. I've had hours-long with most major domestic inverter manufacturers, and comms is like 80 on their list of things to fix. Or, they say "Well, you have to work with our Integration Partner, and pay them $12,000 for their Magic Box, which converters Modbus/IP to Modbus IP for some reason, and has their own firmware update cycle, and blah blah blah" Once the Next New Machine comes out, the tech support for previous machines instantly dries up, so no one cares about fixing these things any more.

While I appreciate the Meta-pointedness gusto of doing such a thing in a metapoint, I already sink hours and hours into fixing and mapping broken and ill-defined modbus maps on a daily basis, and this includes doing metapoints for some of the most egregious examples of this. It's this use case that made me think that this would be a Feature that might make this a bit less ridiculous.
By the same logic of that argument, why bother with the Discard Extreme Values, or prevent extreme sets options? Both of those could indicate malfunction, and be implemented quite easily as a metapoint as well.

I guess I've kinda given up on "needing proof" that a communications interface / whatever is Broken. It almost always is. I'm trying to get Good Solid Data as best I can to run reports and analysis of the system as a whole, not to fight with disinterested inverter manufacturers about how to fix their kw registers.

Cheers,
-Greg

[MattFox]

I see, sounds like you've had one hell of a frustrating load just dumped on you...

Sorry Greg, I didn't realise the gravity of it all.
I can see that yes this would definitely benefit you. But somehow I feel that something to help automate what you're doing wouldn't be a bad move either...
Sounds like a lot of settings fiddling and the like. Would there be anything you could do or like to have to streamline your workflow?

Fox

[CraigWeb]

This sort of data smoothening is normally handled by a PLC which will take like 400 samples and then average them before providing them on Modbus. I would suggest that you use average logging and increase your cache to a large enough number to mitigate this.

Basically mango will wait till it has to say 25 values it received from each poll and then average them out. Reducing the effect of one bad poll.

[dovydasz]

@mattfox said in New Feature Request: Expansion of "Discard Extreme Values" feature with a "Discard if Delta > X" argument.:

just have another idea... use an event that fires on update and deletes the latest value if it's greater than n size of your previous value... it'd be no different than infinite coding the exact same behavi

I have a very similar situation as Turbo. Could you please help with an example how to create an event which fires some script for deleting the last value?

[MattFox]

Sure,
You'll have to use the point-values api for the deletion though. ([mango_domain]/swagger-ui.html#/point-value-rest-controller/deletePointValuesUsingDELETE_1)
Refer to my http builder forum post here for using the http builder.
Let's call your context variable: pt

var DELTA_MAX = 10;
var last = pt.lastValue(1,false);

var delta = Math.abs(pt.value - last.value);
if(delta > DELTA_MAX)
{
  //http.delete("/rest/v2/point-values/"+pt.xid+"?from="+(pt.timestamp-10)+"&to="+(pt.timestamp+10));
}

I'm afraid I do not know if there are any other means at this time to delete point values without the API.

Fox