Greetings:

I gave up on using the built in Units in Mango. They seem to be.. Sort of broken and generally annoying, since there never seems to be the right M/k/u/m type for whatever I'm doing-- Megawatts, kilowatts, whatever. The Best Practice seems to be to just use the Text Renderer Properties, and set the Suffix to Whatever unit you want.

I tried using the built-in units a number of ways, but I got tired of dealing with the little "dot" for multiplied units which really aren't. I don't remember which unit, but it was something like kwh showing as kw<dot>h.. It's not kilowatt TIMES hours.. Anyways. I've run into trouble with import/export also using built-in units, where m^2 doesn't import right under some conditions (for example). Don't have these troubles using the "text renderer properties" Suffix box

Yeah. The general practice seems to be "use the text renderer properties"... Makes your life easier.

Thanks for the quick updates: @CraigWeb @Jared-Wiltshire

This fixed the Modbus tools, and I just tested it, so we're good to go. Thanks for that. These tools are Super Useful, as even thought Modbus has been around for longer than I've been alive, people who implement it still can't agree if it's 0-indexed or 1-indexed, or what the actual difference is between holding and input registers are.

Also, I verified the flashing save Icon issue with multiple tabs... I think I noticed this before, but in my exhausted stupor the previous night I was sort of not thinking straight.

Thanks!

Greetings, all:

I've been using USB to Serial converters on my little Linux Based Mango devices for my business. I've had varying luck by brand, and I figured this is as good a place as any to talk about by mest brand so far, and some troubles they've had.

The ones I use are the DTech USB 2.0 to RS 422/ RS 485 adapters. These units are based on the FTDI Chipset, and I generally have Good Luck. I've probably installed clost to 50 of these things, with so far 3 failures- One due to a direct lightning hit, and the other two Very Odd, Indeed.

It's this "very odd indeed" one that I want to comment on, as a sort of Use Case for how Super Useful the "Log IO" function can be in Mango. I was doing final testing on a product that was too ship, and found that the Modbus Serial data source was behaving all weirdly-- Sometimes polling correctly, but generally Failing Wholesale.

I was just about geared about to post a mango thread about wondering if the latest Modbus module had some bugs in it, but lo and behold, the "Log IO" showed me something was weird.

So:
The poll should look like this (Using the CAS Modbus Scanning Tool, if you ever need a Modbus Doodad, that's a great little test tool)
[22:21:24] => Poll: C8 04 00 00 00 14 E1 9C
[22:21:25] <= Response: C8 04 28 00 6E 00 9A 08 8D 08 84 08 81 08 8F 08 87 08 A1 00 D4 01 AF 20 E2 00 0E 00 21 00 20 00 00 00 6C 00 00 00 00 00 00 09 28 AD 11

Mango was seeing this:
2019/08/29-22:12:21,943 O c80400000014e19c
2019/08/29-22:12:21,955 I 6feeffffffd73d8efe
2019/08/29-22:12:22,023 I c80428006e00a708da08d808da
2019/08/29-22:12:22,025 I 08e708da
2019/08/29-22:12:22,026 I 08f5
2019/08/29-22:12:22,027 I 00
2019/08/29-22:12:22,027 I d401
2019/08/29-22:12:22,028 I c320
2019/08/29-22:12:22,049 I e1000a002a00250000007000000000000008f8eb71

I was thinking-- OKay, maybe there's a bug in the serial driver? You can see Mango originating the proper request "c80400000014e19c " which is a read input registers starting at 0 for 20.

Then, Mess starts coming back into the device almost immediatly after. You can see the start of my Slave device talking back, "c80428006e00a708da08d808da" .. But before that is this "6feeffffffd73d8efe"

Turns out that Mess was coming from the RS485 chip-- It looks almost as if the Serial device server isn't actually swapping RX/TX direction (It's a single duplex, 19200 link for this connection), and is scrambling itself up. So I plugged in that USB adapter into my Windows Box, and CAS Modbus Scanner says the same mess:
[22:19:12] => Poll: C8 04 00 00 00 14 E1 9C
[22:19:12] <= Response: 6F EE FF FF FF D7 3D 8E FE C8 04 28 00 6E 00 8F 08 6E 08 69 08 5E 08 71 08 65 08 7E 00 D4 01 B6 20 E2 00 0B 00 11 00 0D 00 00 00 6E 00 00 00 00 00 00 09 20 B2 34

There's the valid poll, followed by the Huge Mess of Whatever.

This was a brand new converter, right out of the box. If I have a chance, I'll look at it under the microscope and see if I can figure out why it's broken, just for my own edification.

So: Anyways, lesson learned here: If Mango is showing Bizarre Things with Serial, check your USB adapter first.. I had another one of these adapters fail sort of randomly, and that one is here on the floor next to me. I'll see if this one has the same random problem.

Speaking of this, does anyone have any other trustworthy but sub $50 USB to Serial 485 devices? There seems to be a certain lack of these things out there that don't suck, and I use a LOT of 485... I've used several brands, and these DTech ones are the least achy-breaky of the ones I've tried. If my projects had higher budgets, I'd be getting Name Branded Units , but I can't justify $300 converters on these devices...

Cheers,
-Greg Linder

@MattFox For other folks who come to this thread, I was able to use the Prologix GPIB Ethernet Adapter: https://prologix.biz/product/gpib-ethernet-controller/

I have a working set of Mango Configs that I've got working to talk SCPI using the GPIB <-> Ethernet device servers to make this work. I originally tried to get this working with the Serial data source, but ran into a host of issues related to this.

Mango's quite versatile, but apparently the Serial data source chokes on SCPI-style commands- One reason was noted by @MattFox earlier in the thread, but it's also part of a problem with SCPI: Mango expects all data sources to return data, and some (many) SCPI commands don't actually generate an acknowledgement.

The GPIB <-> Ethernet adapter has the same problem, but the timeouts are faster, and you can use a trick in the Prologix device (++auto 1) to basically stream data into mango in Real Time.

The Mango config I put together for this is quite large, but if you're looking for SCPI interfacing, I'd imagine this script would work well for Native Ethernet as well as Vintage GPIB devices using an Etherent <-> GPIB device.

The Prologix one I linked above is the more affordable of the choices- you can just telnet into it, and issue commands, and it spits stuff back. It's actually quite cool.

@nino-kurtalj

I calculate this using rollups- You can use the integral rollup type on watts over the time you want. Mango will give you this in watt seconds. Then you can divide that by 3,600 to get watt hours, then scale it as needed.

Better is to read the kwh raw value out of your metering device, though, if that works for you.

Here's an example:

// This runs every hour on the hour to calculate last hours' energy
var energy = 0;


// This sets up and returns 1 hour integral on power
var values = [];
var endDate = CONTEXT.getTimestamp();
var startDate = endDate - CONTEXT.millisInPast(HOURS, 1); // Last 1 HOUR
var pointToQuery = [ POWER.getDataPointWrapper().getId() ];
PointValueQuery.rollupQuery( pointToQuery, startDate, endDate, 
  function( value, index ) {
    values.push(value);
  }, com.serotonin.m2m2.Common.Rollups.INTEGRAL, 1, com.serotonin.m2m2.Common.TimePeriods.HOURS);
    

energy = values[0] / 3600000;


my = energy;

return my;

I'm not sure if this is the right way to do this, but it works for me.

@mihairosu Heya: Is there a reason you're not using the "no update" event style type built into the Mango event creation stuff? That will let you just let Mango do that- We use these alarms extensively to detect "stuck" data points:

@Turbo Well hot damn. The thing works.

So now I can socat data in as a TCP point.. That's cool.

For folks following along:
First, do this to forward the port:

socat -d -d -d -d tcp-listen:4141,reuseaddr,fork file:/dev/ttyUSB0,nonblock,cs8,b2400,cstopb=0,raw,echo=0

This is a 2400 baud device.
I set the timeout for 250, no delimeter, and then port 4141 (as above) and host of Localhost.

Then, in the data point, I created a general placeholder with a read command of "00" and a value regex of (.*), value index 0

This appears to update properly.

Now to write a parser- This sure would be easier with the original M-Bus device.

Thanks for your help, @MattFox .

There also appears to be a bug (still) in the serial data source, at least when running on Linux- This test install is a raspberry pi, but we'll be deploying this on our IMX8's as usual, I hpoe.

@tungthanh500 Budgets. The M-Bus to USB converter is $20, the M-Bus to modbus device is considerably more expensive. Our goal is also to keep the number of things "needing configuration" to an absolutely minimum:If we can do it in Mango, it saves us time and money (and spare parts inventory costs).

@MattFox Fair enough- @nino-kurtalj said "Let's assume I have values in the W.".. Which I understood to mean he was using "watts" and it looks like he was trying to integrate watts over seconds in the metapoint he posted, using the now-last values as a sort of way to integrate watts into kwh.

Also; @nino-kurtalj if you are integrating watts over time to get kwh, realize that you're accumulating errors as you go. It's much much better to use the raw kwh (or wh, or energy point) out of your metering device. This is because even integrated watts to kwh in mango, you're still limited by polling rate of your device. I run 1000's of points of energy and power in my system, and we can see the errors building up over time: Particularly when the power signal changes rapidly.

The internal power meter kwh points (the "kwh" register you read out of your electrical meter) integrates at "electrical speed", so they catch those wiggles in real time: They basically sum every update of some high-speed V*A calculation.

If you're reading watts and integrating that into wh youself, you can only get those values every time you ask the meter for it, so you'll miss the little wiggles that happen in between polls. If you're polling every second, this will probably be a fairly small error, but it grows over time, so if you go true up your integrated watt -> kwh point by comparing it to like the power company's billing meter, you'll always be off by a few percent. The slower the watt polling interval, the larger that energy error is at the end of the month.

I run into this (and deal with this) all the time in the business I'm in, since some people insist on taking (for example) 15-minute average kilowatt, summing them up, calling it kwh, and then complaining to me that my energy points are wrong- They aren't.

They're just integrating a low-frequency discrete time signal and comparing it with an integral at real time as the signal changes: They will not be the same. That error can cause lots of confusion, if you're (for example) trying to calculate your AC line losses over time- The error in going from 15-minute average power into kwh is generally larger than the actual losses in the cable you're trying to measure.

We do use "integrated watts" from Mango for certain applications, but it's in places where that error is easy to deal with (such as by hourly comparisons, where you don't have time for that error to accumulate over months before comparing).

@CraigWeb Yup. Thanks for getting back to me.

I actually have (6) Scripts running in a loop: I'm doing it this way because I'm working around the fact that I can't (seem) to get the raw serial data source to work in Mango V4.5 (no RX under any circumstances, near as I can tell). So, I'm using TCP <-> GPIB, then using the Raw TCP driver to talk to that, which works well. However, there's lots of random delays needed to make this work, since we're sending SCPI commands and polling big arrays back out of our equipment, which has a very non-standard and irregular lag. This means I need to use the Raw TCP source using the timeout (rather than the EOL marker, since not al SCPI commands return values with known terminations). This means I had to break my script into into (6) scripts instead, basically:

The first script sets up the sourcemeter for measuring something
Second one takes a measurement
Third one parses that result and calculates some more stuff
Fourth programs the sourcemeter for the next measurement and sets it to run
Fifth one reads the raw data out of the instrument and calculates a bunch of stuff
Sixth one stores those values into points.

The handling of one to the next is done via "update context" points (that are internal points in the first script and refernced in the other scripts) and lots of RuntimeManager.sleep() style things within the scripts to give the RAW tcp points time to get data back so it's available on the next step in the script.

@CraigWeb said in Scripting point running all on its own?:

Does the data source have its own internal points?

yes- Quite a lot
Are you updating those points in the script ?
Yes- We read value from the raw TCP source, do some calcs to it, and store those in the internal scripted points. But those point are not read again in that same script
and do they re-update the context?
No: This is why I busted my original script out into (six) steps, to avoid writing (and reading) points within the same script, since I noticed this created odd behaviors (using point.set(value) then using point.value() doesn't seem to work consistently within the same context- I ended up having to do this because of the timing issues with the raw TCP point depending on timeouts to get responses back).

Thus, I use point.set(value) in one script. Then I use that value in the next script which is set to execute "context update" on a boolean point that exists only to pass control from one script to the next script.

Originally, I had this written in a single script, but that failed because I wasn't able to wait on the TCP/IP readbacks from the raw TCP point using our instrument. So I broke it out into blocks, to make that work, and then ended up passing control from one script to the next using "supervisor" points (internal points to the "supervisory" script) that just exist to trigger a context update on the next script.

The behavior you described that causes the script to execute continuously; Is this by design? If so, can you explain how that works?

Got another one for you: This is Mango V4.5:

It looks some of our scripted data points are running despite not having anything set to make them run. There's a a lot of backstory on this- Originally, I wanted something to run every 10 minutes- Thus, I set up a scripted data source to run every 10 minutes.

This data source does some things, and then sets another point (which is used by the next "step" in the process) to do more calculating; That data point is set in that script to "update context" in the next script.

Correct my understanding here, but, for the sake of this post:
"Updates context" in the world of mango scripting / metapoint means "If this value updates, run this thing. If this value doesn't update, no need to run this thing", according to the setting of the "context update event" box in the data source (update, logged, or change). If you don't have any points selected to "update context" in the points in the script, then that drop down doesn't have any effect; Is that right?

"polling" in the world of Mango scripting means "Do this thing based time settings, either every XX minutes or as a cron pattern"

If you do polling and have "updates context" points selected, then it should do both (run on the Polling time and run when the point updates within the script).

What I have are points that run with no context and no polling enabled. My original script should run every ten minutes- meaning, every 10 minutes, the script should run, do some stuff, and use the set directive to make the next "step" work in this set of chained calculations. That original scripted point should then wait 10 minutes before Doing Thing again, and restarting the process.

Our little thing is working, but it's running continuously, and doesn't seem to care about the settings of the "polling" box (currently set to off, but the script is still running) or the "updates context" checkboxes in the data points in the script (all are "unchecked").

I've noticed this quite a lot in behaviors in my scripting: I generally thought that the script would run under the conditions above (timers, based on CRON or the time setting) and / or the "updates context" next to the points in the script, as informed by the "context update event".

It appears that the scripts are running effectively continuously: This means that in practice (regardless of the settings I have for "polling" or "context updates") I need to bracket my code with "If (somedatapoint.value == true {stuff}" to keep things from updating on their own.

So what am I missing here? I want this thing to run every 10 minutes, let the cycle complete, then stop and wait for the cron (or timer setting) to restart the task 10 minutes later. Instead, it's just looping through endlessly, as if I had "updates context" selected for all the datpoints in the scripted data source.

I've attached a screen grab of one of the points in question- What setting here would indicate to you that this should run effectively constantly?

The script in that scripted point is running, and appears to be updating as if any point updates, despite none of them being checked to "update context". Have I been using "update context" wrong the whole time so far?

If so, how do I get a Mango scripted data point to run, then just sit there and do nothing until the next 10 minute interval elapses. I thought that's what "polling enabled" with either "update period" or "cron pattern" would do?

@MattFox It's not delta. It's integral. Delta between two powers doesn't given you kwh, unless mango use some different understanding of "delta".

If you do delta on power over 1 hour (for example), and you have 1000W at time = 0 and 1000 W at time = 1 hour, your delta is 0 (since 1000-1000 = 0), but your integral should be 1000w * 1hr = 1kwh.

If you do delta on power that's what you get: A delta on power (power at start of time minus power and end of time). You can use that to estimate energy, if the time period is short enough, but energy is the integral of power over time, so you need to use integral (if you're going from watts -> kwh).

@CraigWeb I appreciate that; We're going to be shipping a mountain of these things internationally use M-Bus (I hope), so for now I'm using my TCP silly workaround for this (since the Serial data source still appears to not work properly for whatever reason on our Arm-based machines).

Some other folks have piped up here as well about M-Bus- This exists basically nowhere in the US (as I've found now that I'm testing things using M-Bus) but apparently is quite common in IOT purposes in Europe- Every meter in Norway (for example) has an M-Bus port for interfacing HAN devices directly to your buildings' meter.

Surely someone must have had a reason that this got added in the first place; Although I do understand not updating things if nobody is using it. Since we're going to be using this A Lot (I think), we may be able to chat NRE to get it put back- I'm still chatting with Michael about Mango for our 2023 project pipeline, and this could be a good business discussion point to have.

@CraigWeb Sure:

This is running the latest Mango V4.5. I'm using the Raw TCP data source to chat with a TCP <-> GPIB Adapter, but this bug (also) appears in the serial data source as well. It mentions it in the documentaiton somehwere, too, that you should be able to put \r\n in the "delimiter" and have it sense that as the end of message.

This does not work.

To make it work, you have to download the JSON file and manually put in "\r\n" and then re-import it, at which point the datasource "delimiter" blank shows up blank.

Here's the screen grab of our (working) data source:

Notice: Nothing in the "Delimiter" field.

If I download the JSON for that data source, however, I see this:

      "updatePeriodType":"SECONDS",
      "delimiter":"\r\n",
      "hex":false,

which shows the proper delimiter. This works (btw) so mango is processing the delimiter properly internally, it just doesn't display right in the UI.

"acceptance criteria" would be "The actual delimiter value should be visible in the Delimiter blank, somehow, even of it consists of nonprintable characters"

In the datasource help, attached below as a screen grab, with my highlighting shown, it says " Delimiters are unescaped before sending, so "\r" translates to carrriage return.":

If you put "\n" in that blank, you get this in the JSON:

      "updatePeriodType":"MINUTES",
      "delimiter":"\"\\n\"",
      "hex":false,

If you put \n (no quotes) you get this:

      "updatePeriodType":"MINUTES",
      "delimiter":"\\n",
      "hex":false,

Which seems to mean that the parser expects \\n (not a \n)

The only way I was able to get the TCP data source to store the data using \r\n was to edit the JSON file as above, which seems to work properly. But then the \r\n doesn't show up properly in the "delimiter" blank

@nino-kurtalj

I calculate this using rollups- You can use the integral rollup type on watts over the time you want. Mango will give you this in watt seconds. Then you can divide that by 3,600 to get watt hours, then scale it as needed.

Better is to read the kwh raw value out of your metering device, though, if that works for you.

Here's an example:

// This runs every hour on the hour to calculate last hours' energy
var energy = 0;


// This sets up and returns 1 hour integral on power
var values = [];
var endDate = CONTEXT.getTimestamp();
var startDate = endDate - CONTEXT.millisInPast(HOURS, 1); // Last 1 HOUR
var pointToQuery = [ POWER.getDataPointWrapper().getId() ];
PointValueQuery.rollupQuery( pointToQuery, startDate, endDate, 
  function( value, index ) {
    values.push(value);
  }, com.serotonin.m2m2.Common.Rollups.INTEGRAL, 1, com.serotonin.m2m2.Common.TimePeriods.HOURS);
    

energy = values[0] / 3600000;


my = energy;

return my;

I'm not sure if this is the right way to do this, but it works for me.

@cwangv That's true- I'm happy to send you my code, if you want, when I get it working- It's just a scripted point that searches raw strings for the OBIS code and saves it in datapoints. This for energy meters, and so far, it works OK.

@CraigWeb any plans on bringing the M-Bus module in Mango V4? It's interesting, since it's still stated as "supported" on Mango's website: https://radixiot.com/supported-protocols

You only find out it's missing from V4 when you look at the documentation for Upgrade from V03 -> V04: https://docs-v4.radixiot.com/upgrade-to-v4 and scroll down to "unsupported modules"

@tungthanh500 Budgets. The M-Bus to USB converter is $20, the M-Bus to modbus device is considerably more expensive. Our goal is also to keep the number of things "needing configuration" to an absolutely minimum:If we can do it in Mango, it saves us time and money (and spare parts inventory costs).

@MattFox So: Turns out the socat solution works, but it's not really as straightforward as using the Serial device natively:
Can you poke around and see if the Serial data source needs some hugs? Using a TCP forward for a USB converter serial port seems a bit silly to me, when Mango has a raw Serial device interface that (should) do the same thing.

I've now had no luck with the serial data source on my ARM based IMX8 machine over hardware serial or USB to 485 serial under V3 or V4, and now am seeing the same thing under V4 on a raspberry pi.

I'm happy to set up a screen share / demo video / something if you want to see what I'm seeing with this.

Thanks for your help, in any event, as we now have a (mostly) working MBus data source again, although in a somewhat ridiculously rube-goldbergesque way.