Greetings:

I gave up on using the built in Units in Mango. They seem to be.. Sort of broken and generally annoying, since there never seems to be the right M/k/u/m type for whatever I'm doing-- Megawatts, kilowatts, whatever. The Best Practice seems to be to just use the Text Renderer Properties, and set the Suffix to Whatever unit you want.

I tried using the built-in units a number of ways, but I got tired of dealing with the little "dot" for multiplied units which really aren't. I don't remember which unit, but it was something like kwh showing as kw<dot>h.. It's not kilowatt TIMES hours.. Anyways. I've run into trouble with import/export also using built-in units, where m^2 doesn't import right under some conditions (for example). Don't have these troubles using the "text renderer properties" Suffix box

Yeah. The general practice seems to be "use the text renderer properties"... Makes your life easier.

Thanks for the quick updates: @CraigWeb @Jared-Wiltshire

This fixed the Modbus tools, and I just tested it, so we're good to go. Thanks for that. These tools are Super Useful, as even thought Modbus has been around for longer than I've been alive, people who implement it still can't agree if it's 0-indexed or 1-indexed, or what the actual difference is between holding and input registers are.

Also, I verified the flashing save Icon issue with multiple tabs... I think I noticed this before, but in my exhausted stupor the previous night I was sort of not thinking straight.

Thanks!

Greetings, all:

I've been using USB to Serial converters on my little Linux Based Mango devices for my business. I've had varying luck by brand, and I figured this is as good a place as any to talk about by mest brand so far, and some troubles they've had.

The ones I use are the DTech USB 2.0 to RS 422/ RS 485 adapters. These units are based on the FTDI Chipset, and I generally have Good Luck. I've probably installed clost to 50 of these things, with so far 3 failures- One due to a direct lightning hit, and the other two Very Odd, Indeed.

It's this "very odd indeed" one that I want to comment on, as a sort of Use Case for how Super Useful the "Log IO" function can be in Mango. I was doing final testing on a product that was too ship, and found that the Modbus Serial data source was behaving all weirdly-- Sometimes polling correctly, but generally Failing Wholesale.

I was just about geared about to post a mango thread about wondering if the latest Modbus module had some bugs in it, but lo and behold, the "Log IO" showed me something was weird.

So:
The poll should look like this (Using the CAS Modbus Scanning Tool, if you ever need a Modbus Doodad, that's a great little test tool)
[22:21:24] => Poll: C8 04 00 00 00 14 E1 9C
[22:21:25] <= Response: C8 04 28 00 6E 00 9A 08 8D 08 84 08 81 08 8F 08 87 08 A1 00 D4 01 AF 20 E2 00 0E 00 21 00 20 00 00 00 6C 00 00 00 00 00 00 09 28 AD 11

Mango was seeing this:
2019/08/29-22:12:21,943 O c80400000014e19c
2019/08/29-22:12:21,955 I 6feeffffffd73d8efe
2019/08/29-22:12:22,023 I c80428006e00a708da08d808da
2019/08/29-22:12:22,025 I 08e708da
2019/08/29-22:12:22,026 I 08f5
2019/08/29-22:12:22,027 I 00
2019/08/29-22:12:22,027 I d401
2019/08/29-22:12:22,028 I c320
2019/08/29-22:12:22,049 I e1000a002a00250000007000000000000008f8eb71

I was thinking-- OKay, maybe there's a bug in the serial driver? You can see Mango originating the proper request "c80400000014e19c " which is a read input registers starting at 0 for 20.

Then, Mess starts coming back into the device almost immediatly after. You can see the start of my Slave device talking back, "c80428006e00a708da08d808da" .. But before that is this "6feeffffffd73d8efe"

Turns out that Mess was coming from the RS485 chip-- It looks almost as if the Serial device server isn't actually swapping RX/TX direction (It's a single duplex, 19200 link for this connection), and is scrambling itself up. So I plugged in that USB adapter into my Windows Box, and CAS Modbus Scanner says the same mess:
[22:19:12] => Poll: C8 04 00 00 00 14 E1 9C
[22:19:12] <= Response: 6F EE FF FF FF D7 3D 8E FE C8 04 28 00 6E 00 8F 08 6E 08 69 08 5E 08 71 08 65 08 7E 00 D4 01 B6 20 E2 00 0B 00 11 00 0D 00 00 00 6E 00 00 00 00 00 00 09 20 B2 34

There's the valid poll, followed by the Huge Mess of Whatever.

This was a brand new converter, right out of the box. If I have a chance, I'll look at it under the microscope and see if I can figure out why it's broken, just for my own edification.

So: Anyways, lesson learned here: If Mango is showing Bizarre Things with Serial, check your USB adapter first.. I had another one of these adapters fail sort of randomly, and that one is here on the floor next to me. I'll see if this one has the same random problem.

Speaking of this, does anyone have any other trustworthy but sub $50 USB to Serial 485 devices? There seems to be a certain lack of these things out there that don't suck, and I use a LOT of 485... I've used several brands, and these DTech ones are the least achy-breaky of the ones I've tried. If my projects had higher budgets, I'd be getting Name Branded Units , but I can't justify $300 converters on these devices...

Cheers,
-Greg Linder

@MattFox For other folks who come to this thread, I was able to use the Prologix GPIB Ethernet Adapter: https://prologix.biz/product/gpib-ethernet-controller/

I have a working set of Mango Configs that I've got working to talk SCPI using the GPIB <-> Ethernet device servers to make this work. I originally tried to get this working with the Serial data source, but ran into a host of issues related to this.

Mango's quite versatile, but apparently the Serial data source chokes on SCPI-style commands- One reason was noted by @MattFox earlier in the thread, but it's also part of a problem with SCPI: Mango expects all data sources to return data, and some (many) SCPI commands don't actually generate an acknowledgement.

The GPIB <-> Ethernet adapter has the same problem, but the timeouts are faster, and you can use a trick in the Prologix device (++auto 1) to basically stream data into mango in Real Time.

The Mango config I put together for this is quite large, but if you're looking for SCPI interfacing, I'd imagine this script would work well for Native Ethernet as well as Vintage GPIB devices using an Etherent <-> GPIB device.

The Prologix one I linked above is the more affordable of the choices- you can just telnet into it, and issue commands, and it spits stuff back. It's actually quite cool.

@nino-kurtalj

I calculate this using rollups- You can use the integral rollup type on watts over the time you want. Mango will give you this in watt seconds. Then you can divide that by 3,600 to get watt hours, then scale it as needed.

Better is to read the kwh raw value out of your metering device, though, if that works for you.

Here's an example:

// This runs every hour on the hour to calculate last hours' energy
var energy = 0;


// This sets up and returns 1 hour integral on power
var values = [];
var endDate = CONTEXT.getTimestamp();
var startDate = endDate - CONTEXT.millisInPast(HOURS, 1); // Last 1 HOUR
var pointToQuery = [ POWER.getDataPointWrapper().getId() ];
PointValueQuery.rollupQuery( pointToQuery, startDate, endDate, 
  function( value, index ) {
    values.push(value);
  }, com.serotonin.m2m2.Common.Rollups.INTEGRAL, 1, com.serotonin.m2m2.Common.TimePeriods.HOURS);
    

energy = values[0] / 3600000;


my = energy;

return my;

I'm not sure if this is the right way to do this, but it works for me.

@mihairosu Heya: Is there a reason you're not using the "no update" event style type built into the Mango event creation stuff? That will let you just let Mango do that- We use these alarms extensively to detect "stuck" data points:

@Turbo Well hot damn. The thing works.

So now I can socat data in as a TCP point.. That's cool.

For folks following along:
First, do this to forward the port:

socat -d -d -d -d tcp-listen:4141,reuseaddr,fork file:/dev/ttyUSB0,nonblock,cs8,b2400,cstopb=0,raw,echo=0

This is a 2400 baud device.
I set the timeout for 250, no delimeter, and then port 4141 (as above) and host of Localhost.

Then, in the data point, I created a general placeholder with a read command of "00" and a value regex of (.*), value index 0

This appears to update properly.

Now to write a parser- This sure would be easier with the original M-Bus device.

Thanks for your help, @MattFox .

There also appears to be a bug (still) in the serial data source, at least when running on Linux- This test install is a raspberry pi, but we'll be deploying this on our IMX8's as usual, I hpoe.

@tungthanh500 Budgets. The M-Bus to USB converter is $20, the M-Bus to modbus device is considerably more expensive. Our goal is also to keep the number of things "needing configuration" to an absolutely minimum:If we can do it in Mango, it saves us time and money (and spare parts inventory costs).

@Turbo

Okay: I've figured something out here-
It seems (to me) like the files in /opt/mango/bin (that you use to set up certbot/ etc) don't seem to be aware of the /opt/mango-data/ files, and don't appear to be updated based on the settings in the

This just cost me a huge number of hours:

The instructions for getting the certbot set up (here)
https://docs-v4.radixiot.com/ssl

talks about running "/opt/mango/bin/genkey.sh"

but Genkey doesn't seem to know where the (new) mango.properties default file is (in /opt/mango-data/ by default install from the install script).

This seems to be present in all the SSL config scripts in /opt/mango/bin- They don't seem to know what is in /opt/mango-data/mango.service (which calls out Environment="mango_config=/opt/mango-data/mango.properties")

This means if you do anything manually (like installing CertBot), it seems that the shell scripts all seem to default to /opt/mango , and put the key files there instead of in /opt/mango-data/. When you run a manual key creation ("/opt/mango/bin/genkey.sh") it doesn't appear to know what you put in for the password / etc in mango.properties.

To get us back online, I was able to get the scripts to work by just doing:
export MA_ENV_PROPERTIES=/opt/mango-data/mango.properties
then running the genkey.sh, and the files were in the right place.

Without exporting the variable, the genkey.sh (and the certbot deploy) uses Something Else (not /opt/mango-data/mango.properties, which the Mango V4.5 install script creates).

I think the scripts may need some tweaking to work with the new Mango file pathnames.

I (still) don't have my certbot working, since when it runs, it puts the keystore file in /opt/mango, and not in /opt/mango-data/, which is where the service start/stop script wants to find it. I used up all my certs (you have a limit of 5 per few day time period to renew), so I'll try this again once I get this done.

I'm running this on Ubuntu 22.04 LTS, using BASH as my shell, and I'm running these scripts as superuser.

This is relevant, as I think the certbot scripting inside Mango uses the "renew" functions in /etc/letsencrypt-

and each site has a conf file that includes the renew_hook, so it's going to expect to be able to run that certbot-deploysh script. How can I make sure make the certbot stuff works now that that config file is moved to /opt/mango-data?

Or has this been addressed already?
This may be as simple as hook="EXPORT MANGO_CONFIG=/opt/mango-data/mango.properties /opt/mango/bin/certbot-deploy.sh

.. But I don't know. The Mango V4.5 documentation for setting up CERTBOT seems to have very little to do with how this actually needs to be done.

Cheers,
-Greg

@CraigWeb Heya Craig. Thanks for that! That's exactly what I was thinking. It's absolutely great now that Mango runs under its own user, but that user can't be logged into (at least, at least not without a fair amount of headache). Once CertBot is deployed, Mango will (likely) have no trouble updating them, but on the first run, almost everyone using those scripts will be logging in or SUDOing into root or whatever to make that happen.

The easiest solution (I think) is to juts do something like:
export mango_paths_data=/opt/mango-data
export mango_paths_home=/opt/mango

Or something when you sudo in to do these scripts. or you have to enable the mango user account to be loggable-innable, but that's the whole point (I think) of creating a mango-user in the first place, to not have it be a login account.. Which is sound design practice.

I'm probably going to try that over the weekend again- Thanks so much for looking through those scripts. When I first was trying this, I thought I was out of my mind, but it turns out there's both an easy solution (at least temporarily) to make this work, and no problem when the certbot hook runs (since it's already running as the mango user).

This problem only popped up because the new mango install script does everything under a user it creates, and the interactive config stuff happens because users are logging in and modifying another users's (mangos) settings. It all makes pretty solid sense, but only once you sit down and think through it.

Who knew environment variables would continue to be such a headache?

@CraigWeb That's correct. My confusion comes from the fact that this is a bog standard new install on raw fresh new hardware: I followed the steps in the install guide ("https://docs-v4.radixiot.com/install-script"), using the script provided, and am sort of surprised that the install script (which clearly needs to know these paths to install) didn't check or define those variables, which seems like maybe something it should do.

Also, in (https://docs-v4.radixiot.com/installation-overview), step 4, it says "Set the mango_paths_home environment variable to the where you installed Mango (recommended but not required)."- This does not mention the other variable, "mango_paths_data", and seems to be an outdated reference to the new requirement for both paths to be defined.

However, it also sounds like we have an Easy Solution:
Just set the environment variable for the path for mango_paths_home (which I assume defaults to /opt/mango) and mango_paths_data (which I assume defaults to /opt/mango-data).

So, I guess:
(1) Why does the script not set those variables, if they are sort of key to everything else, when doing a Fresh install of Mango according to the documentation?
(2) Where does it say in the manual that I need to set those environment variables manually?
(3) What are the "standard" values for default mango installs for those paths above?

I appreciate your help, and all I have to do is set some environment variables as mentioned above- But the fact those variables weren't set by the factory install script, and the install guide doesn't mentioned them, either, seems like something that maybe can be fixed- The manual talks all about the JAVA_HOME environment variable, and setting that, which makes a ton of sense, but no mentioned of the (2) env variables that need to be set for the scripting to work.

The install script, btw, is very nice- It sets all kinds of other things, but why not the mango paths for new install?

Cheers,
-Greg

@MattFox I agree- If that's the standard way of doing this, there should be a nice writeup on the Manual / Docs for setting that up- I realize that's an Option, but I'm trying to keep our systems installed "per the documentation", to keep things consistent going forward- I tried the nginx approach on 3.7, and found that it didn't work well due to issues with headers and such. I've seen your posts here on this topic, but is there a nice writeup somewhere that goes step by step to get this going?

@Turbo

Okay: I've figured something out here-
It seems (to me) like the files in /opt/mango/bin (that you use to set up certbot/ etc) don't seem to be aware of the /opt/mango-data/ files, and don't appear to be updated based on the settings in the

This just cost me a huge number of hours:

The instructions for getting the certbot set up (here)
https://docs-v4.radixiot.com/ssl

talks about running "/opt/mango/bin/genkey.sh"

but Genkey doesn't seem to know where the (new) mango.properties default file is (in /opt/mango-data/ by default install from the install script).

This seems to be present in all the SSL config scripts in /opt/mango/bin- They don't seem to know what is in /opt/mango-data/mango.service (which calls out Environment="mango_config=/opt/mango-data/mango.properties")

This means if you do anything manually (like installing CertBot), it seems that the shell scripts all seem to default to /opt/mango , and put the key files there instead of in /opt/mango-data/. When you run a manual key creation ("/opt/mango/bin/genkey.sh") it doesn't appear to know what you put in for the password / etc in mango.properties.

To get us back online, I was able to get the scripts to work by just doing:
export MA_ENV_PROPERTIES=/opt/mango-data/mango.properties
then running the genkey.sh, and the files were in the right place.

Without exporting the variable, the genkey.sh (and the certbot deploy) uses Something Else (not /opt/mango-data/mango.properties, which the Mango V4.5 install script creates).

I think the scripts may need some tweaking to work with the new Mango file pathnames.

I (still) don't have my certbot working, since when it runs, it puts the keystore file in /opt/mango, and not in /opt/mango-data/, which is where the service start/stop script wants to find it. I used up all my certs (you have a limit of 5 per few day time period to renew), so I'll try this again once I get this done.

I'm running this on Ubuntu 22.04 LTS, using BASH as my shell, and I'm running these scripts as superuser.

This is relevant, as I think the certbot scripting inside Mango uses the "renew" functions in /etc/letsencrypt-

and each site has a conf file that includes the renew_hook, so it's going to expect to be able to run that certbot-deploysh script. How can I make sure make the certbot stuff works now that that config file is moved to /opt/mango-data?

Or has this been addressed already?
This may be as simple as hook="EXPORT MANGO_CONFIG=/opt/mango-data/mango.properties /opt/mango/bin/certbot-deploy.sh

.. But I don't know. The Mango V4.5 documentation for setting up CERTBOT seems to have very little to do with how this actually needs to be done.

Cheers,
-Greg

Greetings, all:

I'm trying to get my SSL certs setup according to the Mango V4 Help:
https://docs-v4.radixiot.com/ssl

And, well, that document doesn't seem quite right to me- It looks like it hasn't been updated entirely from Mango 3.7 (or an earlier version of Mango 4).

Notably:
"nano /opt/mango/mango.properties" is not where that files is anymore (it's in /opt/mango-data/mango.properties)

ssl.keystore.location=/opt/mango-data/keystore.jks is no longer jks, it's now p12

/opt/mango/bin/mango-start.sh > /dev/null 2>&1 no longer works if you use 4.5 installed as a service- You need to now use "mango service stop/start"

There's many more- Can someone update this documentation? I'm having a mess of a time getting this to come up right- It feels like Mango's CERTBOT deploy script is still updating the .p12 file in /opt/mango, and then mango is starting looking for the .p12 in /opt/mango-data/, so the cert-bot updated cert never gets into the new java keystore.

Presumably, someone else already has this figured out- Can someone point me to what I'm missing?

@MattFox The wires we are sniffing carry modbus/RTU- The format within mango doesn't really matter. I was considering knocking another thing up using that SOCAT -> TCP device to just get an idea of what the points are being moved back and forth, then develop that into a Modbus read store/forward type of affair.

We basically have a need to eavesdrop on other systems for whom the documentation is a bit minimal- It's a way to "get around" some legacy systems that have controllers that we can't just swap out for Mango, but still need to gather the data for display.

So long as we can get the data into Mango, that'd be swell.

I like @terrypacker 's idea of bypassing Modbus4J's waiting room.. I think I can get something working using the serial data source. I'm doing this for the first time next week, so we'll see how complex this is. I don't even have a full understanding (yet) of exactly how dense / fast / complicated the Modbus/RTU data implementation here actually is- It could only be a handful of points. I was just going to sniff first, using SOCAT to listen on the wire, and try and work out a solution based on what I see.

I can't think I'm the only one trying to do Modbus Sniffing-type stuff with legacy systems, but then again, I see a lot of stuff that I do seems to be"I've never seen this type of thing before"..

Cheers,
-Greg

Greetings, Smart People at the Forum:

I wonder if anyone has (yet) deployed Mango as a Modbus Sniffer. We have some applications for this, and I may end doing this via Serial data source / SOCAT to TCP/IP, but I'd rather use Mango natively for this.

We have several applications where we have to "sniff" existing modbus traffic to feed into other systems. In the past, I've used Modbus repeater devices and other types of doodads to make this happen, but it seems like something that would be pretty easy to implement in Mango.

Ideally, this would be "just plug in Mango to a Modbus network, and it would create points and fill them with values" (not names, scaling, or whatever else) by listening on the RS-485 line.

This would be an amazing feature for the industry I am working in, and would simplify our life a lot as compared to the way I've done this in the past (Configure Mango Modbus data source, get data in, then publish data back out again as a Modbus publisher).

The difference is in the configuration time- If I can sniff the line, I can populate all the data automatically, and then just assign tag names / point names since Mango would already know the Modbus register indices that we would need, and the raw values they contain.

Has anyone ever done this / asked about this before?

@MattFox Yeah- I tried that, too: I just can't get the the thing to receive anything over the serial port.

It would be really nice to have a thread / template file available somewhere that says "Here's how you just get Raw Serial into Mango" to overcome the initial confusion over REGEX, which is absolutely common in these threads.

People do what I did: They turn it on, try and get a "wildcard" RegEx to make sure mango is receiving something then go develop the interface from there.

I have lots of serial customization coming in for our projects, and although the SOCAT <-> Serial TCP thing works okay, it would all be much simpler if I knew that the Mango Serial Device had a good easy starting point.

Greetings, all:

I've posted here before related to the Serial Data Source, and I'd love to understand what I'm doing wrong / how to make this work.

First, questions:
(1) Can the Mango Serial Data Source just act as a passive input? Just storing everything that comes in over the serial port into a point?

(2) If not, how do I force it to "trigger" a read?

Here's my Application:
I have a device that spits out hex code as 2400 baud every few seconds. I want to ingest this into mango, and parse it.

I have this (kinda) working using SOCAT and the raw TCP data source, but that seems like an awful lot of messing about.

The values are hex, fixed length, and have known terminating characters at the start (and end) of the message.

I've been re-reading back through lots of stuff on this forum before about this, and haven't gotten closer to having it work.

More details:
This is a raw serial device attached via a USB to Serial adapter. I have verified my Mango machine is getting the Right Data via using Minicom, and got this almost entirely working with SOCAT and a Raw TCP data source.

I'm just trying to keep this whole thing inside Mango. I keep trying to use the Virtual Serial Port for various things, but so far, I hvaen't been able to get it to work at all- I've tried this on V3.7, and now am running our system here on 4.5, still with no luck.

Are there any demo files / docs available for how to "just get raw data into Mango over a Serial port" so I can then parse it using a scripted data source?

I'd love some guidance on this. The TCP/IP data source seems to allow this functionality, albeit with a bit of goofing around (see my post about interfacing with my HP-IB gear for this thread), but I'd love to be able to know how to use the Raw Serial Data Source, as we have many uses for that now.

Cheers,
-Greg

@MattFox Fair enough- @nino-kurtalj said "Let's assume I have values in the W.".. Which I understood to mean he was using "watts" and it looks like he was trying to integrate watts over seconds in the metapoint he posted, using the now-last values as a sort of way to integrate watts into kwh.

Also; @nino-kurtalj if you are integrating watts over time to get kwh, realize that you're accumulating errors as you go. It's much much better to use the raw kwh (or wh, or energy point) out of your metering device. This is because even integrated watts to kwh in mango, you're still limited by polling rate of your device. I run 1000's of points of energy and power in my system, and we can see the errors building up over time: Particularly when the power signal changes rapidly.

The internal power meter kwh points (the "kwh" register you read out of your electrical meter) integrates at "electrical speed", so they catch those wiggles in real time: They basically sum every update of some high-speed V*A calculation.

If you're reading watts and integrating that into wh youself, you can only get those values every time you ask the meter for it, so you'll miss the little wiggles that happen in between polls. If you're polling every second, this will probably be a fairly small error, but it grows over time, so if you go true up your integrated watt -> kwh point by comparing it to like the power company's billing meter, you'll always be off by a few percent. The slower the watt polling interval, the larger that energy error is at the end of the month.

I run into this (and deal with this) all the time in the business I'm in, since some people insist on taking (for example) 15-minute average kilowatt, summing them up, calling it kwh, and then complaining to me that my energy points are wrong- They aren't.

They're just integrating a low-frequency discrete time signal and comparing it with an integral at real time as the signal changes: They will not be the same. That error can cause lots of confusion, if you're (for example) trying to calculate your AC line losses over time- The error in going from 15-minute average power into kwh is generally larger than the actual losses in the cable you're trying to measure.

We do use "integrated watts" from Mango for certain applications, but it's in places where that error is easy to deal with (such as by hourly comparisons, where you don't have time for that error to accumulate over months before comparing).