@CraigWeb Heya Craig. Thanks for that! That's exactly what I was thinking. It's absolutely great now that Mango runs under its own user, but that user can't be logged into (at least, at least not without a fair amount of headache). Once CertBot is deployed, Mango will (likely) have no trouble updating them, but on the first run, almost everyone using those scripts will be logging in or SUDOing into root or whatever to make that happen.

The easiest solution (I think) is to juts do something like:
export mango_paths_data=/opt/mango-data
export mango_paths_home=/opt/mango

Or something when you sudo in to do these scripts. or you have to enable the mango user account to be loggable-innable, but that's the whole point (I think) of creating a mango-user in the first place, to not have it be a login account.. Which is sound design practice.

I'm probably going to try that over the weekend again- Thanks so much for looking through those scripts. When I first was trying this, I thought I was out of my mind, but it turns out there's both an easy solution (at least temporarily) to make this work, and no problem when the certbot hook runs (since it's already running as the mango user).

This problem only popped up because the new mango install script does everything under a user it creates, and the interactive config stuff happens because users are logging in and modifying another users's (mangos) settings. It all makes pretty solid sense, but only once you sit down and think through it.

Who knew environment variables would continue to be such a headache?

@CraigWeb That's correct. My confusion comes from the fact that this is a bog standard new install on raw fresh new hardware: I followed the steps in the install guide ("https://docs-v4.radixiot.com/install-script"), using the script provided, and am sort of surprised that the install script (which clearly needs to know these paths to install) didn't check or define those variables, which seems like maybe something it should do.

Also, in (https://docs-v4.radixiot.com/installation-overview), step 4, it says "Set the mango_paths_home environment variable to the where you installed Mango (recommended but not required)."- This does not mention the other variable, "mango_paths_data", and seems to be an outdated reference to the new requirement for both paths to be defined.

However, it also sounds like we have an Easy Solution:
Just set the environment variable for the path for mango_paths_home (which I assume defaults to /opt/mango) and mango_paths_data (which I assume defaults to /opt/mango-data).

So, I guess:
(1) Why does the script not set those variables, if they are sort of key to everything else, when doing a Fresh install of Mango according to the documentation?
(2) Where does it say in the manual that I need to set those environment variables manually?
(3) What are the "standard" values for default mango installs for those paths above?

I appreciate your help, and all I have to do is set some environment variables as mentioned above- But the fact those variables weren't set by the factory install script, and the install guide doesn't mentioned them, either, seems like something that maybe can be fixed- The manual talks all about the JAVA_HOME environment variable, and setting that, which makes a ton of sense, but no mentioned of the (2) env variables that need to be set for the scripting to work.

The install script, btw, is very nice- It sets all kinds of other things, but why not the mango paths for new install?

Cheers,
-Greg

@MattFox I agree- If that's the standard way of doing this, there should be a nice writeup on the Manual / Docs for setting that up- I realize that's an Option, but I'm trying to keep our systems installed "per the documentation", to keep things consistent going forward- I tried the nginx approach on 3.7, and found that it didn't work well due to issues with headers and such. I've seen your posts here on this topic, but is there a nice writeup somewhere that goes step by step to get this going?

@Turbo

Okay: I've figured something out here-
It seems (to me) like the files in /opt/mango/bin (that you use to set up certbot/ etc) don't seem to be aware of the /opt/mango-data/ files, and don't appear to be updated based on the settings in the

This just cost me a huge number of hours:

The instructions for getting the certbot set up (here)
https://docs-v4.radixiot.com/ssl

talks about running "/opt/mango/bin/genkey.sh"

but Genkey doesn't seem to know where the (new) mango.properties default file is (in /opt/mango-data/ by default install from the install script).

This seems to be present in all the SSL config scripts in /opt/mango/bin- They don't seem to know what is in /opt/mango-data/mango.service (which calls out Environment="mango_config=/opt/mango-data/mango.properties")

This means if you do anything manually (like installing CertBot), it seems that the shell scripts all seem to default to /opt/mango , and put the key files there instead of in /opt/mango-data/. When you run a manual key creation ("/opt/mango/bin/genkey.sh") it doesn't appear to know what you put in for the password / etc in mango.properties.

To get us back online, I was able to get the scripts to work by just doing:
export MA_ENV_PROPERTIES=/opt/mango-data/mango.properties
then running the genkey.sh, and the files were in the right place.

Without exporting the variable, the genkey.sh (and the certbot deploy) uses Something Else (not /opt/mango-data/mango.properties, which the Mango V4.5 install script creates).

I think the scripts may need some tweaking to work with the new Mango file pathnames.

I (still) don't have my certbot working, since when it runs, it puts the keystore file in /opt/mango, and not in /opt/mango-data/, which is where the service start/stop script wants to find it. I used up all my certs (you have a limit of 5 per few day time period to renew), so I'll try this again once I get this done.

I'm running this on Ubuntu 22.04 LTS, using BASH as my shell, and I'm running these scripts as superuser.

This is relevant, as I think the certbot scripting inside Mango uses the "renew" functions in /etc/letsencrypt-

and each site has a conf file that includes the renew_hook, so it's going to expect to be able to run that certbot-deploysh script. How can I make sure make the certbot stuff works now that that config file is moved to /opt/mango-data?

Or has this been addressed already?
This may be as simple as hook="EXPORT MANGO_CONFIG=/opt/mango-data/mango.properties /opt/mango/bin/certbot-deploy.sh

.. But I don't know. The Mango V4.5 documentation for setting up CERTBOT seems to have very little to do with how this actually needs to be done.

Cheers,
-Greg

Greetings, all:

I'm trying to get my SSL certs setup according to the Mango V4 Help:
https://docs-v4.radixiot.com/ssl

And, well, that document doesn't seem quite right to me- It looks like it hasn't been updated entirely from Mango 3.7 (or an earlier version of Mango 4).

Notably:
"nano /opt/mango/mango.properties" is not where that files is anymore (it's in /opt/mango-data/mango.properties)

ssl.keystore.location=/opt/mango-data/keystore.jks is no longer jks, it's now p12

/opt/mango/bin/mango-start.sh > /dev/null 2>&1 no longer works if you use 4.5 installed as a service- You need to now use "mango service stop/start"

There's many more- Can someone update this documentation? I'm having a mess of a time getting this to come up right- It feels like Mango's CERTBOT deploy script is still updating the .p12 file in /opt/mango, and then mango is starting looking for the .p12 in /opt/mango-data/, so the cert-bot updated cert never gets into the new java keystore.

Presumably, someone else already has this figured out- Can someone point me to what I'm missing?

@MattFox The wires we are sniffing carry modbus/RTU- The format within mango doesn't really matter. I was considering knocking another thing up using that SOCAT -> TCP device to just get an idea of what the points are being moved back and forth, then develop that into a Modbus read store/forward type of affair.

We basically have a need to eavesdrop on other systems for whom the documentation is a bit minimal- It's a way to "get around" some legacy systems that have controllers that we can't just swap out for Mango, but still need to gather the data for display.

So long as we can get the data into Mango, that'd be swell.

I like @terrypacker 's idea of bypassing Modbus4J's waiting room.. I think I can get something working using the serial data source. I'm doing this for the first time next week, so we'll see how complex this is. I don't even have a full understanding (yet) of exactly how dense / fast / complicated the Modbus/RTU data implementation here actually is- It could only be a handful of points. I was just going to sniff first, using SOCAT to listen on the wire, and try and work out a solution based on what I see.

I can't think I'm the only one trying to do Modbus Sniffing-type stuff with legacy systems, but then again, I see a lot of stuff that I do seems to be"I've never seen this type of thing before"..

Cheers,
-Greg

Greetings, Smart People at the Forum:

I wonder if anyone has (yet) deployed Mango as a Modbus Sniffer. We have some applications for this, and I may end doing this via Serial data source / SOCAT to TCP/IP, but I'd rather use Mango natively for this.

We have several applications where we have to "sniff" existing modbus traffic to feed into other systems. In the past, I've used Modbus repeater devices and other types of doodads to make this happen, but it seems like something that would be pretty easy to implement in Mango.

Ideally, this would be "just plug in Mango to a Modbus network, and it would create points and fill them with values" (not names, scaling, or whatever else) by listening on the RS-485 line.

This would be an amazing feature for the industry I am working in, and would simplify our life a lot as compared to the way I've done this in the past (Configure Mango Modbus data source, get data in, then publish data back out again as a Modbus publisher).

The difference is in the configuration time- If I can sniff the line, I can populate all the data automatically, and then just assign tag names / point names since Mango would already know the Modbus register indices that we would need, and the raw values they contain.

Has anyone ever done this / asked about this before?

@MattFox Yeah- I tried that, too: I just can't get the the thing to receive anything over the serial port.

It would be really nice to have a thread / template file available somewhere that says "Here's how you just get Raw Serial into Mango" to overcome the initial confusion over REGEX, which is absolutely common in these threads.

People do what I did: They turn it on, try and get a "wildcard" RegEx to make sure mango is receiving something then go develop the interface from there.

I have lots of serial customization coming in for our projects, and although the SOCAT <-> Serial TCP thing works okay, it would all be much simpler if I knew that the Mango Serial Device had a good easy starting point.

Greetings, all:

I've posted here before related to the Serial Data Source, and I'd love to understand what I'm doing wrong / how to make this work.

First, questions:
(1) Can the Mango Serial Data Source just act as a passive input? Just storing everything that comes in over the serial port into a point?

(2) If not, how do I force it to "trigger" a read?

Here's my Application:
I have a device that spits out hex code as 2400 baud every few seconds. I want to ingest this into mango, and parse it.

I have this (kinda) working using SOCAT and the raw TCP data source, but that seems like an awful lot of messing about.

The values are hex, fixed length, and have known terminating characters at the start (and end) of the message.

I've been re-reading back through lots of stuff on this forum before about this, and haven't gotten closer to having it work.

More details:
This is a raw serial device attached via a USB to Serial adapter. I have verified my Mango machine is getting the Right Data via using Minicom, and got this almost entirely working with SOCAT and a Raw TCP data source.

I'm just trying to keep this whole thing inside Mango. I keep trying to use the Virtual Serial Port for various things, but so far, I hvaen't been able to get it to work at all- I've tried this on V3.7, and now am running our system here on 4.5, still with no luck.

Are there any demo files / docs available for how to "just get raw data into Mango over a Serial port" so I can then parse it using a scripted data source?

I'd love some guidance on this. The TCP/IP data source seems to allow this functionality, albeit with a bit of goofing around (see my post about interfacing with my HP-IB gear for this thread), but I'd love to be able to know how to use the Raw Serial Data Source, as we have many uses for that now.

Cheers,
-Greg

@MattFox Fair enough- @nino-kurtalj said "Let's assume I have values in the W.".. Which I understood to mean he was using "watts" and it looks like he was trying to integrate watts over seconds in the metapoint he posted, using the now-last values as a sort of way to integrate watts into kwh.

Also; @nino-kurtalj if you are integrating watts over time to get kwh, realize that you're accumulating errors as you go. It's much much better to use the raw kwh (or wh, or energy point) out of your metering device. This is because even integrated watts to kwh in mango, you're still limited by polling rate of your device. I run 1000's of points of energy and power in my system, and we can see the errors building up over time: Particularly when the power signal changes rapidly.

The internal power meter kwh points (the "kwh" register you read out of your electrical meter) integrates at "electrical speed", so they catch those wiggles in real time: They basically sum every update of some high-speed V*A calculation.

If you're reading watts and integrating that into wh youself, you can only get those values every time you ask the meter for it, so you'll miss the little wiggles that happen in between polls. If you're polling every second, this will probably be a fairly small error, but it grows over time, so if you go true up your integrated watt -> kwh point by comparing it to like the power company's billing meter, you'll always be off by a few percent. The slower the watt polling interval, the larger that energy error is at the end of the month.

I run into this (and deal with this) all the time in the business I'm in, since some people insist on taking (for example) 15-minute average kilowatt, summing them up, calling it kwh, and then complaining to me that my energy points are wrong- They aren't.

They're just integrating a low-frequency discrete time signal and comparing it with an integral at real time as the signal changes: They will not be the same. That error can cause lots of confusion, if you're (for example) trying to calculate your AC line losses over time- The error in going from 15-minute average power into kwh is generally larger than the actual losses in the cable you're trying to measure.

We do use "integrated watts" from Mango for certain applications, but it's in places where that error is easy to deal with (such as by hourly comparisons, where you don't have time for that error to accumulate over months before comparing).

@CraigWeb Yup. Thanks for getting back to me.

I actually have (6) Scripts running in a loop: I'm doing it this way because I'm working around the fact that I can't (seem) to get the raw serial data source to work in Mango V4.5 (no RX under any circumstances, near as I can tell). So, I'm using TCP <-> GPIB, then using the Raw TCP driver to talk to that, which works well. However, there's lots of random delays needed to make this work, since we're sending SCPI commands and polling big arrays back out of our equipment, which has a very non-standard and irregular lag. This means I need to use the Raw TCP source using the timeout (rather than the EOL marker, since not al SCPI commands return values with known terminations). This means I had to break my script into into (6) scripts instead, basically:

The first script sets up the sourcemeter for measuring something
Second one takes a measurement
Third one parses that result and calculates some more stuff
Fourth programs the sourcemeter for the next measurement and sets it to run
Fifth one reads the raw data out of the instrument and calculates a bunch of stuff
Sixth one stores those values into points.

The handling of one to the next is done via "update context" points (that are internal points in the first script and refernced in the other scripts) and lots of RuntimeManager.sleep() style things within the scripts to give the RAW tcp points time to get data back so it's available on the next step in the script.

@CraigWeb said in Scripting point running all on its own?:

Does the data source have its own internal points?

yes- Quite a lot
Are you updating those points in the script ?
Yes- We read value from the raw TCP source, do some calcs to it, and store those in the internal scripted points. But those point are not read again in that same script
and do they re-update the context?
No: This is why I busted my original script out into (six) steps, to avoid writing (and reading) points within the same script, since I noticed this created odd behaviors (using point.set(value) then using point.value() doesn't seem to work consistently within the same context- I ended up having to do this because of the timing issues with the raw TCP point depending on timeouts to get responses back).

Thus, I use point.set(value) in one script. Then I use that value in the next script which is set to execute "context update" on a boolean point that exists only to pass control from one script to the next script.

Originally, I had this written in a single script, but that failed because I wasn't able to wait on the TCP/IP readbacks from the raw TCP point using our instrument. So I broke it out into blocks, to make that work, and then ended up passing control from one script to the next using "supervisor" points (internal points to the "supervisory" script) that just exist to trigger a context update on the next script.

The behavior you described that causes the script to execute continuously; Is this by design? If so, can you explain how that works?

Got another one for you: This is Mango V4.5:

It looks some of our scripted data points are running despite not having anything set to make them run. There's a a lot of backstory on this- Originally, I wanted something to run every 10 minutes- Thus, I set up a scripted data source to run every 10 minutes.

This data source does some things, and then sets another point (which is used by the next "step" in the process) to do more calculating; That data point is set in that script to "update context" in the next script.

Correct my understanding here, but, for the sake of this post:
"Updates context" in the world of mango scripting / metapoint means "If this value updates, run this thing. If this value doesn't update, no need to run this thing", according to the setting of the "context update event" box in the data source (update, logged, or change). If you don't have any points selected to "update context" in the points in the script, then that drop down doesn't have any effect; Is that right?

"polling" in the world of Mango scripting means "Do this thing based time settings, either every XX minutes or as a cron pattern"

If you do polling and have "updates context" points selected, then it should do both (run on the Polling time and run when the point updates within the script).

What I have are points that run with no context and no polling enabled. My original script should run every ten minutes- meaning, every 10 minutes, the script should run, do some stuff, and use the set directive to make the next "step" work in this set of chained calculations. That original scripted point should then wait 10 minutes before Doing Thing again, and restarting the process.

Our little thing is working, but it's running continuously, and doesn't seem to care about the settings of the "polling" box (currently set to off, but the script is still running) or the "updates context" checkboxes in the data points in the script (all are "unchecked").

I've noticed this quite a lot in behaviors in my scripting: I generally thought that the script would run under the conditions above (timers, based on CRON or the time setting) and / or the "updates context" next to the points in the script, as informed by the "context update event".

It appears that the scripts are running effectively continuously: This means that in practice (regardless of the settings I have for "polling" or "context updates") I need to bracket my code with "If (somedatapoint.value == true {stuff}" to keep things from updating on their own.

So what am I missing here? I want this thing to run every 10 minutes, let the cycle complete, then stop and wait for the cron (or timer setting) to restart the task 10 minutes later. Instead, it's just looping through endlessly, as if I had "updates context" selected for all the datpoints in the scripted data source.

I've attached a screen grab of one of the points in question- What setting here would indicate to you that this should run effectively constantly?

The script in that scripted point is running, and appears to be updating as if any point updates, despite none of them being checked to "update context". Have I been using "update context" wrong the whole time so far?

If so, how do I get a Mango scripted data point to run, then just sit there and do nothing until the next 10 minute interval elapses. I thought that's what "polling enabled" with either "update period" or "cron pattern" would do?

@MattFox It's not delta. It's integral. Delta between two powers doesn't given you kwh, unless mango use some different understanding of "delta".

If you do delta on power over 1 hour (for example), and you have 1000W at time = 0 and 1000 W at time = 1 hour, your delta is 0 (since 1000-1000 = 0), but your integral should be 1000w * 1hr = 1kwh.

If you do delta on power that's what you get: A delta on power (power at start of time minus power and end of time). You can use that to estimate energy, if the time period is short enough, but energy is the integral of power over time, so you need to use integral (if you're going from watts -> kwh).

@CraigWeb I appreciate that; We're going to be shipping a mountain of these things internationally use M-Bus (I hope), so for now I'm using my TCP silly workaround for this (since the Serial data source still appears to not work properly for whatever reason on our Arm-based machines).

Some other folks have piped up here as well about M-Bus- This exists basically nowhere in the US (as I've found now that I'm testing things using M-Bus) but apparently is quite common in IOT purposes in Europe- Every meter in Norway (for example) has an M-Bus port for interfacing HAN devices directly to your buildings' meter.

Surely someone must have had a reason that this got added in the first place; Although I do understand not updating things if nobody is using it. Since we're going to be using this A Lot (I think), we may be able to chat NRE to get it put back- I'm still chatting with Michael about Mango for our 2023 project pipeline, and this could be a good business discussion point to have.

@CraigWeb Sure:

This is running the latest Mango V4.5. I'm using the Raw TCP data source to chat with a TCP <-> GPIB Adapter, but this bug (also) appears in the serial data source as well. It mentions it in the documentaiton somehwere, too, that you should be able to put \r\n in the "delimiter" and have it sense that as the end of message.

This does not work.

To make it work, you have to download the JSON file and manually put in "\r\n" and then re-import it, at which point the datasource "delimiter" blank shows up blank.

Here's the screen grab of our (working) data source:

Notice: Nothing in the "Delimiter" field.

If I download the JSON for that data source, however, I see this:

      "updatePeriodType":"SECONDS",
      "delimiter":"\r\n",
      "hex":false,

which shows the proper delimiter. This works (btw) so mango is processing the delimiter properly internally, it just doesn't display right in the UI.

"acceptance criteria" would be "The actual delimiter value should be visible in the Delimiter blank, somehow, even of it consists of nonprintable characters"

In the datasource help, attached below as a screen grab, with my highlighting shown, it says " Delimiters are unescaped before sending, so "\r" translates to carrriage return.":

If you put "\n" in that blank, you get this in the JSON:

      "updatePeriodType":"MINUTES",
      "delimiter":"\"\\n\"",
      "hex":false,

If you put \n (no quotes) you get this:

      "updatePeriodType":"MINUTES",
      "delimiter":"\\n",
      "hex":false,

Which seems to mean that the parser expects \\n (not a \n)

The only way I was able to get the TCP data source to store the data using \r\n was to edit the JSON file as above, which seems to work properly. But then the \r\n doesn't show up properly in the "delimiter" blank

@nino-kurtalj

I calculate this using rollups- You can use the integral rollup type on watts over the time you want. Mango will give you this in watt seconds. Then you can divide that by 3,600 to get watt hours, then scale it as needed.

Better is to read the kwh raw value out of your metering device, though, if that works for you.

Here's an example:

// This runs every hour on the hour to calculate last hours' energy
var energy = 0;


// This sets up and returns 1 hour integral on power
var values = [];
var endDate = CONTEXT.getTimestamp();
var startDate = endDate - CONTEXT.millisInPast(HOURS, 1); // Last 1 HOUR
var pointToQuery = [ POWER.getDataPointWrapper().getId() ];
PointValueQuery.rollupQuery( pointToQuery, startDate, endDate, 
  function( value, index ) {
    values.push(value);
  }, com.serotonin.m2m2.Common.Rollups.INTEGRAL, 1, com.serotonin.m2m2.Common.TimePeriods.HOURS);
    

energy = values[0] / 3600000;


my = energy;

return my;

I'm not sure if this is the right way to do this, but it works for me.

@cwangv That's true- I'm happy to send you my code, if you want, when I get it working- It's just a scripted point that searches raw strings for the OBIS code and saves it in datapoints. This for energy meters, and so far, it works OK.

@CraigWeb any plans on bringing the M-Bus module in Mango V4? It's interesting, since it's still stated as "supported" on Mango's website: https://radixiot.com/supported-protocols

You only find out it's missing from V4 when you look at the documentation for Upgrade from V03 -> V04: https://docs-v4.radixiot.com/upgrade-to-v4 and scroll down to "unsupported modules"

@tungthanh500 Budgets. The M-Bus to USB converter is $20, the M-Bus to modbus device is considerably more expensive. Our goal is also to keep the number of things "needing configuration" to an absolutely minimum:If we can do it in Mango, it saves us time and money (and spare parts inventory costs).

@MattFox So: Turns out the socat solution works, but it's not really as straightforward as using the Serial device natively:
Can you poke around and see if the Serial data source needs some hugs? Using a TCP forward for a USB converter serial port seems a bit silly to me, when Mango has a raw Serial device interface that (should) do the same thing.

I've now had no luck with the serial data source on my ARM based IMX8 machine over hardware serial or USB to 485 serial under V3 or V4, and now am seeing the same thing under V4 on a raspberry pi.

I'm happy to set up a screen share / demo video / something if you want to see what I'm seeing with this.

Thanks for your help, in any event, as we now have a (mostly) working MBus data source again, although in a somewhat ridiculously rube-goldbergesque way.

@Turbo Ok. One more quick question on here: Since this device is (effectively) streaming data in - There isn't even a RX pin on the device we're talking to, is there a way to set up the tcpip connector to just store data as it comes in, rather than requiring the use of the "read" command before awaiting a response?

Right now, I'm just sending "00" out as a read command, but the device has no read command; It just spews data out. The read command forces a sync where none exists. I'd rather just have the tcpip point record the point when data comes into the point, rather than after the read command is issued.

If I disable "queryable" I get no data at all, but I enable queries, I end up missing every third packet, since there's no way to sync the "output only" nature of the M-Bus device with Mango's need to poll and record a response.

It kinda works, but I wonder if there's some secret to have Mango just store whatever comes in over that port?