Stuff data from cloud back to remote? + Backups

[mrlucretius]

Hi Guys,

So once we had our first glorious SolarSCADA demonstration site on a real 1.6MW solar power plant, everyone was happy, all was well, and the sun was shining.

Then, a few weeks later some clouds blew in, and lightning struck something near or in the site, and blew a bunch of things up. While the electronics were variously protected and isolated, this was a freak occurrence, as lightning is well known and tolerated by solar plant design and operation.

So the Mango PC got smoked. And then I was like, oh snap. What about backups? Of course I did not do any. I was planning to down the road.

So, sad story aside, two questions:

Can I backfill data on a remote site from a central cloud mango that was logging persistent TCP points? Is this a pretty manual process?

I know you have an article about backups that I will read. I was wondering if someone could write a few sentences about how I should have managed this mango install, and best practices for going forward.

Thanks,
Alex

[phildunlap]

Hi Alex,

Can I backfill data on a remote site from a central cloud mango that was logging persistent TCP points? Is this a pretty manual process?

Yes you can, and its manual-ness is really about how efficient you want to be. On the /mango_no_sql.shtm page there is a 'Mergration' tab which is like a merge (bring in data from another MangoTSDB) and a migrate (bring in data from another Mango SQL database) to bring across data from another Mango based on XID matching (because the IDs, which are the TSDB second level folder names, Mango/databases/mangoTSDB/*/{dataPointId}, are not user assigned). So, if the XIDs were not prefixed by the publisher (or you don't mind adding the publisher prefix during the mergration and removing it afterward, probably at the SQL console) it is as simple as having the Mango/databases/ files available directly to the restoring unit and for the XIDs to match.

Barring that, it starts to get more manual. You can drop the shardNumber.data.rev file from one Mango/databases/mangotSDB/*/{dataPointId} into another and at Mango restart or sooner the data will be there to be consumed. Usually this involves some use of the XID to figure out the IDs (usually I do this sort of task in a python or bash script, and I have maps built ahead of time for data point XID to ID by going to the sql console in each system and SELECT id, xid FROM dataPoints

I know you have an article about backups that I will read. I was wondering if someone could write a few sentences about how I should have managed this mango install, and best practices for going forward

You should definitely be saving your SQL backup offsite. With an SQL backup, it's a pain but the work to recover the data from a central server is much easier (no recreating points). With an SQL backup, you can do clever solutions like create a publisher from the cloud and use a history sync from inception time to restore the data into new points on a persistent receiver, then you can stop Mango and shuffle things around the Mango/databases/mangoTSDB directory as I suggested (shuffle the data files from the sync'ed points into their source point directories) and then delete the persistent receiver. Even a JSON backup would make this easily achieved.

if you do not have a cloud serving as a data warehouse, then the importance of the data history will determine the extent you need to concern yourself with backing up the data. For a lot of our single-site commercial users, we've found IDrive backups of incremental NoSQL backups (and all the SQL and JSON backups too) have saved us some bacon and kept some clients happy. Intuitively you should backup the configuration whenever you change it. You should back up the data as often as makes sense. We've definitely had some users who don't care if their data history exists on any restored or replaced system, since they're really only using it for control.

SQL backups differ from JSON backups in containing things like event data and ensuring data points have the same IDs upon restore.

[mrlucretius]

Awesome, thanks for the detailed reply. I will carefully read this a few times.

It turns out that it appears that the hard disk is still alive (I hear it mounts and is readable).

So, I think I can setup a new PC (network/security/VPN/services), and just copy the /opt/mango folder over?

Is it really going to be that simple?

Thanks,
Alex

[phildunlap]

Yes. If the hard drive is good then restoring the Mango directory would be sufficient to keep your data and configuration (and anything like excel report templates, data file templates, custom ftl files, overridden properties, overridden classes, etc) and is indeed that simple. You can use this form to request we transfer the license from the old GUID (which you could find through your store account or in the /opt/mango/m2m2.license.xml file) once you've got it running on the new system: https://infiniteautomation.com/license-help/

[mrlucretius]

Hi Guys,

I was able to restore the mango on new hardware by copying the /opt/mango dir from the old hard drive. That was easy! Mango starts up fine after the sysadmin work. I don't see my dashboards but I think that may be a license thing?

I submitted a license change request.

Thanks,
-Alex

[phildunlap]

License transferred!

[mrlucretius]

Hey,

For some reason my dashboard did not come back after copying old /opt/mango to new PC.

How to diagnose / restore? Any thoughts / pointers?

Thanks,
Alex

[phildunlap]

Hmm. How was your dashboard created? All files by default are in the /opt/mango directory, and "Dashboard Designer" or "Edit Pages" are stored in the jsonData table in the SQL database. I wonder if what happened is that the pages got lost for some reason in the jsonData table entry that track pages. I

would check the /ui/administration/json-store page to see the "UI Menu" and "UI Pages" items, and also to see if your missing pages are shown in the JSON store. If so, you can probably find a backup of the right UI Menu and Pages in a configuration backup (in jsonData item, XIDs "mangoUI-menu" and "mangoUI-pages", in /opt/mango/backup/Mango-Configuration-*.json) then import that., which should restore your dashboards to appearing in menus and having menu items and whatnot.

[mrlucretius]

Hi,

I will dig into the dashboard stuff as you suggest. However in the mean time I just added a new dashboard, just cut and pasted from a local xml copy.

Another issue, I think copying /opt/mango was not enough...

Getting weird errors. Guessing they are file permissions issues?

Can't create new data sources. Can't configure data sources. HTTP 500 errors?

-Alex

[phildunlap]

Another issue, I think copying /opt/mango was not enough...

That directory definitely contains all things configured in Mango (by default). You would have had to consciously store files elsewhere for that not to be sufficient.

There is almost certainly more information in your log file. If I had to take a shot from the hip though, the JSPs need to be recompiled which means deleting your Mango/work/jsp directory (can be done while Mango is running).

[mrlucretius]

Hi,

I was thinking that copying the /opt/mango folder may need some permissions adjustments perhaps? I did not mean to say that the folder is not a self contained mango installation, as it certainly is.

I deleted the JSP folder. Apparently, the same errors as above still.

Which logs should I look at?

I see /opt/mango/logs

tsdb-series-74-shard-714-scan-report.....

I can post some log stuff here if you like.

As always, appreciate the help and guidance, I am learning new things all the time.

-Alex

[phildunlap]

Ah, sorry, I meant the ma.log file. That's the normal Mango log and usually what i mean if I'm not relative-path levels of specific. Mango/logs/ma.log and on some Windows machines Mango/bin/logs/ma.log

[phildunlap]

I was thinking that copying the /opt/mango folder may need some permissions adjustments perhaps?

Certainly possible. You can easily sudo chown -R user:group /path/to/Mango if that's the case.

[mrlucretius]

It looks like both my functional cloud mango and the remote mango we are trying to revive have root:root ownership of the /opt/mango folder... That should be OK right?

I am getting my hands on the log file here in a minute. Will edit this post.

[phildunlap]

The files inside the folder will matter too and so shall the user running Mango. As it sits I would wager only your root user can run Mango without having permissions issues.

If you change Mango to run under a less privileged user (better practice) you'll likely need to add that user to the dialout group so that you can use serial devices, and you'll probably need to give your Java executable the bind capability if you want to have restricted ports available for use.

[mrlucretius]

I emailed a zip with logs to support@infiniteautomation.com

-Alex

[phildunlap]

Some of those errors look like you may need to add the dialout group to the user launching Mango. If you're running as the root user or have already added the 'dialout' group, then I'd wonder what operating system is this? The native library it's looking for in the

com.serotonin.util.LifecycleException: Comm configuration error. Check that your serial port DLL or libraries have been correctly installed. Could not initialize class jssc.SerialNativeInterface

log message that is spamming the ma.log you sent in should be included in the JSSC jar in Mango/lib/jssc-2.8.0.jar file.

There wasn't really much of interest in the three dated files. In one it looked like your DNS was acting up, in one it looked like there was a client error while using the dashboard designer (certainly can happen), and then one was just a normal boot.

[mrlucretius]

Ok this thing is debian-esque:

https://www.mediawiki.compulab.com/index.php/IOT-GATE-iMX7_and_SBC-IOT-iMX7_NXP_i.MX7_SW_Resources

I will dig in.

-Alex

[Turbo]

Okay. I figured this out:

(1) This is a small sort of Debian install, supported by Compulab, that is missing some Important Things.
(2) Part of this is an old (or incomplete) java that doesn't have all the bits to make Mango happy.

I fixed this by:
(1) apt-get update--
Which failed, because the people at Compulab didn't register their certs right, so I had to put the cert data in manually.
(2) apt-get update worked.
(3) apt-get install java then did the business.

We can now configure our data sources without the 500 error.

However, that opened up the next thing, which seems to be something odd in the serial ports for this device. Was hoping I'd have more time to figure this out, but I'll post this as another thread.

Cheers,
-Greg Linder

[phildunlap]

Hi Greg, thanks for sharing what resolved it!