Feature request - filter for config export

[shaun]

Hi Guys,

I find myself more and more exporting configuration from mango and hand/bulk editing JSON, then re-importing it to make additions to the system.

Normally I would use the UI to create a data source, and then a set of data points that I would then iterate over in a text editor to make more similar points or duplicate devices. I use this like a template.

As the configuration grows this process takes longer cull data out of tens-of-thousands of lines of export.

Feature request: Can we get an additional feature on the configuration export tool to filter by a few fields? eg: parent device, XID, text pattern match/regex match on a field?

Something really simple here is probably all we need to make this method of template-export-edit-bulk add easier to manage.

Unless there is already an easier way to bulk add that I'm not aware of?

Cheers!
-Shaun

[Jared Wiltshire]

@shaun its not a bad idea. I can see the usefulness. In the meantime you could use the REST API to query for data points and then save new ones. Are you familiar with any scripting languages? If you know Javascript you could use our Mango client for Node.js

[phildunlap]

Hi shaun,

Yes! You may not have noticed it, but there's an export button tucked into the top right corner of the data sources / data points table on /data_sources.shtm. This will export all the items in the upper table, so you can go to the data points tab and filter down to the point set you want. These filters do mostly support regex I believe. This will completely suffice if you only wish to export data source or data points

---

In Mango 3.2, things got pretty crazy with your options here. Script environments in Mango got a new context object, "JsonEmport" which allows you to work on JSON through the JavaScript. There was also an endpoint added to the api for either executing or testing a script.

//Let's copy the point DP_40000 to read some other modbus offests, say
String baseJson = JsonEmport.dataPointQuery("eq(xid,DP_40000)"); //or like(deviceName,Coffee*)&like(name,*ater*)
for( var k = 1; k < 10; k += 1 ) { //we'll make nine new points
  var newDp = JSON.parse(baseJson)["dataPoints"][0];
  //We could have done JSON.parse outside the loop on this one, but it's important to remember 
  //if you put them into a list they're references, so if you put the same item in a list twice and 
  //change it in one place it will be different in both place. Since JavaScript doesn't come with an 
  //object copier as part of the language (to my knowledge) it can be easy to let JSON.parse do the 
  //object copy work (or you can look up an object copy function or write one)
  newDp["xid"] = "DP_4000" + k;
  newDp["pointLocator"]["offset"] = 40000 + k;
  //What will be imported is the object as it looks when stringified
  JsonEmport.doImport( JSON.stringify( {"dataPoints": [ newDp ]} ));
}
//Note imports will not happen from script validation
RuntimeManager.disableDataSource("This script's xid"); //So let's turn this script off if it were a data source.

I suppose while working with my previous python recommendations, I would say run them on the machine against the backup file is a way to escape the size of the file becoming burdensome. You can't do regex matching in RQL queries, but it does support wildcards as exampled.

[shaun]

Thanks for the details guys...

I'm familiar with the CSV export from the data points/sources page, but that doesn't give you the event detectors that I can get from the config export.

When I build a reusable template based on json config I usually include the data source, data points, event detectors, event handlers, and the point hierarchy... I try to make it as complete as possible.

Most of the deployments I work on have many of the same device, so fast duplication with just changes of names, IPs, modbus IDs etc makes life easy.

I'll have to explore the API and scripting as you both suggest soon!

Cheers!
-Shaun

[phildunlap]

Definitely!

I was referring to the JSON export button,

The event detectors are exported with the data points. Given one export of data points and one whole system config json, the following script will combine them with the handlers, sources and the point hierarchy...

#!/bin/python
import json

dataPointsFile = open("/path/to/dataPoints.json")
dataPoints = json.load( dataPointsFile )
dataPointsFile.close()

dataPoints = dataPoints["dataPoints"] #it exports as a list of data points keyed to "dataPoints"

configFile = open("/path/to/Mango-Configuration.json")
config = json.load( configFile )
configFile.close()

dsXidMap = {} #we'll want to see if we added a data source to our list of found sources already
dpXidMap = {} #we'll use this to prune the point hierarchy
dataSources = []
eventHandlers = []

def addDataSourceIfMissing(dp) :
    if dp["dataSourceXid"] in dsXidMap :
        return
    for ds in config["dataSources"] :
        if ds["xid"] == dp["dataSourceXid"] :
            dataSources.append(ds)
            dsXidMap[ds["xid"]] = True
            return
    print "Failed to find data source for dp: " + dp["deviceName"] + " - " + dp["name"]

def addHandlers(ed, dp) :
    for eh in config["eventHandlers"] :
        if eh["eventType"]["sourceType"] == "DATA_POINT" and eh["eventType"]["dataPointXID"] == dp["xid"] \
                and eh["eventType"]["detectorXID"] == ed["xid"] :
            eventHandlers.append(eh)

for dp in dataPoints :
    dpXidMap[dp["xid"]] = True
    addDataSourceIfMissing(dp)
    for ed in dp["eventDetectors"] :
        addHandlers(ed, dp)

def pruneRecursively(node) :
    newSubfolders = []
    for s in node["subfolders"] :
        pruneRecursively(s)
        if len(s["points"]) > 0 or len(s["subfolders"]) > 0 :
            newSubfolders.append(s)
    node["subfolders"] = newSubfolders
    newDps = []
    for pnt in node["points"] :
        if pnt in dpXidMap :
            newDps.append(pnt)
    node["points"] = newDps

newPh = []
for root in config["pointHierarchy"] :
    pruneRecursively(root)
    if len(root["points"]) > 0 or len(root["subfolders"]) > 0 :
        newPh.append(root)

outputFile = open("/path/to/output.json", "w+")
outputFile.write( json.dumps( {"dataPoints": dataPoints, "dataSources": dataSources, "eventHandlers": eventHandlers, "pointHierarchy":newPh }, \
    sort_keys=False, indent=4, separators=(",",": ")) )
outputFile.close()

[shaun]

Oh! @phildunlap I didn't realise thats what that button was for.. haha!

Thanks again, and thanks very much for the script too!

Cheers!
-Shaun