How to bulk-import event detectors?

[shaun]

Hi All,

I'm used to the idea of using a csv full of data points for import, but I was wondering is there an easy way to do this for event detectors too?

I've got a bunch of simple formulas in an excel document that auto generate things like the XIDs, names, etc for data points from a pasted in list, but I can't seem to find a way to do it for event detectors against the data points.

I see that if I do a configuration export for data points, included in the export are the event detectors, but i'm finding it hard to follow the JSON in a text editor.. is there a way to either do this as a CSV as well, or is there some tool that I should be looking a to help with generating the JSON for a re-import?

Cheers!
-Shaun

[JoelHaggar]

This is a bit of an issue. I think the only way right now is to use a script to find and replace the event detectors you want into the JSON of your data points. This is something we plan to change but don't have a great solution for you at the moment.

[phildunlap]

Hi Shaun,

There is not a CSV importer for event detectors / handlers yet, so you'll have to find another method if you wish to do bulk. Here's a little Python script like those I use for this kind of thing, where I'll be taking a CSV as the input. It'll also require the data points to exist, then navigating to the System Settings page and doing a configuration backup, such that we have the latest JSON to work with (or you can export just the dataPoints from the export page, if you're only trying to work with point detectors).

import json
from StringIO import StringIO

configFile = open("/path/to/Mango/backup/Mango-Configuration.json")
config = json.load(configFile)
configFile.close()

baseLowLimitEventDetector = """{
               "xid":"%(eventDetectorXid)s",
               "type":"LOW_LIMIT",
               "alarmLevel":"URGENT",
               "limit":%(limit)s,
               "durationType":"SECONDS",
               "duration":0,
               "notLower":false,
               "alias":"%(alias)s"
            }"""

dpXidDict = {}
for dp in config["dataPoints"] :
    dpXidDict[dp["xid"]] = dp

csvConfigFile = open("/path/to/csvFile.csv")
#if it had headers, you could build a headerDict or disregard the first row.
csvConfigFile.readline() #just consume headers line, assume positions are known.

#let's assume it's comma delimited, there are no loose commas
#alias = index 7, eventDectectorXid = index 8, limit = index 9, dpxid = index 1
for line in csvConfigFile :
    data = line.split(",")
    if data[7] != "" and data[1] in dpXidDict :
        #the dpXidDict holds references to the config's objects, so changes there will be reflected in config, we can use a formatting string to transform our CSV into JSON, then load the json and append the object
        dpXidDict[data[1]]["eventDetectors"].append( json.load( StringIO( baseLowLimitEventDetector % { "eventDetectorXid": data[8], "limit":data[9], "alias":data[7] } ) ) )

csvConfigFile.close()

#write our new JSON out
outputFile = open("/path/to/output/Adding-point-detectors.json", "w+")
outputFile.write( json.dumps( config, sort_keys=False, indent=4, separators=(",", ": ") ) )
outputFile.close()

I didn't test this much or check if I had a syntax error or anything, but that's the gist of a method to speed along that process.

[shaun]

Thanks guys...

I'll give your script a go Phil... thanks...

Joel, I'm not sure what the plan might be for developing an 'import' feature for this, but perhaps just something simple like extending the csv export/import for data points with some more columns.... you could cover the case for multiple event detectors by just duplicating rows for the data point (it could update/ignore and add the extra event detectors at the same time). Programmatically, that would hopefully be simple to implement.

Cheers!
-Shaun

[Mircea]

Hi Joel/Phil,
Any new solution on updating the event detector in bulk ?
What about Mango 3 ?
Thanks
Mircea

[phildunlap]

Hi Mircea,

The next release of 3 does include API endpoints for event detectors specifically. We're trying to get the release out this Wednesday. It will have the following endpoints:

GET /v2/event-detectors Query Event Detectors
POST /v2/event-detectors Create an Event Detector
GET /v2/event-detectors/data-point/{xid} Get all Event Detectors for a given data point
GET /v2/event-detectors/data-source/{xid} Get all Event Detectors for a given data source
GET /v2/event-detectors/explain-query Get Explaination For Query
DELETE /v2/event-detectors/{id} Delete an Event Detector
GET /v2/event-detectors/{id} Get an Event Detector
PUT /v2/event-detectors/{id} Update an Event Detector

We have not added any sort of CSV solution.

[Mircea]

Thanks Phil,
The CSV solution was quite good for bulk updates.
Will be a CSV solution to update event detector ?
Thanks
Mircea

[terrypacker]

Mircea,

There is not a CSV solution for event detectors yet in 3.1, but we can look at adding this into a release in the near future.

[ricardo]

Hi Philip,

Is there an update on this topic? Is it csv import of event detectors available yet?

BR,
Ricardo

[JoelHaggar]

This is going to be added into Mang0 v4 at some point. The backend API is there but there is not yet any UI for it. Keep an eye out for when Mango v4 is release and the release notes. It should be coming in the next couple months.