configuration of/filtering email alerts

[shaun]

Hi All,

I'm trying to configure email alerts in an efficient (in terms of configuration) way.

So far the only way I can see to send emails is to either pick the event level to send from (none, info, urgent, etc) in the user account, or configure a mailing-list and use event handlers per-event-detector to send emails.

Ideally I'd like to be able to do something simple like send alerts to a mailing list where the alerts have come from a range of configured devices (could either filter on names or have a list?).

Is there a way to do this that I've missed? Can this be scripted somehow with a global script?
It'd be great if the points hierarchy could be used as part of the filtering perhaps?

While its great to have granular control, It seems fairly cumbersome to have to configure an event handler against each event detector of each data point.

Cheers!
-Shaun

[phildunlap]

Hi Shaun,

I understand what you mean about it being cumbersome, and we are trying to improve the ability to handle tasks like this through the UI. It is, however, quite a solvable problem if you are up for some Python (or JavaScript, I suppose) scripting. Here's a script that will take a point folder name, search for unhandled events on those points, and then output a file of JSON to create all those email event handlers.

import json
import copy
from StringIO import StringIO

folderNameToCheck = "Device 1 Folder"

myConfigFile = open("/path/to/Mango/backup/Mango-Configuration.json")
myConfig = json.load(myConfigFile)
myConfigFile.close()

def findPHFolderByNameRecursive( root, name ) :
	if root["name"] == name :
		return root
	for folder in root["subfolders"] :
		f = findPHFolderByNameRecursive( folder, name )
		if f is not None :
			return f
	return None

#DFS Greedy
def findPHFolderByName( name ) :
	for folder in myConfig["pointHierarchy"] :
		fldr = findPHFolderByNameRecursive( folder, name )
		if fldr is not None :
			return fldr
	return None
	
def folderContainsPoint( folder, dpXid ) :
	for xid in folder["points"] :
		if xid == dpXid :
			return True
	for sf in folder["subfolders"] :
		if folderContainsPoint( sf, dpXid ) :
			return True
	return False

baseEventHandler = json.load(StringIO( """{
         "eventType":{
            "sourceType":"DATA_POINT",
            "dataPointXID":"DP_790074",
            "detectorXID":"PED_764719"
         },
         "xid":"EH_207436",
         "handlerType":"EMAIL",
         "activeRecipients":[
            {
               "recipientType":"USER",
               "username":"admin"
            }
         ],
         "sendEscalation":false,
         "sendInactive":false,
         "includeSystemInformation":false,
         "includePointValueCount":10,
         "includeLogfile":false,
         "alias":"Generated email handler: ",
         "disabled":false
      }"""))
	  
#We'll check if points are in this folder, then create handlers for their detectors
folderToHandle = findPHFolderByName( folderNameToCheck )

handlerDict = {} #we need to know if some may already have handlers
for eh in myConfig["eventHandlers"] :
	if eh["eventType"]["sourceType"] != "DATA_POINT" :
		continue
	handlerDict[eh["eventType"]["dataPointXID"] + "-" + eh["eventType"]["detectorXID"]] = eh
	
outputConfig = {"eventHandlers":[]}
generatedHandlers = 0
for dp in myConfig["dataPoints"] :
	if folderContainsPoint( folderToHandle, dp["xid"] ) :
		for ed in dp["eventDetectors"] :
			if dp["xid"]+"-"+ed["xid"] not in handlerDict :
				#Found an unhandled detector, let's create a handler!
				newHandler = copy.deepcopy( baseEventHandler )
				newHandler["eventType"]["dataPointXID"] = dp["xid"]
				newHandler["eventType"]["detectorXID"] = ed["xid"]
				newHandler["xid"] = "EH_11-17-16_" + str(generatedHandlers)
				newHandler["alias"] += dp["deviceName"] + " - " + dp["name"]
				generatedHandlers += 1
				outputConfig["eventHandlers"].append( newHandler )

outputFile = open("/path/to/create-handlers-output.json", "w+")
outputFile.write( json.dumps( outputConfig, sort_keys=False, indent=4, separators=(",",": ") ))
outputFile.close()

The advantage to keeping it simple and granular is that power is never sacrificed to maintain a non-granular interface tool, as well as spending time elsewhere than trying to come up with every possible angle someone could seek such a tool from. There are ways of doing anything en masse

To use this for a mailing list instead of the admin user, I would create an event handler that emails the mailing list in question, export that, and update the "activeRecipients" section of the baseEventHandler

[shaun]

Hi Phil,

Thanks for your help... I'll give this method a go...

I agree, granular control is important, but so is having a common configuration that can apply to a large selection of 'things'.

What'd be great to see in a future release is perhaps a "common event handler" section where you can define a number of common event handlers that can use a list, regular expression, or select a section in the points hierarchy to apply to.

Cheers!
-Shaun

[phildunlap]

I hear you. I get a little fidgetity though because once again, scripting up a tool for this purpose wouldn't take very long, especially seeing what my script is doing. You could just create those basic event handlers, replace the baseEventHandler (which is an awfully general name when it's really an Email handler of a particular type in that base json), and then you only have to write a for dp in config["dataPoints"] loop to

1. Regex for names: re.search("expressionPattern", dp["name"]) is not None && print "Found: " + dp["name"] (you'd have to import re above)
2. Points in folder using the code above
3. Reference the "dataPoints" list of a watchlist to identify points
4. Check any other json property, or any other relation to another json property

You can make lots of modifications easily with an understanding of this. Say we could ignore the reading of the file and the reimport of the output file. Perhaps our 'config' object just takes care of that itself. Suddenly, alphabetizing all the watchlists in a system by "deviceName - name" becomes:

dpXidDict = {}
for dp in config["dataPoints"] :
    dpXidDict[dp["xid"]] = dp

for wl in config["watchLists"] : #consider `if re.search("pattern", wl["name"]) is not None :`
    wl["dataPoints"].sort(key=lambda x: dpXidMap[x]["deviceName"] + " - " + dpXidMap[x]["name"])

We are definitely working on making configuring larger systems more dynamic, but the power of a script can't be underestimated in responding to the ways life phrases the restrictions of a particular task. And it presents a great deal of control over generating things like the aliases or xids if you intend to use them for anything later. Python is fun, even if that is a legitimate barrier to entry. There's really not that much more to know to start pondering things like

1. cross validating similar but idiosyncratic systems
2. generating points from a CSV or Excel file (I think I've shown "String replacement %s" % ("rocks",) examples elsewhere on the forum), using that in a JSON object that is wrapped by StringIO() and json.load()
3. using the config to decide API endpoints to call and record data from

And so on. Scripting can only prevail in a versatility competition, so it's good to consider what's possible, in my opinion.

[shaun]

Hi Phil,

Your first script for generating Event handlers works great.. except for one thing that I can't figure out...

After I import an Event Handler, if I use the web UI to try and edit (even making no change), when i click save, I'm see an error message:

Unsupported time period: 0

Sample of my generated/imported configuration:

{
    "eventHandlers": [
        {
            "includePointValueCount": 10,
            "activeRecipients": [
                {
                    "recipientType": "MAILING_LIST",
                    "mailingList": "ML_Generator"
                }
            ],
            "xid": "EH_Gen1_Active_Fault_Type_Warning",
            "includeSystemInformation": false,
            "eventType": {
                "dataPointXID": "DP_Gen1_Active_Fault_Type",
                "sourceType": "DATA_POINT",
                "detectorXID": "PED_Gen1_Active_Fault_Type_Warning"
            },
            "sendInactive": true,
            "sendEscalation": false,
            "disabled": false,
            "alias": "Bank 1 PDU0 Frequency FaultGen1 - Active Fault Type",
            "handlerType": "EMAIL",
            "inactiveOverride": false,
            "includeLogfile": false
        },
        {
            "includePointValueCount": 10,
            "activeRecipients": [
                {
                    "recipientType": "MAILING_LIST",
                    "mailingList": "ML_Generator"
                }
            ],
            "xid": "EH_Gen1_Active_Fault_Type_Shutdown",
            "includeSystemInformation": false,
            "eventType": {
                "dataPointXID": "DP_Gen1_Active_Fault_Type",
                "sourceType": "DATA_POINT",
                "detectorXID": "PED_Gen1_Active_Fault_Type_Shutdown"
            },
            "sendInactive": true,
            "sendEscalation": false,
            "disabled": false,
            "alias": "Bank 1 PDU0 Frequency FaultGen1 - Active Fault Type",
            "handlerType": "EMAIL",
            "inactiveOverride": false,
            "includeLogfile": false
        }
    ]
}

If I manually create the event handler in the UI, i don't get any such error on save, or subsequent re-save.

Another related question... If I wanted to bulk update my event handlers, whats the best way to do this?
I'm guessing I can use the additive nature of the configuration import and remove the section in your script that looks for an existing event hander? That wouldn't remove things like old email addresses if I wanted to remove an email address/mailing list from the Event Handler.

What about a bulk delete of all handlers for a specific device?

Cheers!
-Shaun

[shaun]

Hi @phildunlap

Can you provide further assistance with the above error I'm seeing and questions?

Cheers!
-Shaun

[phildunlap]

Hi Shaun

That's strange! I was not able to reproduce. Did it print a stack trace out to your console or log? The only thing I can imagine that being is the escalation scheduling. If you check the escalation box using the UI, what does it display for the time period of the escalation delay? I suspect we've fixed the underlying issue in the latest core version.

Bulk updating can be tricky to pick the 'best' way. Find/replace is often sufficient and fast, sometimes going through the UI will be fastest. Doing this JSON scripting stuff, you can use the same code to load your configuration, and the same code to write the output (only use myConfig instead of outputConfig) and you will write out the whole of your configuration again, but altered by whatever loops come in between.

To run a bulk delete (which doesn't happen often, usually it's a data source that needs deleting, where the UI will do well), I generate SQL.

ouputSQL = "DELETE FROM dataPoints WHERE xid IN ("
for dp in myConfig["dataPoints"] :
    if dp["deviceName"] == "So long, this device!" :
        outputSQL += "'" + dp["xid"] + "',"

if "," in outputSQL :
    outputSQL = outputSQL[:-1] + ");"
    print outputSQL

else :
    print "No Points matching device name"

and while that would be the same as...

DELETE FROM dataPoints WHERE deviceName='So long, this device!';

... it's a more flexible abstraction (but it's rare you need granular delete control in my experience, unless you make a bad mistake during a big generation!)

[shaun]

Hi Phil,

Thanks for your help...

The export; find/replace; import method is something I've done before to good effect... The SQL for bulk deletion is something that I was really not sure of... thanks!

The error I'm seeing does produce something in the log:...

com.serotonin.ShouldNeverHappenException: Unsupported time period: 0
	at com.serotonin.m2m2.Common.getPeriodDescription(Common.java:266)
	at com.serotonin.m2m2.rt.event.type.AuditEventType.maybeAddPeriodChangeMessage(AuditEventType.java:189)
	at com.serotonin.m2m2.vo.event.EventHandlerVO.addPropertyChanges(EventHandlerVO.java:566)
	at com.serotonin.m2m2.vo.event.EventHandlerVO.addPropertyChanges(EventHandlerVO.java:46)
	at com.serotonin.m2m2.rt.event.type.AuditEventType.raiseChangedEvent(AuditEventType.java:94)
	at com.serotonin.m2m2.db.dao.EventDao.updateEventHandler(EventDao.java:685)
	at com.serotonin.m2m2.db.dao.EventDao$8.doInTransactionWithoutResult(EventDao.java:665)
	at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:34)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
	at com.serotonin.m2m2.db.dao.EventDao.saveEventHandler(EventDao.java:659)
	at com.serotonin.m2m2.db.dao.EventDao.saveEventHandler(EventDao.java:654)
	at com.serotonin.m2m2.web.dwr.EventHandlersDwr.save(EventHandlersDwr.java:286)
	at com.serotonin.m2m2.web.dwr.EventHandlersDwr.saveEmailEventHandler(EventHandlersDwr.java:255)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.directwebremoting.impl.ExecuteAjaxFilter.doFilter(ExecuteAjaxFilter.java:34)
	at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
	at com.serotonin.m2m2.web.dwr.util.TranslationsFilter.doFilter(TranslationsFilter.java:37)
	at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
	at com.serotonin.m2m2.web.dwr.util.ExceptionDetectionFilter.doFilter(ExceptionDetectionFilter.java:26)
	at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
	at com.serotonin.m2m2.web.dwr.util.DwrPermissionFilter.doFilter(DwrPermissionFilter.java:45)
	at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
	at org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:431)
	at org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:283)
	at org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:52)
	at org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)
	at org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

Screenshot of what I submitted to generate the error is below. This was just editing the existing event handler.

I tried setting the escalation on as you suggest (in the screenshot), and I get the exact same error.
Email alerts do seem to be working though... I've had a few come through over the last few days via the mailing list.

This is the event handler configuration I imported to create the initial event handler:

{
            "includePointValueCount": 10,
            "activeRecipients": [
                {
                    "recipientType": "MAILING_LIST",
                    "mailingList": "ML_CRAC"
                }
            ],
            "xid": "EH_CRAC1_U1_Airflow_Alarm",
            "includeSystemInformation": false,
            "eventType": {
                "dataPointXID": "DP_CRAC1_U1_Airflow_Alarm",
                "sourceType": "DATA_POINT",
                "detectorXID": "PED_CRAC1_U1_Airflow_Alarm"
            },
            "sendInactive": true,
            "sendEscalation": false,
            "disabled": false,
            "alias": "CRAC1 - Unit 1 Airflow Alarm",
            "handlerType": "EMAIL",
            "inactiveOverride": false,
            "includeLogfile": false
        },

Cheers!
-Shaun

[phildunlap]

Thanks for providing so much detail!

It does look like that issue was fixed in 2.8 so upgrading is one path to resolution.

I see it does indeed have to do with the escalation period! You could try reimporting your generated JSON with "escalationDelayType": "HOURS" but that may not work. If it doesn't, you'll unfortunately have to import new handlers with that property set to begin with, and probably delete the first batch (which is one of the reasons I sometimes put things like dates in my generated XIDs, one could have said DELETE FROM eventHandlers WHERE xid LIKE '%11-27-16%;

Sorry for the inconvenience!

[shaun]

Thanks Phil,

I'll do the upgrade out of hours and let you know the results.

Cheers!
-Shaun

[shaun]

Hi Phil,

Just did the upgrade... so far everything looks fine, but I notice all my event handlers have disappeared..

All my data sources, data points, event detectors, mailing lists, dashboards, etc are all there and working..

Tomorrow I'll have a go at re-importing my event handlers and see how it goes.

Cheers!
-Shaun

[phildunlap]

Oh no!

Can you email in the contents of your Mango/logs/Upgrade12.log and ma.log files?

[shaun]

Hi Phil,

Its not really a big deal... if thats all thats missing, and thats all I've found so far. I suspect they may have been 'cleaned up' if they were invalid configuration perhaps?

I've just emailed the logs into support@ with your name in the subject.

Cheers!
-Shaun

[shaun]

Hi Phil,

Just to update you.. I re-imported all my event handlers, and the issue from before where I couldn't edit is now gone.

Cheers!
-Shaun

[phildunlap]

Thanks!

It looks like the upgrade went fine, so I'm all the more curious how they disappeared. It says in your upgrade log: 'Upgraded 131 event handlers.' Would this have been the number of other handlers, not counting the ones we generated? Does the result from running the SQL SELECT count(id) FROM eventHandlers; give a seemingly sensible number?