Do you want to erase alarms from the database, or simply acknowledge them so they won't be displayed by default in the Pending Alarms page?

The "Acknowledge All" option can only acknowledge alarms one page at a time. This can be time consuming if there are many pages of alarms. Note that alarms/events that occur on a regular schedule can be acknowledged automatically by creating a Maintenance Event in the Maintenance Events module.

I agree that sometimes there are too many pending alarms, and I often find that I want to acknowledge or silence some, but not all alarms. Currently the only way to acknowledge them is a one-or-all choice, so that if you want to acknowledge all but one of 100 alarms, you must acknowledge 99 alarms individually. It would be nice if we could filter the list before selecting "acknowledge all." It simply takes too long to acknowledge them individually, especially if we're connected over a high-latency link.

I think a good solution would be to implement the following:

• Rename the "Acknowledge All" link to "Acknowledge All This Page"
• Create a true "Acknowledge All" link
• Implement a filter to select which alarms to display, and acknowledge only those
• When selecting any of the "Acknowledge All" links, allow choosing the alarm level so that, for instance, you can acknowledge all Information in one go, but then acknowledge the Urgent events individually.

Extreme or incorrect values in the Mango data history have caused me a lot of problems (see the [url=http://mango.serotoninsoftware.com/forum/posts/list/1375.page Incorrect 0 point value...). We must have a way to locate and purge or modify selected data values in the PointValue history. I have deleted extreme values in the past via SQL, but these deletions are subject to errors (i.e. dangerous) and have often resulted in high-CPU usage problems that could only be rectified by restarting the Mango service.

While the "Discard extreme values" datapoint function is useful, it has limitations:

• It only applies to new values; it does not remove extreme values from the point history
• In most cases the Discard feature is not enabled until extreme values have already been stored in the history
• The discard value will not discard invalid 0.00000 values; it will only discard extreme values
• If the Discard function is enabled, there is no way to detect the time, frequency, or value of the discarded data, which makes it difficult to troubleshoot and prevent extreme values from occurring in the future.[/list]
  It would be helpful if an optional alarm event were initiated each time an extreme value is discarded. The alarm would include the time stamp, the discarded value, and the discard threshold.

What we need most of all is a point history editor that provides a means to select points based on their value and timestamp meeting certain criteria, and then either purge or modify those values (e.g. multiply or add to the point).

Extreme or incorrect values in the Mango data history have caused me a lot of problems (see the [url=http://mango.serotoninsoftware.com/forum/posts/list/1375.page Incorrect 0 point value...). We must have a way to locate and purge or modify selected data values in the PointValue history. I have deleted extreme values in the past via SQL, but these deletions are subject to errors (i.e. dangerous) and have often resulted in high-CPU usage problems that could only be rectified by restarting the Mango service.

While the "Discard extreme values" datapoint function is useful, it has limitations:

• It only applies to new values; it does not remove extreme values from the point history
• In most cases the Discard feature is not enabled until extreme values have already been stored in the history
• The discard value will not discard invalid 0.00000 values; it will only discard extreme values
• If the Discard function is enabled, there is no way to detect the time, frequency, or value of the discarded data, which makes it difficult to troubleshoot and prevent extreme values from occurring in the future.[/list]
  It would be helpful if an optional alarm event were initiated each time an extreme value is discarded. The alarm would include the time stamp, the discarded value, and the discard threshold.

What we need most of all is a point history editor that provides a means to select points based on their value and timestamp meeting certain criteria, and then either purge or modify those values (e.g. multiply or add to the point).

The nightly 'Medium Priority Work Items' that are causing 'datasource time value unchanged for 1 minute' alarms imply that there is a slowdown in the 1 second interval reading of the modbus data source. The problem isn't just the alarm, it is the data dropout that the alarm implies. This appears to be triggered every night by the automatic data purge:

INFO  2013-05-04 03:05:00,001 (com.serotonin.m2m2.rt.maint.DataPurge.executeImpl:60) - Data purge started  
INFO  2013-05-04 03:08:31,832 (com.serotonin.m2m2.rt.maint.DataPurge.executeImpl:70) - Data purge ended, 2001716 point samples deleted  

A simple solution may be to assign a low (or lower) priority to the data purge process, or assign it with ionice so that it does not tax the disk IO. When could something like this be implemented?
I have roughly 300 datapoints using Mango core 2.0.6
Thanks.

Whether I create a new meta datapoint or edit an existing one, when I click on Generate History that icon blinks for a while, and then a pop-up window displays the number of points generated. The number of points generated seems to correspond to the "Update event" interval and the inception of the first input context point; the wait time also seems to correlate to the number of points generated. However, the "Point Details" page does not show the newly generated points: neither in the point history nor in the graph. I suspect that the point history is being generated but not stored in the database. I use this feature a lot, both when fine tuning formulas or when creating new meta datapoints. Can you please look into this ASAP?

Relevant modules: core 2.0.6, meta 1.4.0

On a related note, when I was using Mango 1.12.4, it seemed that I often had to purge all data before generating history. Does "generate history" always generate from Inception to Latest, or do I have to purge old data first? What time interval does it generate over if I already have points in the history?
Thanks.

Whether I create a new meta datapoint or edit an existing one, when I click on Generate History that icon blinks for a while, and then a pop-up window displays the number of points generated. The number of points generated seems to correspond to the "Update event" interval and the inception of the first input context point; the wait time also seems to correlate to the number of points generated. However, the "Point Details" page does not show the newly generated points: neither in the point history nor in the graph. I suspect that the point history is being generated but not stored in the database. I use this feature a lot, both when fine tuning formulas or when creating new meta datapoints. Can you please look into this ASAP?

Relevant modules: core 2.0.6, meta 1.4.0

On a related note, when I was using Mango 1.12.4, it seemed that I often had to purge all data before generating history. Does "generate history" always generate from Inception to Latest, or do I have to purge old data first? What time interval does it generate over if I already have points in the history?
Thanks.

I have an event detector that raises an alarm if the time stamp register value obtained from a data source does not change in more than one minute (I read that source every second, over Ethernet). Shortly after 3am almost every night, the following occurs:

1. The time stamp unchanged alarm is raised,
2. The CPU idle goes down
3. CPU I/O wait goes up.
4. Points to write goes way up
5. Disk block writes goes up, but disk block reads goes way up
6. Medium Priority Work Items shoots up from zero into the thousands

Questions:

1. What are these Medium Priority Work Items? Why is the peak number so different each night?
2. How can I minimize the load or quantity of that 3am thread?
3. Can Mango be modified so the Medium Priority Work Items are assigned a priority that is low enough to not interfere with data source reading?
4. Can the Medium Priority Work items be staggered so they're not all submitted at 3am?
5. Can Mango be modified so that reading a data source is assigned a higher priority thread?

Thanks

I have an event detector that raises an alarm if the time stamp register value obtained from a data source does not change in more than one minute (I read that source every second, over Ethernet). Shortly after 3am almost every night, the following occurs:

1. The time stamp unchanged alarm is raised,
2. The CPU idle goes down
3. CPU I/O wait goes up.
4. Points to write goes way up
5. Disk block writes goes up, but disk block reads goes way up
6. Medium Priority Work Items shoots up from zero into the thousands

Questions:

1. What are these Medium Priority Work Items? Why is the peak number so different each night?
2. How can I minimize the load or quantity of that 3am thread?
3. Can Mango be modified so the Medium Priority Work Items are assigned a priority that is low enough to not interfere with data source reading?
4. Can the Medium Priority Work items be staggered so they're not all submitted at 3am?
5. Can Mango be modified so that reading a data source is assigned a higher priority thread?

Thanks

I'm using the free Fail2ban (https://en.wikipedia.org/wiki/Fail2ban) software to ban brute force ssh login attemps, and it is banning 1-3 IP addresses every day.

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs -- too many password failures, seeking for exploits, etc. Generally Fail2Ban then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email, or ejecting CD-ROM tray) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, curier, ssh, etc).
http://www.fail2ban.org/
If you can implement the first feature:

• Write failed login attempts to a log file, including originating IP address
  ... then we can use [url=http://www.fail2ban.org/] fail2ban to watch the Mango log file for repeated login failures. Fail2ban will take care of banning the IP address if the number of failed login entries exceeds the allowed number. This would be a fast track to improved security.
  How about it?
  Source code at https://github.com/fail2ban/fail2ban (not that you would need it).

Thanks.

I must determine whether it is safe to grant write access to Internet Mango users without creating unnecessary risks. To implement this without adding too much danger, I must know whether common security features have been implemented.

Which of the security features below have been implemented in Mango Automation 2.x?

• Write failed login attempts to a log file, including originating IP address
• Ban IP addresses with repeated login failures to prevent brute force attacks
• Salt the passwords to prevent a break-in via a user with a weak password from making all other users vulnerable
• Sanitize inputs to prevent SQL back door attacks as illustrated in [url=http://xkcd.com/327/]this cartoon

Thanks

I must determine whether it is safe to grant write access to Internet Mango users without creating unnecessary risks. To implement this without adding too much danger, I must know whether common security features have been implemented.

Which of the security features below have been implemented in Mango Automation 2.x?

• Write failed login attempts to a log file, including originating IP address
• Ban IP addresses with repeated login failures to prevent brute force attacks
• Salt the passwords to prevent a break-in via a user with a weak password from making all other users vulnerable
• Sanitize inputs to prevent SQL back door attacks as illustrated in [url=http://xkcd.com/327/]this cartoon

Thanks

Thanks! Are you referring to preventing the zero insertion during import, or to adding an option to the datapoint's "purge now" box to purge points meeting specified value and date criteria?

The latter would be nice, especially since the "Discard extreme values" option is often not enabled until after invalid data has been stored in the point history. Additionally, enabling the discard function will prevent data problems from being detected and solved.

Either way, I need to find out what SQL queries to use to purge or to modify points that fall meet value and date/age criteria.

Thanks,
PM

The problem is more likely to be due to the interval logging than being a meta point. Can you confirm?
That datapoint is set to log the average on a 10 minute interval. However, it was set this way with M2M 1.12.3, and I don't recall this being a problem with that version. I can change to log all changes and test, but that would gather data very quickly, and I don't want to add that to the dataset, especially since they'll be hard to delete with my poor SQL skills.

You can use the sql console to delete point values. If you're unfamiliar with SQL i can provide some hints.
The scariest part of using SQL on a Mango database is not knowing whether making changes in one table must be accompanied by making changes to another table to avoid breaking things. I have little SQL experience, so it would help immensely if you could provide a specific example that deletes points that fall within a specific min-max value range, (1) for the entire dataset, (2) over a specified date range, and (3) over a specified age range. Getting that syntax just right in a Mango context would take me a long time.

It would also be helpful if you could give me an example that shows how to modify all points in a dataset (e.g. x=x+80, or y=y*10). This has come up when, for instance, I have modified a modbus additive or a modbus multiplier. I need to be able to make the datapoint history consistent with the change. I have scoured the forums looking for mango-specific SQL syntax in the past, but not found anything. It would be so handy if there were a mango wiki, or if SQL commands could be saved and applied to any point, like the report page. In the latter case, you could have some pre-saved example SQL commands on the SQL page that serve as examples of likely operations.

Before import, here is the point history. Note the 10 minute intervals.

History 	(as of 2013/01/14 19:10:25) 	Show most recent records Get data
Value 	Time 	Annotation
59 %V	19:01:13	
58 %V	18:51:13	
57 %V	18:41:13	
57 %V	18:31:13	
56 %V	18:21:13	
54 %V	18:11:13	
55 %V	18:01:13	
55 %V	17:51:13	
55 %V	17:41:13	
57 %V	17:31:13	

The data point definition is then exported from MA 2.0.3, pasted below, and re-imported into the same Mango installation without modification:

{
   "dataPoints":[
      {
         "loggingType":"INTERVAL",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"AVERAGE",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"NUMERIC",
            "updateEvent":"MINUTES",
            "context":[
               {
                  "dataPointXid":"DP_VPC1",
                  "varName":"VPC1"
               }
            ],
            "executionDelaySeconds":0,
            "script":"return (((VPC1.value) - 1.75) \/ (2.25 -1.7)) * 100;",
            "settable":false,
            "updateCronPattern":""
         },
         "eventDetectors":[
         ],
         "engineeringUnits":"Percent",
         "plotType":"LINE",
         "chartColour":"",
         "chartRenderer":{
            "type":"IMAGE",
            "timePeriodType":"WEEKS",
            "numberOfPeriods":1
         },
         "dataSourceXid":"DS_Battery",
         "defaultCacheSize":1,
         "deviceName":"Battery",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623158E30,
         "discardLowLimit":-1.7976931348623158E30,
         "enabled":true,
         "intervalLoggingPeriod":10,
         "name":"Battery % Voltage 1.75V-2.25V (HPC1)",
         "purgeOverride":true,
         "purgePeriod":1,
         "textRenderer":{
            "type":"ANALOG",
            "format":"##0",
            "suffix":" %V"
         },
         "tolerance":1.0,
         "xid":"CDP_HPC1Batt%volt"
      }
   ]
}

Import
Paste import data into the field below and click
Data point 'CDP_HPC1Batt%volt': saved
Import complete

Note the history after import, and the irregular time stamp:

History (as of 2013/01/14 19:14:27) Show most recent records Get data
Value Time Annotation
0 %V 19:13:49
63 %V 19:11:13
59 %V 19:01:13
58 %V 18:51:13
57 %V 18:41:13
57 %V 18:31:13
56 %V 18:21:13
54 %V 18:11:13
55 %V 18:01:13
55 %V 17:51:13

This is occurring with at least the meta-data points. It impacts a lot of points. Since it has occurred many times with many different datapoints, in addition to fixing the problem, I need to know how to delete the data from the point history.

Thanks.

I added the message directive as you suggested (line 6 below), but now all the email subject heading I receive are blank.

/opt/mango/ftl/html$ **cat active.ftl **

<#--
    Copyright (C) 2006-2011 Serotonin Software Technologies Inc. All rights reserved.
    @author Matthew Lohbihler
-->
<#include "include/eventHeader.ftl">
<@subject message=message/>
<table>
  <tr><td colspan="3" class="smallTitle">${instanceDescription} - <@fmt key="ftl.eventActive"/></td></tr>
  <#include "include/eventData.ftl">
  <#include "include/eventMessage.ftl">
</table>
<#include "include/eventFooter.ftl">

Can you please paste a tested 'active.ftl' file that adds both the alarm level and the message to the subject field?
How do you feel about making that the default subject heading template for 2.0.4?
Thanks

You write: > When i import a new enabled point into a running meta data source
The zero insertions occur when importing an existing datapoint definition into a running metadata source.

Aside from troubleshooting this problem, how do you recommend I purge point values that meet certain criteria?
The datapoint details purge section purges all points, and the "Discard extreme values" feature only works for new values, not for values already in the database. e.g. What SQL query can I use to delete points where a < x < b?

Can you add a routine to the datapoint "purge now" section that allows us to delete stored point values that fall between certain specified numbers within a specified date range, or older or newer than a specified age?

Thanks.

The invalid 0 point values are not coming from an SQL data source (I have no SQL data sources; mostly modbus data sources). The zero values are occurring in metadata points, and based on the graphs they have occurred during datapoint definition imports.

Great, thanks. I edited that file, now I just have to wait for alert emails to confirm that it works. Quick questions:

1. How can I add the alarm level to the message subject?
2. Why not make this the default email subject (more intuitive and convenient) in the next release?
3. Does changing a file under ftl/ require a Mango restart?
4. The greatest part of Mango is the context-sensitive help. Where will I find email customization instructions in the help?

It would be nice if there were a help search feature.
If you think email customization would clutter the help, then your help could be a one-liner:
"To customize the help, read <MANGO_DIR>/ftl/html/README"

Thanks again.

Great, thanks! Would that be 2.0.4? Where can I find the release notes or planned release date?

How can I get the event email message subject to show the "alarm level" and "event message" (or event alias) rather than the id number?
By message, I'm referring to whatever text is displayed on the Alarms page Message field:

Pending alarms
Acknowledge all Acknowledge all Silence all Silence all
Id Alarm level Time Message Inactive time
The problem is that all the subjects of emails sent via "User Details / Send alarm emails" currently look the same:
Mango event active notification - id:<some_number>
This is not informative, and I must open each email to find out what event occurred and what is the severity. It's too tedious to create an email event handler each time I add an event detector to a point; I would like the "Send alarm emails" setting in "User Details" to send the event "Alarm level" and "Message" in the subject field by default.

I already copied
/opt/mango/classes/i18n.properties to /opt/mango/overrides/properties/i18n.properties, but it is not obvious how I should change line 277-283.

ftl.subject.default=Mango event {0} notification
ftl.subject.default.id=Mango event {0} notification - id:{1}
ftl.subject.alias={0} ({1})
ftl.subject.alias.id={0} ({1}) - id:{2}
ftl.subject.active=active
ftl.subject.escalation=escalation
ftl.subject.inactive=inactive

Thanks.