How to change the http sending value parameter

phildunlap

There is no way to delete points through import/export functions. You can generate the SQL from a CSV really easily, though, just get the XID from a column in the CSV...

#!/bin/python

xidColumn = 3 #change this!
points = open("/path/to/points.csv")
outputFile = open("/path/to/output.sql", "w+")
for line in points.readlines() :
    data = line.replace("\r","").replace("\n","").split(",")
    outputFile.write("DELETE FROM dataPoints WHERE xid='"+data[xidColumn]+"';\n");
outputFile.close();

jmatos

@phildunlap

The project is now installed and working on the client. We are in the real test and pay attention to how it behaves.

To try workaround the issue on C) (c) - In the Virtual DATASOURCE there is one "update period" that makes the systems send the last alarm in that period time. Is there some way to deactivate it?) I set the update period to 300000h (about 34 years). Do you see any inconvenience on that?

Thank you very much

phildunlap

Yeah that's fine. What'll happen is every time the data source is enabled or disabled it will poll once, and then it'll be negligible load on the system.

In Mango 3 I added a feature to the virtual data source to disable its polling entirely.

jmatos

Hi Phildunlap,

As you know the solution is already installed at client facilities.

unfortunately I have cross with a "huge" problem that I can't figure out. :-(

When there is more than two (2) - say six - simultaneous alarm I can't get them all out on the http publisher. The alarms are registered in historic but only two reach the destination. It seems that they are lost between the publisher and from this to network. I've used wireshark and I dont see they getting out.

I've try to increase the cache size (is necessary to restart the all mango instance or a simple save is enough?)

May you please help me on this one?

Thanks in advance,
Jose

jmatos

@jmatos said in How to change the http sending value parameter:

Hi Phildunlap,

As you know the solution is already installed at client facilities.

unfortunately I have cross with a "huge" problem that I can't figure out. :-(

When there is more than two (2) - say six - simultaneous alarm I can't get them all out on the http publisher. The alarms are registered in historic but only two reach the destination. It seems that they are lost between the publisher and from this to network. I've used wireshark and I dont see they getting out.

I've try to increase the cache size (is necessary to restart the all mango instance or a simple save is enough?)

May you please help me on this one?

Thanks in advance,
Jose

Do you have any answer to me regarding this issue?

phildunlap

Hi jmatos,

I would wonder,

What is the update event for the publisher?
What is the logging type for the Alarm out point?
Are you using the NoSQL database?

My expectation is that the logging type for the alarm point is discarding some seemingly-backdated data in the concurrent update situation, or possible that the data is occurring at the same millisecond, and the NoSQL database can't store more than one value at one timestamp.

jmatos

Hi phildunlap

All
When point value changes
I think so (How can I be sure?)

More info: When I go to menu [Data Point Details] history all the alarm out are registered OK.

Here on lab I try an simulated an alarm out send through the http publisher to the same destination software with a virtual source generating a random number every 1ooms and all them reach the destination.

Tomorrow I'll go to the client and try this virtual send.

updating after visit the client:

Test results with real BACnet objects on client:
At two times I had to restart Mango and five of five(!) alarms get to the destination! But just on the startup... :-( All the test afterwards only get the "old same" two.

So, the alarms are registered at entry level and reported in the Data Point Details. But only two reach the destination.

I think you maybe right concerning the same millisecond, but in this case something has to explain the five-of-five that reach the destination on startup.

Is there some delay we can insert to workaround the possible same timestamp?

Important: As I told you before, I'm not sure what DB is in use. How can I be sure and if not using NoSQL how do I change it? the problem can be in here... :-(

Thanks in advance,
Jose

phildunlap

Hmm.

Before working on a workaround for the same milliseconds, let's try to confirm that's actually occurring (but maybe this would be the workaround). One thing you could try is putting a couple manual offsets in the point links' times, like:

TIMESTAMP = source.time+1; //one millisecond later than the source's TS
//you could use +2 on another one, and so on

It's possible on startup it registered a COV notification, got that point values (everything would have different timestamps), but for some reason reverted to polling those points. In which case, all new values would have the timestamp of the poll.

jmatos

Hi Phil,

Where do I put it? Sorry but I can figure it out. Probably because I have 1800 point links... I'm lost here. :-(

As instructed previously by you I put a Global script

function getPrefix() { /*So that you may change the prefix for all scripts later if required */
    return "4";
}

this is running on all point links. Is any way to add the time code here? Or am I confused? (its not the first time :D )

thank you

phildunlap

Wow 1800! Are they all to the same target point? At a certain number we may want to rethink the general tactic of wedging lots of points' alarms out one point. But, if it works, ok.

Off hand I can't think of a way to do what I was describing in the global script, but here's a python script to mass modify your point links. You probably want to keep a JSON backup of them, in case you wish to revert.

import json

configFile = open("/path/to/Mango/backup/Mango-Configuration.json")
config = json.load( configFile )
configFile.close()

x = 0
for pl in config["pointLinks"] :
  if pl["script"] != "" :
    pl["script"] = "TIMESTAMP=source.time+" + str(x) + ";\\n" + pl["script"]
    x += 1

outputFile = open("/path/to/output.json", "w+")
outputFile.write( json.dumps( {"pointLinks": config["pointLinks"]}, indent=4, sort_keys=False, separators=(",",": ") )
outputFile.close()

jmatos

I'm hopping to test the first code in six or 8 link points. That I can do one by one... ;-)

Thanks in advance for the second code.

I'll feedback to you.

Cheers

jmatos

@phildunlap said in How to change the http sending value parameter:

Wow 1800! Are they all to the same target point? At a certain number we may want to rethink the general tactic of wedging lots of points' alarms out one point. But, if it works, ok.

Off hand I can't think of a way to do what I was describing in the global script, but here's a python script to mass modify your point links. You probably want to keep a JSON backup of them, in case you wish to revert.
import json

configFile = open("/path/to/Mango/backup/Mango-Configuration.json")
config = json.load( configFile )
configFile.close()

x = 0
for pl in config["pointLinks"] :
  if pl["script"] != "" :
    pl["script"] = "TIMESTAMP=source.time+" + str(x) + ";\\n" + pl["script"]
    x += 1

outputFile = open("/path/to/output.json", "w+")
outputFile.write( json.dumps( {"pointLinks": config["pointLinks"]}, indent=4, sort_keys=False, separators=(",",": ")
outputFile.close()

Sorry. I get an invalid syntax at outputFile.close()

phildunlap

Whoops. Looks like there was a missing parenthesis. Fixed.

jmatos

@phildunlap said in How to change the http sending value parameter:

Whoops. Looks like there was a missing parenthesis. Fixed.

As the syntax error persists I try one more missed parenthesis. Your answer and Notepad++ made me see that conclusion. So 2 added.

outputFile.write( json.dumps( {"pointLinks": config["pointLinks"]}, indent=4, sort_keys=False, separators=(",",": ") ) )

Thank you

phildunlap

Oof, sorry. I've convinced myself I can write these little python things and not test them, but I really should have the preamble and output copied in so I can't do little things like that!

jmatos

Hi Phil,

I'm not figure it out... the code

TIMESTAMP = source.time+1; //one millisecond later than the source's TS
//you could use +2 on another one, and so on

Adds only a timestamp or in fact inserts a delay? Or both?

Thank you

phildunlap

The idea was to alter the timestamp set to the almnr point by some number of milliseconds into the future. They are varied because using the NoSQL database you cannot have two values at the same millisecond, and if these point links are triggered by one data source's polling (or just multiple points on some of the data sources), there's a high probability there are duplicate timestamps on the target point.

I'm not sure that's going to solve the issue you're describing, but it was something to try.

Edit: no delay should have been inserted.

Edit again, to explain:
TIMESTAMP is a variable name checked after the script returns its value, and if defined it is used as the timestamp for the value on the point set by the point link (or also the value assigned to a meta point, in that case)

jmatos

Hi Phil,

It does not work... :(

So I have cross my fingers and do a version upgrade to MANGO 3...

Now MANGO's not responding! (snif!). "ERR_CONNECTION_REFUSED" is all I get as browser response. The Webmin is working fine. Is there something else I can try before I restore the Mango directory ?

Thank you

phildunlap

Hi Jose,

I wouldn't expect the issue you're describing to be resolved by upgrading. But, you probably have an error in startup that we could see if you attempted to start Mango manually on the command line.

So navigate to Mango/bin and run either ./ma.sh start or ma-start.bat

Actually figuring out what's going on in your situation may be too difficult through the forum. I understand the issue to be something like...

Mango 1 polls a data source
Many points links set a point on Mango 1 many times.
The values of that point are published through an HTTP publisher
The receiving end is not ending up with the same set of data as the sending end, but no errors are being recorded on either Mango

There's a lot to explore there.

jmatos

here is the result

mango@mangoES3249:/opt/mango/bin$sudo ./ma.sh start
sudo: unable to resolve host mangoES3249
[sudo] password for mango:
sudo: ./ma.sh: command not found
mango@mangoES3249:/opt/mango/bin$ma.sh start
-bash: ma.sh: command not found
mango@mangoES3249:/opt/mango/bin$dir
ext-available  ext-enabled  h2-web-console.bat  h2-web-console.sh  ma-init.sh  ma.pid  ma-restart.sh  ma.sh  ma-start.bat  ma-start.sh  ma-stop.sh  upgrade.bat  upgrade.sh