@Pedro thanks for replying. As I recall, this reset didn't work for serial data sources so a manual reboot was needed. Also if radix bothered to answer these questions in the forum they wouldn't receive so many tickets.
Majority are how to questions I'd wager

Fox

@terrypacker thanks, but this is very unfortunate, as I am not thinking of "a much nicer page" in that I probably have 20 pages: most have dozens or hundreds of points. Converting by hand would be very tedious.

Is there a reference that explains how to recreate dashboard structures that are equivalent to existing structures used in the graphical view? For instance, the "Simple point" data structure normally shows the point value. On mouseover, a pop-up point links to the point details, and shows a pop-up default graph. Would I have to recreate that behavior in the dashboard from scratch?

Similarly, the "complex point" structure shows a table of Simple points, each with the Simple point mouseover behavior; binary points show pop-up default tables on mouseover. Would I have to write all that too, or are there equivalent data structures? Last time I looked at the dashboard a couple of years ago, I did not see a built-in way to replicate this behavior without going low-level. Even if there is an equivalent structure, mousing and clicking around to recreate each point would take ages because there are so many.

What is available to speed this conversion?

@phildunlap said in Times series database storage format, point arrays, and downsampling:

quite a lot going on there!

Yes, I was considering whether to split my post into different topics. Thank you for all your answers.

The POST /rest/v2/script/ run allows you to submit a script and get the results of its run.

That sounds very interesting. I just completed a python script using XSRF token. Nice: I like how it eliminated the need to login.

One could easily downsample their data via script, which I can provide a simple or efficient example of if desired.

If you already have an existing script, it would be nice if you could post it under its own forum topic, as I'm sure many of us would like to downsample old data. In addition to downsampling old data, I have numerous points that were logged much too often due to initially setting a log tolerance threshold that was too small. However, I won't have time to run this script right away due to my other project.

there is no data type that is "array of data type: numeric" and handled in Mango as such.

You understood me correctly. I'm looking to store an array of readings (as in multiple channels for each timestamp). Basically, a 2D numerical array where the rows are for different timestamps and the columns are the same type of data type but from different sources (channels). If it were stored in a CSV or spreadsheet, it would look like this:

timestamp t+0, channel[1],channel[2],channel[3],channel[4],channel[5],...,channel[1000] timestamp t+1, channel[1],channel[2],channel[3],channel[4],channel[5],...,channel[1000] timestamp t+2, channel[1],channel[2],channel[3],channel[4],channel[5],...,channel[1000] timestamp t+3, channel[1],channel[2],channel[3],channel[4],channel[5],...,channel[1000] timestamp t+4, channel[1],channel[2],channel[3],channel[4],channel[5],...,channel[1000] ...

It seems to me that in order to reduce data redundancy (by not storing the same timestamp multiple times) I could store the data in HDF5 format. HDF5 includes the metadata for the stored information, so the data can be retrieved into a meaningful format using generic tools, even without the source code that stored it. Additionally, it can efficiently compress and decompress binary data such as numerical arrays. My array elements could be any number of bytes. HDF5 is also extremely fast.

Summary Points - Benefits of HDF5

Self-Describing The datasets with an HDF5 file are self describing. This allows us to efficiently extract metadata without needing an additional metadata document. Supports Heterogeneous Data: Different types of datasets can be contained within one HDF5 file. Supports Large, Complex Data: HDF5 is a compressed format that is designed to support large, heterogeneous, and complex datasets. Supports Data Slicing: "Data slicing", or extracting portions of the dataset as needed for analysis, means large files don't need to be completely read into the computers memory or RAM. Open Format - wide support in the many tools: Because the HDF5 format is open, it is supported by a host of programming languages and tools, including open source languages like R and Python and open GIS tools like QGIS.

I also found TsTables, a PyTables wrapper that will enable storing time stamped arrays into HDF5 files in daily shards and seamlessly stitch them together during queries. Appends are also efficient. The HDF5 tools will also help for debugging whether any inconsistency is occurring during the read or write operation.

I would not expect a history generation to cause the issue again with the settings changed.

I see your point about the delay in shutdown. You can wait in the shutdown until it is reporting that it is waiting for batch write behind tasks to finish, and then use a kill -9 to stop Mango, this should not cause issues. It is possible to reflect out the counter and manually set it to something, but I think it would be better to restart Mango.

https://forum.infiniteautomation.com/topic/3991/release-notes-for-scripting-3-5-1

All the sweeter!

@phildunlap @craigweb

Thank you both. This time the error was mine. I thought the point was multistate, but I now realize it was numeric rendered as a range. So I exported its historic values, changed the point type, then imported the values back in. It works now.

The containsKey() example in the Mango Javascript help also works. In the past some of the examples did not work or were not consistent with new back-end functionality. I have not checked the other examples.

@joelhaggar Thanks for that the link to the manual upgrade. The upgrade worked, but there are still problems. The upgrade from 3.3.4 to 3.4.4 broke numerous metadata scripts (a problem someone else posted in https://forum.infiniteautomation.com/topic/3627/not-a-function). Also, the internal upgrade is still not working.

I just tried to upgrade the two modules listed on /ui/administration/modules/upgrade:

mangoNoSqlDatabase Time Series Data Store 3.4.0 3.4.1 MangoUI Provides an AngularJS user interface, 3.4.6 3.4.7

However, the automated download failed once again:

WARN 2018-07-11T20:40:26,229 (com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run:503) - Upgrade download error java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_131] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131] at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.9.jar:4.4.9] at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.9.jar:4.4.9] at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:206) ~[httpcore-4.4.9.jar:4.4.9] at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) ~[httpcore-4.4.9.jar:4.4.9] at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[httpclient-4.5.5.jar:4.5.5] at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:148) ~[httpclient-4.5.5.jar:4.5.5] at com.serotonin.io.StreamUtils.transfer(StreamUtils.java:34) ~[mango-3.4.4.jar:?] at com.serotonin.io.StreamUtils.transfer(StreamUtils.java:27) ~[mango-3.4.4.jar:?] at com.serotonin.web.http.HttpUtils4.execute(HttpUtils4.java:105) ~[mango-3.4.4.jar:?] at com.serotonin.m2m2.web.dwr.ModulesDwr$UpgradeDownloader.run(ModulesDwr.java:499) [mango-3.4.4.jar:?] at com.serotonin.timer.Task.runTask(Task.java:179) [mango-3.4.4.jar:?] at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) [mango-3.4.4.jar:?] at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:336) [mango-3.4.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

The ping time from my Mango server to infiniteautomation.com is close to 700ms. Perhaps there is a timeout in the Mango download function that is too strict to meet that loop time?

Hi Pedro,

Which System Setting changes can eliminate this error?

You could slow down the point data mover and status provider by increasing the "Batch process manager period (ms)" and the "Status update period (ms) " in the NoSQL section of the system settings. Also, if you are fully up to date, these messages have been removed entirely since they're mostly clutter.

What symptoms would this error lead to?

None. This would be more indicative that a garbage collection lasted longer than the intervals specified for the two tasks in part 1 of this response.

We are sometimes not receiving emails of critical events. Could this be the cause?

No. You should have an error in your log file about being unable to send an email if the error was known before the actual relay of the mail was handed on to the SMTP server. If you do not have an error about sending a test email when you try to send a test email, then either your SMTP provider isn't forwarding it or it's being filtered out of your inbox.

Thanks. The charts are working after I deleted those files and restarted.

BACnet is working now:

I upgraded the eGauge firmware and set each eGauge broadcast mask back to what I had before the V3 upgrade (192.200.255.255) per eGauge recommendations. I did not change the netmask. eGauge does not think the firmware upgrade would affect the BACnet discovery. Nonetheless, Mango subsequently found the eGauges via BACnet WhoIs, but the old BACnet points were still not updating. Mango allowed me to add new BACnet points from the eGauges, and the new (redundant) BACnet point updated, but none of the old points updated.

Disabling/enabling points, or disabling/enabling the BACnet datasoure, also did not cause the old BACnet points to update. The old BACnet points finally started updating after I saved the BACnet datasource, even though I had not made any changes to it. The only LAN switch/router configuration settings changed for troubleshooting were "IGMP snooping" and "Flood unknown multicast traffic," but they were subsequently re-enabled, and BACnet WhoIs still works. Neither LAN setting had any effect on BACnet.

Thanks for your help.

@phildunlap Phillip, Thanks for your help. I added the Dashboard module via the Module menu, with restart selected. After a restart, I logged in and the graphical view and the point detail pages rendered properly. The graphical view did not show the point values at first. Mango is usually slow to render values after a restart. The values rendered after I deleted /opt/mango/work/jsp, but that may have been coincidence.

Since I had not used Dashboard, I never expected its removal to break those two views.

Thanks. I would like to move the devices to separate data sources, provided the point value history can be moved with it. I'm glad to hear that I can cross that bridge some time after the upgrade.

The bug was in the Meta module. I didn't put it in the store yet, nor did I backport to the 2.8 Meta module.

Glad to hear it! But, I think you may wish to get rid of the LOG.debug portion. print() doesn't return anything so you're logging undefined. My script looks like this:

LOG.debug(print("test"));

which produces the output:

test
DEBUG 2017-12-21 09:39:40,795 - undefined

Hi Pedro,

This was by design to keep compatibility with the old permissions system on the graphical views, I believe. The difference in code is simply adding

Permissions.ensureDataPointSetPermission(user, point);

at GraphicalViewDwr::372

Thanks, that's a good idea. I created and saved a new view. The point still renders as a grey Lego block with a yellow triangle to its left. A mouseover of the yellow triangle still displays "The point for this component is missing or invalid" as it did in my other view described above.

When I mouseover the grey block, I see the valid point rendered as text, displaying the expected Multistate text string, time stamp, set dialog, and table history.

Thank you for your advice. I got this working last week, but as you indicate, it requires tree structures in Mango 2.8.8:

one metadata point used for creating a context change from user input one metadata point to detect the context change then execute the script one scripting datasource to set multiple points each time the context of the first point changes

I'm glad to hear this has been simplified in Mango 3.x.

@phildunlap No: user id = 53. The blank fields for that user are homeUrl and timezone. All other fields are filled in.