Brilliant, thanks Phil. Appreciate your explanation. That second regex works perfectly!

Hi

I am attempting to use the HTTP retriever to 'scrape' a HTML page for a number of values/datapoints. The page contains a table that looks like this, and my objective is to capture all of the numerical values as separate datapoints:

<table>

	<td>Gewicht Volk 1</td>
        <td><b><p class="right">44.6</p></b></td>
        <td>kg</td>
        <td>Zu/Abnahme</td>          
                    <td><b><p class="right">44.6</p></b></td>
         <td>kg</td></tr>
	<tr>
		<td>Gewicht Volk 2</td>
		<td><b><p class="right">29.4</p></b></td>
        <td>kg</td>
        <td>ab 00.00 Uhr</td> 
                    <td><b><p class="right">29.4</p></b></td>
        <td>kg</td></tr>
	<tr>
		<td>Luftdruck</td>
		<td><b><p class="right">1015</p></b></td>
        <td>mbar</td>	
	</tr>
        <td>Temperatur Drucksensor</td> 
        <td><b><p class="right">1.7</p></b></td>
        <td>°C</td>
	<tr>
        <td>Temperatur</td>
        <td><b><p class="right">6.2</p></b></td>
        <td>°C</td>
        <td>Tagesmin.</td> 
        <td><b><p class="right">0.0</p></b></td>
        <td>°C</td>
        <td>, Tagesmax.</td> 
        <td><b><p class="right">0.0</p></b></td>
        <td>°C</td>
</tr>
        <td>Brutraumtemperatur</td>
        <td><b><p class="right">0.0</p></b></td>
        <td>°C</td>
</tr>
        <td>Regensensor</td>
        <td><b><p class="right">17</p></b></td>
        <td>mm</td>   
        <td>Tagesmenge</td> 
        <td><b><p class="right">0.0</p></b></td>
        <td>mm</td>
</tr>
        <td>Luftfeuchtigkeit</td>
        <td><b><p class="right">80.6</p></b></td>
        <td>%</td>
	<tr>
        <td>Akku</td>
        <td><b><p class="right">12.0</p></b></td>
        <td>V</td>
    </tr>
     <tr>
        <td>CSQ (Signalqualität Antenne)</td>
        <td><b><p class="right">-1</p></b></td>
        <td> </td>
	 </tr>

If I use a regex along the lines of (?<=<p class="right">)(.*)(?=<\/p) I can match the first value (44.6), but I cannot get the second value or nth value by incrementing the 'value capture group' value in the data point properties. Adding a {n} index to the end of the regex to get the nth match doesn't seem to help either.

If I use a regex like <td>Luftfeuchtigkeit<\/td> <td><b><p class="right">(.*?)< (the forum has stripped the additional whitespaces) or even (?<=Luftdruck<\/td> <td><b><p class="right">)(.*)(?=<\/p>) I don't get any matches at all.

Attempts to 'learn' regex have come up short, so I am limited to copying examples from others and messing about by trial and error. Any suggestions about how to accomplish this would be very much appreciated!

@hayden_aus Great to hear you got things sorted out! Never used Digi but have previously looked at their kit (before landing on Mikrotik, actually) and it seems like nice gear.

Great, thanks Joel – I'll check that out. Mango reports that the JVM free memory is 144 MB and JVM maximum available memory is 241 MB.

Hey hayden_AUS,

Hmm - 1.178.148.xxx is not a private network (it is publicly routable) so if that is the IP address your Dlink device is getting (and as you likely already know, you'd need to look at the actual address on the device's 4G interface - using a service like 'whatismyipaddress.com' won't show you if you are behind NAT), carrier NAT should not be an issue here.

Because you're able to access it from your mobile hotspot I don't think that port blocking from Virgin is the issue either. Based on what you've said so far, my guess would be that it's either some strange issue with the Dlink's firewall (in which case, I'd try disabling any firewall functionality completely), some issue with the configuration of your Dlink itself (port forwarding rules or similar) or with your Dynamic DNS service (either the Dynamic DNS provider or the resolver that you use at your office might be slow to update).

I would continue troubleshooting by first logging in to your router from your (working) hotspot setup, confirming the current 4G/WAN IP address, then probing/attempting to connect to that IP address directly - without using DNS - from another connection.

I am not working in the industry right now but I have previously used m2mone (sister company of m2mconnectivity) and can't recommend them highly enough.

The pricing is comparable to what you might pay as a retail customer of Telstra (but you have access to very cheap, low data plans - like a few hundred MB for $3 per month) but of course you have the additional benefit of being able to immediately get access to a support person who is an expert in the field.

As a m2mone customer you also get access to an extremely powerful and useful web portal that you can use to manage your fleet of SIMs, run diagnostic checks, and even send some commands to the cell tower.

It's a real no-brainer in my opinion.

For a remote m2m application like yours, I would also go straight past Dlink and select something ruggedised and industrial from m2mcommunications. The additional investment pays itself off after it saves you making one trip back to your site. The Maestro E228 is probably a good entry-level unit. I personally used many Routerboard RB912 units (with Sierra MC8705 3G module) and while you may need to do some fiddling to get it set up, you cannot beat the value.

If you want to directly message me some additional details (as much as you are comfortable sharing) I'd be happy to take a quick look in to your networking issue from my end. As you can probably tell, I'm interested in this stuff!

Hi hayden_AUS

I have some experience with running Mango units behind 3G services (Telstra) and accessing remotely and via VPN.

I have a really vague recollection of running in to the same update issue and I think that it was due to the DNS resolver being unavailable. I would perhaps check the DNS settings on the MangoES (it may not be getting the correct one automatically, or may have previously been configured with a DNS server that is not reachable from your 4G connection).

If that doesn't help, one other thing might be to make sure that the system clock is set correctly on the MangoES. If the date/time is off, this could break SSL connections and that might explain why the IA store is 'timing out'.

The other connectivity issues you're having (from your office) are mysterious. What equipment are you using for your 4G connection? Are you using an APN that gives you a public IP or are you behind carrier NAT? If you can share a bit more detail about the set up of your remote site and your office I might be able to throw around a couple of ideas.

Hi there,

I have a test install of Mango 3.1.1 running in an AWS t2.micro (free tier) instance, with 512MB memory. Approx every week or so Mango will crash because Java runs out of memory:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f65c452d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/mango/hs_err_pid25504.log

I am running Ubuntu 14.04.05 LTS with:

# JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)

Is there any suggested way I can configure Java (or Mango) so that it runs more happily on this server?

I should also add that I am running another application on this server (a PHP-based network monitor called LibreNMS) which I'm sure is not helping things.

Thanks!

Hi Mircea,

Is this for a MangoES? If so, the following link may help: https://wiki.debian.org/NetworkConfiguration#Setting_the_speed_and_duplex

As noted in that link, do be careful (particularly if this device is remote!) that you do not inadvertently lock yourself out of the device. Unstable link speeds can often be symptomatic of bad cabling, faulty termination, or intermittent noise and identifying and addressing the root cause may be called for.

One other thing to note is that many switches and other bits of network gear allow you to set a 'manual' speed by effectively disabling some rates, eg. you could disable Gigabit on the switch port and auto-negotiation would max out at 100Base-TX.

Hey Phil, the workaround didn't seem to work, but I got sidetracked with some other stuff and didn't try again.

I'm now running envcands 3.1.0 but don't see the date picker on the datasource page. Should it be there/is there anything special I need to do to see it?

Thanks Jared! I had somehow overlooked that in the examples - I will give that one a shot. Cheers!

Hi,

Is it currently possible (or will it be possible in the future) to graph a binary or multistate point (or points) similar to the following (from home-assistant.io)?

It looks like amcharts has a 'gantt' type chart which may achieve this, but I really like the way that it appears above.

Thanks!

This is really straightforward (once you know how!)

First, make sure your system is set up to send emails under System Settings > Email. For reliable sending of email, I strongly recommend that you use a service such as Sendgrid (free for up to 10k emails per month, I think).

Then, under your data point details, scroll down to the very bottom and under 'event detector', add a detector of type 'high limit' (and 'low limit', if you like). The fields should be self-explanatory, but check the 'use reset limit' box if you want to receive another alarm when the point returns to normal.

Save the point, and finally, browse to your 'event handlers' page, where you will see your point and alarm/s listed under 'point event detectors'. Here you can add recipients and customise the email further.

Best
Jeremy

@Balistar That BME280 module is really interesting! Includes a barometer and according to this article, is much more accurate than the DHT type sensors:

https://www.kandrsmith.org/RJS/Misc/Hygrometers/calib_many.html

@JoelHaggar said in ESP8266:

With your setup did you use the Mango MQTT Data Source

Yes; The ESP8266 is publishing temperature, humidity and an analogue value (from a light dependent resistor) to a simple free online broker (cloudmqtt.com) and then I configured the MQTT data source to subscribe to messages from the broker.

@bitreaper said in ESP8266:

I was curious, did you use the Arduino environment and the pubsubclient library? Also, which sensor did you use? The DHT-11?

Yes, I used the pubsubclient library and a DHT22 sensor. I just used a basic boilerplate sketch that I found online. I think that my code may need tweaking as it looks like it may create a new connection/socket each time it pushes an update but it's otherwise working fine.

I had hoped that MQTT, or at least the pubsubclient library, would include some sort of provision for store-and-forward or buffering in case of a dropped connection, but that doesn't seem to be the case. Functionally, it seems to offer few advantages over other protocols, but I have yet to really dig in to it. My next small project will be to drive a relay using MQTT and that will pose some interesting questions with respect to maintaining state, discarding old commands, etc.

I recently learned about the ESP8266 and have become quite fascinated. I was able to pick up a board locally for about $10 (but online, they are only a few dollars) and after connecting a simple temperature and humidity sensor was able to get it pushing sensor data over WiFi to Mango running in AWS (via a free online MQTT broker) in really no time.

Has anyone else been playing with these units and done anything interesting? As a standalone sensor, or remote IO, they are very interesting, but I wonder if anyone has come up with any really creative applications - like using them as a modbus-TCP to Serial gateway, or implementing some sort of logging and syncing with Mango?

Best
Jeremy

Hey, nice one. Thanks Phil. I'll investigate your work-around!

Hi,

Is the Enviro Canada data source still working? I am using the Envcands 3.0.1 module on a 3.0.2 system but am receiving no values for most of my datapoints, and a single value of 3.0 for the temperature data point (It's cold here, but not that cold).

I am using Station id 51442 which appears to have current data (link).

When I toggle the data source on and off, the log only shows the module stopping and starting, and doesn't show any errors.

Any ideas? Thanks!

Jeremy

After switching back to H2 and importing my data, is it OK to remove the entire /databases/mangoTSDB folder? (as well as the rolling backups in /backup/?)

Mango reported my NoSQL db was quite large so I'd like to clean it up. Thanks!

Perfect, thanks Joel.

Hi,

I've been using the evaluation NoSQL module 3.0.1 (on Mango 3.0.2) but would now like to disable NoSQL and return to storing point values in the H2 tables.

Is it OK to simply uninstall/delete the NoSQL module from the module admin page? And if so, will my point history be converted to H2, or would I need to export/import my data point values?

Thankyou!