• jeremyh

    Brilliant, thanks Phil. Appreciate your explanation. That second regex works perfectly!

    posted in User help read more
  • jeremyh

    Hi

    I am attempting to use the HTTP retriever to 'scrape' a HTML page for a number of values/datapoints. The page contains a table that looks like this, and my objective is to capture all of the numerical values as separate datapoints:

    <table>
    
    	<td>Gewicht Volk 1</td>
            <td><b><p class="right">44.6</p></b></td>
            <td>kg</td>
            <td>Zu/Abnahme</td>          
                        <td><b><p class="right">44.6</p></b></td>
             <td>kg</td></tr>
    	<tr>
    		<td>Gewicht Volk 2</td>
    		<td><b><p class="right">29.4</p></b></td>
            <td>kg</td>
            <td>ab 00.00 Uhr</td> 
                        <td><b><p class="right">29.4</p></b></td>
            <td>kg</td></tr>
    	<tr>
    		<td>Luftdruck</td>
    		<td><b><p class="right">1015</p></b></td>
            <td>mbar</td>	
    	</tr>
            <td>Temperatur Drucksensor</td> 
            <td><b><p class="right">1.7</p></b></td>
            <td>°C</td>
    	<tr>
            <td>Temperatur</td>
            <td><b><p class="right">6.2</p></b></td>
            <td>°C</td>
            <td>Tagesmin.</td> 
            <td><b><p class="right">0.0</p></b></td>
            <td>°C</td>
            <td>, Tagesmax.</td> 
            <td><b><p class="right">0.0</p></b></td>
            <td>°C</td>
    </tr>
            <td>Brutraumtemperatur</td>
            <td><b><p class="right">0.0</p></b></td>
            <td>°C</td>
    </tr>
            <td>Regensensor</td>
            <td><b><p class="right">17</p></b></td>
            <td>mm</td>   
            <td>Tagesmenge</td> 
            <td><b><p class="right">0.0</p></b></td>
            <td>mm</td>
    </tr>
            <td>Luftfeuchtigkeit</td>
            <td><b><p class="right">80.6</p></b></td>
            <td>%</td>
    	<tr>
            <td>Akku</td>
            <td><b><p class="right">12.0</p></b></td>
            <td>V</td>
        </tr>
         <tr>
            <td>CSQ (Signalqualität Antenne)</td>
            <td><b><p class="right">-1</p></b></td>
            <td> </td>
    	 </tr>
    

    If I use a regex along the lines of (?<=<p class="right">)(.*)(?=<\/p) I can match the first value (44.6), but I cannot get the second value or nth value by incrementing the 'value capture group' value in the data point properties. Adding a {n} index to the end of the regex to get the nth match doesn't seem to help either.

    If I use a regex like <td>Luftfeuchtigkeit<\/td> <td><b><p class="right">(.*?)< (the forum has stripped the additional whitespaces) or even (?<=Luftdruck<\/td> <td><b><p class="right">)(.*)(?=<\/p>) I don't get any matches at all.

    Attempts to 'learn' regex have come up short, so I am limited to copying examples from others and messing about by trial and error. Any suggestions about how to accomplish this would be very much appreciated!

    posted in User help read more
  • jeremyh

    @hayden_aus Great to hear you got things sorted out! Never used Digi but have previously looked at their kit (before landing on Mikrotik, actually) and it seems like nice gear.

    posted in MangoES Hardware read more
  • jeremyh

    Great, thanks Joel – I'll check that out. Mango reports that the JVM free memory is 144 MB and JVM maximum available memory is 241 MB.

    posted in Hardware read more
  • jeremyh

    Hey hayden_AUS,

    Hmm - 1.178.148.xxx is not a private network (it is publicly routable) so if that is the IP address your Dlink device is getting (and as you likely already know, you'd need to look at the actual address on the device's 4G interface - using a service like 'whatismyipaddress.com' won't show you if you are behind NAT), carrier NAT should not be an issue here.

    Because you're able to access it from your mobile hotspot I don't think that port blocking from Virgin is the issue either. Based on what you've said so far, my guess would be that it's either some strange issue with the Dlink's firewall (in which case, I'd try disabling any firewall functionality completely), some issue with the configuration of your Dlink itself (port forwarding rules or similar) or with your Dynamic DNS service (either the Dynamic DNS provider or the resolver that you use at your office might be slow to update).

    I would continue troubleshooting by first logging in to your router from your (working) hotspot setup, confirming the current 4G/WAN IP address, then probing/attempting to connect to that IP address directly - without using DNS - from another connection.

    I am not working in the industry right now but I have previously used m2mone (sister company of m2mconnectivity) and can't recommend them highly enough.

    The pricing is comparable to what you might pay as a retail customer of Telstra (but you have access to very cheap, low data plans - like a few hundred MB for $3 per month) but of course you have the additional benefit of being able to immediately get access to a support person who is an expert in the field.

    As a m2mone customer you also get access to an extremely powerful and useful web portal that you can use to manage your fleet of SIMs, run diagnostic checks, and even send some commands to the cell tower.

    It's a real no-brainer in my opinion.

    For a remote m2m application like yours, I would also go straight past Dlink and select something ruggedised and industrial from m2mcommunications. The additional investment pays itself off after it saves you making one trip back to your site. The Maestro E228 is probably a good entry-level unit. I personally used many Routerboard RB912 units (with Sierra MC8705 3G module) and while you may need to do some fiddling to get it set up, you cannot beat the value.

    If you want to directly message me some additional details (as much as you are comfortable sharing) I'd be happy to take a quick look in to your networking issue from my end. As you can probably tell, I'm interested in this stuff!

    posted in MangoES Hardware read more
  • jeremyh

    Hi hayden_AUS

    I have some experience with running Mango units behind 3G services (Telstra) and accessing remotely and via VPN.

    I have a really vague recollection of running in to the same update issue and I think that it was due to the DNS resolver being unavailable. I would perhaps check the DNS settings on the MangoES (it may not be getting the correct one automatically, or may have previously been configured with a DNS server that is not reachable from your 4G connection).

    If that doesn't help, one other thing might be to make sure that the system clock is set correctly on the MangoES. If the date/time is off, this could break SSL connections and that might explain why the IA store is 'timing out'.

    The other connectivity issues you're having (from your office) are mysterious. What equipment are you using for your 4G connection? Are you using an APN that gives you a public IP or are you behind carrier NAT? If you can share a bit more detail about the set up of your remote site and your office I might be able to throw around a couple of ideas.

    posted in MangoES Hardware read more
  • jeremyh

    Hi there,

    I have a test install of Mango 3.1.1 running in an AWS t2.micro (free tier) instance, with 512MB memory. Approx every week or so Mango will crash because Java runs out of memory:

    Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f65c452d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
    # An error report file with more information is saved as:
    # /opt/mango/hs_err_pid25504.log
    

    I am running Ubuntu 14.04.05 LTS with:

    # JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
    

    Is there any suggested way I can configure Java (or Mango) so that it runs more happily on this server?

    I should also add that I am running another application on this server (a PHP-based network monitor called LibreNMS) which I'm sure is not helping things.

    Thanks!

    posted in Hardware read more
  • jeremyh

    Hi Mircea,

    Is this for a MangoES? If so, the following link may help: https://wiki.debian.org/NetworkConfiguration#Setting_the_speed_and_duplex

    As noted in that link, do be careful (particularly if this device is remote!) that you do not inadvertently lock yourself out of the device. Unstable link speeds can often be symptomatic of bad cabling, faulty termination, or intermittent noise and identifying and addressing the root cause may be called for.

    One other thing to note is that many switches and other bits of network gear allow you to set a 'manual' speed by effectively disabling some rates, eg. you could disable Gigabit on the switch port and auto-negotiation would max out at 100Base-TX.

    posted in User help read more
  • jeremyh

    Hey Phil, the workaround didn't seem to work, but I got sidetracked with some other stuff and didn't try again.

    I'm now running envcands 3.1.0 but don't see the date picker on the datasource page. Should it be there/is there anything special I need to do to see it?

    posted in User help read more
  • jeremyh

    Thanks Jared! I had somehow overlooked that in the examples - I will give that one a shot. Cheers!

    posted in User help read more