Additional Data Source for UK Met Office weather using DataPoint API service

[bux236]

Hello,

I've recently been trying to pull weather data from the UK Met office API, whilst I can get some values in, things get complicated (for me any way!) when you start trying to look at the forecast weather as T for temperature occurs many times throughout the output.

Would it be possible to have a Data source built in to Mango, as with the Environment Canada data source, when you simply select your weather station ID, and then whether you would like today's values, tomorrows or the day after etc (free API gives a 3 day forecast)

If this isn't possible, could I get some advice on how to read the values please?

Here is a sample of the data received in XML (JSON also available)

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<SiteRep>
<Wx>
<Param name="F" units="C">Feels Like Temperature</Param>
<Param name="G" units="mph">Wind Gust</Param>
<Param name="H" units="%">Screen Relative Humidity</Param>
<Param name="T" units="C">Temperature</Param>
<Param name="V" units="">Visibility</Param>
<Param name="D" units="compass">Wind Direction</Param>
<Param name="S" units="mph">Wind Speed</Param>
<Param name="U" units="">Max UV Index</Param>
<Param name="W" units="">Weather Type</Param>
<Param name="Pp" units="%">Precipitation Probability</Param>
</Wx>
<DV dataDate="2017-11-05T09:00:00Z" type="Forecast">
<Location i="310006" lat="52.4065" lon="-1.5182" name="COVENTRY" country="ENGLAND" continent="EUROPE" elevation="79.0">
<Period type="Day" value="2017-11-05Z">
<Rep D="W" F="2" G="20" H="90" Pp="53" S="4" T="4" V="GO" W="9" U="0">360</Rep>
<Rep D="WSW" F="3" G="16" H="93" Pp="34" S="4" T="4" V="GO" W="10" U="1">540</Rep>
<Rep D="WNW" F="4" G="18" H="81" Pp="5" S="11" T="7" V="VG" W="7" U="1">720</Rep>
<Rep D="NW" F="5" G="18" H="74" Pp="0" S="11" T="8" V="VG" W="1" U="1">900</Rep>
<Rep D="NW" F="4" G="20" H="84" Pp="0" S="7" T="6" V="GO" W="0" U="0">1080</Rep>
<Rep D="NW" F="2" G="16" H="87" Pp="0" S="7" T="5" V="GO" W="0" U="0">1260</Rep>
</Period>
<Period type="Day" value="2017-11-06Z">
<Rep D="WNW" F="1" G="9" H="92" Pp="0" S="4" T="3" V="GO" W="0" U="0">0</Rep>
<Rep D="WSW" F="1" G="9" H="94" Pp="0" S="4" T="2" V="GO" W="0" U="0">180</Rep>
<Rep D="SW" F="0" G="9" H="94" Pp="1" S="4" T="2" V="GO" W="0" U="0">360</Rep>
<Rep D="SSW" F="2" G="13" H="90" Pp="0" S="4" T="4" V="GO" W="1" U="1">540</Rep>
<Rep D="SSW" F="7" G="13" H="76" Pp="1" S="7" T="9" V="VG" W="1" U="1">720</Rep>
<Rep D="SSW" F="8" G="16" H="70" Pp="5" S="9" T="9" V="VG" W="7" U="1">900</Rep>
<Rep D="S" F="6" G="20" H="79" Pp="5" S="9" T="8" V="VG" W="7" U="0">1080</Rep>
<Rep D="S" F="5" G="20" H="79" Pp="8" S="11" T="8" V="VG" W="7" U="0">1260</Rep>
</Period>
<Period type="Day" value="2017-11-07Z">
<Rep D="S" F="6" G="25" H="80" Pp="11" S="11" T="9" V="VG" W="8" U="0">0</Rep>
<Rep D="S" F="6" G="27" H="82" Pp="16" S="13" T="9" V="VG" W="8" U="0">180</Rep>
<Rep D="S" F="7" G="29" H="85" Pp="22" S="16" T="10" V="VG" W="8" U="0">360</Rep>
<Rep D="S" F="7" G="29" H="84" Pp="86" S="16" T="10" V="GO" W="15" U="1">540</Rep>
<Rep D="SSW" F="7" G="25" H="85" Pp="62" S="13" T="10" V="MO" W="12" U="1">720</Rep>
<Rep D="NW" F="5" G="25" H="85" Pp="39" S="13" T="8" V="MO" W="10" U="1">900</Rep>
<Rep D="WNW" F="4" G="16" H="84" Pp="12" S="9" T="7" V="VG" W="7" U="0">1080</Rep>
<Rep D="WNW" F="3" G="16" H="89" Pp="2" S="7" T="5" V="VG" W="0" U="0">1260</Rep>
</Period>
<Period type="Day" value="2017-11-08Z">
<Rep D="WNW" F="1" G="11" H="89" Pp="0" S="7" T="4" V="VG" W="0" U="0">0</Rep>
<Rep D="WNW" F="0" G="11" H="90" Pp="0" S="7" T="3" V="VG" W="0" U="0">180</Rep>
<Rep D="WNW" F="0" G="11" H="90" Pp="0" S="7" T="2" V="VG" W="0" U="0">360</Rep>
<Rep D="WNW" F="1" G="13" H="88" Pp="0" S="7" T="4" V="VG" W="1" U="1">540</Rep>
<Rep D="WNW" F="6" G="13" H="77" Pp="0" S="7" T="8" V="VG" W="1" U="1">720</Rep>
<Rep D="W" F="7" G="13" H="72" Pp="0" S="7" T="9" V="VG" W="1" U="1">900</Rep>
<Rep D="WSW" F="3" G="11" H="86" Pp="4" S="7" T="6" V="VG" W="7" U="0">1080</Rep>
<Rep D="SW" F="3" G="16" H="86" Pp="6" S="9" T="5" V="VG" W="7" U="0">1260</Rep>
</Period>
<Period type="Day" value="2017-11-09Z">
<Rep D="SW" F="4" G="20" H="87" Pp="14" S="11" T="7" V="GO" W="7" U="0">0</Rep>
<Rep D="WSW" F="5" G="22" H="92" Pp="16" S="13" T="8" V="GO" W="8" U="0">180</Rep>
<Rep D="WNW" F="6" G="22" H="87" Pp="28" S="13" T="9" V="VG" W="9" U="0">360</Rep>
<Rep D="WNW" F="5" G="25" H="78" Pp="5" S="13" T="8" V="VG" W="7" U="1">540</Rep>
<Rep D="WNW" F="7" G="31" H="70" Pp="7" S="16" T="10" V="VG" W="3" U="1">720</Rep>
<Rep D="WNW" F="6" G="29" H="69" Pp="6" S="16" T="9" V="VG" W="3" U="1">900</Rep>
<Rep D="WNW" F="5" G="22" H="77" Pp="2" S="11" T="8" V="VG" W="2" U="0">1080</Rep>
<Rep D="W" F="4" G="20" H="82" Pp="2" S="11" T="7" V="VG" W="2" U="0">1260</Rep>
</Period>
</Location>
</DV>
</SiteRep>```
//

Many Thanks,

Chris

[phildunlap]

Hi Chris,

It would be possible to write such a data source, probably. You didn't link to the API document. If you're asking if it's possible for us to write such a data source, this would undoubtably have a cost.

However, you probably don't need a full module to get what you're after. You can almost certainly get it done with a Scripting data source. Here's a script with some level of detail: https://forum.infiniteautomation.com/topic/2999/intesis-integration

The short of it would be that likely the JSON API will be easier to work with, as you could do something like,

HttpBuilder.get("https://ukmetofficeapiurl.extension/data", 
  {"AuthHeader": "Value"}, //or just {} if there are no custom headers
  {"ParameterKey": "Value", "stationID": stationIdContextPoint.value} //or hardcode, loop, etc
  ).err(function(status, headers, content) {
    throw content;
  }).resp(function(status, headers, content) {
    var response = JSON.parse(content);
    for(var k = 0; k < response.data.length; k++)
      someContextPoint.set(response.data[k].value, response.data[k].time);
  }).execute();

[bux236]

Hi Phil,

Thanks for your reply, here is a link to the API reference documentation.
I've also sent you a message with API keys in etc.

Would you be able to give me a start of a script to be able to acquire the following please?
Say for temperature "T="XX" for hour 0, 180, 360.... though to 1260, could
you show me how write each one to an individual data point please?

Many Thanks,
Chris

[phildunlap]

Hi Chris,

Because you're working with forecast data, you'll need to either increase the future dated discard to be past the five-day forecast (last field in Purge Settings, on the System Settings page) or use a virtual point to store the data and check the "Saves historic" checkbox on the Scripting data source.

I did it with a virtual point set to log all data and the script set to "Saves historic" with the script...

//For simplicity I am not going to do point creation in the script. You can see
// an example of that in the Intesis Integration thread I linked to

//The API key is a simple token, so we don't need to do any complex authentication

var headers = {}; //No custom headers needed
var parameters = {"res": "3hourly", "key": "your-private-api-key"};
var station = "12345";
var requestUrl = "http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/" + station;

HttpBuilder.get(requestUrl, headers, parameters)
    .err(function(status, headers, content) {
        throw content;
    }).resp(function(status, headers, content) {
        var response = JSON.parse(content);
        var periods = response.SiteRep.DV.Location.Period;
        for(var k = 0; k < periods.length; k+=1) {
            var period = periods[k];
            var baseDate = new Date(period.value);
            for(var j = 0; j < period.Rep.length; j+=1) {
                forecast.set(period.Rep[j].T, baseDate.getTime() + 
                    parseInt(period.Rep[j]["$"])*60000); //Convert to milliseconds offset
            }
        }
    }).execute();

And here's the JSON for the virtual source and the scripting source...

{
   "dataSources":[
      {
         "xid":"DS_MetOfficeWeatherScript",
         "name":"Met Office Weather Script",
         "enabled":false,
         "type":"SCRIPTING",
         "alarmLevels":{
            "SCRIPT_ERROR":"URGENT",
            "DATA_TYPE_ERROR":"URGENT",
            "POLL_ABORTED":"URGENT",
            "LOG_ERROR":"URGENT"
         },
         "purgeType":"YEARS",
         "updateEvent":"CONTEXT_UPDATE",
         "context":[
            {
               "varName":"forecast",
               "dataPointXid":"DP_244b7739-3f87-48cd-8590-fb7e11e8654b",
               "updateContext":false
            }
         ],
         "logLevel":"NONE",
         "cronPattern":"0 0 * * * ?",
         "executionDelaySeconds":0,
         "historicalSetting":true,
         "script":"\/\/For simplicity I am not going to do point creation in the script. You can see\n\/\/ an example of that in the Intesis Integration thread I linked to\n\n\/\/The API key is a simple token, so we don't need to do any complex authentication\n\nvar headers = {}; \/\/No custom headers needed\nvar parameters = {\"res\": \"3hourly\", \"key\": \"920895ce-6661-4131-b10f-7c169ea22542\"};\nvar station = \"310006\";\nvar requestUrl = \"http:\/\/datapoint.metoffice.gov.uk\/public\/data\/val\/wxfcs\/all\/json\/\" + station;\n\nHttpBuilder.get(requestUrl, headers, parameters)\n    .err(function(status, headers, content) {\n        throw content;\n    }).resp(function(status, headers, content) {\n        var response = JSON.parse(content);\n        var periods = response.SiteRep.DV.Location.Period;\n        for(var k = 0; k < periods.length; k+=1) {\n            var period = periods[k];\n            var baseDate = new Date(period.value);\n            for(var j = 0; j < period.Rep.length; j+=1) {\n                forecast.set(period.Rep[j].T, baseDate.getTime() + \n                    parseInt(period.Rep[j][\"$\"])*60000); \/\/Convert to milliseconds offset\n            }\n        }\n    }).execute();",
         "scriptPermissions":{
            "customPermissions":"",
            "dataPointReadPermissions":"superadmin",
            "dataPointSetPermissions":"superadmin",
            "dataSourcePermissions":"superadmin"
         },
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      },
      {
         "xid":"DS_Vrt_DataHouse",
         "name":"Data House",
         "enabled":true,
         "type":"VIRTUAL",
         "alarmLevels":{
            "POLL_ABORTED":"URGENT"
         },
         "purgeType":"YEARS",
         "updatePeriodType":"MINUTES",
         "polling":false,
         "updatePeriods":5,
         "editPermission":"",
         "purgeOverride":false,
         "purgePeriod":1
      }
   ],
   "dataPoints":[
      {
         "xid":"DP_244b7739-3f87-48cd-8590-fb7e11e8654b",
         "name":"Temperature Forecast",
         "enabled":true,
         "loggingType":"ALL",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"AVERAGE",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"NUMERIC",
            "changeType":{
               "type":"NO_CHANGE",
               "startValue":"0"
            },
            "settable":true
         },
         "eventDetectors":[
         ],
         "plotType":"SPLINE",
         "rollup":"NONE",
         "unit":"",
         "path":"",
         "chartColour":"",
         "chartRenderer":{
            "type":"IMAGE",
            "timePeriodType":"DAYS",
            "numberOfPeriods":1
         },
         "dataSourceXid":"DS_Vrt_DataHouse",
         "defaultCacheSize":1,
         "deviceName":"Data House",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":1,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "textRenderer":{
            "type":"ANALOG",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "format":"0.00"
         },
         "tolerance":0.0
      },
      {
         "xid":"DP_594c3747-9f29-4c5d-9a0e-e171a8cef10f",
         "name":"temperatureForecast",
         "enabled":true,
         "loggingType":"ALL",
         "intervalLoggingPeriodType":"MINUTES",
         "intervalLoggingType":"AVERAGE",
         "purgeType":"YEARS",
         "pointLocator":{
            "dataType":"NUMERIC",
            "contextUpdate":false,
            "settable":true,
            "varName":"temperatureForecast"
         },
         "eventDetectors":[
         ],
         "plotType":"SPLINE",
         "rollup":"NONE",
         "unit":"",
         "path":"",
         "chartColour":"",
         "chartRenderer":{
            "type":"IMAGE",
            "timePeriodType":"DAYS",
            "numberOfPeriods":1
         },
         "dataSourceXid":"DS_MetOfficeWeatherScript",
         "defaultCacheSize":1,
         "deviceName":"Met Office Weather Script",
         "discardExtremeValues":false,
         "discardHighLimit":1.7976931348623157E308,
         "discardLowLimit":-1.7976931348623157E308,
         "intervalLoggingPeriod":1,
         "intervalLoggingSampleWindowSize":0,
         "overrideIntervalLoggingSamples":false,
         "preventSetExtremeValues":false,
         "purgeOverride":false,
         "purgePeriod":1,
         "readPermission":"",
         "setExtremeHighLimit":1.7976931348623157E308,
         "setExtremeLowLimit":-1.7976931348623157E308,
         "setPermission":"",
         "textRenderer":{
            "type":"ANALOG",
            "useUnitAsSuffix":true,
            "unit":"",
            "renderedUnit":"",
            "format":"0.00"
         },
         "tolerance":0.0
      }
   ]
}

[bux236]

Hi Phil,

Thanks for taking the time to produce that for me.
I will have a look through and see what I can do

Many Thanks,

Chris

[phildunlap]

Certainly!

I noticed when I was writing this that the API endpoints for "latest point values" don't like returning future dated values (so the table on the data point details page may be mostly empty, for instance). This will probably be re-assessed. But, I was able to chart them without issue, or query on them otherwise if there was some analysis to do.

[Jared Wiltshire]

I'd also just suggest looking at just using the HTTP JSON retriever data source. This is assuming that you don't need to future date the values you retrieve.

[phildunlap]

The Http JSON retriever would work well if you wanted to collect a specific offset, Like, if instead of having a point that meant 'Temperature Forecast' which is what I tried to provide, the Http JSON retriever would likely provide points that meant "The forecasted temperature two days from now" which could then gain history and you could chart the projected temperature two days out and compare it to the projected temperature one or three days out over time. I'm not sure why you would, but this does sound like a fine angle for data analysis. It could be interesting to time-shift those graphs over one another and see how the projections narrow to the reality, or something.

The tradeoff would by that the Http JSON retriever doesn't support getting multiple values for the same point from the same message, so you couldn't chart predicted temperature as you could with what I provided.