Help needed (newbie) Accessing an API with a Scripting Datasource Mango 4.4.2

[raylatbasix]

If someone could point me in the right direction getting the following sample code working as a scripting datasource in Mango 4, it would be appreciated. I have the need to connect to DegreeDays.net API so I can import a history of Heating Degree Days into a Mango Datapoint. Here is the following sample code from the DegreeDays.net API developer's page:

const crypto = require('crypto'), // built-in module, no need to npm install
  fetch = require('node-fetch'); // install with npm install node-fetch

// The test API access keys are described at www.degreedays.net/api/test
// They will let you access data from the Cape Cod area only.
// To fetch data from locations worldwide, sign up for a proper API account at
// www.degreedays.net/api/ and copy your API access keys here.
const accountKey = 'test-test-test';
const securityKey = 'test-test-test-test-test-test-test-test-test-test-test-test-test';
// You can call the API over HTTP using http://apiv1.degreedays.net/json or
// over HTTPS using https://apiv1.degreedays.net/json - set the endpoint URL
// below as appropriate.
const endpoint = 'http://apiv1.degreedays.net/json';



// ************* STEP 1: Create the request ************************************
// First we create a JSON request that specifies what we want from the API.
// See www.degreedays.net/api/json#request for more on this.

// You can fetch data from a station ID, a longitude/latitude position, or a 
// postal/zip code.
const location = {
  type: 'PostalCodeLocation',
  postalCode: '02532',
  countryCode: 'US'
};
// In this example we fetch both HDD and CDD, using the same breakdown (daily
// data covering the last 7 days) for both. For more breakdown options see
// www.degreedays.net/api/json#breakdown
const breakdown = {
  type: 'DailyBreakdown',
  period: {
    type: 'LatestValuesPeriod',
    numberOfValues: 7
  }
};
const locationDataRequest = {
  type: 'LocationDataRequest',
  location: location,
  dataSpecs: {
    // Here we specify 2 DataSpec items: one for HDD and one for CDD.  You
    // can specify up to 100 DataSpec items in one request (e.g. to fetch
    // data in lots of base temperatures).  With an API Standard+ account you
    // can have a DataSpec for hourly temperature data too.
    // Give each DataSpec a unique name so you can get the corresponding
    // DataSet from the response.
    myHDD: {
      type: 'DatedDataSpec',
      calculation: {
        type: 'HeatingDegreeDaysCalculation',
        baseTemperature: {
          unit: 'F',
          value: 60
        }
      },
      breakdown: breakdown
    },
    myCDD: {
      type: 'DatedDataSpec',
      calculation: {
        type: 'CoolingDegreeDaysCalculation',
        baseTemperature: {
          unit: 'F',
          value: 70
        }
      },
      breakdown: breakdown
    }
  }
};
const fullRequest = {
  securityInfo: {
    endpoint: endpoint,
    accountKey: accountKey,
    timestamp: new Date().toISOString(),
    random: Buffer.from(crypto.randomBytes(12)).toString('hex')
  },
  request: locationDataRequest
};
const fullRequestJson = JSON.stringify(fullRequest);
// Now our JSON request is ready.  Uncomment the line below to log the JSON:
//console.log(fullRequestJson);



// ************* STEP 2: Send the request to the API ***************************
// Next we sign the JSON request and package everything together into an HTTP
// request which we send to the Degree Days.net API.  This follows the spec at
// www.degreedays.net/api/json#send
const signatureBytes = 
  crypto.createHmac('sha256', securityKey).update(fullRequestJson).digest();
// The API requires the JSON request and the signature to be base64url encoded.
function base64urlEncode(unencoded) {
  return Buffer.from(unencoded).toString('base64')
    .replace(/=/g, '').replace(/\+/g, '-').replace(/\//g, '_');
}
// Send the HTTP request to the API servers using node-fetch.  You could change
// this to use another HTTP library if you wanted.
fetch(endpoint, {
    method: 'POST',
    body: 'request_encoding=base64url' +
      '&signature_method=HmacSHA256' +
      '&signature_encoding=base64url' +
      '&encoded_request=' + base64urlEncode(fullRequestJson) +
      '&encoded_signature=' + base64urlEncode(signatureBytes),
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
    }
  })
  .then(response => {
    if (!response.ok) {
      throw new Error(response.status + ': ' + response.statusText);
    }
    return response.json();
  })
  .catch(err => console.error('Problem connecting to API:', err)
  .then(parsedObject => handleResponse(parsedObject))
  .catch(err => console.error('Problem handling API response:', err));



// ************* STEP 3: Process the response from the API *********************
// The JSON response is explained at www.degreedays.net/api/json#response
function handleResponse(fullResponse) {
  // fullResponse.metadata has some rate limit info, but we are mainly
  // interested in fullResponse.response (a LocationDataResponse in this case).
  const response = fullResponse.response;
  if (response.type === 'Failure') {
    // See www.degreedays.net/api/json#failure for more about failures.
    console.log('Request failure:', response);
  } else {
    // The response contains a lot of useful info, as shown in the JSON docs
    // at www.degreedays.net/api/json#response
    console.log('Station ID: ', response.stationId);
    // "myHDD" is the name we gave the HDD DataSpec in our request.
    const hddData = response.dataSets.myHDD;
    if (hddData.type === 'Failure') {
      console.log('Failure for HDD DataSet:', hddData);
    } else {
      console.log('HDD:');
      for (var v of hddData.values) {
        console.log(v.d + ': ' + v.v);
      }
    }
    const cddData = response.dataSets.myCDD;
    if (hddData.type === 'Failure') {
      console.log('Failure for CDD DataSet:', cddData);
    } else {
      console.log('CDD:');
      for (var v of cddData.values) {
        console.log(v.d + ': ' + v.v);
      }
    }
  }
}

I appreciate the help. Thanks.

[MattFox]

@raylatbasix I can confidently say first off, if you can go with nodejs and their example, do so. The code will be more up to date and more likely to have less security vulnerabilities.
However, if you must use mango alone, then create a global script and paste the contents of this library into it:
https://simplecrypto.js.org/
It must be version 2.5, that's the last Vanilla js version available.
After that if you refer to other posts in the forum you will be able to see how http requests are coded and be able to write something that fulfills the requirements of your api.
The library mentioned above should have the sha256 hashing tool you need to encrypt your calls.

Hope this is a good start.

Fox

[raylatbasix]

@MattFox Thanks for the info Fox! Being new to this, If I install NodeJS on the same Windows installation that Mango 4 resides on, will I be able to write to a Mango datapoint through the Mango Rest API? I've searched the forum, and see NodeJS referred to quite a bit, but not being a web developer, I'm afraid I would need a set of instructions on how to implement that with Mango.

I've got other Mango scripting data sources grabbing weather data from an outside API and successfully writing to Mango data points just by finding resources on this Forum of which you seem to contribute quite a bit to. Thanks for that!

I've also tried copy/pasting the simplecrypto.js you mentioned into a Global Script but it will not validate. I'll keep trying things on my own, but if you have any further pointers they are much appreciated. Thank you for your time!

[MattFox]

@raylatbasix rather than writing to the api, why not keep it simple and use the http json receiver data source?
You can definitely run node on the same box as mango. I've done it for making mango mqtt publisher-ready.
I'll click into my mango instance tonight after work and will see if I can get this crypto library playing ball. If so, happy to work with you towards a solution.
Otherwise PM me and we can arrange something to help you get things over the line

Fox

[MattFox]

Managed to access a remote unit...
Yep, things don't seem to work. NodeJS club it is!

Fox

[raylatbasix]

@MattFox Thank you for trying Fox! It's the end of my day here in the US, but I will plan on getting in touch with you for some NodeJS help via PM soon! Thanks again.

[MattFox]

@raylatbasix Jammy, I'll be here

[MattFox]

After working with Ray, I threw together a repository with a script that allows either CLI or mango to fire data into mango 4 by stating the xid, user, pass/bearer, host, and port.

https://github.com/riagominota/degreedays-mangoclient
This repo however, can be rejigged to work with other third party api sources you like.

After that, Ray has kindly sent back his config here for future people to use as a guideline for their own work:

MS Windows environment NodeJS Mango Client Steps for DegreeDays.net NodeJS API:
On your PC where Mango is installed:
Install Node.js 18.12.1 (x64) which includes npm:
https://nodejs.org/dist/v18.12.1/node-v18.12.1-x64.msi

---

The Following are Optional for NodeJS programming development environment purposes:

	Install Visual Studio Code (optional):
 			https://code.visualstudio.com/sha/download?build=stable&os=win32-x64-user
 
	Install WSL Subsystem for Linux (optional):
 		https://learn.microsoft.com/en-us/windows/wsl/
 		From Windows PowerShell: wsl --install
 
	Install WSL Visual Studio code Extension (optional):
 			https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl

• Create a new Empty Project folder, then from a command prompt CD into that folder,
  (Important!) Run: "npm init" without quotes to initialize a new nodeJS package.json,
  and then run the following npm installs from the command line in that folder:

   npm install mango-client
   
   npm install luxon
   
   npm install form-data
   
   npm install node-fetch@2

• In the same project folder, create a new file called "ddMango.js", and place the following code in the file, then save:

//Begin Code *********************************************************
/**
 * Date: 2/1/23 - DD/MM/YYYY
 * Author: Matt 'Fox' Fox
 * Description: Logs into mango, and fires data into system for a given datapoint.
 * Call the file by running `node ddMango.js --xid=... --user=... --pass=...` - ideal for evented calls
 * Optionally, host and port can be added if not localhost or 8080 etc.
 */
const process = require('process');
const  MangoClient  = require('mango-client');
const { DateTime } = require("luxon");

const REQ_ARG_LIST = ["xid","user","pass"];
const ARG_LIST = ["bearer","host","port"]; // A bearer token can be used instead if desired

let ARG_VALS = {xid:"",user:"",pass:"",bearer:"",host:"localhost",port:8080};

// Args are provided in the form of --xid=... --user=... --pass=...
//This means we can call this from mango itself
const hasArgVal = key => {
    // Return true if the key exists and a value is defined
    for( let str of process.argv)
        if ( str.includes( `--${ key }` ) ) return true;

    return false;
}

const getArgVal = arg => {

    let val = null;
    if( hasArgVal(arg) ){
      val = process.argv.find( element => element.startsWith( `--${arg}=` ) );
      val = val.replace( `--${arg}=` , '' );
    }

    // Return null if the key does not exist and a value is not defined
    return val;
}

 // Ensure all args are properly populated before firing the script...
const VerifyArgs = () =>{
    for(let k in ARG_LIST)
    {
        if( hasArgVal(ARG_LIST[k]) )
        {
            ARG_VALS[ ARG_LIST[k] ] = getArgVal(ARG_LIST[k]);
        }
    }

    for(let k in REQ_ARG_LIST)
    {
        if( hasArgVal(REQ_ARG_LIST[k]) )
        {
            ARG_VALS[ REQ_ARG_LIST[k] ] = getArgVal(REQ_ARG_LIST[k]);
        }
        else
        {
            if(REQ_ARG_LIST[k]=="pass" && ARG_VALS.bearer!==""){continue;}

            console.log(`ERROR: argument --${REQ_ARG_LIST[k]} has not been passed as an argument to this script. Exiting.`);
            process.exit(1);
        }
    }
}

VerifyArgs();

// Prepare new Mango Client Instance to receive ARG_VALS
const MangoClientInstance = new MangoClient({...ARG_VALS});
if(ARG_VALS.bearer!=="")
    MangoClientInstance.setBearerAuthentication(ARG_VALS.user,ARG_VALS.bearer);
else
    MangoClientInstance.setBasicAuthentication(ARG_VALS.user,ARG_VALS.pass);

    // Save point values Array of:
    //  {xid,value,dataType,timestamp,annotation}
    // to Mango Client instance.
const insertValuesV4 = (xid,values) => {
    return MangoClientInstance.restRequest({
        path: `/rest/latest/point-values/${xid}`,
        method: 'PUT',
        data: values
    }).then(response => {
        return response.data;
    });
}

// Get degree days data here... then let's send it to mango.
// You will need to parse the date into a timestamp using luxon
const crypto = require('crypto'), // built-in module, no need to npm install
  fetch = require('node-fetch'); // install with npm install node-fetch@2

//Test DegreeDays API account info for Cape Cod MA, will need your own DegreeDays.net Account
const accountKey = 'test-test-test';
const securityKey = 'test-test-test-test-test-test-test-test-test-test-test-test-test';
const endpoint = 'http://apiv1.degreedays.net/json';

const location = {
  type: 'PostalCodeLocation',
 //Cape Cod MA Test API zip code
  postalCode: '02532',
  countryCode: 'US'
};

const breakdown = {
  type: 'DailyBreakdown',
  period: {
    type: 'LatestValuesPeriod',
    numberOfValues: 1 //This number (days) can be increased for a larger set of historic data.
  }
};
const locationDataRequest = {
  type: 'LocationDataRequest',
  location: location,
  dataSpecs: {
    myHDD: {
      type: 'DatedDataSpec',
      calculation: {
        type: 'HeatingDegreeDaysCalculation',
        baseTemperature: {
          unit: 'F',
          value: 65
        }
      },
      breakdown: breakdown
    }
  }
};
const fullRequest = {
  securityInfo: {
    endpoint: endpoint,
    accountKey: accountKey,
    timestamp: new Date().toISOString(),
    random: Buffer.from(crypto.randomBytes(12)).toString('hex')
  },
  request: locationDataRequest
};
const fullRequestJson = JSON.stringify(fullRequest);

const signatureBytes = 
  crypto.createHmac('sha256', securityKey).update(fullRequestJson).digest();
function base64urlEncode(unencoded) {
  return Buffer.from(unencoded).toString('base64')
    .replace(/=/g, '').replace(/\+/g, '-').replace(/\//g, '_');
}

fetch(endpoint, {
    method: 'POST',
    body: 'request_encoding=base64url' +
      '&signature_method=HmacSHA256' +
      '&signature_encoding=base64url' +
      '&encoded_request=' + base64urlEncode(fullRequestJson) +
      '&encoded_signature=' + base64urlEncode(signatureBytes),
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
    }
  })
  .then(response => {
    if (!response.ok) {
      throw new Error(response.status + ': ' + response.statusText);
    }
    return response.json();
  })
  .catch(err => console.error('Problem connecting to API:', err))
  .then(parsedObject => handleResponse(parsedObject))
  .catch(err => console.error('Problem handling API response:', err));

function handleResponse(fullResponse) {
  const response = fullResponse.response;
  if (response.type === 'Failure') {
    console.log('Request failure:', response);
  } else {
 //   console.log('Station ID: ', response.stationId);
    const hddData = response.dataSets.myHDD;
    if (hddData.type === 'Failure') {
      console.log('Failure for HDD DataSet:', hddData);
    } else {
       // console.log('HDD:');
      for (var v of hddData.values) {
       // let degreeDaysValue = console.log(v.d + ': ' + v.v);
      let degreeDaysValue = v.v;
      let degreeDaysDate = DateTime.fromISO(v.d).toUnixInteger()*1000
       // console.log(degreeDaysValue);
       // console.log(degreeDaysDate);

// FIRE!! (Insert Values into Mango XID)
       insertValuesV4(ARG_VALS.xid,
        {
            value:degreeDaysValue,
            dataType:"NUMERIC",
            timestamp:degreeDaysDate
        }
);
      }
    }
  }
}
// End Code ************************************************

• Now log into Mango as admin, and create a new user - being sure to set the role to match read and set point permissions of the datapoint you want to write to. In my case it was 'DegreeDaysUser'

Create a new Scripting Data Source ie. "DegreeDays.net API", and Paste the following script in the data source:

souce: https://forum.mango-os.com/topic/3818/invoke-shell-commands-from-a-scripting-environment

//Begin Code
com.serotonin.m2m2.rt.maint.work.ProcessWorkItem.queueProcess("C:/Program Files/nodejs/node C:/NodeJSApps/degreedaysAPI/ddMango.js --xid=DP_DegreeDaysHDD --user=dduser --pass=ddpassword --host=localhost --port=8080", 10);
//End Code

***PLEASE NOTE: the full paths to the node binary (C:/Program Files/nodejs/node.exe) as well as the full path to the ddMango.js script e.g. C:/NodeJSApps/degreedaysAPI/ddMango.js Make sure these match your specific folder path environment. Also change the ddMango.js script --xid,--user,--pass,--host,--port argument values to match your environment.

• On this same Scripting data source, select "Polling" and then set the polling to use the desired schedule (In this case CRON Pattern: 0 30 0 ? * * * was used)
  The CRON pattern above represents running this script everyday at 12:30AM. Validate script, then save.

• (Important!) Now re-open this same Scripting Data Source you just created, and then create a new DataPoint with the following XID:
  DP_DegreeDaysHDD (or whatever suits your naming conventions...)

• Change the new datapoint to Data Type "Numeric", and Logging Type: "All Data", also fill out the point name, and Variable name field with names of your choosing, then assign the "DegreeDaysUser" role to have Read, edit, set permissions, then save the point.

---

To Test to see if the script works:

• Click "Validate" for the script again on this scripting data source, you should see the newly added DP_DegreeDaysHDD datapoint update with values.
• Also check the next day after 12:30AM to see if CRON fired the script to update the DP_DegreeDaysHDD value as well.

Thanks for your contribution Ray!

-Fox

[raylatbasix]

@MattFox Thanks for posting this Fox!! And thank you again for all your help! Your an invaluable asset to this forum!

Note: For others that try to access this data from the DegreeDays.net API as shown above, you may need to adjust the CRON polling time to something a little later than 12:30AM as listed above. Some weather stations do not fully update yesterdays available Heating Degree Data until a couple hours or so after midnight, so experiment with the CRON polling time accordingly.