• Recent
    • Tags
    • Popular
    • Register
    • Login

    Please Note This forum exists for community support for the Mango product family and the Radix IoT Platform. Although Radix IoT employees participate in this forum from time to time, there is no guarantee of a response to anything posted here, nor can Radix IoT, LLC guarantee the accuracy of any information expressed or conveyed. Specific project questions from customers with active support contracts are asked to send requests to support@radixiot.com.

    Radix IoT Website Mango 3 Documentation Website Mango 4 Documentation Website Mango 5 Documentation Website

    Scraping an entire data set

    Scheduled Pinned Locked Moved User help
    12 Posts 3 Posters 3.0k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      psysak
      last edited by

      I was wondering if someone wouldn't mind pointing me in the right direction, the first step of which would be to confirm this is even possible.

      I would like to be able to scrape an entire set of data points via XML and then be able to plot those in an excel report. The data I will be pulling in is hourly predicted consumption rates that some energy entity posts. They post an entire week of predictions at a time and I'd like to be able to pull those into Mango, analyze each reading to see if it crosses some set value, and then plot the entire data set in a report.

      I'm not asking for a solution to the problem but rather some ideas on how I would be able to store that data set. Alternatively I'm thinking that instead of needing to plot it I could just read in the points one by one, compare to my threshold and then just report about what time and day that threshold was exceeded.

      Thank you forum!

      1 Reply Last reply Reply Quote 0
      • phildunlapP Offline
        phildunlap
        last edited by phildunlap

        Hi psysak,

        It sounds like a task for the Data File data source. That's the primary tool for an arbitrary XML task. You can generate the parsing class after generating an .xsd for this XML. Then you have to write a class that implements the function to get data out of the parsed XML. You could fetch the file with either a wget on a cron or write a poll class to load into the Data File data source to fetch the data.

        Alternatively, you could use the HttpBuilder utility in a script. On a cron, your script does something like...

        HttpBuilder.get("sptth://api.extension/demand-predictions", { /*object of headers*/ }, { /*object of parameters*/ }).err(
        function(status, headers, content) {
          throw "Got non-200 response: " + status;
        }).resp(function(status, headers, content) {
          print(content);
          //Okay, now we have to parse the XML. You can do regex /(pattern)/.exec(content)[1]
          // Or you can do it to any degree of parsing (i.e. indexOf --> substr, new java.io.StringReader(content), etc)
        }).execute();
        1 Reply Last reply Reply Quote 0
        • P Offline
          psysak
          last edited by

          Thanks Phil! Super helpful as always.

          That HttpBuilder function, what language is that part of? I ask because I'd like to see the documentation for it. Learning on the fly here :)

          1 Reply Last reply Reply Quote 0
          • phildunlapP Offline
            phildunlap
            last edited by

            The HttpBuilder is a utility specific to Mango. You can find some notes about it in the "Mango JavaScript" help dialogue within Mango. It's linked to by the contextual help for all things that use scripts in the 'related items' section. Probably github is the only source that's always at least up to date for this file: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/web/WEB-INF/dox/mangoJavaScript.htm

            1 Reply Last reply Reply Quote 0
            • P Offline
              psysak
              last edited by

              Just FYI, I can't find any mention of HttpBuilder in that Mango Scripting help.

              Last question... for today :) While I'm on the topic, how about JSON.parse documentation?

              1 Reply Last reply Reply Quote 0
              • P Offline
                psysak
                last edited by

                I think I may have got it, I found some stuff in that Rhino syntax help :)

                1 Reply Last reply Reply Quote 0
                • phildunlapP Offline
                  phildunlap
                  last edited by

                  Nice! Yeah, it's a builtin object for JavaScript, generally. Just an FYI but the references to Rhino are dated. Mango 3 only runs on Java 8 which uses Nashorn as the engine. Most things should be very similar, but it may matter at some point.

                  1 Reply Last reply Reply Quote 0
                  • P Offline
                    psysak
                    last edited by

                    OK thanks

                    1 Reply Last reply Reply Quote 0
                    • P Offline
                      psysak
                      last edited by

                      Interesting, the context help in the mango jumps from RQL Examples to Logging and totally misses the HttpBuilder and Context sections. Thanks for the GIT link

                      1 Reply Last reply Reply Quote 0
                      • Jared WiltshireJ Offline
                        Jared Wiltshire
                        last edited by

                        MDN is a very good reference for all things JavaScript
                        https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse

                        Developer at Radix IoT

                        1 Reply Last reply Reply Quote 0
                        • P Offline
                          psysak
                          last edited by

                          Thanks @Jared-Wiltshire For anyone else possibly reading this, one of the issues I ran into there was the versions of the libraries, for example the RegExp command. The way it's documented there I don't think makes the Mango very happy

                          1 Reply Last reply Reply Quote 0
                          • Jared WiltshireJ Offline
                            Jared Wiltshire
                            last edited by

                            @psysak MDN will definitely contain references to a lot of new JavaScript features which aren't available in the Java 8 Nashorn engine which we leverage. Nashorn should be compliant with ECMAScript 5.1. The MDN pages tell you which version of ECMAScript (JavaScript) the feature is supported in. e.g. look under specifications on the JSON.parse() page, it says

                            ECMAScript 5.1 (ECMA-262)
                            The definition of 'JSON.parse' in that specification. Standard Initial definition. Implemented in JavaScript 1.7."

                            Developer at Radix IoT

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post