Mango REST API only returning 100 data-points

[adamlevy]

I'm running the latest version of Mango with all modules up to date.

When I perform a GET request to the data-points end point I get back a JSON with two top level fields: "total" and "items". The "total" value is 334, however the number of items in the "items" array however is always only 100.

For example using my curl wrapper called mangocli:

$ mangocli data-points | head -n5
{
  "total": 334,
  "items": [
    {
      "id": 1,
$ mangocli data-points | jq '.items | length'
100

This is true for both the v1 and v2 APIs and when I use the swagger interface and curl to make the requests.

The user account that the JWT token is associated with has full admin privileges in Mango.

It seems like some sort of pagination but I can't find any mention of this behavior in the forum or in swagger. I also tried adding a query parameter for page but that resulted in a 500 error.

[phildunlap]

Hi adamlevy,

I suspect you are using an RQL endpoint. There is a default limit of 100, or whatever is currently here as DEFAULT_LIMIT: https://github.com/infiniteautomation/ma-core-public/blob/main/Core/src/com/serotonin/m2m2/db/dao/AbstractBasicDao.java

You can put limit(-1) in your query string to have no limit.

[adamlevy]

Philip

Thanks for your prompt reply. I'm a little confused still though.

"I suspect you are using an RQL endpoint."

I'm querying Mango's REST API V2. The same one that the Swagger interface shows you. Is there some other option here?

"You can put limit(-1) in your query string to have no limit."

Could you be more specific about how to specify the limit in a REST query?

To clarify, with mangocli I am using curl to send a GET request to this endpoint: /rest/v2/data-points

I tried adding it as a url param ``https://nus.ia3.io/rest/v2/data-points?limit=-1` but that fails with 500.

Attempting to put it in the body of a GET request also fails.

Edit: I just looked back at the swagger interface and saw the note "Use RQL formatted query"
I'm looking into RQL. I'm not yet familiar with it.

Edit 2: After reading up on RQL, I played around with my query and got the following URI to work: /rest/v2/data-points?limit(400). Unfortunately limit(-1) causes a SQL error with MariaDB. I get back the following

{
  "message": "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '-1' at line 1",
  "stackTrace": "sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\nsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
...

Thank you for pointing me in the right direction!
Where can I read up on what RQL mango accepts other than reading the source code directly?

[Jared Wiltshire]

@adamlevy ?limit=x wont work I believe as you noticed, it should be ?limit(x). You can read more about RQL here - https://github.com/persvr/rql

There is a default limit of 100 so that if you don't explicitly set a limit it doesn't dump 100,000 data points back.

Probably the easiest way to have a play with some RQL for querying data points is to use the watch list builder. Create a new "Point query" type watch list and play around on the query tab.

[phildunlap]

Interesting, I had thought there was special handling somewhere in the RQL for nonsense limits, but it turns out that's H2 grammar that LIMIT -1 is the same as no limit specified. http://www.h2database.com/html/grammar.html

[adamlevy]

@Jared-Wiltshire @phildunlap

Thanks again for your help!