Question about point query performance and filtering

iperry

Heya,

I am curious about the performance of using ma-point-query. Is it better to:
Use ma-point-query which returns 100 data points and then filter for the 5 I want
OR
Use 5 ma-point-query(s) to get the specific data points

For example, I have a device for which there are 78 data points defined, a mixture of modbus, meta, and virtual points. I can either do a ma-point-query which returns all 78 and then filter for the temperature, inclination, etc points I want or query for those points explicitly. What is the better approach? Is there a best-practice? Currently I am using both approaches, based on the situation, but was thinking of standardizing my coding.

Thanks
Ian

phildunlap

Hi Ian,

Performance questions are often determined by finer details and other information. For instance, are you selecting 100 points out of 100000 ? Or are you selecting all the points and then filtering? The other question is which element of performance is the one you're seeking to optimize, as there are a few stages that could bottleneck in what you're describing.

Searching the database for the datapoints. Work is done by the server, selecting by id or data source id would be fastest always, xids pretty fast, wildcard RQL matching in miscellaneous fields would take the longest, but would probably still be extremely fast if your system isn't huge.
Serializing them into a response. This doesn't take too much time, but more points would likely linearly scale the work it had to do. The bandwidth of the gateway may be a consideration, in that if you pay per byte you may not want to include an extra 20 points to filter out later on. This work is also done by the server.
Filtering after the fact. Work is done in the client's browser instead of the server.

If you submit multiple queries, you have to do any overhead associated with processing those queries, but as the old mantra goes,

...Be wary of premature optimization

because until you know that it's a meaningful load for the system to service these requests, it might be more effort than its worth to say one approach is always best. Do what is clear to you!

iperry

Hi Phil,

Thanks for the reply. Some good stuff to think about moving forward. Questions about performance are usually a bit ethereal as there many factors that can determine performance as you point out. heh

I was more curious if there were things I should be considering as we increased the number of devices and thus the number of data points.

Ian

phildunlap

For sure!

I was more curious if there were things I should be considering as we increased the number of devices and thus the number of data points.

Have you looked into tagging points? Then you can just select for the tag in the RQL like eq(tags.tagKey,tagValue) and get exactly the points you're looking for.

Also, not to encourage overthinking about performance, but wildcards at the end of strings are much more efficient than wildcards at the middle or beginning of the string. So, if your points have a similar prefix on one or another field you can select in a like(deviceName,Site 1*)

iperry

I have reviewed the point tags but haven't looked into using them. So far, a point hierarchy seems to be sufficient for organizing data points. I do have a future task to explore tags further to see how they work and if they may be better to use.

CraigWeb

Hi @iperry

I can guarantee, that you wont look back, once you get comfortable using tags.

iperry

Hi @phildunlap ,

I am circling back to looking at tags again for defining our device hierarchy and would like to use tags in a ma-point-query. However, I cannot find an example (if it exists) of the syntax.

For example:
<ma-point-query query="{tags.PCU: 'PCU'}" sort="'name'" points="pcuTest"></ma-point-query>

This does not work of course. What is the syntax when using this directive?

Thanks
Ian

MattFox

@iperry your query needs either in(tags.tagname, searchItem) //( eq(tags.tagname,searchItem) works too ) or like(tags.tagname,*searchItem*)
Oh I see you don't use RQL. I think using RQL is easier IMO. You can generate RQL via the build watchlist editor under administration.

<ma-point-query query="'in(tags.PCU, PCU)&sort(name)&limit(200)'" points="pcuTest"></ma-point-query>
Do note that tags.PCU can have different values. ie tags.PCU="myFlatCluster"

iperry

Hi @MattFox ,

Thanks for the example. I hadn't used RQL yet in the <ma-point-query> directive yet so that is cool to see.

When I first made the tag, I thought it was just that value, kind of like a hashtag. I clued in afterwards that there was a difference between the name of the tag and it's values heh

Ian

MattFox

Glad I could be of help, feel free to ask if you hit any snags with any RQL strings.