Quick Links
Introduction
The goal of this article is not to dive into how to architect your code in the best possible way but to show why leveraging search templates can help you better organize your code and separate the duties of your front-end and back-end code.
Depending on your role within your team and your experience crafting Elasticsearch queries, you have probably come across a few different situations. At one point, you might have been in control of both the front end and the back end. Another time, you might have been in charge of either one and had to cooperate with another engineer or engineering team to interact with Elasticsearch.
Moreover, some people thinking of Elasticsearch as THE back end don’t bother having a custom back-end layer and send DSL queries directly from their front-end code. That’s like sending SQL queries directly from your web front end to your RDBMS, i.e., a very bad practice, to say the least, for many reasons.
With great power…
It should come as no surprise that building efficient DSL queries is not only an art in itself but also of paramount importance when it comes to guaranteeing your service-level agreements and maintaining your cluster stability. Such power comes with great responsibility, which is why the norm should be to not let anyone on your team craft their own half-baked queries and throw them at your cluster. There are so many ways that can go wrong and harm your cluster performance.
If you heard about design by contract when learning object-oriented programming, you know the importance of delineating a clear boundary between software components and properly defining their duties. DSL queries are no exception to that rule!
Enter search templates
A great feature in Elasticsearch that allows you to design your queries so that they honor a specific contract is search templates. The main advantage of search templates is that they allow you to define DSL queries that can be stored in your cluster state. That means that they stay under the control of whoever is in charge of your cluster and not the developer team building the front end or back end. Also, since the cluster state can be saved in your snapshots, this provides automatic backups of your queries.
Search templates can be compared to SQL stored procedures that clearly define the input parameters but don’t necessarily provide any information as to how they are implemented. The code below shows what a search template looks like in practice:
PUT _scripts/my-search-template
{
"script": {
"lang": "mustache",
"source": {
"from": "{{from}}",
"size": "{{size}}",
"query": {
"match": {
"content": "{{query_string}}"
}
}
}
}
}In the above code, we can see that a search template is stored as a script called `my-search-template` (line 1). The templating language used for specifying the query is called Mustache (line 4). The query itself is defined in the `source` section (lines 5–14) and parametrized by three different variables denoted by double curly brackets (e.g., `{{variable}}`).
Let’s now look at the code below that shows how to invoke such a search template and run the parameterized query:
POST my-index/_search/template
{
"id": "my-search-template",
"params": {
"from": 0,
"size": 10,
"query_string": "hello world"
}
}Here lies the beauty of search templates. Regardless of the complexity of your queries, you will always call them the same way through the `<index>/search/template` API endpoint, i.e., by specifying the name of your search template in the `id` parameter (line 3) and providing a map containing all the parameters expected by your query (lines 4–8). At runtime, Elasticsearch will substitute the variable names in the template with the values provided in the `params` map. We can also consider this as a remote procedure call of the following form:
my-search-template(from = 0, size = 10, query_string = "hello world")
To see what the query would look like with a specific set of parameters, we can render the template using the `_render/template` API endpoint and the exact same payload:
POST _render/template
{
"id": "my-search-template",
"params": {
"from": 0,
"size": 10,
"query_string": "hello world"
}
}
=>
{
"template_output" : {
"from" : "0",
"size" : "10",
"query" : {
"match" : {
"message" : "hello world"
}
}
}
}Whether you have a single DSL query or one hundred, they will all be invoked the same way, i.e., by providing a template name and a parameters map containing the input values of your query. Pretty powerful, isn’t it?
Also, it goes without saying that when communicating with your users, a search template should always be accompanied by some documentation detailing what each parameter does, its type and/or valid value(s), whether it is required or not, etc., in exactly the same way as you would document such things as your code, test cases, HTTP API endpoints, or SQL queries.
The benefits of search templates
Search templates provide many benefits. For example, they give you the power to properly craft your DSL queries as you see fit and define a clear contract with the developers who will leverage them. That contract is defined by the set of parameters that your users will have to pass when calling the search template. Your search query is not exposed anymore and your cluster is now shielded from any suboptimal queries coming from “the outside.”
Another big benefit is that you are now free to evolve your query without impacting the users and their application code. That’s because any change or improvement to your query will be completely transparent to them and will not require any change in their application code, except if you add new parameters, of course.
One of the main reasons why developers are usually given the opportunity to build queries themselves is that DSL queries are rarely static and are normally built dynamically depending on varying conditions. As we will see shortly, that excuse is not valid either since this is also possible with search templates, which provide the flexibility to build queries dynamically depending on the values of the input parameters.
Advanced usage
Even though Mustache is a logic-less templating language, it does much more than just allow you to substitute variables. For example, it also provides support for building your queries dynamically as hinted at in the previous section. We will now review all the possible ways you can craft powerful templated queries using Mustache.
Default values
Some parameters are optional and don’t have to be specified on every single call. For instance, that’s the case for the `from` and `size` query parameters that Elasticsearch defaults to 0 and 10, respectively, if they are not provided. Mustache supports defining default values if some variable is not passed into the parameters map using the syntax below:
PUT _scripts/my-search-template
{
"script": {
"lang": "mustache",
"source": {
"from": "{{from}}{{^from}}0{{/from}}",
"size": "{{size}}{{^size}}10{{/size}}",
"query": {
"match": {
"content": "{{query_string}}"
}
}
}
}
}In the above code, we declare that we use the value of the `{{from}}` parameter, but if it is not specified, then `{{from}}` will be replaced with an empty string followed by 0, i.e., the content of the`{{^from}}…{{/from}}` inverted section, which is activated if the `from` parameter is missing.
Note that this article does not intend to be a comprehensive guide to the Mustache syntax, hence we invite you to check out the Mustache official documentation to learn more about the specificities of the language.
Conditions
The ability to construct a query dynamically by including or excluding constraints based on the presence or the value of input parameters can be achieved with another kind of section delimited by a pound sign to start a block (i.e., `{{#…}}`) and a slash to end it (i.e., `{{/…}}`), as shown in the command below:
PUT _scripts/my-search-template
{
"script": {
"lang": "mustache",
"source": """
{
"from": "{{from}}{{^from}}0{{/from}}",
"size": "{{size}}{{^size}}10{{/size}}",
"query": {
"bool": {
"must": [
{{#date_range}}
{
"range": {
"date": {
"gte": "{{date_range.min}}",
"lt": "{{date_range.max}}"
}
}
},
{{/date_range}}
{
"match": {
"message": "{{query_string}}"
}
}
]
}
}
}
"""
}
}First, it is worth noting that we switched from specifying a JSON formatted query in the source to a string one using the triple quotes feature from Kibana (i.e., the `”””` on lines 5 and 31). This is needed because of the `{{#date_range}}` block which is not valid JSON. Since the HTTP request payload must contain valid JSON and JSON literal strings and cannot contain newlines, we can either pass a JSON-escaped string within double quotes or a triple-quoted string that will resolve to a valid JSON query once rendered. We prefer the latter as it is more flexible than having to edit and read JSON query one-liners.
Back to our query above, an additional `range` constraint on the `date` field will be included depending on the presence or absence of the `date_range` parameter. If such a parameter is specified, its `min` and `max` values will be used to specify the valid date interval of the query.
For instance, we can invoke the query using the following parameters to dynamically include an additional constraint on the `date` field that returns only documents from the year 2023:
POST my-index/_search/template
{
"id": "my-search-template",
"params": {
"query_string": "hello world"
"date_range": {
"min": "2023-01-01",
"max": "2024-01-01"
}
}
}There are virtually no limits as to how many conditions you can specify, so you can build queries of any arbitrary complexity. Here is another example of using a boolean flag to switch between returning search hits or aggregations:
PUT _scripts/my-search-template
{
"script": {
"lang": "mustache",
"source": """
{
{{#aggs}}
"size": 0,
{{/aggs}}
{{^aggs}}
"from": "{{from}}{{^from}}0{{/from}}",
"size": "{{size}}{{^size}}10{{/size}}",
{{/aggs}}
"query": {
"match": {
"content": "{{query_string}}"
}
}
{{#aggs}}
, "aggregations": {
"categories": {
"terms": {
"field": "categories"
}
}
}
{{/aggs}}
}
"""
}
}In the above code, we can reuse the same search template to switch between a normal search returning hits matching a given message or an aggregation search returning a list of top categories for the same set of matching documents. We see that if the `aggs` flag is set to true, we set the size to 0 and include the `terms` aggregations query. In the opposite case, a normal search with the default from/size parameters will be run. The code below shows what it takes to invoke this search template with aggregations enabled:
POST _render/template
{
"id": "my-search-template",
"params": {
"query_string": "hello world"
"aggs": true
}
}
=>
{
"template_output" : {
"size" : 0,
"query" : {
"match": {
"content": "hello world"
}
},
"aggs" : {
"categories" : {
"terms" : {
"field" : "categories"
}
}
}
}
}Mustache sections offer a very powerful way to create dynamic DSL queries. Make sure to check out the official Mustache documentation to learn more about sections.
JSON conversion
Another interesting feature is the ability to inject JSON into your queries. Note that this is not part of Mustache but is provided by Elasticsearch as an extension. For instance, adding a condition to match a list of categories can be done using the `{{toJson}}` function, which will convert a variable in the parameters map as JSON into the query.
PUT _scripts/my-search-template
{
"script": {
"lang": "mustache",
"source": """
{
"from": "{{from}}{{^from}}0{{/from}}",
"size": "{{size}}{{^size}}10{{/size}}",
"query": {
"bool": {
"must": [
{{#categories}}
{
"terms": {
"categories": {{#toJson}}categories.values{{/toJson}}
}
},
{{/categories}}
{
"match": {
"message": "{{query_string}}"
}
}
]
}
}
}
"""
}
}The above query can be invoked using the following parameters in order to include a constraint on the `categories` field:
POST _render/template
{
"id": "my-search-template",
"params": {
"query_string": "hello world",
"categories": {
"values": [
"clothes", "electronics", "music"
]
}
}
}
=>
{
"template_output" : {
"from" : "0",
"size" : "10",
"query" : {
"bool" : {
"must" : [
{
"terms" : {
"categories" : [ "clothes", "electronics", "music" ]
}
},
{
"match" : {
"message" : "hello world"
}
}
]
}
}
}
}The `{{toJson}}` function is flexible enough to also let you include a JSON constraint specified directly by your users. However, you need to be very careful if you go down that road as JSON injection can induce security risks in much the same way as SQL injection does in RDBMS databases.
There are also two other functions that Elasticsearch added on top of Mustache: `{{url}}` to URL encode strings and `{{join}}` to concatenate array values into a comma-separated string.
Conclusion
This article started by emphasizing the importance of clearly separating the definition of your DSL queries from their use by creating a contract using search templates. Doing so allows you to keep the responsibility for cleverly crafting DSL queries with the engineers who are in charge of your Elasticsearch cluster and not the developers who query it.
We then showed how to create and render search templates and how to run them by simply specifying their name and a map of input parameters.
Finally, we showed some advanced usages of search templates that illustrated how flexible templates can be when it comes to dynamically creating DSL queries. Specifically, we showed how to set default values in cases where input parameters are not explicitly specified and how to shape your query to include dynamic constraints based on the presence of input parameters or the value of flags. We wrapped up by showing some Elasticsearch extensions to the Mustache language that allow you to include JSON parameters, concatenate array values, and URL encode strings.
