Elasticsearch Elasticsearch Aggregations Pagination: Efficient Techniques for Large Result Sets

By Opster Team

Updated: Jul 20, 2023

| 2 min read

Introduction 

Aggregations in Elasticsearch are a powerful way to analyze and summarize your data. However, when dealing with large result sets, it’s essential to paginate the results efficiently to avoid performance issues and ensure a smooth user experience.

In this article, we will discuss various techniques for paginating aggregations in Elasticsearch. If you want to learn about Elasticsearch aggregation, check out this guide. You should also take a look at this guide, which contains a detailed explanation on Elasticsearch pagination – which technique to use depending on your use case.  

1. Using Composite Aggregation

Composite aggregation is a powerful tool for paginating through large result sets. It allows you to retrieve a fixed number of buckets at a time, making it ideal for pagination. Here’s a step-by-step guide on how to use composite aggregation for pagination:

Step 1: Define the composite aggregation query

Create a composite aggregation query with the desired sources (fields) and size (number of buckets per page).

GET /your_index/_search
{
  "size": 0,
  "aggs": {
    "composite_agg": {
      "composite": {
        "size": 10,
        "sources": [
          {
            "field1": {
              "terms": {
                "field": "field1.keyword"
              }
            }
          },
          {
            "field2": {
              "terms": {
                "field": "field2.keyword"
              }
            }
          }
        ]
      }
    }
  }
}

Step 2: Retrieve the first page of results

Execute the query to get the first page of results.

Step 3: Extract the after_key

In the response, extract the `after_key` value. This value is used to fetch the next page of results.

Step 4: Fetch the next page of results

To fetch the next page, add the `after` parameter to the composite aggregation query and set its value to the extracted `after_key`.

GET /your_index/_search
{
  "size": 0,
  "aggs": {
    "composite_agg": {
      "composite": {
        "size": 10,
        "sources": [
          {
            "field1": {
              "terms": {
                "field": "field1.keyword"
              }
            }
          },
          {
            "field2": {
              "terms": {
                "field": "field2.keyword"
              }
            }
          }
        ],
        "after": {
          "field1": "value_from_after_key",
          "field2": "value_from_after_key"
        }
      }
    }
  }
}

Step 5: Repeat steps 3 and 4

Continue fetching pages by repeating steps 3 and 4 until no more buckets are returned.

2. Using Partitioning with Terms Aggregation

Another approach to paginate aggregations is by using partitioning with terms aggregation. This method divides the field’s values into a specified number of partitions, allowing you to retrieve a portion of the results at a time. Here’s how to use partitioning for pagination:

Step 1: Define the terms aggregation query with partitioning

Create a terms aggregation query with the desired field, number of partitions, and the current partition.

GET /your_index/_search
{
  "size": 0,
  "aggs": {
    "partitioned_terms_agg": {
      "terms": {
        "field": "field1.keyword",
        "include": {
          "partition": 0,
          "num_partitions": 10
        }
      }
    }
  }
}

Step 2: Retrieve the first partition of results

Execute the query to get the first partition of results.

Step 3: Fetch the next partition of results

To fetch the next partition, update the `partition` value in the `include` parameter and execute the query again.

GET /your_index/_search
{
  "size": 0,
  "aggs": {
    "partitioned_terms_agg": {
      "terms": {
        "field": "field1.keyword",
        "include": {
          "partition": 1,
          "num_partitions": 10
        }
      }
    }
  }
}

Step 4: Repeat step 3

Continue fetching partitions by repeating step 3 until all partitions have been retrieved.

Conclusion

In conclusion, paginating aggregations in Elasticsearch can be achieved using composite aggregation or partitioning with terms aggregation. Both methods provide efficient ways to handle large result sets and ensure optimal performance. Choose the technique that best suits your use case and requirements.