Elasticsearch Elasticsearch Search Unique Values: Aggregations and Cardinality

By Opster Team

Updated: Jun 18, 2023

| 2 min read

Introduction

When working with Elasticsearch, you may encounter situations where you need to retrieve unique values from your dataset. This can be useful for various purposes, such as identifying distinct categories, tags, or user IDs. In this article, we will explore how to use Elasticsearch aggregations and the cardinality metric to search for unique values in your dataset. If you want to learn about how to improve your Elasticsearch aggregation performance, check out this guide.

Using Terms Aggregation

Terms aggregation is a powerful technique for grouping and counting unique values in Elasticsearch. It works by creating buckets for each unique value in the specified field and counting the number of documents that fall into each bucket. Here’s a step-by-step guide on how to use terms aggregation to search for unique values:

1. Choose the field you want to aggregate on. This field should be of keyword type or have a keyword subfield if it’s a text field.

2. Create a search query with an aggregation clause. In the aggregation clause, specify the type of aggregation as “terms” and provide the field name.

Here’s an example query that retrieves unique categories from a dataset of products:

GET /products/_search
{
  "size": 0,
  "aggs": {
    "unique_categories": {
      "terms": {
        "field": "category.keyword"
      }
    }
  }
}

In this example, we set the size to 0 because we’re only interested in the aggregation results, not the actual documents. The response will include a list of unique categories along with their document counts:

{
  ...
  "aggregations": {
    "unique_categories": {
      "buckets": [
        {
          "key": "Electronics",
          "doc_count": 100
        },
        {
          "key": "Clothing",
          "doc_count": 80
        },
        ...
      ]
    }
  }
}

Using Cardinality Aggregation

Cardinality aggregation is another technique for finding unique values in Elasticsearch. It estimates the number of distinct values in a field using the HyperLogLog++ algorithm, which provides a trade-off between accuracy and memory usage. Here’s how to use cardinality aggregation:

1. Choose the field you want to aggregate on. As with terms aggregation, this field should be of keyword type or have a keyword subfield if it’s a text field.

2. Create a search query with an aggregation clause. In the aggregation clause, specify the type of aggregation as “cardinality” and provide the field name.

Here’s an example query that estimates the number of unique user IDs in an events dataset:

GET /events/_search
{
  "size": 0,
  "aggs": {
    "unique_user_ids": {
      "cardinality": {
        "field": "user_id.keyword"
      }
    }
  }
}

The response will include an estimated count of unique user IDs:

{
  ...
  "aggregations": {
    "unique_user_ids": {
      "value": 12345
    }
  }
}

Keep in mind that cardinality aggregation provides an approximation, not an exact count. You can control the accuracy of the estimation by adjusting the `precision_threshold` parameter. Higher values will result in more accurate counts but consume more memory.

GET /events/_search
{
  "size": 0,
  "aggs": {
    "unique_user_ids": {
      "cardinality": {
        "field": "user_id.keyword",
        "precision_threshold": 1000
      }
    }
  }
}

In this example, we set the `precision_threshold` to 1000, which means that the estimation will be accurate up to 1000 unique values. Beyond that, the algorithm will start to trade accuracy for memory efficiency.

Conclusion

Elasticsearch provides powerful aggregation techniques for searching unique values in your dataset. Terms aggregation allows you to group and count unique values, while cardinality aggregation estimates the number of distinct values with adjustable accuracy. Choose the appropriate method based on your use case and requirements, and leverage the power of Elasticsearch to analyze and explore your data.