Elasticsearch Elasticsearch Truncate Field

By Opster Team

Updated: Jan 28, 2024

| 2 min read

Quick links

• Introduction
• Using Ingest Pipeline to Truncate Fields
• Using Scripted Fields in Kibana
• Conclusion

Introduction

When working with Elasticsearch, there might be situations where you need to truncate fields to limit the length of the stored data. This can be useful for reducing storage space, improving search performance, or ensuring compliance with specific data storage requirements. In this article, we will discuss various techniques and best practices for truncating fields in Elasticsearch.

Using Ingest Pipeline to Truncate Fields

One of the best ways to truncate fields in Elasticsearch is by using an ingest pipeline. Ingest pipelines allow you to pre-process documents before they are indexed. You can use the truncate processor to limit the length of a field to a specific number of characters. Here’s how to create an ingest pipeline with a script processor that can truncate a field to a given number of characters:

1. Define the ingest pipeline:

PUT _ingest/pipeline/truncate_fields
{
  "description": "Truncate fields to a specific length",
  "processors": [
    {
      "script": {
        "source": "ctx.field_name = ctx.field_name.substring(0, (int) Math.min(params.size, ctx.field_name.length()))",
        "params": {
          "size": 15
        }
      }
    }
  ]
}

Replace “field_name” with the name of the field you want to truncate and set the length to the desired number of characters, which is defined in the “params” object.

2. Next, index a document using the ingest pipeline:

PUT my-index/_doc/1?pipeline=truncate_fields
{
  "field_name": "This is a long text that needs to be truncated to fit the storage requirements and improve search performance."
}

The document will be indexed with the “field_name” truncated to 15 characters (if you kept the parameters as we proposed in the given example), as shown below.

GET my-index/_doc/1

{
  "_index": "my-index",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "message": "This is a long "
  }
}

Using Scripted Fields in Kibana

If you want to truncate fields only for display purposes in Kibana, you can use scripted fields. Scripted fields allow you to create custom fields that are computed on-the-fly based on the existing fields in your documents. Here’s how to create a scripted field to truncate a field in Kibana:

1. Go to Kibana and navigate to the “Management” section.
2. Click on “Index Patterns” and select the index pattern you want to modify.
3. Click on the “Scripted fields” tab and click “Add scripted field.”
4. Enter a name for the new scripted field, e.g., “truncated_field_name.”
5. In the “Script” field, enter the following Painless script:

doc[‘field_name’].value.substring(0, Math.min(5, doc[‘field_name’].value.length()))

Replace “field_name” with the name of the field you want to truncate and set the number 5 to the desired number of characters.

6. Save the scripted field.

Now, when you create visualizations or view documents in Kibana, you can use the “truncated_field_name” field to display the truncated version of the original field.

Conclusion

Truncating fields in Elasticsearch can be achieved using different techniques, depending on your requirements. Using an ingest pipeline with a truncate processor is a powerful method for limiting the length of fields during indexing, while scripted fields in Kibana provide a flexible way to display truncated fields without modifying the underlying data. By applying these techniques, you can optimize your Elasticsearch storage and search performance while adhering to data storage requirements.

If you want to learn more about ingest pipelines and how to use them take a look at this guide.