Elasticsearch Optimizing Bulk Operations with OpenSearch-Py

By Opster Team

Updated: Nov 14, 2023

| 2 min read

Introduction

Efficiently indexing and updating large volumes of data is a common requirement in many OpenSearch and Elasticsearch applications. The OpenSearch-Py library provides a convenient way to perform bulk operations, which can significantly improve the performance of data ingestion and updates.

In this article, we will discuss how to use the OpenSearch-Py library to perform bulk operations and provide some tips for optimizing performance. If you want to learn about OpenSearch Bulk, check out this guide.  

Using OpenSearch-Py for Bulk Operations

The OpenSearch-Py library provides a `bulk` helper function that allows you to perform multiple index, update, and delete operations in a single request. This can greatly reduce the overhead of making individual requests for each operation, leading to improved performance.

Here’s a step-by-step guide on how to use the `bulk` helper function:

1. Install the OpenSearch-Py library:

pip install opensearch-py

2. Import the necessary modules:

python
from opensearchpy import OpenSearch
from opensearchpy.helpers import bulk

3. Create an instance of the OpenSearch client:

python
client = OpenSearch("http://localhost:9200")

4. Define the data and actions for the bulk operation:

python
actions = [
    {"_op_type": "index", "_index": "test-index", "_id": 1, "_source": {"field1": "value1"}},
    {"_op_type": "index", "_index": "test-index", "_id": 2, "_source": {"field1": "value2"}},
    {"_op_type": "update", "_index": "test-index", "_id": 1, "doc": {"field1": "updated_value1"}},
    {"_op_type": "delete", "_index": "test-index", "_id": 2}
]

In this example, we are performing two index operations, one update operation, and one delete operation. The `_op_type` field specifies the type of operation, and the other fields provide the necessary information for each operation.

5. Execute the bulk operation:

python
success, failed = bulk(client, actions)

The `bulk` function returns the number of successful and failed operations.

Optimizing Performance

Here are some tips for optimizing the performance of bulk operations using OpenSearch-Py:

1. Use the appropriate batch size: The optimal batch size for bulk operations depends on various factors, such as the size of the documents, the available resources, and the performance characteristics of the cluster. Experiment with different batch sizes to find the best balance between the number of requests and the size of each request.

2. Use parallel processing: If you have multiple CPU cores available, you can use parallel processing to improve the performance of bulk operations. The OpenSearch-Py library provides a `parallel_bulk` helper function that automatically handles parallel processing for you. Simply replace the `bulk` function with the `parallel_bulk` function in the example above.

3. Monitor and adjust the refresh interval: The refresh interval determines how often the changes made by indexing, updating, or deleting operations become visible to search queries. By default, the refresh interval is set to 1 second. For bulk operations, you may want to increase the refresh interval to reduce the overhead of refreshing the index or even set it to -1 to disable it entirely while the bulk operations are underway. However, keep in mind that increasing the refresh interval will also increase the latency of search results.

4. Use the right number of shards and replicas: The number of shards and replicas in your cluster can have a significant impact on the performance of bulk operations. Make sure to choose the right number of shards and replicas based on your specific use case and requirements.

Conclusion

Bulk operations are an essential tool for efficiently indexing and updating large volumes of data in OpenSearch and Elasticsearch. By using the OpenSearch-Py library and following the optimization tips provided in this article, you can significantly improve the performance of your data ingestion and update processes.