Elasticsearch How to Migrate from Elasticsearch to OpenSearch

By Opster Expert Team - Gustavo

Updated: Feb 20, 2024

| 8 min read

Quick links

• Initial words
• Version considerations when migrating to OpenSearch
• Migration methods
  1. Rolling upgrades
  2. Snapshots
  3. Reindex API
  4. OpenSearch upgrade tool
• Final words

Initial words

If you’re looking to migrate from Elasticsearch to OpenSearch, or vice versa, there are multiple considerations to take into account. One such consideration is that OpenSearch supports migration seamlessly using rolling upgrades, but  users have to consider their cluster version.

You can check our related articles about system migrations/upgrades: 

• Elasticsearch Upgrade – How to Upgrade Versions Without Data Loss (Opster)

• How To: Upgrade from Open Distro to OpenSearch (OpenSearch)

Version considerations when migrating to OpenSearch

OpenSearch 1.0 can be upgraded from Elasticsearch 6.8.13 to 7.10.2.

If you are working with an older Elasticsearch version, you need to upgrade to 6.8.13 first, at the very least. OpenSearch recommends upgrading ES from 6.8.13 to 7.10.2 before upgrading to OpenSearch 1.0.

Another important element to consider is that the minimum supported index version is 6.0, so all 5.0 indices need to be reindexed before the migration.

You can follow this upgrade path table:

To check your Elasticsearch version you can run: curl localhost:9200

{
  "name" : "Gustavo’s Machine",
  "cluster_name" : "elasticsearch_brew",
  "cluster_uuid" : "JTt304FgSw6itSFPmMtvrg",
  "version" : {
    "number" : "7.10.2-SNAPSHOT",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-01-16T01:41:27.115673Z",
    "build_snapshot" : true,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

In this example, the Elasticsearch version is 7.10.2, so it’s possible to migrate directly.

Can the 7.11+ versions (after Opensearch fork) be migrated to OpenSearch? 

It requires more work as it is not possible using rolling upgrades. However there are other methods for migration, as detailed below, that will enable you to migrate from Elasticsearch 7.11+ versions.

Migration methods

Rolling upgrades and Snapshots take care of the data and cluster configuration. The Reindex API is useful if you only need to move your data from one cluster to another without worrying about cluster configuration or mappings. The Opensearch Upgrade tool makes the process itself easier.

1.Rolling upgrades

Rolling upgrades are the official way to upgrade, or in this case migrate the cluster without interrupting service.

Rolling upgrades are supported:

• Between minor versions
• From 5.6 to 6.8
• From 6.8 to 7.14.1
• From any version since 7.14.0 to 7.14.1

Upgrading directly to 7.14.1 from 6.7 or earlier requires a full cluster restart.

As mentioned at the beginning of this article, we can upgrade directly to Opensearch if the Elasticsearch version is between 6.8 and 7.10.2. Otherwise we will have to upgrade Elasticsearch to 6.8 first (Opensearch even recommends 7.10.2).

The full documentation is available here. 

Follow the steps below to migrate to OpenSearch:

Upgrade Elasticsearch

1. Disable shard allocation:

PUT _cluster/settings
{
 "persistent": {
   "cluster.routing.allocation.enable": "primaries"
 }
}

2. Stop ES in a single node (rolling upgrade) or all nodes (cluster restart upgrade):

sudo systemctl stop elasticsearch.service

3. Upgrade the single node or all nodes.

The exact command will depend on what system are you using, but in general is similar to this:

sudo yum install elasticsearch-oss-7.10.2 --enablerepo=elasticsearch

Make sure to NOT override your data, logs and config directories.

4. Restart Elasticsearch and wait for the nodes to rejoin:

sudo systemctl start elasticsearch.service

5. Rolling repeat steps 2-4 until all nodes are using the same version.

Verify your versions using:

# Elasticsearch OSS
curl -XGET 'localhost:9200/_nodes/_all?pretty=true'
# Open Distro for Elasticsearch with security plugin enabled
curl -XGET 'https://localhost:9200/_nodes/_all?pretty=true' -u 'admin:admin' -k

You can also use: GET _cat/indices?v  to verify that all indices are on green status.

6. Re-enable shard allocation:

PUT _cluster/settings
{
 "persistent": {
   "cluster.routing.allocation.enable": "all"
 }
}

7. If upgrading from 5.x to 6.x reindex all indices.

8. Repeat until you get to the desired version.

The process to upgrade to OpenSearch is similar, as explained below.

Upgrade OpenSearch

When your Elasticsearch nodes are in the right version (6.8.0 to 7.10.2), you can proceed to upgrading to OpenSearch.

1. Disable shard allocation:

PUT _cluster/settings
{
 "persistent": {
   "cluster.routing.allocation.enable": "primaries"
 }
}

2. Stop Elasticsearch on a single node or all nodes:

sudo systemctl stop elasticsearch.service

3. Upgrade the single node or all nodes:

3.a) Extract the OpenSearch tarball to a new directory to ensure you do not overwrite your Elasticsearch OSS config, data, and logs directories.

3.b) (Optional) Copy or move your Elasticsearch OSS data and logs directories to new paths. For example, you might move /var/lib/elasticsearch to /var/lib/opensearch.

3.c) Set the OPENSEARCH_PATH_CONF environment variable to the directory that contains opensearch.yml (e.g. /etc/opensearch).

3.d) In opensearch.yml, set path.data and path.logs. You might also want to disable the security plugin for now. 

It’s important to note that disabling the security plugin might have implications with your services connected through SSL, namely Logstash, Metricbeat, etc.

opensearch.yml might look something like this:

path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
plugins.security.disabled: true

3.e) Port your settings from elasticsearch.yml to opensearch.yml. Most settings use the same names. At a minimum, specify cluster.name, node.name, discovery.seed_hosts, and cluster.initial_master_nodes.

4. Start OpenSearch in the single node or all nodes.

5. Wait for all the nodes to rejoin the cluster in OpenSearch.

Confirm using this curl:

# Security plugin disabled
curl -XGET 'localhost:9200/_nodes/_all?pretty=true'
# Security plugin enabled
curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_nodes/_all?pretty=true'

6. Repeat until all nodes have OpenSearch.

7. Re-enable shard allocation.

PUT _cluster/settings
{
 "persistent": {
   "cluster.routing.allocation.enable": "all"
 }
}

2.Snapshots

Snapshots are backups of the entire cluster state at a given point in time, including cluster settings, nodes settings and index metadata. Snapshots are useful for recovery from a failure, but also for migrations.

We will generate a Snapshot in Elasticsearch and then restore it in OpenSearch. In this case, we will save the snapshot in a shared directory between Elasticsearch and Opensearch.

Keep in mind that snapshots must meet the same criteria as normal rolling upgrades. If the snapshot to restore does contain indices having an incompatible version, you will have to reindex your indices in the new version and then generate the snapshot.

More information about snapshot versions compatibility can be found here

This method is supported up to Elasticsearch 7.12, versions beyond 7.12 are not supported yet.

There are many ways to register a repository. You can follow the instructions in this guide.

After registering the repository we can confirm everything is as it should be:

GET _snapshot/backups

We should see something like this:

{
 "backups" : {
   "type" : "fs",
   "settings" : {
     "location" : "/mnt/snapshots"
   }
 }
}

We can use this same setting in our OpenSearch cluster:

PUT localhost:9090/_snapshot/backups
{
 "type": "fs",
 "settings": {
   "location": "/mnt/snapshots"
 }
}

Now Elasticsearch and Opensearch share the same Snapshots folder.

To create a snapshot run the following in Elasticsearch:

PUT _snapshot/backups/es_bk
{
 "indices": "snapshot_data"
}

The name of the backup is “es_bk” and we are restoring the “snapshot_data” index.

Confirm you can see the backup running the following in Elasticsearch:

GET _snapshot/backups/es_bk

Once confirmed, restore the es_bk snapshot into Opensearch:

POST _snapshot/backups/es_bk/_restore
{
 "accepted": true
}

You can monitor the progress of the restore using the recovery API:

GET _cat/recovery?v&active_only=true

3.Reindex API

The Reindex API is pretty straightforward. You just set a source and a destination index, and documents will be indexed in that destination. (See some tips for improving reindexing performance here.)

The source index can be remote, so we can take advantage of this feature to move our data from Elasticsearch to Opensearch. This strategy is recommended for small datasets/testing. 

Reindexing is slower than Snapshots, and will not take into account cluster settings or index mappings.

The biggest pro of this approach is you can migrate data from Elasticsearch 5.x/6.x or 7.12.0+ to Opensearch 1.0.

Another advantage of this method is we can provide a query to filter out the documents we want to migrate.

As we mentioned, the reindex will not copy our mappings/settings, so it’s a good idea to generate an empty destination index with the same settings as our source index.

Let’s quickly create an index so we can explore its configuration.

POST test_index

{
 "name": "Example name",
 "otherproperty": "foo"
}

This will generate a new index with dynamic mappings associated with the document we created. It can be explored running a GET call:

GET test_index

And we get a response like this:

{
 "test_index": {
   "aliases": {},
   "mappings": {
     "properties": {
       "name": {
         "type": "text",
         "fields": {
           "keyword": {
             "type": "keyword",
             "ignore_above": 256
           }
         }
       },
       "otherproperty": {
         "type": "text",
         "fields": {
           "keyword": {
             "type": "keyword",
             "ignore_above": 256
           }
         }
       }
     }
   },
   "settings": {
     "index": {
       "number_of_shards": "1",
       "auto_expand_replicas": null,
       "provided_name": "test_index",
       "creation_date": "1632096818796",
       "priority": "0",
       "number_of_replicas": "1",
       "uuid": "eTux87PyRT6WZoOQrMJVVQ",
       "version": {
         "created": "7100299"
       }
     }
   }
 }
}

From here we have to delete some auto generated properties:

"provided_name": "test_index",
"creation_date": "1632096818796",
"uuid": "eTux87PyRT6WZoOQrMJVVQ",
"version": {
   "created": "7100299"
 }

And finally we can create our destination index in Opensearch:

PUT test_index_2
{
 "mappings": {
   "properties": {
     "name": {
       "type": "text",
       "fields": {
         "keyword": {
           "type": "keyword",
           "ignore_above": 256
         }
       }
     },
     "otherproperty": {
       "type": "text",
       "fields": {
         "keyword": {
           "type": "keyword",
           "ignore_above": 256
         }
       }
     }
   }
 },
 "settings": {
   "index": {
     "number_of_shards": "1",
     "auto_expand_replicas": null,
     "priority": "0",
     "number_of_replicas": "1"
   }
 }
}

 Now we can run in our OpenSearch index:

POST _reindex
{
 "source": {
   "remote": {
     "host": "http://localhost:9200"
   },
   "index": "test_index",
   "query": {
     "match_all": {}
   }
 },
 "dest": {
   "index": "test_index_2"
 }
}

Note the match_all query added to illustrate how to narrow the documents being reindexed. Now test_index_2 in OpenSearch will have the same data as test_index in Elasticsearch. You can set some parameters like maximum documents to ingest, or request per second, as explained here.

4.OpenSearch upgrade tool [NEW ON Opensearch 1.1] 

In the latest Opensearch 1.1 release, there is a new upgrade tool to automate some of the steps we described in this article in the “Upgrading Opensearch” section.

The full documentation can be found here. 

Limitations of the upgrade tool

• Needs to be run in every node of the cluster individually.
• Has no rollback option, so backups need to be made.
• Community plugins (if any) must be installed manually.
• You must remove any unsupported setting manually.

How it works

• Connects to the running Elasticsearch installation.
• Verifies if the version is compatible for migration.
• Imports settings from elasticsearch.yml to opensearch.yml.
• Copies settings for JVM (jvm.options.d) and logging (log4j2.properties) from ES to OS.
• Installs core plugins. Third-party ones must be installed manually.
• Imports secure settings from elasticsearch.keystore to opensearch.keystore.

How to use the upgrade tool

1. Disable shard allocation:

PUT _cluster/settings
{
 "persistent": {
   "cluster.routing.allocation.enable": "primaries"
 }
}

2. Download and extract the OpenSearch tarball to a new directory.

3. Make sure Environment variables are set properly:

4. Run the upgrade tool:

./bin/opensearch-upgrade

5. Stop Elasticsearch in the node:

sudo systemctl stop elasticsearch.service

6. Start OpenSearch in the node:

./bin/opensearch -d

7. Repeat in all nodes.

8. After all nodes are updated, re-enable shard allocation:

PUT _cluster/settings
{
 "persistent": {
   "cluster.routing.allocation.enable": "all"
 }
}

Final words

There are many ways to migrate your data between Elasticsearch and OpenSearch, and the best approach for you will depend on the amount of data you have, whether you want to restore your cluster settings, whether you care about losing uptime in the process, and your versions.