Quick links
- Can the 7.11+ Elasticsearch versions be migrated to OpenSearch?
- Migration methods for 7.11+ Elasticsearch versions:
- Summary
Can the 7.11+ versions (after Opensearch fork) be migrated to OpenSearch?
Rolling upgrades cannot be used to migrate from ES to OS if you’re using Elasticsearch version 7.11 or later. For these cases there are other methods for migration, as detailed below, that will enable you to migrate from Elasticsearch 7.11+ versions.
Migration methods
When you are using later Elasticsearch versions, there are essentially 3 methods for migrating to OpenSearch:
1. Snapshots
2. Reindex API
3. OpenSearch upgrade tool
1. Snapshots
Snapshots are backups of the entire cluster state at a given point in time, including cluster settings, nodes settings and index metadata. Snapshots are useful for recovery from a failure, but also for migrations.
We will generate a Snapshot in Elasticsearch and then restore it in OpenSearch. In this case, we will save the snapshot in a shared directory between Elasticsearch and OpenSearch.
Keep in mind that snapshots must meet the same criteria as normal rolling upgrades. If the snapshot to restore does contain indices having an incompatible version, you will have to reindex your indices in the new version and then generate the snapshot.
More information about snapshot versions compatibility can be found here.
This method is supported up to Elasticsearch 7.12, versions beyond 7.12 are not supported yet.
There are many ways to register a repository. You can follow the instructions in this guide.
After registering the repository we can confirm everything is as it should be:
GET _snapshot/backups
We should see something like this:
{
"backups" : {
"type" : "fs",
"settings" : {
"location" : "/mnt/snapshots"
}
}
}We can use this same setting in our OpenSearch cluster:
PUT localhost:9090/_snapshot/backups
{
"type": "fs",
"settings": {
"location": "/mnt/snapshots"
}
}Now Elasticsearch and OpenSearch share the same Snapshots folder.
To create a snapshot, run the following in Elasticsearch:
PUT _snapshot/backups/es_bk
{
"indices": "snapshot_data"
}The name of the backup is “es_bk” and we are restoring the “snapshot_data” index.
Confirm you can see the backup running the following in Elasticsearch:
GET _snapshot/backups/es_bk
Once confirmed, restore the es_bk snapshot into OpenSearch:
POST _snapshot/backups/es_bk/_restore
{
"accepted": true
}You can monitor the progress of the restore using the recovery API:
GET _cat/recovery?v&active_only=true
2. Reindex API
The Reindex API is pretty straightforward. You just set a source and a destination index, and documents will be indexed in that destination.
The source index can be remote, so we can take advantage of this feature to move our data from Elasticsearch to OpenSearch. This strategy is recommended for small datasets/testing.
Reindexing is slower than Snapshots, and will not take into account cluster settings or index mappings.
The biggest pro of this approach is you can migrate data from Elasticsearch 5.x/6.x or 7.12.0+ to OpenSearch 1.0.
Another advantage of this method is we can provide a query to filter out the documents we want to migrate.
The reindex will not copy our mappings/settings, so it’s a good idea to generate an empty destination index with the same settings as our source index.
Let’s quickly create an index so we can explore its configuration.
POST test_index
{
"name": "Example name",
"otherproperty": "foo"
}This will generate a new index with dynamic mappings associated with the document we created. It can be explored running a GET call:
GET test_index
We get a response like this:
{
"test_index": {
"aliases": {},
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"otherproperty": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": "1",
"auto_expand_replicas": null,
"provided_name": "test_index",
"creation_date": "1632096818796",
"priority": "0",
"number_of_replicas": "1",
"uuid": "eTux87PyRT6WZoOQrMJVVQ",
"version": {
"created": "7100299"
}
}
}
}
}
From here we have to delete some auto generated properties:
"provided_name": "test_index",
"creation_date": "1632096818796",
"uuid": "eTux87PyRT6WZoOQrMJVVQ",
"version": {
"created": "7100299"
}And finally we can create our destination index in OpenSearch:
PUT test_index_2
{
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"otherproperty": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": "1",
"auto_expand_replicas": null,
"priority": "0",
"number_of_replicas": "1"
}
}
}Now we can run in our OpenSearch index:
POST _reindex
{
"source": {
"remote": {
"host": "http://localhost:9200"
},
"index": "test_index",
"query": {
"match_all": {}
}
},
"dest": {
"index": "test_index_2"
}
}Note the match_all query added to illustrate how to narrow the documents being reindexed.
Now test_index_2 in OpenSearch will have the same data as test_index in Elasticsearch.
You can set some parameters like maximum documents to ingest, or request per second, as explained here.
3. OpenSearch upgrade tool [NEW ON OpenSearch 1.1]
In the latest OpenSearch 1.1 release, there is a new upgrade tool to automate some of the steps for upgrading OpenSearch.
The full documentation can be found here.
Limitations of the upgrade tool
- Needs to be run in every node of the cluster individually.
- Has no rollback option, so backups need to be made.
- Community plugins (if any) must be installed manually.
- You must remove any unsupported setting manually.
How it works
- The upgrade tool connects to the running Elasticsearch installation.
- Verifies if the version is compatible for migration.
- Imports settings from elasticsearch.yml to opensearch.yml.
- Copies settings for JVM (jvm.options.d) and logging (log4j2.properties) from ES to OS.
- Installs core plugins. Third-party ones must be installed manually.
- Imports secure settings from elasticsearch.keystore to opensearch.keystore.
How to use the upgrade tool
- Disable shard allocation:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}- Download and extract the OpenSearch tarball to a new directory.
- Make sure Environment variables are set properly:
- ES_HOME – Path to the existing Elasticsearch installation home.
- export ES_HOME = /home/workspace/upgrade-demo/node1/elasticsearch-7.10.2
- ES_PATH_CONF – Path to the existing Elasticsearch config directory.
- export ES_PATH_CONF = /home/workspace/upgrade-demo/node1/es-config
- OPENSEARCH_HOME – Path to the OpenSearch installation home.
- export OPENSEARCH_HOME = /home/workspace/upgrade-demo/node1/opensearch-1.0.0
- OPENSEARCH_PATH_CONF – Path to the OpenSearch config directory.
- export OPENSEARCH_PATH_CONF = /home/workspace/upgrade-demo/node1/opensearch-config
- Run the upgrade tool:
./bin/opensearch-upgrade
- Stop Elasticsearch in the node:
sudo systemctl stop elasticsearch.service
- Start OpenSearch in the node:
./bin/opensearch -d
- Repeat in all nodes.
- After all nodes are updated, re-enable shard allocation:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}
Summary
After the ES/OS fork, differences between Elasticsearch and OpenSearch internals will start growing, and migrating will start getting more tricky.
The Snapshot method is useful only until the 7.12 version because Elasticsearch snapshots change from that version onwards.
The Reindex API is more stable because it relies on the Index API, which is a core functionality and should not be changing in the mid/long term. It can be used to move files from an ES cluster to an OS one without hassle. This method is focused on the data, and does not carry any cluster configuration.
To migrate all settings you can use the OpenSearch migration tool, which will also tell you if your Elasticsearch version can be migrated with that tool or not.
