Allow upgrading from ES6 to OpenSearch #4444

akshaymankar · 2025-02-05T13:22:27Z

Ticket: https://wearezeta.atlassian.net/browse/WPB-16066

Checklist

Add a new entry in an appropriate subdirectory of changelog.d
Read and follow the PR guidelines

supersven · 2025-02-10T16:03:11Z

Rebased.

supersven · 2025-02-10T17:16:56Z

ES 7

concourse-ci/integration-tests-kube-ci with ES7

brig-index-migrate-data crash-loops with:

D, No TLS cert file path configured.
D, Connecting to 10.244.12.110:9042
I, Known hosts: [datacenter1:rack1:10.244.12.110:9042]
I, New control connection: datacenter1:rack1:10.244.12.110:9042#<socket: 71>
I, request=brig-index, Migration necessary., expectedVersion=6, foundVersion=0
D, Connection established: datacenter1:rack1:10.244.12.110:9042#<socket: 73>
D, Connection established: datacenter1:rack1:10.244.12.110:9042#<socket: 74>
brig-index: PersistVersionFailed (MigrationVersion {migrationVersion = 6}) "Response {responseStatus = Status {statusCode = 400, statusMessage = \"Bad Request\"}, responseVersion = HTTP/1.1, responseHeaders = [(\"X-elastic-product\",\"Elasticsearch\"),(\"Warning\",\"299 Elasticsearch-7.17.27-0f88dde84795b30ca0d2c0c4796643ec5938aeb5 \\\"[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).\\\"\"),(\"content-type\",\"application/json; charset=UTF-8\"),(\"content-length\",\"259\")], responseBody = \"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Invalid type: expecting [_doc] but got [wire_brig_migrations]\\\"}],\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Invalid type: expecting [_doc] but got [wire_brig_migrations]\\\"},\\\"status\\\":400}\", responseCookieJar = CJ {expose = ]}, responseClose' = ResponseClose, responseOriginalRequest = Request {\n  host                 = \"elasticsearch-ephemeral\"\n  port                 = 9200\n  secure               = True\n  requestHeaders       = [(\"Authorization\",\"<REDACTED>\"),(\"Content-Type\",\"application/json\")]\n  path                 = \"/wire_brig_migrations/wire_brig_migrations/6\"\n  queryString          = \"\"\n  method               = \"PUT\"\n  proxy                = Nothing\n  rawBody              = False\n  redirectCount        = 10\n  responseTimeout      = ResponseTimeoutDefault\n  requestVersion       = HTTP/1.1\n  proxySecureMode      = ProxySecureWithConnect\n}\n, responseEarlyHints = ]}"
stream closed EOF for test-ehvzsslclb3c/brig-index-migrate-data-9hncv (brig-index)

ES 6

concourse-ci/integration-tests-kube-ci

brig-index-migrate-data crash-loops with:

D, No TLS cert file path configured.
D, Connecting to 10.244.9.179:9042
I, Known hosts: [datacenter1:rack1:10.244.9.179:9042]
I, New control connection: datacenter1:rack1:10.244.9.179:9042#<socket: 71>
I, request=brig-index, Migration necessary., expectedVersion=6, foundVersion=0
D, Connection established: datacenter1:rack1:10.244.9.179:9042#<socket: 73>
D, Connection established: datacenter1:rack1:10.244.9.179:9042#<socket: 74>
brig-index: PersistVersionFailed (MigrationVersion {migrationVersion = 6}) "Response {responseStatus = Status {statusCode = 400, statusMessage = \"Bad Request\"}, responseVersion = HTTP/1.1, responseHeaders = [(\"content-type\",\"application/json; charset=UTF-8\"),(\"content-length\",\"395\")], responseBody = \"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Rejecting mapping update to [wire_brig_migrations] as the final mapping would have more than 1 type: [_doc, wire_brig_migrations]\\\"}],\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Rejecting mapping update to [wire_brig_migrations] as the final mapping would have more than 1 type: [_doc, wire_brig_migrations]\\\"},\\\"status\\\":400}\", responseCookieJar = CJ {expose = ]}, responseClose' = ResponseClose, responseOriginalRequest = Request {\n  host                 = \"elasticsearch-ephemeral\"\n  port                 = 9200\n  secure               = True\n  requestHeaders       = [(\"Authorization\",\"<REDACTED>\"),(\"Content-Type\",\"application/json\")]\n  path                 = \"/wire_brig_migrations/wire_brig_migrations/6\"\n  queryString          = \"\"\n  method               = \"PUT\"\n  proxy                = Nothing\n  rawBody              = False\n  redirectCount        = 10\n  responseTimeout      = ResponseTimeoutDefault\n  requestVersion       = HTTP/1.1\n  proxySecureMode      = ProxySecureWithConnect\n}\n, responseEarlyHints = ]}"
stream closed EOF for test-s2kasm620frz/brig-index-migrate-data-zxjhg (brig-index)

supersven · 2025-02-11T05:44:08Z

ES 6

Integration tests succeeded: concourse-ci/integration-tests-kube-ci

Even better, if we can directly jump to OpenSearch.

We need wireapp/bloodhound@9ce5c7a to have a putMapping that is compatible with ES 6.

Ensure we didn't break backwards compatibility. It would be nice to tests ES 6 and 7 support in parallel. However, to identify the pain points, this is good enough.

Otherwise, there are two around. Which leads to failures.

underscore prefix At least, OS 1.3 and ES 6 did not complain.

There may be index settings set by the ES instance itself (by default or configuration.) So, do not expect only our settings.

supersven · 2025-02-14T13:18:38Z

Rebased on develop to resolve conflicts with the Redis upgrade (6fe0220).

This is useful to flip back and forth between versions to validate with the integration tests.

The older version threw weird SSL errors like opensearch-project/security#3299 .

…nstance - Make the migrationIndexName configurable to test scenarios where the migrations haven't run. - Add the second index search instance's URL to config

To enable index migration tests to new instances.

The additonal ElasticSearch host is required by brig, not integration.

Otherwise, we may end up with two mappings in ES 6 and may have trouble finding users.

The new name explains better what happens.

This contains ES 6 and ES 7 (OS 1.3) support.

charts/elasticsearch-ephemeral/values.yaml

hack/helmfile.yaml

supersven

It's a bit odd that I'm approving, though I wrote most of the code. However, this was reviewed by @akshaymankar (who cannot approve because he initially opened this PR.)

zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Feb 5, 2025

supersven force-pushed the es-7 branch from b6b78cf to ba228a1 Compare February 10, 2025 16:02

This was referenced Feb 11, 2025

Index document supersven/wire-bloodhound-tests#3

Merged

[Do not merge]: Create index migration images #4453

Closed

akshaymankar and others added 9 commits February 14, 2025 12:15

Upgrade Elasticsearch to 7.17.27

b5418a5

WIP: Get tests to be green with ES7

3969fbe

docker-compose: Use OS 1.3 instead of ES 7

088660e

Even better, if we can directly jump to OpenSearch.

Update bloodhound

5e8ee6c

We need wireapp/bloodhound@9ce5c7a to have a putMapping that is compatible with ES 6.

Run the CI with ES 6

0103032

Ensure we didn't break backwards compatibility. It would be nice to tests ES 6 and 7 support in parallel. However, to identify the pain points, this is good enough.

Be consistent regarding the brig migration mapping name

0442c1a

Otherwise, there are two around. Which leads to failures.

Run integration tests with ES 7

ab783aa

Bulk request versions: _version and _version_type lose their

72805ff

underscore prefix At least, OS 1.3 and ES 6 did not complain.

Cope with default index settings

51a2cd3

There may be index settings set by the ES instance itself (by default or configuration.) So, do not expect only our settings.

supersven force-pushed the es-7 branch from f3c05a0 to 2f3f6b0 Compare February 14, 2025 13:17

Provide helmfile profiles for ElasticSearch and OpenSearch

2f775d9

This is useful to flip back and forth between versions to validate with the integration tests.

supersven force-pushed the es-7 branch from 7f60431 to 2f775d9 Compare February 14, 2025 18:59

supersven added 4 commits February 17, 2025 18:33

docker-compose: Upgrade OpenSearch to 1.3.20

aad9c61

The older version threw weird SSL errors like opensearch-project/security#3299 .

Add test to create and fill a new index on a different index search i…

f1d8deb

…nstance - Make the migrationIndexName configurable to test scenarios where the migrations haven't run. - Add the second index search instance's URL to config

WIP: CI: Adjust intgration test setup for two index search service

13d97cd

To enable index migration tests to new instances.

Fix kube setup

4904f35

The additonal ElasticSearch host is required by brig, not integration.

supersven force-pushed the es-7 branch from 988768d to 4904f35 Compare February 18, 2025 08:27

supersven added 5 commits February 18, 2025 09:37

Fix TLS in local ElasticSearch tests

9f644a4

Always putNamedMapping with the old mapping name

d498da9

Otherwise, we may end up with two mappings in ES 6 and may have trouble finding users.

Add changelogs

927ea75

Rename: createCommand -> createEsIndexCommand

35d0320

The new name explains better what happens.

Nix: Update bloodhound pin to latest wire-fork

127094c

This contains ES 6 and ES 7 (OS 1.3) support.

supersven marked this pull request as ready for review February 19, 2025 09:54

akshaymankar changed the title ~~Upgrade Elasticsearch to 7.17.27~~ Allow upgrading from ES6 to OpenSearch Feb 19, 2025

Remove TODO: We gained confidence that this change is correct

c99e3a2

supersven requested review from a team as code owners February 19, 2025 12:11

akshaymankar commented Feb 19, 2025

View reviewed changes

charts/elasticsearch-ephemeral/values.yaml Outdated Show resolved Hide resolved

hack/helmfile.yaml Outdated Show resolved Hide resolved

supersven and others added 4 commits February 19, 2025 13:13

Delete obsolete TODO

fa6c8d2

Delete obsolete TODO

67e64b2

Update charts/elasticsearch-ephemeral/values.yaml

31733f8

Update hack/helmfile.yaml

ae06aae

supersven approved these changes Feb 20, 2025

View reviewed changes

supersven merged commit fa54154 into develop Feb 20, 2025
10 checks passed

supersven deleted the es-7 branch February 20, 2025 08:14

zebot mentioned this pull request Mar 6, 2025

Release 2025-03-06 - (expected chart version 5.12.0) #4483

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow upgrading from ES6 to OpenSearch #4444

Allow upgrading from ES6 to OpenSearch #4444

akshaymankar commented Feb 5, 2025 •

edited by supersven

Loading

supersven commented Feb 10, 2025

supersven commented Feb 10, 2025 •

edited

Loading

supersven commented Feb 11, 2025

supersven commented Feb 14, 2025

supersven left a comment

Allow upgrading from ES6 to OpenSearch #4444

Allow upgrading from ES6 to OpenSearch #4444

Conversation

akshaymankar commented Feb 5, 2025 • edited by supersven Loading

Checklist

supersven commented Feb 10, 2025

supersven commented Feb 10, 2025 • edited Loading

ES 7

ES 6

supersven commented Feb 11, 2025

ES 6

supersven commented Feb 14, 2025

supersven left a comment

Choose a reason for hiding this comment

akshaymankar commented Feb 5, 2025 •

edited by supersven

Loading

supersven commented Feb 10, 2025 •

edited

Loading