Add ensureCustomSerialization to ensure that headers are serialized correctly with multiple transport hops

[cwperks]

Description

This PR fixes multiple issues:

• [BUG] Unexpected failure while sending request, -84 is not a valid id #4494
• [BUG] DLS Performance with 2.16 #4670

When upgrading from 2.11/2.12/2.13 to >= 2.14, there can be an issue when deserializing threadContext headers on the receiving node due to the transmitting node re-using the serialization format from a previous transport hop.

This was particularly seen in scenarios where there is a chain of transport hops, such as 2.14 (Coordinator node) -> 2.14 node -> 2.12 node

These scenarios can happen on ingestion during rolling upgrades where a replica shard may temporarily be on an older node than a primary shard.

This PR adds logic in the SecurityInterceptor to re-serialize the headers if they have previously been populated from a prior transport hop for backwards compatibility.

This PR adds a version check to the logic around ensureJdkSerialization and ensureCustomSerialization to only execute the logic in a mixed cluster containing nodes < 2.14. If all nodes are >= 2.14 then this logic is redundant and adds additional overhead.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Bug fix

Issues Resolved

• Resolves [BUG] Unexpected failure while sending request, -84 is not a valid id #4494
• Resolves [BUG] DLS Performance with 2.16 #4670

Testing

Creating this PR in Draft until testing details are added.

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.

Project coverage is 65.54%. Comparing base (2dbc508) to head (efa5c1e).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/security/configuration/ClusterInfoHolder.java	60.00%	1 Missing and 1 partial ⚠️
...search/security/transport/SecurityInterceptor.java	86.66%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4741      +/-   ##
==========================================
- Coverage   65.54%   65.54%   -0.01%     
==========================================
  Files         319      319              
  Lines       22448    22470      +22     
  Branches     3602     3604       +2     
==========================================
+ Hits        14714    14728      +14     
- Misses       5927     5933       +6     
- Partials     1807     1809       +2     

Files with missing lines	Coverage Δ
...org/opensearch/security/filter/SecurityFilter.java	65.87% <100.00%> (ø)
.../org/opensearch/security/support/Base64Helper.java	94.11% <100.00%> (+3.20%)	⬆️
...arch/security/configuration/ClusterInfoHolder.java	65.38% <60.00%> (-1.29%)	⬇️
...search/security/transport/SecurityInterceptor.java	80.68% <86.66%> (+0.91%)	⬆️

... and 9 files with indirect coverage changes

[peternied]

This was particularly seen in scenarios where there is a chain of transport hops, such as 2.14 (Coordinator node) -> 2.14 node -> 2.12 node

What a scenario - great job tracking this down @cwperks

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-2.x
# Create a new branch
git switch --create backport/backport-4741-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8ae88a7049291abc3be2003552a04fbc6d092fb1
# Push it to GitHub
git push --set-upstream origin backport/backport-4741-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-4741-to-2.x.

[willyborankin]

@cwperks Could you plz prepare a manual backport?

[cwperks]

Opened a backport here: #4743