Backport/backport 6280 to 2.x

[dreamer-89]

Manual backport of #6280 to 2.x

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/11025/
• CommitID: a781dc0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/11025/
• CommitID: a781dc0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

#6248

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreIT.testRemoteSegmentStoreRestore" -Dtests.seed=6C75DC5D95F409A0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-LY -Dtests.timezone=Asia/Chungking -Druntime.java=17

org.opensearch.remotestore.RemoteStoreIT > testRemoteSegmentStoreRestore FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([6C75DC5D95F409A0:F88BD9C4AFBD1A70]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1007)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:938)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:927)
        at org.opensearch.remotestore.RemoteStoreIT.testRemoteSegmentStoreRestore(RemoteStoreIT.java:165)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock

• URL: https://build.ci.opensearch.org/job/gradle-check/11027/
• CommitID: a781dc0
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov-commenter]

Codecov Report

Merging #6288 (a781dc0) into 2.x (c5eb96a) will increase coverage by 0.23%.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##                2.x    #6288      +/-   ##
============================================
+ Coverage     70.23%   70.47%   +0.23%     
- Complexity    58881    59111     +230     
============================================
  Files          4785     4785              
  Lines        283490   283490              
  Branches      41259    41259              
============================================
+ Hits         199123   199789     +666     
+ Misses        67679    66997     -682     
- Partials      16688    16704      +16     

Impacted Files	Coverage Δ
...main/java/org/opensearch/node/NodeMocksPlugin.java	0.00% <0.00%> (-100.00%)	⬇️
...rch/test/junit/listeners/ReproduceInfoPrinter.java	9.52% <0.00%> (-68.26%)	⬇️
...a/org/opensearch/client/cluster/SniffModeInfo.java	0.00% <0.00%> (-62.50%)	⬇️
...a/org/opensearch/tasks/TaskCancelledException.java	50.00% <0.00%> (-50.00%)	⬇️
.../index/shard/IndexShardNotRecoveringException.java	0.00% <0.00%> (-50.00%)	⬇️
...cluster/coordination/PendingClusterStateStats.java	20.00% <0.00%> (-48.00%)	⬇️
...adcast/BroadcastShardOperationFailedException.java	55.55% <0.00%> (-44.45%)	⬇️
...ndex/seqno/RetentionLeaseBackgroundSyncAction.java	37.50% <0.00%> (-42.86%)	⬇️
.../search/aggregations/pipeline/HoltLinearModel.java	20.33% <0.00%> (-42.38%)	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	22.22% <0.00%> (-40.75%)	⬇️
... and 524 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[dreamer-89]

@mch2 @Rishikesh1159 @reta : Request for review.