HotToWarmTieringService changes and changes in shard balancer to tier shards #144

neetikasinghal · 2024-07-16T22:41:57Z

Description

Adds in the service side of the changes needed for hot to warm tiering.
Also, adds in the logic to run shard allocation for the shards accepted for tiering - it does not handle tiering of replicas yet (Allocator changes to support tiering with replicas opensearch-project/OpenSearch#14850)
Also tested the tiering changes end-to-end by pulling in recovery changes: Writable warm relocation recovery opensearch-project/OpenSearch#14670

Related Issues

opensearch-project#14545
opensearch-project#13980

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

server/src/main/java/org/opensearch/common/settings/IndexScopedSettings.java

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

kotwanikunal · 2024-07-17T17:01:05Z

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

+     * pool.
+     */
+    public void tierShards() {
+        List<ShardRouting> shardsPendingTiering = new ArrayList<>();


It might be better to break this down into replicas and primary.
We can run the operation on replicas first, and the migration for primary would handle the scenario of flipping replica to primary.
(There would be a case where you have no replica - which is fine. But when you do, it is safer)

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/ShardsBalancer.java

server/src/main/java/org/opensearch/node/Node.java

server/src/main/java/org/opensearch/tiering/HotToWarmTieringService.java

...c/internalClusterTest/java/org/opensearch/remotestore/tiering/HotToWarmTieringServiceIT.java

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

server/src/test/java/org/opensearch/indices/tiering/HotToWarmTieringServiceTests.java

server/src/main/java/org/opensearch/indices/tiering/HotToWarmTieringService.java

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

server/src/main/java/org/opensearch/node/Node.java

…arch-project#14851) Signed-off-by: RS146BIJAY <[email protected]>

…roject#14812) * Use default value when index.number_of_replicas is null Signed-off-by: Liyun Xiu <[email protected]> * Add integration test Signed-off-by: Liyun Xiu <[email protected]> * Add changelog Signed-off-by: Liyun Xiu <[email protected]> --------- Signed-off-by: Liyun Xiu <[email protected]>

…ile. (opensearch-project#14684) * Implement write and read flow to upload/download shard diff file. Signed-off-by: Shailendra Singh <[email protected]>

…oject#14426) * Optimize Cluster Stats Indices to precomute node level stats Signed-off-by: Pranshu Shukla <[email protected]>

…ary shards per index (opensearch-project#14908) Signed-off-by: Gaurav Bafna <[email protected]>

…-project#14761) Signed-off-by: RS146BIJAY <[email protected]>

…ch-project#14918) The test would previously fail if the randomness led to only a single indexing request being included in the bulk payload. This change guarantees multiple indexing requests in order to ensure the batch logic kicks in. Also replace some unneeded mocks with real classes. Signed-off-by: Andrew Ross <[email protected]>

…shAndMergeHeaders (opensearch-project#15051) Signed-off-by: Craig Perkins <[email protected]>

Signed-off-by: bowenlan-amzn <[email protected]>

…ch distribution (opensearch-project#15082) Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

Resolves: opensearch-project#14706 Signed-off-by: Peter Nied <[email protected]>

…ensearch-project#14711) --------- Signed-off-by: kkewwei <[email protected]>

…ensearch-project#15072) Signed-off-by: Jay Deng <[email protected]>

…ject#15037) Signed-off-by: Gaurav Bafna <[email protected]>

…ta stream but is unused (opensearch-project#15080) * Fix delete not-using index template failed when the index pattern matches a data stream Signed-off-by: Gao Binlong <[email protected]> * modify change log Signed-off-by: Gao Binlong <[email protected]> * Fix version check Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]>

Signed-off-by: Craig Perkins <[email protected]>

…e failed (opensearch-project#15101) Signed-off-by: Gao Binlong <[email protected]>

…pensearch-project#15110) * Bump org.tukaani:xz from 1.9 to 1.10 in /plugins/ingest-attachment Bumps [org.tukaani:xz](https://github.com/tukaani-project/xz-java) from 1.9 to 1.10. - [Release notes](https://github.com/tukaani-project/xz-java/releases) - [Changelog](https://github.com/tukaani-project/xz-java/blob/master/NEWS.md) - [Commits](tukaani-project/xz-java@v1.9...v1.10) --- updated-dependencies: - dependency-name: org.tukaani:xz dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…NDEX_SETTING feature (opensearch-project#15076) Signed-off-by: Neetika Singhal <[email protected]>

* Bump actions/setup-java from 1 to 4 Bumps [actions/setup-java](https://github.com/actions/setup-java) from 1 to 4. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@v1...v4) --- updated-dependencies: - dependency-name: actions/setup-java dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…#15086) GitHub doesn't suport dynamic days since created/modified, I've created a simple redirect on my website that will support this use case. See https://peternied.github.io/redirect/issue_search.html for full context on what is avaliable. Source is available on https://github.com/peternied/peternied.github.io Signed-off-by: Peter Nied <[email protected]>

…y-hdfs (opensearch-project#15119) * Bump org.apache.avro:avro from 1.11.3 to 1.12.0 in /plugins/repository-hdfs Signed-off-by: Craig Perkins <[email protected]> * Add to CHANGELOG Signed-off-by: Craig Perkins <[email protected]> --------- Signed-off-by: Craig Perkins <[email protected]>

…astle:bc-fips from 1.0.2.5 to 2.0.0 (opensearch-project#15122) * Bump org.bouncycastle:bcpg-fips from 1.0.7.1 to 2.0.8 and org.bouncycastle:bc-fips from 1.0.2.5 to 2.0.0 in /distribution/tools/plugin-cli Signed-off-by: Craig Perkins <[email protected]> * Add to CHANGELOG Signed-off-by: Craig Perkins <[email protected]> --------- Signed-off-by: Craig Perkins <[email protected]>

Signed-off-by: David Zane <[email protected]>

…onstant_keyword field type (opensearch-project#15127) Signed-off-by: Gao Binlong <[email protected]>

…y-azure (opensearch-project#15111) * Bump com.azure:azure-core in /plugins/repository-azure Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.49.1 to 1.51.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core_1.49.1...azure-core_1.51.0) --- updated-dependencies: - dependency-name: com.azure:azure-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ct#15134) Signed-off-by: Rishabh Singh <[email protected]>

Signed-off-by: Liyun Xiu <[email protected]>

Signed-off-by: Jay Deng <[email protected]>

--------- Signed-off-by: Bharathwaj G <[email protected]>

…-project#13897) * initial code for the sandbox resource tracking and cancellation framework Signed-off-by: Kiran Prakash <[email protected]> * Fix Failing Tests Signed-off-by: Kiran Prakash <[email protected]> * spotless Apply Signed-off-by: Kiran Prakash <[email protected]> * Update SandboxService.java Signed-off-by: Kiran Prakash <[email protected]> * Update SandboxService.java Signed-off-by: Kiran Prakash <[email protected]> * Update SandboxTask.java Signed-off-by: Kiran Prakash <[email protected]> * Add java docs Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * javadocs Signed-off-by: Kiran Prakash <[email protected]> * javadocs Signed-off-by: Kiran Prakash <[email protected]> * java docs Signed-off-by: Kiran Prakash <[email protected]> * Update AbstractTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update SandboxModule.java Signed-off-by: Kiran Prakash <[email protected]> * Some tests and stubs Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * :server:testingConventions Signed-off-by: Kiran Prakash <[email protected]> * Update AbstractTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * more tests Signed-off-by: Kiran Prakash <[email protected]> * addressing comments Signed-off-by: Kiran Prakash <[email protected]> * revert some accidentally pushed files Signed-off-by: Kiran Prakash <[email protected]> * resolve flakiness Signed-off-by: Kiran Prakash <[email protected]> * renaming sandbox to querygroup and adjusting code based on merged PRs Signed-off-by: Kiran Prakash <[email protected]> * jvm to memory Signed-off-by: Kiran Prakash <[email protected]> * missing java docs Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * pluck cancellation changes out of this PR Signed-off-by: Kiran Prakash <[email protected]> * remove unused Signed-off-by: Kiran Prakash <[email protected]> * remove cancellation related code and add more tests coverage Signed-off-by: Kiran Prakash <[email protected]> * us only memory and not jvm Signed-off-by: Kiran Prakash <[email protected]> * test conventions Signed-off-by: Kiran Prakash <[email protected]> * Bring back enum Signed-off-by: Kiran Prakash <[email protected]> * Update SearchBackpressureService.java Signed-off-by: Kiran Prakash <[email protected]> * revert changes Signed-off-by: Kiran Prakash <[email protected]> * revert changes Signed-off-by: Kiran Prakash <[email protected]> * all required changes Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * cleanups Signed-off-by: Kiran Prakash <[email protected]> * Delete QueryGroupService.java Signed-off-by: Kiran Prakash <[email protected]> * cleanups Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupLevelResourceUsageViewTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupLevelResourceUsageViewTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * rebasing with latest main Signed-off-by: Kiran Prakash <[email protected]> * remove experimental Signed-off-by: Kiran Prakash <[email protected]> * remove queryGroupId Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <[email protected]> * change code comments Signed-off-by: Kiran Prakash <[email protected]> * remmove QueryGroupUsageTracker Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <[email protected]> * remove QueryGroupTestHelpers Signed-off-by: Kiran Prakash <[email protected]> * cleanups Signed-off-by: Kiran Prakash <[email protected]> * remove queryGroupHelper Signed-off-by: Kiran Prakash <[email protected]> * Update ResourceTypeTests.java Signed-off-by: Kiran Prakash <[email protected]> * extend OpenSearchTestCase Signed-off-by: Kiran Prakash <[email protected]> * pr comments Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update QueryGroupResourceUsageTrackerServiceTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update ResourceTypeTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update ResourceTypeTests.java Signed-off-by: Kiran Prakash <[email protected]> * Update ResourceType.java Signed-off-by: Kiran Prakash <[email protected]> * Update ResourceType.java Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]>

Signed-off-by: Jay Deng <[email protected]>

…search-project#14234) Signed-off-by: Andrew Ross <[email protected]>

Signed-off-by: Neetika Singhal <[email protected]>

neetikasinghal requested a review from sohami as a code owner July 16, 2024 22:41

neetikasinghal commented Jul 16, 2024

View reviewed changes

server/src/main/java/org/opensearch/common/settings/IndexScopedSettings.java Outdated Show resolved Hide resolved

kotwanikunal reviewed Jul 17, 2024

View reviewed changes

jed326 reviewed Jul 17, 2024

View reviewed changes

neetikasinghal mentioned this pull request Jul 17, 2024

Add rest, transport layer changes for Tiering opensearch-project/OpenSearch#13980

Merged

9 tasks

harishbhakuni reviewed Jul 17, 2024

View reviewed changes

neetikasinghal force-pushed the tiering branch from 5569b43 to da7c5c2 Compare July 18, 2024 07:35

neetikasinghal force-pushed the tiering-service branch from 044ea5b to 0a64950 Compare July 19, 2024 00:22

harishbhakuni reviewed Jul 19, 2024

View reviewed changes

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/node/Node.java Show resolved Hide resolved

neetikasinghal force-pushed the tiering-service branch from 0a64950 to dbb74c8 Compare July 19, 2024 19:12

neetikasinghal force-pushed the tiering branch 3 times, most recently from 9f55e78 to b1107ec Compare July 22, 2024 17:29

neetikasinghal force-pushed the tiering-service branch from dbb74c8 to fde4f91 Compare July 22, 2024 18:33

neetikasinghal force-pushed the tiering branch from b1107ec to 31256f3 Compare July 22, 2024 22:38

neetikasinghal changed the base branch from tiering to main July 23, 2024 05:08

neetikasinghal changed the base branch from main to tiering July 23, 2024 05:09

neetikasinghal marked this pull request as draft July 23, 2024 05:10

neetikasinghal force-pushed the tiering-service branch from fde4f91 to 2e9f80d Compare July 23, 2024 05:12

neetikasinghal force-pushed the tiering branch from 31256f3 to 8ff3bcc Compare July 23, 2024 05:16

neetikasinghal mentioned this pull request Jul 23, 2024

HotToWarmTieringService changes to tier shards opensearch-project/OpenSearch#14891

Open

3 tasks

Caching avg total bytes and avg free bytes inside ClusterInfo (opense…

1305002

…arch-project#14851) Signed-off-by: RS146BIJAY <[email protected]>

neetikasinghal force-pushed the tiering-service branch from 2e9f80d to 052d551 Compare July 23, 2024 05:38

chishui and others added 6 commits July 23, 2024 08:14

[Remote Routing Table] Implement write and read flow for shard diff f…

f85a58f

…ile. (opensearch-project#14684) * Implement write and read flow to upload/download shard diff file. Signed-off-by: Shailendra Singh <[email protected]>

Optimized ClusterStatsIndices to precomute shard stats (opensearch-pr…

5026af6

…oject#14426) * Optimize Cluster Stats Indices to precomute node level stats Signed-off-by: Pranshu Shukla <[email protected]>

Fix constraint bug which allows more primary shards than average prim…

3497081

…ary shards per index (opensearch-project#14908) Signed-off-by: Gaurav Bafna <[email protected]>

Optmising AwarenessAllocationDecider for hashmap.get call (opensearch…

e46d1d8

…-project#14761) Signed-off-by: RS146BIJAY <[email protected]>

cwperks and others added 23 commits July 31, 2024 20:23

Add javadoc about ThreadContextPermission for stashWithOrigin and sta…

67a2e4c

…shAndMergeHeaders (opensearch-project#15051) Signed-off-by: Craig Perkins <[email protected]>

Add 2.17.0 in main branch (opensearch-project#15053)

d4e7766

Signed-off-by: bowenlan-amzn <[email protected]>

Add MacOS aarch64 to precommit since we rolled out the support for su…

7c471a0

…ch distribution (opensearch-project#15082) Signed-off-by: Andriy Redko <[email protected]>

Bump Netty to 4.1.112.Final (opensearch-project#15081)

48634bd

Signed-off-by: Andriy Redko <[email protected]>

Decommission the Core Triage meeting (opensearch-project#15085)

f829a9f

Resolves: opensearch-project#14706 Signed-off-by: Peter Nied <[email protected]>

support rangeQuery and regexpQuery in constant_keyword field type (op…

bd226c2

…ensearch-project#14711) --------- Signed-off-by: kkewwei <[email protected]>

Support scripting for composite aggs in concurrent segment search (op…

a785073

…ensearch-project#15072) Signed-off-by: Jay Deng <[email protected]>

Fix RemoteCloneIndex flaky test by using sync FS repo (opensearch-pro…

47171f8

…ject#15037) Signed-off-by: Gaurav Bafna <[email protected]>

Update README to 2.17.0 (opensearch-project#15099)

e8c6f0f

Signed-off-by: Craig Perkins <[email protected]>

Fix version check in yml test for the bug fix of delete index templat…

b911b6f

…e failed (opensearch-project#15101) Signed-off-by: Gao Binlong <[email protected]>

Fix NODE_SEARCH_CACHE_SIZE_SETTING initialization for TIERED_REMOTE_I…

f0ef14d

…NDEX_SETTING feature (opensearch-project#15076) Signed-off-by: Neetika Singhal <[email protected]>

Add took time to request nodes stats (opensearch-project#15054)

76b9931

Signed-off-by: David Zane <[email protected]>

Fix version check for adding rangeQuery and regexpQuery support for c…

c725431

…onstant_keyword field type (opensearch-project#15127) Signed-off-by: Gao Binlong <[email protected]>

Add baseline-cluster-config key to benchmark config (opensearch-proje…

f980924

…ct#15134) Signed-off-by: Rishabh Singh <[email protected]>

Fix bulk ingest NPE with empty pipeline (opensearch-project#15033)

b47b401

Signed-off-by: Liyun Xiu <[email protected]>

CODEOWNERS personalizations for jed326 (opensearch-project#15137)

212597e

Signed-off-by: Jay Deng <[email protected]>

neetikasinghal force-pushed the tiering-service branch from 4774394 to 7da1aa6 Compare August 7, 2024 00:41

bharath-techie and others added 5 commits August 7, 2024 12:27

Add changes to build star tree in off heap (opensearch-project#14817)

a918530

--------- Signed-off-by: Bharathwaj G <[email protected]>

Fix CHANGELOG for opensearch-project#15054 (opensearch-project#15150)

348c04e

Signed-off-by: Jay Deng <[email protected]>

Add 'ShardManagement:*' labels to Cluster Manager triage search (open…

7f72a6e

…search-project#14234) Signed-off-by: Andrew Ross <[email protected]>

HotToWarmTieringService changes to tier shards

d99f55f

Signed-off-by: Neetika Singhal <[email protected]>

neetikasinghal force-pushed the tiering-service branch from 7da1aa6 to d99f55f Compare August 7, 2024 23:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

neetikasinghal commented Jul 16, 2024 •

edited

Loading

kotwanikunal Jul 17, 2024

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

Are you sure you want to change the base?

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

Conversation

neetikasinghal commented Jul 16, 2024 • edited Loading

Description

Related Issues

Check List

kotwanikunal Jul 17, 2024

Choose a reason for hiding this comment

neetikasinghal commented Jul 16, 2024 •

edited

Loading