DAOS-14225 control: Ignore duplicate call to SetRank #13169

tanabarr · 2023-10-12T11:09:28Z

Tests are occasionally failing during server start-up due to duplicate
SetRank calls in the control-plane. The reason for the offending call
is related to a race existing on rank 0 which bootstraps the MS and
manually triggers SetRank rather than waiting for the NotifyReady
callback mechanism triggered by a dRPC from the engine.

The previous attempt to remove the special case for rank 0 resulted in
the failure that is presumed to be the reason for implementing the
special case in the first place. This subsequent fix attempt simply
returns early from SetRank function if a rank has already been set on
the engine instance but retains the special case rank setting process
for the bootstrapping rank.

DAOS-14528 test: Fix PoolCreateCapacityTests for md-on-ssd (#13308)

Miscellaneous fixes allowing to run the test for md-on-scm or md-on-ssd.

Co-authored-by: Cedric Koch-Hofer [email protected]
Signed-off-by: Tom Nabarro [email protected]

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2023-10-12T11:09:49Z

Bug-tracker data:
Ticket title is 'pool/create_capacity.py: engine rank 0 killed by local daos_server unexpectly'
Status is 'Resolved'
Labels: 'triaged'
https://daosio.atlassian.net/browse/DAOS-14225

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-10-14T22:45:54Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13169/2/testReport/

daosbuild1 · 2023-10-16T11:28:30Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13169/3/testReport/

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-29T14:14:58Z

Test stage NLT on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/4/execution/node/839/log

tanabarr · 2023-11-30T12:09:55Z

Squashed @knard-intel 's #13308 into this so we can get CI PoolCreateCapacityTests passing with the combined changes.
Have pushed with features pool to ensure coverage of the SetRank change as well as PoolCreateCapacityTests.

GATEKEEPER: Please use the PR title and description as the commit message when merging with master, thanks in advance.

daltonbohning · 2023-11-30T15:20:44Z

src/control/server/instance.go

+	if ei.IsReady() {
+		ei.log.Errorf("SetupRank called on an already set-up instance %d", ei.Index())
+		return nil
+	}
+


Does this PR address the race condition, or do we just log an error on duplicate calls?

Duplicate SetRank() calls get ignored and logged as an error to enable further debug.

Should it be an error or warning? I'm just thinking from the perspective of an Admin wondering why they see this error if it's not really an issue

I will change to debug, @daltonbohning would you mind if I moved the fix for this to #13385 so as to not hold up CI progress, it's been running for a while.

Sure, not an issue from me

I've updated the log level for this entry to Debug in this PR

src/tests/ftest/pool/create_capacity.py

daosbuild1

LGTM. No errors found by checkpatch.

kjacque

Go and doc changes LGTM

daosbuild1 · 2023-12-05T05:00:09Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/5/execution/node/1420/log

knard38 · 2023-12-07T15:16:04Z

@phender please could you review the functional test part of this PR.
From what I understood, you historically significantly contribute to the fixed test.
Thanks in advance.

daosbuild1 · 2023-12-07T17:47:35Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/6/execution/node/354/log

daosbuild1 · 2023-12-07T22:51:29Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/6/execution/node/446/log

daosbuild1 · 2024-01-12T18:49:45Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13169/15/testReport/

daosbuild1 · 2024-01-14T07:01:01Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/16/execution/node/686/log

daosbuild1 · 2024-01-14T10:39:26Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/16/execution/node/732/log

knard38 · 2024-01-15T08:52:58Z

@phender @knard-intel I've rebased this PR on #13594 to attempt to fix the pool create failures which I believe are due to the NVMe storage query key lookup issue. Also running with MD-on-SSD coverage and pool features.

My last commits were lost with your force push (from what I understand).
I am going to add them and restart the CI.

Integrate reviewers comments: - Add message in exception - fix use of min() - Fix spelling issue Features: pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

Increase test timeout with the time needed to destory all the pools. Features: pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

Integrate reviewers comments: - Strengthen and simplify the pools creation step - Check minimal quantity of pools Skip-func-hw-medium-md-on-ssd: false Skip-func-hw-medium-verbs-provider-md-on-ssd: false Skip-func-hw-large-md-on-ssd: false Features: control pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

Fix spelling and coding standard issue. Skip-func-hw-medium-md-on-ssd: false Skip-func-hw-medium-verbs-provider-md-on-ssd: false Skip-func-hw-large-md-on-ssd: false Features: control pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

tanabarr · 2024-01-15T13:03:27Z

@phender @knard-intel I've rebased this PR on #13594 to attempt to fix the pool create failures which I believe are due to the NVMe storage query key lookup issue. Also running with MD-on-SSD coverage and pool features.

My last commits were lost with your force push (from what I understand). I am going to add them and restart the CI.

I'm very sorry about that, I'm not sure how that happened.

knard38 · 2024-01-15T14:22:18Z

@phender @knard-intel I've rebased this PR on #13594 to attempt to fix the pool create failures which I believe are due to the NVMe storage query key lookup issue. Also running with MD-on-SSD coverage and pool features.

My last commits were lost with your force push (from what I understand). I am going to add them and restart the CI.

I'm very sorry about that, I'm not sure how that happened.

No problem, hopefully the CI will report less errors ;)

daosbuild1 · 2024-01-15T19:40:40Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/17/execution/node/1479/log

daosbuild1 · 2024-01-15T22:14:43Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13169/17/execution/node/1417/log

daosbuild1 · 2024-01-16T01:11:10Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13169/17/testReport/

tanabarr · 2024-01-16T11:41:24Z

CI results for run no. 17 with control and pool features failed for the following known issues:

Requesting forced landing

knard38

GO part LGTM

src/tests/ftest/pool/create_capacity.py

…trank-twice-fail-simple Required-githooks: true

Integrating reviewers comments: - Remove useless custom tearDown() - Prefer local variable over class attributes - Code refactoring of the pools creation loop Skip-func-hw-medium-md-on-ssd: false Skip-func-hw-medium-verbs-provider-md-on-ssd: false Skip-func-hw-large-md-on-ssd: false Features: control pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

Fix pylint and speel checks. Skip-func-hw-medium-md-on-ssd: false Skip-func-hw-medium-verbs-provider-md-on-ssd: false Skip-func-hw-large-md-on-ssd: false Features: control pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

Removing useless constant. Skip-func-hw-medium-md-on-ssd: false Skip-func-hw-medium-verbs-provider-md-on-ssd: false Skip-func-hw-large-md-on-ssd: false Features: control pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

daosbuild1 · 2024-01-22T23:27:54Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13169/20/testReport/

tanabarr requested review from a team as code owners October 12, 2023 11:09

tanabarr requested review from mjmac and removed request for a team October 12, 2023 11:09

tanabarr self-assigned this Oct 12, 2023

tanabarr added bug control-plane work on the management infrastructure of the DAOS Control Plane labels Oct 12, 2023

daosbuild1 reviewed Oct 12, 2023

View reviewed changes

tanabarr force-pushed the tanabarr/control-setrank-twice-fail-simple branch from afe722e to 7f9d5c6 Compare November 29, 2023 13:29

daosbuild1 reviewed Nov 29, 2023

View reviewed changes

tanabarr requested a review from a team as a code owner November 30, 2023 11:52

tanabarr force-pushed the tanabarr/control-setrank-twice-fail-simple branch from 38721ba to 484562b Compare November 30, 2023 12:03

tanabarr changed the title ~~DAOS-14225 control: Ignore duplicate call to SetRank~~ DAOS-14225 control,test: Ignore duplicate call to SetRank Nov 30, 2023

tanabarr requested review from kjacque and knard38 November 30, 2023 12:04

daltonbohning reviewed Nov 30, 2023

View reviewed changes

daosbuild1 reviewed Dec 1, 2023

View reviewed changes

tanabarr changed the title ~~DAOS-14225 control,test: Ignore duplicate call to SetRank~~ DAOS-14225 control: Ignore duplicate call to SetRank Dec 4, 2023

kjacque previously approved these changes Dec 5, 2023

View reviewed changes

tanabarr requested a review from daltonbohning December 5, 2023 10:14

knard38 requested a review from phender December 7, 2023 15:11

kanard38 added 4 commits January 15, 2024 08:55

DAOS-14528 test: Fix PoolCreateCapacityTests for md-on-ssd

bec659a

Integrate reviewers comments: - Add message in exception - fix use of min() - Fix spelling issue Features: pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-14528 test: Fix PoolCreateCapacityTests for md-on-ssd

f101151

Increase test timeout with the time needed to destory all the pools. Features: pool Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

tanabarr added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jan 16, 2024

knard38 previously approved these changes Jan 19, 2024

View reviewed changes

phender requested changes Jan 19, 2024

View reviewed changes

kanard38 added 2 commits January 22, 2024 09:06

Merge remote-tracking branch 'origin/master' into tanabarr/control-se…

e279835

…trank-twice-fail-simple Required-githooks: true

knard38 dismissed their stale review via 72578e8 January 22, 2024 15:12

kanard38 added 2 commits January 22, 2024 15:16

phender approved these changes Jan 24, 2024

View reviewed changes

knard38 approved these changes Jan 24, 2024

View reviewed changes

tanabarr removed the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jan 24, 2024

mjmac approved these changes Jan 24, 2024

View reviewed changes

mjmac merged commit 54c54af into master Jan 24, 2024
50 of 51 checks passed

mjmac deleted the tanabarr/control-setrank-twice-fail-simple branch January 24, 2024 21:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-14225 control: Ignore duplicate call to SetRank #13169

DAOS-14225 control: Ignore duplicate call to SetRank #13169

tanabarr commented Oct 12, 2023 •

edited

Loading

github-actions bot commented Oct 12, 2023 •

edited

Loading

daosbuild1 left a comment

daosbuild1 commented Oct 14, 2023

daosbuild1 commented Oct 16, 2023

daosbuild1 left a comment

daosbuild1 commented Nov 29, 2023

tanabarr commented Nov 30, 2023

daltonbohning Nov 30, 2023

tanabarr Dec 1, 2023

daltonbohning Dec 1, 2023

tanabarr Dec 4, 2023 •

edited

Loading

daltonbohning Dec 5, 2023

tanabarr Jan 12, 2024

daosbuild1 left a comment

kjacque left a comment

daosbuild1 commented Dec 5, 2023

knard38 commented Dec 7, 2023

daosbuild1 commented Dec 7, 2023

daosbuild1 commented Dec 7, 2023

daosbuild1 commented Jan 12, 2024

daosbuild1 commented Jan 14, 2024

daosbuild1 commented Jan 14, 2024

knard38 commented Jan 15, 2024 •

edited

Loading

tanabarr commented Jan 15, 2024

knard38 commented Jan 15, 2024

daosbuild1 commented Jan 15, 2024

daosbuild1 commented Jan 15, 2024

daosbuild1 commented Jan 16, 2024

tanabarr commented Jan 16, 2024

knard38 left a comment

daosbuild1 commented Jan 22, 2024

DAOS-14225 control: Ignore duplicate call to SetRank #13169

DAOS-14225 control: Ignore duplicate call to SetRank #13169

Conversation

tanabarr commented Oct 12, 2023 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Oct 12, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Oct 14, 2023

daosbuild1 commented Oct 16, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 29, 2023

tanabarr commented Nov 30, 2023

daltonbohning Nov 30, 2023

Choose a reason for hiding this comment

tanabarr Dec 1, 2023

Choose a reason for hiding this comment

daltonbohning Dec 1, 2023

Choose a reason for hiding this comment

tanabarr Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

daltonbohning Dec 5, 2023

Choose a reason for hiding this comment

tanabarr Jan 12, 2024

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

kjacque left a comment

Choose a reason for hiding this comment

daosbuild1 commented Dec 5, 2023

knard38 commented Dec 7, 2023

daosbuild1 commented Dec 7, 2023

daosbuild1 commented Dec 7, 2023

daosbuild1 commented Jan 12, 2024

daosbuild1 commented Jan 14, 2024

daosbuild1 commented Jan 14, 2024

knard38 commented Jan 15, 2024 • edited Loading

tanabarr commented Jan 15, 2024

knard38 commented Jan 15, 2024

daosbuild1 commented Jan 15, 2024

daosbuild1 commented Jan 15, 2024

daosbuild1 commented Jan 16, 2024

tanabarr commented Jan 16, 2024

knard38 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jan 22, 2024

tanabarr commented Oct 12, 2023 •

edited

Loading

github-actions bot commented Oct 12, 2023 •

edited

Loading

tanabarr Dec 4, 2023 •

edited

Loading

knard38 commented Jan 15, 2024 •

edited

Loading