DAOS-16312 control: Always use --force for dmg system stop #15799

tanabarr · 2025-01-27T23:08:12Z

Whenever stopping an engine process from within the control-plane, use
SIGKILL rather than asking nicely (SIGTERM). This has been requested
to try to avoid situations that could result in dataloss.

This change preserves the behaviour where ds_mgmt_drpc_prep_shutdown()
and then ds_pool_disable_exclude() will be called during a controlled
shutdown where dmg system stop is called without other arguments.

Notable behavior changes with this PR:

Always performs SIGKILL on dmg system stop regardless of command
options supplied.
Will attempt prepare shutdown to disable exclusions across cluster
during “controlled” shutdown where dmg system stop is called without
options.
It is now recommended to call dmg system stop without options if
attempting to shutdown an entire cluster without triggering rebuilds.
Force option can be used to skip “disable exclusions” prepare
shutdown dRPC to each rank during dmg system stop

Allow-unstable-test: true
Features: control

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

Allow-unstable-test: true Features: control Signed-off-by: Tom Nabarro <[email protected]>

github-actions · 2025-01-27T23:09:52Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/DAOS-16312

daosbuild1 · 2025-01-27T23:55:14Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15799/1/testReport/

mjmac · 2025-01-28T02:00:19Z

src/control/cmd/dmg/system.go

@@ -177,7 +177,7 @@ func (cmd *systemEraseCmd) Execute(_ []string) error {
 // systemStopCmd is the struct representing the command to shutdown DAOS system.
 type systemStopCmd struct {
 	baseRankListCmd
-	Force bool `long:"force" description:"Force stop DAOS system members"`
+	Force bool `long:"force" description:"Currently ignored"`


IMO this changed line isn't helpful. It will have to be changed back, and it doesn't really tell the admin anything useful. "Oh, it's ignored? So then force stop doesn't work? Well then, how do I forcibly stop the system?"

You can see how this change may have the opposite effect to what you intended... I think the description should be the same and the flag should just be a no-op so that everyone doesn't have to change their scripts.

Ok, I will revert the change

mjmac · 2025-01-28T02:04:57Z

src/control/cmd/dmg/system.go

@@ -191,7 +191,8 @@ func (cmd *systemStopCmd) Execute(_ []string) (errOut error) {
 	if err := cmd.validateHostsRanks(); err != nil {
 		return err
 	}
-	req := &control.SystemStopReq{Force: cmd.Force}
+	// DAOS-16312: Always use force when stopping ranks.
+	req := &control.SystemStopReq{Force: true}


This is not the best place to make this change. It means that only dmg users will benefit from it. Control API users will not. Better to just set it in the SystemStop RPC invoker. As an added benefit, changing it there will minimize the blast radius of this change, so that you don't have to modify the dmg tests.

daosbuild1 · 2025-01-28T12:29:41Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15799/1/testReport/

Features: control Signed-off-by: Tom Nabarro <[email protected]>

mjmac · 2025-01-28T15:59:28Z

src/control/server/ctl_ranks_rpc.go

-	signal := syscall.SIGINT
-	if req.Force {
-		signal = syscall.SIGKILL
-	}


Why make all of these changes? You are (or someone else is) just going to have to change everything back later. This could have been a one-line change, maybe with some extra comments. What you could do is define a const, e.g. DefaultStopSignal, and then when things change back you only need to change it in one place.

Because I'm not convinced it should be that simple. If we simply force all the time then we also break the call to "ds_pool_disable_exclude()" which is required for controlled shutdown as discussed here. I'm waiting for response from those that initially requested that feature and in the meantime will push both solutions.

The simple version that you suggest is: #15803

Looks like this is the version @gnailzenh is in favour of where prep_shutdown/disable_exclude behaviour is preserved for the non-force and no-ranks-specified dmg system stop controlled shutdown case.

@mjmac can we go with this one as an urgent fix please?

tanabarr · 2025-01-28T17:43:20Z

build 4 triggered at P2 with allow unstable pragma after build 3 failed NLT memcheck with unrelated issues

daosbuild1 · 2025-01-28T18:24:29Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15799/4/testReport/

daosbuild1 · 2025-01-28T23:58:52Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15799/4/testReport/

knard38

LGTM

tanabarr · 2025-01-29T11:21:38Z

Not all hardware stages started in run 4, restart from stage hardware test -> run 5.

tanabarr · 2025-01-29T12:37:41Z

Gatekeeper please use PR title and description in commit message when landing, TIA.

mjmac · 2025-01-29T13:21:13Z

@tanabarr: It looks like run #4 had a ftest failure that needs to be addressed (can't copy/paste from the remote desktop session), in the FTEST_control.ControlLogEntry test.

Test-tag: vm,ControlLogEntry Allow-unstable-test: true Signed-off-by: Tom Nabarro <[email protected]>

daltonbohning

ftest LGTM

tanabarr · 2025-01-29T18:06:33Z

build 7 test only fix should be sufficient with build 5&6 results to get this landed, what say you @daltonbohning
@mjmac @kjacque @knard38 can I get reviews please? urgent priority issue

daltonbohning · 2025-01-29T18:58:03Z

build 7 test only fix should be sufficient with build 5&6 results to get this landed, what say you @daltonbohning @mjmac @kjacque @knard38 can I get reviews please? urgent priority issue

Build 4
- NLT valgrind failures - I don't know if related
- Failed ControlLogEntry - being fixed + ran in Build 7
- Failed HW Medium - Build 5 passed this
Build 5 - only ran HW Medium with pr and control - all pass
Build 6 - Aborted before build stages
Build 7 - Presumably will only run the ControlLogEntry test

Assuming Build 7 passes, that just leaves the NLT valgrind failure in Build 4:
https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-15799/4/pipeline
Which probably isn't related?

tanabarr · 2025-01-29T20:58:03Z

PR should be nearly ready to land after build 7, Cedric approved and only one small test fix since he did. mike has also reviewed. NLT memcheck definitely unrelated as no C code changed. @daltonbohning @phender thoughts?

tanabarr · 2025-01-30T10:32:31Z

@phender @daltonbohning ControlLogEntry passed on build 7, NLT failures all seem to be existing and not contributed to by this PR https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15799/7/NLT_server/
Can you see if this can be landed?

DAOS-16312 control: Always use --force for dmg system stop

8543255

Allow-unstable-test: true Features: control Signed-off-by: Tom Nabarro <[email protected]>

tanabarr requested review from a team as code owners January 27, 2025 23:08

mjmac reviewed Jan 28, 2025

View reviewed changes

use sigkill only but suppress exclude unless forced

061d971

Features: control Signed-off-by: Tom Nabarro <[email protected]>

tanabarr requested a review from mjmac January 28, 2025 13:38

mjmac reviewed Jan 28, 2025

View reviewed changes

tanabarr requested review from mjmac, liw, johannlombardi, gnailzenh, kjacque and knard38 January 28, 2025 17:30

knard38 previously approved these changes Jan 29, 2025

View reviewed changes

fix failure in log_entrye.py ftest

9a49883

Test-tag: vm,ControlLogEntry Allow-unstable-test: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr dismissed knard38’s stale review via 9a49883 January 29, 2025 17:03

tanabarr requested review from a team as code owners January 29, 2025 17:03

daltonbohning reviewed Jan 29, 2025

View reviewed changes

tanabarr requested review from knard38 and daltonbohning January 29, 2025 18:03

mjmac approved these changes Jan 29, 2025

View reviewed changes

tanabarr self-assigned this Jan 29, 2025

tanabarr added control-plane work on the management infrastructure of the DAOS Control Plane usability Changes specific to user facing tools or behaviour. labels Jan 29, 2025

knard38 approved these changes Jan 30, 2025

View reviewed changes

tanabarr requested a review from a team January 30, 2025 10:23

tanabarr added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jan 30, 2025

kjacque approved these changes Jan 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16312 control: Always use --force for dmg system stop #15799

DAOS-16312 control: Always use --force for dmg system stop #15799

tanabarr commented Jan 27, 2025 •

edited

Loading

github-actions bot commented Jan 27, 2025

daosbuild1 commented Jan 27, 2025

mjmac Jan 28, 2025

tanabarr Jan 28, 2025

mjmac Jan 28, 2025

tanabarr Jan 28, 2025

daosbuild1 commented Jan 28, 2025

mjmac Jan 28, 2025

tanabarr Jan 28, 2025

tanabarr Jan 28, 2025

tanabarr Jan 29, 2025

tanabarr Jan 29, 2025

tanabarr commented Jan 28, 2025

daosbuild1 commented Jan 28, 2025

daosbuild1 commented Jan 28, 2025

knard38 left a comment

tanabarr commented Jan 29, 2025

tanabarr commented Jan 29, 2025

mjmac commented Jan 29, 2025

daltonbohning left a comment

tanabarr commented Jan 29, 2025

daltonbohning commented Jan 29, 2025

tanabarr commented Jan 29, 2025

tanabarr commented Jan 30, 2025

DAOS-16312 control: Always use --force for dmg system stop #15799

Are you sure you want to change the base?

DAOS-16312 control: Always use --force for dmg system stop #15799

Conversation

tanabarr commented Jan 27, 2025 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Jan 27, 2025

daosbuild1 commented Jan 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 commented Jan 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tanabarr commented Jan 28, 2025

daosbuild1 commented Jan 28, 2025

daosbuild1 commented Jan 28, 2025

knard38 left a comment

Choose a reason for hiding this comment

tanabarr commented Jan 29, 2025

tanabarr commented Jan 29, 2025

mjmac commented Jan 29, 2025

daltonbohning left a comment

Choose a reason for hiding this comment

tanabarr commented Jan 29, 2025

daltonbohning commented Jan 29, 2025

tanabarr commented Jan 29, 2025

tanabarr commented Jan 30, 2025

tanabarr commented Jan 27, 2025 •

edited

Loading