Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAOS-16766 container: rebuid and container destruction race #15971

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

wangshilong
Copy link
Contributor

daos_lru_ref_evict_wait() may yield, potentially creating a race condition with rebuild operations. During rebuild migration, the container could be reopened and restarted, which could result in EBUSY errors from subsequent vos_cont_destroy() calls.

To resolve this issue:

  1. We avoid container eviction during waiting periods
  2. Container lookup failures are guaranteed by checking the @sc_destroying flag before proceeding This design ensures consistency by preventing concurrent access to containers marked for destruction.

Test-tag: test_ec_single_target_rank_failure pr

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

daos_lru_ref_evict_wait() may yield, potentially creating a race condition with
rebuild operations. During rebuild migration, the container could be reopened
and restarted, which could result in EBUSY errors from subsequent vos_cont_destroy() calls.

To resolve this issue:
  1. We avoid container eviction during waiting periods
  2. Container lookup failures are guaranteed by checking the @sc_destroying flag before
     proceeding This design ensures consistency by preventing concurrent access
     to containers marked for destruction.

Test-tag: test_ec_single_target_rank_failure pr
Signed-off-by: Wang Shilong <[email protected]>
Copy link

github-actions bot commented Feb 25, 2025

Ticket title is 'erasurecode/multiple_rank_failure.py:EcodOnlineMultiRankFail.test_ec_multiple_rank_failure - daos container destroy DER_TIMEDOUT'
Status is 'In Review'
Labels: '2.6.3rc2,2.6.3rc3,2.7.101tb,md_on_ssd,weekly_test'
https://daosio.atlassian.net/browse/DAOS-16766

@wangshilong wangshilong marked this pull request as ready for review February 26, 2025 01:20
@wangshilong wangshilong requested review from a team as code owners February 26, 2025 01:20
jolivier23
jolivier23 previously approved these changes Feb 26, 2025
NiuYawei
NiuYawei previously approved these changes Feb 27, 2025
@wangshilong wangshilong requested a review from a team February 27, 2025 02:34
* to containers marked for destruction.
*/
daos_lru_ref_noevict_wait(tls->dt_cont_cache, &cont->sc_list);
daos_lru_ref_evict(tls->dt_cont_cache, &cont->sc_list);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check "if (!llink->ll_evicted)" before calling daos_lru_ref_evict() since we may yield for wait?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory this is possible,but currently there is no other caller which might evict container, and calling daos_lru_ref_evict() again here is not harmful now. will add extra check if PR need be refreshed for any reason.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the check inside daos_lru_ref_evict()

@Nasf-Fan Nasf-Fan self-requested a review February 27, 2025 09:38
Test-tag: test_ec_single_target_rank_failure pr
Signed-off-by: Wang Shilong <[email protected]>
@daltonbohning daltonbohning removed the request for review from a team February 27, 2025 19:32
@daltonbohning
Copy link
Contributor

Removing gatekeeper until there are 2 approvals

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants