Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16971 cart: add RPC origin address #15820

Open
wants to merge 1 commit into
base: google/2.6
Choose a base branch
from
Open

DAOS-16971 cart: add RPC origin address #15820

wants to merge 1 commit into from
+148 −29

Conversation

wangdi1
Copy link
Contributor

@wangdi1 wangdi1 commented Jan 30, 2025

Add cart RPC origin address to error handling.

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@github-actions GitHub Actions
Copy link

Ticket title is 'bulk cancel causing libfabric progress failure.'
Status is 'Awaiting Verification'
Labels: 'GCP,google-cloud-daos'
https://daosio.atlassian.net/browse/DAOS-16971

@wangdi1 wangdi1 requested a review from soumagne January 30, 2025 23:02
@wangdi1
DAOS-16971 cart: add RPC origin address
24f7c0c 
Add cart RPC origin address to error handling.

Run-GHA: true
Allow-unstable-test: true
Signed-off-by: Di Wang <[email protected]>
@wangdi1 wangdi1 force-pushed the wangdi/google_26_rpc_debug branch from db7cb67 to 24f7c0c Compare January 30, 2025 23:24
frostedcmos
frostedcmos reviewed Jan 31, 2025
View reviewed changes
Copy link
Contributor

@frostedcmos frostedcmos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments inline. Perhaps for the error cases we can include 'origin info' as part of RPC_ERROR and RPC_WARN as well - while there is some overhead it might be ok for error cases and help with debug.

src/object/srv_obj.c
uint32_t addr_size = 48;

crt_rpc_get_origin_addr(rpc, addr, &addr_size);
D_ERROR("rpc %p opc %d send reply, pmv %d, epoch " DF_X64 ", status %d from %s\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a case like this it would be better to add a public variant of RPC_ERROR() macro and use it to print additional info, as it will print decoded opcode for example along with rpcid and other fields that are helpful in debug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

src/object/srv_obj.c
D_ERROR("rpc %p opc %d send reply, pmv %d, epoch " DF_X64 ", status %d from %s\n",
rpc, opc_get(rpc->cr_opc), ioc->ioc_map_ver, orwo->orw_epoch, status, addr);
} else {
D_DEBUG(DB_IO, "rpc %p opc %d send reply, pmv %d, epoch " DF_X64 ", status %d\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here but a variant of RPC_DEBUG

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

src/include/cart/api.h
@@ -2375,6 +2376,18 @@ int crt_context_quota_limit_get(crt_context_t crt_ctx, crt_quota_type_t quota, i
int
crt_req_get_proto_ver(crt_rpc_t *req);

/**
* Get the rpc original address.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/original/origin/ ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

src/include/cart/api.h
*
* \param[in] rpc pointer to RPC request
* \param[out] addr pointer to the converted buffer.
* \param[in/out] addr_size the size of the converted buffer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would be more clear for me for this to be 'addr_len' instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

src/object/srv_obj.c
uint32_t addr_size = 48;

crt_rpc_get_origin_addr(cb_info->bci_bulk_desc->bd_rpc, addr, &addr_size);
D_ERROR("bulk transfer failed: %d from %s\n", cb_info->bci_rc, addr);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should then include origin_addr info for any transfer fail inside of crt_bulk_transfer/cb so that it covers other places too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, good suggestion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

src/object/srv_obj.c
uint32_t addr_size = 48;

crt_rpc_get_origin_addr(rpc, addr, &addr_size);
D_ERROR(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, a public variant of RPC_ERROR (e.g. D_RPC_ERROR) would be handy here to add and use

@github-actions GitHub Actions
Copy link

Functional on EL 8.8 Test Results

131 tests   127 ✅  1h 28m 48s ⏱️
 41 suites    4 💤
 41 files      0 ❌

Results for commit 24f7c0c.

johannlombardi
johannlombardi reviewed Jan 31, 2025
View reviewed changes
Copy link
Contributor

@johannlombardi johannlombardi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am personally not a big fan of spreading calls to crt_rpc_get_origin_addr() everywhere in the source code. We cannot use the TLS or even put it in a field of the cart RPC structure?

@frostedcmos
Copy link
Contributor

I am personally not a big fan of spreading calls to crt_rpc_get_origin_addr() everywhere in the source code. We cannot use the TLS or even put it in a field of the cart RPC structure?

yeah, we've been further discussing this and jeff also suggested storing resolved addr on first lookup in rpc_priv

@github-actions GitHub Actions
Copy link

Functional Hardware Large Test Results

64 tests   64 ✅  29m 32s ⏱️
14 suites   0 💤
14 files     0 ❌

Results for commit 24f7c0c.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@frostedcmos frostedcmos frostedcmos left review comments

@johannlombardi johannlombardi johannlombardi left review comments

@jolivier23 jolivier23 Awaiting requested review from jolivier23

@soumagne soumagne Awaiting requested review from soumagne

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wangdi1 @frostedcmos @johannlombardi