DAOS-17151 engine: RPC credits for IO chore task

[Nasf-Fan]

We need to control the in-flight RPCs (via IO chore tasks) count from current target to others. Otherwise, too many in-flight RPCs may occupy a lot of DRAM as to cause out of memory on server.

Introduce new server-side environment DAOS_IO_CHORE_CREDITS to allow admin to configure such credits. The default value is 4096, it may be optimized in the future based on more test results.

Allow-unstable-test: true

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is 'ENOMEM on punch when running mdtest with RF2'
Status is 'In Review'
Labels: 'ALCF,alcf_track'
https://daosio.atlassian.net/browse/DAOS-17151

[daosbuild1]

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15970/3/testReport/

[daltonbohning]

ftest LGTM

[daltonbohning]

FYI clang-format is required now

[Nasf-Fan]

FYI clang-format is required now

Hm... it seems that some required format changes is not related with my patch. For example:
https://github.com/daos-stack/daos/actions/runs/13526109121/job/37796952760?pr=15970

diff --git a/src/include/daos_fs.h b/src/include/daos_fs.h
index ec597ad22..63d0c401f 100644
--- a/src/include/daos_fs.h
+++ b/src/include/daos_fs.h
@@ -42,7 +42,7 @@ extern "C" {
/** Maximum xattr name /
#define DFS_MAX_XATTR_NAME 255
/* Maximum xattr value */
-#define DFS_MAX_XATTR_LEN 65536
+#define DFS_MAX_XATTR_LEN 65536

But my patch has never touched such file. On the other hand, the clang-format requirement for different function clarifications are not consistent, for example:

-int dss_chore_register(struct dss_chore *chore);
-void dss_chore_deregister(struct dss_chore *chore);
-void dss_chore_diy(struct dss_chore *chore);
+int
+dss_chore_register(struct dss_chore *chore);
+void
+dss_chore_deregister(struct dss_chore *chore);
+void

• dss_chore_diy(struct dss_chore *chore);
  

So do you think we have to pass clang-format check currently? @daltonbohning @gnailzenh