-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DAOS-14826 client: reserve low fds #13532
Changes from 2 commits
3dfafc8
b04f4da
4aec3a1
6614ce3
c57faf6
fe93bac
5b8e966
6e94096
c8ed7de
48918a7
b757c30
87f0aa2
96a4cb8
68c5880
80f0dd6
abf91cd
160d04f
d134f7b
b1bf1e9
9836e22
645092e
ebc81e8
5bccea2
af89d48
938936e
7f2cc7d
1030d67
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -913,6 +913,9 @@ child_hdlr(void) | |
context_reset = true; | ||
} | ||
|
||
/* avoid using fd smaller than this number in daos_init() */ | ||
#define DAOS_MIN_FD 10 | ||
|
||
/** determine whether a path (both relative and absolute) is on DAOS or not. If yes, | ||
* returns parent object, item name, full path of parent dir, full absolute path, and | ||
* the pointer to struct dfs_mt. | ||
|
@@ -1007,6 +1010,19 @@ query_path(const char *szInput, int *is_target_path, dfs_obj_t **parent, char *i | |
/* trying to avoid lock as much as possible */ | ||
if (!daos_inited) { | ||
/* daos_init() is expensive to call. We call it only when necessary. */ | ||
int low_fd_list[DAOS_MIN_FD], low_fd_count = 0, fd_kernel, idx; | ||
|
||
fd_kernel = open("/proc/self/maps", O_RDONLY); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's wrong with opening "/"? Presumably this code can only get here after the hooks have been installed so perhaps it would be better to call the redirected open directly? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree. "/" should work too. |
||
while (fd_kernel >= 0) { | ||
if (fd_kernel >= DAOS_MIN_FD) { | ||
close(fd_kernel); | ||
break; | ||
} else { | ||
low_fd_list[low_fd_count] = fd_kernel; | ||
low_fd_count++; | ||
} | ||
fd_kernel = open("/proc/self/maps", O_RDONLY); | ||
} | ||
rc = daos_init(); | ||
if (rc) { | ||
DL_ERROR(rc, "daos_init() failed"); | ||
|
@@ -1024,6 +1040,8 @@ query_path(const char *szInput, int *is_target_path, dfs_obj_t **parent, char *i | |
|
||
daos_inited = true; | ||
atomic_fetch_add_relaxed(&daos_init_cnt, 1); | ||
for (idx = 0; idx < low_fd_count; idx++) | ||
close(low_fd_list[idx]); | ||
ashleypittman marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} | ||
|
||
/* dfs info can be set up after daos has been initialized. */ | ||
|
@@ -2540,6 +2558,10 @@ new_fstatat(int dirfd, const char *__restrict path, struct stat *__restrict stat | |
return new_xstat(1, path, stat_buf); | ||
} | ||
|
||
if (dirfd >= FD_FILE_BASE && dirfd < FD_DIR_BASE && path[0] == 0) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. im not sure i understand what you are doing here, nor i understand this commit message: what is the corner case being fixed? can you be more clear. if this corner case is fixed, can you add a test for the corner case? thanks. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. dirfd is allowed to be a fd of a regular file. I did not expect such situation. ok. I will add a unit test for this corner case. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am trying to add the corner case in dfuse unit test. Looks libioil has the same issue. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry. False alarm. Used wrong flag in fstatat(). |
||
/* same as fstat for a file. May need further work to handle flags */ | ||
return fstat(dirfd, stat_buf); | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this related to the commit message? Same applies for the rest of the code changes in this file. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a corner case of fstatat(). dirfd is allowed to be a fd of a regular file. |
||
idx_dfs = check_path_with_dirfd(dirfd, &full_path, path, &error); | ||
if (error) | ||
goto out_err; | ||
|
@@ -4979,7 +5001,7 @@ dup(int oldfd) | |
int | ||
dup2(int oldfd, int newfd) | ||
{ | ||
int fd, fd_directed, idx, rc, errno_save; | ||
int fd, oldfd_directed, fd_directed, idx, rc, errno_save; | ||
|
||
/* Need more work later. */ | ||
if (next_dup2 == NULL) { | ||
|
@@ -4995,9 +5017,13 @@ dup2(int oldfd, int newfd) | |
else | ||
return newfd; | ||
} | ||
if ((oldfd < FD_FILE_BASE) && (newfd < FD_FILE_BASE)) | ||
oldfd_directed = query_fd_forward_dest(oldfd); | ||
if ((oldfd_directed < FD_FILE_BASE) && (oldfd < FD_FILE_BASE) && (newfd < FD_FILE_BASE)) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I still don't see the relevance of this change? Is it to do with low fds and required for the new test to pass or is it addressing another issue? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is fixing a bug I recently identified. This change is needed to pass the new test of running bash script that uses low fd directly. The new test is added in the PR for exec() interception. bash opens a file on DFS then calls dup2(fake_fd, 5) to get a low fd 5, then call dup2(5, 1) to let fd 1 point to the file on DFS. After that, bash calls exec() to run "cat" to write to the file on DFS. #! /bin/sh exec 5>config.log {
|
||
return next_dup2(oldfd, newfd); | ||
|
||
if (oldfd_directed >= FD_FILE_BASE && oldfd < FD_FILE_BASE) | ||
oldfd = oldfd_directed; | ||
|
||
if (newfd >= FD_FILE_BASE) { | ||
DS_ERROR(ENOTSUP, "unimplemented yet for newfd >= FD_FILE_BASE"); | ||
errno = ENOTSUP; | ||
|
@@ -5015,13 +5041,22 @@ dup2(int oldfd, int newfd) | |
else | ||
fd_directed = query_fd_forward_dest(oldfd); | ||
if (fd_directed >= FD_FILE_BASE) { | ||
rc = close(newfd); | ||
if (rc != 0 && errno != EBADF) | ||
return -1; | ||
fd = allocate_a_fd_from_kernel(); | ||
int fd_tmp; | ||
|
||
fd_tmp = allocate_a_fd_from_kernel(); | ||
if (fd_tmp < 0) { | ||
/* failed to allocate an fd from kernel */ | ||
errno_save = errno; | ||
DS_ERROR(errno_save, "failed to get a fd from kernel"); | ||
errno = errno_save; | ||
return (-1); | ||
} | ||
/* rely on dup2() to get the desired fd */ | ||
fd = next_dup2(fd_tmp, newfd); | ||
if (fd < 0) { | ||
/* failed to allocate an fd from kernel */ | ||
errno_save = errno; | ||
close(fd_tmp); | ||
DS_ERROR(errno_save, "failed to get a fd from kernel"); | ||
errno = errno_save; | ||
return (-1); | ||
|
@@ -5031,6 +5066,9 @@ dup2(int oldfd, int newfd) | |
errno = EBUSY; | ||
return (-1); | ||
} | ||
rc = close(fd_tmp); | ||
if (rc != 0) | ||
return -1; | ||
idx = allocate_dup2ed_fd(fd, fd_directed); | ||
if (idx >= 0) | ||
return fd; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking at this existing code it doesn't appear thread-safe in the case of two threads entering this code-path concurrently, not a regression but it would be good to add locking to this at the same time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. We were aware of this possible issue. daos_init() uses lock already. query_path() is called in many intercepted functions. We hoped to avoid lock if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lock here isn't going to make any difference to performance and should only be needed in the contended case anyway. Without one there's potential for daos_init() to be called multiple times but daos_fini() only being called once leading to resource leaks and incorrect tidyup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case daos_init() is called more than one time, daos_init_cnt has the count. daos_fini() will be called daos_init_cnt times in destruction phase.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like you've re-worked this code since yesterday but it's still not thread safe although as you daos_fini() does look like it'll be called the correct number of times.
pthread_atfork() wants to be called precisely once, and if you're using the daos_inited boolean in this manner then you need to access it atomically.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! Moved pthread_atfork() into the locked region. Could you please check again?