Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle the case where a process is SIGKILLed #85

Open
phargogh opened this issue Apr 7, 2023 · 0 comments
Open

Handle the case where a process is SIGKILLed #85

phargogh opened this issue Apr 7, 2023 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@phargogh
Copy link
Member

phargogh commented Apr 7, 2023

I have a case with the NCI work where my taskgraph-based pipeline is exhausting the available memory and linux's oom-killer is effectively kill -9ing the process. When this happens:

  1. python starts up a new process to replace it, keeping the number of Processes the same as what the Pool expects at all times. The new process started has a different PID.
  2. Because the Task was killed, the self.task_done_executing event is never set, which means that the graph hangs indefinitely.

The behaviorally consistent thing to do here might just be to terminate the graph when we detect that a process has been killed.

I'm labeling this as an enhancement because it's new behavior and because the workaround here is to just not exhaust the memory I'm allowed.

@phargogh phargogh added the enhancement New feature or request label Apr 7, 2023
@phargogh phargogh added this to the 1.0 milestone Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant