Shared tensor mechanism for avoiding tensor serialization/ipc costs #58
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yunseong
suggested changes
Jul 14, 2019
runner_model.py
Outdated
| @@ -17,6 +17,11 @@ def input_shape(self): | |||
| """Returns the expected shape of the input tensor to this model.""" | |||
| raise NotImplementedError | |||
|
|
|||
| @staticmethod | |||
| def output_shape(): | |||
| """Returns the expected shape of the output tensor of this model.""" | |||
There was a problem hiding this comment.
As we discussed previously, could we add the description for the case of None in the input/output shape (i.e., not a Tensor at the dimension)?
There was a problem hiding this comment.
The change should be applied to input_shape() as well.
jsjason
commented
Jul 15, 2019
runner_model.py
Outdated
| @@ -17,6 +17,11 @@ def input_shape(self): | |||
| """Returns the expected shape of the input tensor to this model.""" | |||
| raise NotImplementedError | |||
|
|
|||
| @staticmethod | |||
| def output_shape(): | |||
| """Returns the expected shape of the output tensor of this model.""" | |||
|
Sorry for the extra commits, I left out a few details. I'm really ready for a review now. |
yunseong
reviewed
Jul 15, 2019
yunseong
approved these changes
Jul 16, 2019
This was referenced Jul 16, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #57.
This PR implements the solution described in #57, using shared tensors and integer signals to pass intermediate tensor values instead of directly serializing tensors and sending them through queues (which are linux pipes, internally).
First, the main benchmark process examines the given pipeline and creates all shared tensors in advance. Each runner process is given a certain amount of output tensors to write their outputs. The device of each output tensor is set to be equal to their corresponding processes, and the shape is assumed to be given by the model implementation (
RunnerModel.output_shape()). Note that all output tensors for step N are given to all processes of step N+1 as input tensors. The newly added classBenchmarkTensorsacts as an abstraction for managing such shared tensors.The number of output tensors per process can be configured via the new command line argument
-t/--tensors_per_process, which has been added to the log directory format as well.Now, the information that needs to be passed through queues is an index indicating "which shared tensor is ready." I've implemented the index simply as a tuple of two integers: one for denoting the process index (there can be multiple processes per step), and one for denoting the tensor index (out of
tensors_per_processtensors). Each process writes its outputs to one of itstensors_per_processoutput tensors and sends the correct index afterwards. In case alltensors_per_processtensors have been used up, it goes back to the first tensor and repeats the whole procedure, sort of like a ring buffer. To ensure that the process does not write to an output tensor that has not yet been consumed by a process from the next step, I've added amultiprocessing.Eventobject to every shared tensor so that consumers can notify waiting producers. Check the comments in the newTensorEventclass and the usages inrunner.pyfor more details.Test:
Before this PR, the average time taken between
inference0_finishandrunner1_start(the ipc part) was 13.010609 ms. After applying this PR, the same section takes only 2.661710 ms.