Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]6577-Support-torch-trt-dynamo #6578

Draft
wants to merge 4 commits into
base: dev
Choose a base branch
from

Conversation

binliunls
Copy link
Contributor

Fixes #6577 .

Description

PyTorch introduced the dynamo as a part to compile PyTorch models to different backends. And it also works for the Torch-TensorRT compiler and the Torch-TensorRT team is working on supporting it. Therefore, it would be better to have MONAI trt_export API also support this feature.

Currently the trt_export API only supports exporting models via torchscript models. Therefore a use_torchscript parameter will be added in this PR to the trt_export API to choose whether to use the torchscript as IR. Because the dynamo mode only supports to directly export models from pytorch models instead of the torchscript models.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@binliunls
Copy link
Contributor Author

After discussing with the Torch-TensorRT team, currently there is not a way to serialize/export/save a dynamo model. So if we want to support the dynamo model, the best method as I know is to convert the model at the beginning of the inference. Since the conversion may take some overhead, the total inference latency may not necessary be optimized with this way.
On the other hand, we can create a feature request for the torch-tensorrt team to add the support for the serialization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support TensorRT export with dynamo
1 participant