Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cusparse-hipsparse integration #53

Merged
merged 30 commits into from
Jun 6, 2024

Conversation

bmedishe
Copy link

No description provided.

@bmedishe bmedishe self-assigned this Apr 30, 2024
@bmedishe
Copy link
Author

bmedishe commented May 13, 2024

---------------------- TEST SUMMARY ------------------
tests/cupy_tests/array_api_tests|0| 8 skipped in 0.05s
tests/cupy_tests/binary_tests|15| 15 passed in 1.49s
tests/cupy_tests/core_tests|2872| warnings summary
tests/cupy_tests/creation_tests|506| 408 passed, 93 skipped, 5 deselected in 14.66s
tests/cupy_tests/cuda_tests|268| 167 passed, 96 skipped, 6 deselected in 3.63s
tests/cupy_tests/fft_tests|2711| 1834 passed, 876 skipped, 1 deselected in 143.68s (0:02:23)
tests/cupy_tests/functional_tests|73| 73 passed, 1 warning in 78.55s (0:01:18)
tests/cupy_tests/indexing_tests|406| 405 passed, 1 deselected in 4.87s
tests/cupy_tests/io_tests|42| 42 passed in 1.19s
tests/cupy_tests/lib_tests|576| 534 passed, 42 xfailed, 12 warnings in 81.03s (0:01:21)
tests/cupy_tests/linalg_tests|1395| 1379 passed, 16 skipped, 28 warnings in 102.01s (0:01:42)
tests/cupy_tests/logic_tests|824| 824 passed, 2 warnings in 5.59s
tests/cupy_tests/manipulation_tests|585| 549 passed, 32 skipped, 1 deselected, 3 xfailed in 15.01s
tests/cupy_tests/math_tests|10192| warnings summary
tests/cupy_tests/misc_tests|31| 31 passed in 4.57s
tests/cupy_tests/padding_tests|75| 75 passed in 1.81s
tests/cupy_tests/polynomial_tests|55| 55 passed in 6.29s
tests/cupy_tests/prof_tests|8| 8 passed in 0.26s
tests/cupy_tests/random_tests|1977| 1931 passed, 39 skipped, 4 deselected, 3 xfailed in 53.21s
tests/cupy_tests/sorting_tests|499| 433 passed, 64 skipped, 1 deselected, 1 xfailed in 25.41s
tests/cupy_tests/statistics_tests|733| 730 passed, 3 deselected, 83 warnings in 88.33s (0:01:28)
tests/cupy_tests/test_cublas.py|2700| 2696 passed, 4 xfailed in 14.68s
tests/cupy_tests/testing_tests|293| 260 passed, 23 skipped, 10 xfailed in 7.14s
tests/cupy_tests/test_init.py|18| 17 passed, 1 skipped in 2.49s
tests/cupy_tests/test_ndim.py|12| 12 passed in 1.12s
tests/cupy_tests/test_numpy_interop.py|11| 10 passed, 1 xfailed in 7.25s
tests/cupy_tests/test_type_routines.py|34| 34 passed in 0.79s
tests/cupy_tests/test_typing.py|1 item| 1 passed in 0.25s
tests/cupyx_tests/distributed_tests|59| 23 failed, 36 passed in 466.13s (0:07:46)
tests/cupyx_tests/fallback_mode_tests|107| 106 passed, 1 skipped in 5.89s
tests/cupyx_tests/jit_tests|172| 135 passed, 19 skipped, 18 xfailed, 6 warnings in 3.90s
tests/cupyx_tests/linalg_tests|24| 20 passed, 5 skipped in 1.41s
tests/cupyx_tests/profiler_tests|19| 19 passed in 1.41s
tests/cupyx_tests/scipy_tests/fftpack_tests|901| 901 passed in 59.58s
tests/cupyx_tests/scipy_tests/fft_tests|12237| 11657 passed, 420 skipped, 160 xpassed in 232.62s (0:03:52)
tests/cupyx_tests/scipy_tests/interpolate_tests|592| 471 passed, 120 skipped, 1 deselected in 15.92s
tests/cupyx_tests/scipy_tests/linalg_tests|468| 462 passed, 6 skipped in 8.68s
tests/cupyx_tests/scipy_tests/ndimage_tests|33626| 848 passed, 32775 skipped, 3 deselected in 42.44s
tests/cupyx_tests/scipy_tests/signal_tests|9307| short test summary info
tests/cupyx_tests/scipy_tests/sparse_tests|24205| warnings summary
tests/cupyx_tests/scipy_tests/spatial_tests|536| 536 skipped in 1.01s
tests/cupyx_tests/scipy_tests/special_tests|483| 474 passed, 9 skipped, 34 warnings in 6.61s
tests/cupyx_tests/scipy_tests/stats_tests|201| 201 passed in 3.46s
tests/cupyx_tests/scipy_tests/test_get_array_module.py|2| 2 passed in 1.31s
tests/cupyx_tests/test_cudnn.py|1829| 1829 skipped in 1.81s
tests/cupyx_tests/test_cupyx.py|2| 2 passed in 0.45s
tests/cupyx_tests/test_cusolver.py|648| 32 passed, 616 skipped in 2.19s
tests/cupyx_tests/test_cusparse.py|1326| 808 passed, 146 skipped, 372 xfailed in 20.64s
tests/cupyx_tests/test_cutensor.py|79| 79 skipped in 0.53s
tests/cupyx_tests/test_lapack.py|70| 41 passed, 29 skipped in 6.58s
tests/cupyx_tests/test_optimize.py|9| 9 skipped in 0.50s
tests/cupyx_tests/test_pinned_array.py|76| 72 passed, 4 deselected in 1.92s

Copy link

@pnunna93 pnunna93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check CI failures and my comments below

cupy_backends/cuda/libs/cusparse.pyx Show resolved Hide resolved
cupy_backends/hip/cupy_hipsparse.h Show resolved Hide resolved
@@ -103,9 +103,6 @@ cpdef enum:
CUSPARSE_SOLVE_POLICY_NO_LEVEL = 0
CUSPARSE_SOLVE_POLICY_USE_LEVEL = 1

CUSPARSE_ALG_NAIVE = 0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are only these two enums removed from here and moved under the IF condition?

Copy link

@lcskrishna lcskrishna May 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bmedishe Could you review this one with 6.0_internal_testing branch. Looks like these enum values shouldn't be re-ordered..

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lcskrishna these two enums do not have hip mapping

Copy link

@pnunna93 pnunna93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

@lcskrishna lcskrishna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @bmedishe Could you have a full test run one final time with latest rocm/pytorch and upload the full log here.
Otherwise this PR LGTM.

Copy link

@AdrianAbeyta AdrianAbeyta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending the full log!

@bmedishe
Copy link
Author

bmedishe commented Jun 6, 2024

All tests pass except for 24 nccl tests
---------------------- TEST SUMMARY ------------------
tests/cupy_tests/array_api_tests|0| 8 skipped in 0.05s
tests/cupy_tests/binary_tests|15| 15 passed in 1.25s
tests/cupy_tests/core_tests|2872| warnings summary
tests/cupy_tests/creation_tests|506| 408 passed, 93 skipped, 5 deselected in 12.86s
tests/cupy_tests/cuda_tests|268| 167 passed, 96 skipped, 6 deselected in 3.54s
tests/cupy_tests/fft_tests|2711| 1834 passed, 876 skipped, 1 deselected in 144.60s (0:02:24)
tests/cupy_tests/functional_tests|73| 73 passed, 1 warning in 68.41s (0:01:08)
tests/cupy_tests/indexing_tests|406| 405 passed, 1 deselected in 5.24s
tests/cupy_tests/io_tests|42| 42 passed in 1.20s
tests/cupy_tests/lib_tests|576| 534 passed, 42 xfailed, 12 warnings in 81.70s (0:01:21)
tests/cupy_tests/linalg_tests|1395| 1379 passed, 16 skipped, 28 warnings in 102.38s (0:01:42)
tests/cupy_tests/logic_tests|824| 824 passed, 2 warnings in 5.57s
tests/cupy_tests/manipulation_tests|585| 549 passed, 32 skipped, 1 deselected, 3 xfailed in 16.29s
tests/cupy_tests/math_tests|10192| warnings summary
tests/cupy_tests/misc_tests|31| 31 passed in 4.70s
tests/cupy_tests/padding_tests|75| 75 passed in 1.81s
tests/cupy_tests/polynomial_tests|55| 55 passed in 6.24s
tests/cupy_tests/prof_tests|8| 8 passed in 0.25s
tests/cupy_tests/random_tests|1977| 1931 passed, 39 skipped, 4 deselected, 3 xfailed in 131.16s (0:02:11)
tests/cupy_tests/sorting_tests|499| 433 passed, 64 skipped, 1 deselected, 1 xfailed in 23.02s
tests/cupy_tests/statistics_tests|733| 730 passed, 3 deselected, 83 warnings in 88.45s (0:01:28)
tests/cupy_tests/test_cublas.py|2700| 2696 passed, 4 xfailed in 14.72s
tests/cupy_tests/testing_tests|293| 260 passed, 23 skipped, 10 xfailed in 7.09s
tests/cupy_tests/test_init.py|18| 17 passed, 1 skipped in 2.47s
tests/cupy_tests/test_ndim.py|12| 12 passed in 1.11s
tests/cupy_tests/test_numpy_interop.py|11| 10 passed, 1 xfailed in 6.55s
tests/cupy_tests/test_type_routines.py|34| 34 passed in 0.78s
tests/cupy_tests/test_typing.py|1 item| 1 passed in 0.25s
tests/cupyx_tests/distributed_tests|59| 24 failed, 35 passed in 466.04s (0:07:46)
tests/cupyx_tests/fallback_mode_tests|107| 106 passed, 1 skipped in 5.95s
tests/cupyx_tests/jit_tests|172| 135 passed, 19 skipped, 18 xfailed, 22 warnings in 3415.64s (0:56:55)
tests/cupyx_tests/linalg_tests|24| 20 passed, 5 skipped in 1.38s
tests/cupyx_tests/profiler_tests|19| 19 passed in 1.39s
tests/cupyx_tests/scipy_tests/fftpack_tests|901| 901 passed in 60.14s (0:01:00)
tests/cupyx_tests/scipy_tests/fft_tests|12237| 11657 passed, 420 skipped, 160 xpassed in 231.00s (0:03:51)
tests/cupyx_tests/scipy_tests/interpolate_tests|592| 471 passed, 120 skipped, 1 deselected in 15.69s
tests/cupyx_tests/scipy_tests/linalg_tests|468| 462 passed, 6 skipped in 8.66s
tests/cupyx_tests/scipy_tests/ndimage_tests|33626| 848 passed, 32775 skipped, 3 deselected in 38.16s
tests/cupyx_tests/scipy_tests/signal_tests|9307| short test summary info
tests/cupyx_tests/scipy_tests/sparse_tests|24205| warnings summary
tests/cupyx_tests/scipy_tests/spatial_tests|536| 536 skipped in 1.01s
tests/cupyx_tests/scipy_tests/special_tests|483| 474 passed, 9 skipped, 34 warnings in 4.11s
tests/cupyx_tests/scipy_tests/stats_tests|201| 201 passed in 3.39s
tests/cupyx_tests/scipy_tests/test_get_array_module.py|2| 2 passed in 1.29s
tests/cupyx_tests/test_cudnn.py|1829| 1829 skipped in 1.80s
tests/cupyx_tests/test_cupyx.py|2| 2 passed in 0.45s
tests/cupyx_tests/test_cusolver.py|648| 32 passed, 616 skipped in 2.14s
tests/cupyx_tests/test_cusparse.py|1326| 808 passed, 146 skipped, 372 xfailed in 20.35s
tests/cupyx_tests/test_cutensor.py|79| 79 skipped in 0.51s
tests/cupyx_tests/test_lapack.py|70| 41 passed, 29 skipped in 6.57s
tests/cupyx_tests/test_optimize.py|9| 9 skipped in 0.49s
tests/cupyx_tests/test_pinned_array.py|76| 72 passed, 4 deselected in 1.91s
tests/cupyx_tests/test_rsqrt.py|1 item| 1 passed in 1.30s
tests/cupyx_tests/test_runtime.py|2| 1 passed, 1 skipped in 1.08s
tests/cupyx_tests/test_time.py|9| 9 passed in 1.57s
tests/cupyx_tests/tools_tests|21| 6 passed, 15 deselected in 16.13s
tests/example_tests|12| 7 passed, 5 skipped in 19.78s
tests/install_tests|12| 9 passed, 3 skipped in 0.62s

@bmedishe
Copy link
Author

bmedishe commented Jun 6, 2024

List of failed NCCL tests
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_broadcast FAILED [ 20%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_reduce FAILED [ 22%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_all_reduce FAILED [ 23%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_reduce_scatter FAILED [ 25%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_all_gather FAILED [ 27%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_send_and_recv FAILED [ 28%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_send_recv FAILED [ 30%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_scatter FAILED [ 32%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_gather FAILED [ 33%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_all_to_all FAILED [ 35%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_barrier FAILED [ 37%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_broadcast PASSED [ 38%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_reduce PASSED [ 40%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_all_reduce PASSED [ 42%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_reduce_scatter PASSED [ 44%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_all_gather PASSED [ 45%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_send_and_recv PASSED [ 47%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_send_recv PASSED [ 49%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_scatter PASSED [ 50%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_gather FAILED [ 52%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_all_to_all PASSED [ 54%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_barrier PASSED [ 55%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_send_and_recv FAILED [ 57%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_broadcast FAILED [ 59%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_reduce FAILED [ 61%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_all_reduce FAILED [ 62%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_scatter FAILED [ 64%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_gather FAILED [ 66%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_all_gather FAILED [ 67%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_all_to_all FAILED [ 69%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_reduce_scatter FAILED [ 71%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_send_recv FAILED [ 72%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_send_and_recv PASSED [ 74%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_broadcast PASSED [ 76%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_reduce FAILED [ 77%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_all_reduce PASSED [ 79%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_scatter PASSED [ 81%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_gather PASSED [ 83%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_all_gather PASSED [ 84%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_all_to_all PASSED [ 86%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_reduce_scatter PASSED [ 88%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_send_recv PASSED [ 89%]
tests/cupyx_tests/distributed_tests/test_comm.py::TestInitDistributed::test_init FAILED [ 91%]

@bmedishe
Copy link
Author

bmedishe commented Jun 6, 2024

List of failed NCCL tests tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_broadcast FAILED [ 20%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_reduce FAILED [ 22%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_all_reduce FAILED [ 23%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_reduce_scatter FAILED [ 25%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_all_gather FAILED [ 27%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_send_and_recv FAILED [ 28%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_send_recv FAILED [ 30%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_scatter FAILED [ 32%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_gather FAILED [ 33%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_all_to_all FAILED [ 35%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackend::test_barrier FAILED [ 37%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_broadcast PASSED [ 38%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_reduce PASSED [ 40%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_all_reduce PASSED [ 42%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_reduce_scatter PASSED [ 44%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_all_gather PASSED [ 45%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_send_and_recv PASSED [ 47%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_send_recv PASSED [ 49%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_scatter PASSED [ 50%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_gather FAILED [ 52%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_all_to_all PASSED [ 54%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendWithMPI::test_barrier PASSED [ 55%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_send_and_recv FAILED [ 57%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_broadcast FAILED [ 59%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_reduce FAILED [ 61%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_all_reduce FAILED [ 62%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_scatter FAILED [ 64%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_gather FAILED [ 66%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_all_gather FAILED [ 67%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_all_to_all FAILED [ 69%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_reduce_scatter FAILED [ 71%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparse::test_send_recv FAILED [ 72%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_send_and_recv PASSED [ 74%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_broadcast PASSED [ 76%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_reduce FAILED [ 77%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_all_reduce PASSED [ 79%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_scatter PASSED [ 81%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_gather PASSED [ 83%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_all_gather PASSED [ 84%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_all_to_all PASSED [ 86%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_reduce_scatter PASSED [ 88%] tests/cupyx_tests/distributed_tests/test_comm.py::TestNCCLBackendSparseWithMPI::test_send_recv PASSED [ 89%] tests/cupyx_tests/distributed_tests/test_comm.py::TestInitDistributed::test_init FAILED [ 91%]

Fails with cupy_backends.cuda.api.runtime_hip.CUDARuntimeError: hipErrorNoDevice: no ROCm-capable device is detected

@AdrianAbeyta AdrianAbeyta merged commit 3733ed9 into rocm6.2_internal_testing Jun 6, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants