BUG: dead lock in transfer actor in the case of GPU

[UranusSeven]

What do these changes do?

The current implementation of the transfer function leads to a deadlock when executing Xorbits on multiple GPUs. The issue arises from the StorageHandlerActor.fetch_batch function, which invokes SenderManagerActor.send_batch_data and subsequently calls StorageHandlerActor.request_quota_with_spill. Due to the locking mechanism within the StorageHandlerActor method call, a deadlock arises.

UPDATE:

• fix dead lock on GPU
• fix GPU buffer size 0 issue
• known issue: when ucx is enabled, performance is not improved on multi GPUs.
• known issue: too many actor calls in this implement. May lead to performance down.

Related issue number

Check code requirements

• tests added / passed (if needed)
• Ensure all linting tests pass

[codecov]

Codecov Report

Attention: Patch coverage is 78.26087% with 20 lines in your changes missing coverage. Please review.

Project coverage is 67.93%. Comparing base (b4d70cc) to head (8fea60f).
Report is 62 commits behind head on main.

Files with missing lines	Patch %	Lines
python/xorbits/_mars/services/storage/handler.py	81.94%	10 Missing and 3 partials ⚠️
python/xorbits/_mars/services/storage/transfer.py	61.11%	5 Missing and 2 partials ⚠️

❗ There is a different number of reports uploaded between BASE (b4d70cc) and HEAD (8fea60f). Click for more details.

HEAD has 13 uploads less than BASE

Flag	BASE (b4d70cc)	HEAD (8fea60f)
unittests	24	11

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #488       +/-   ##
===========================================
- Coverage   93.52%   67.93%   -25.59%     
===========================================
  Files        1026     1026               
  Lines       79516    79532       +16     
  Branches    16475    16479        +4     
===========================================
- Hits        74365    54031    -20334     
- Misses       3468    23289    +19821     
- Partials     1683     2212      +529     

Flag	Coverage Δ
unittests	67.85% <78.26%> (-25.56%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[luweizheng]

Fix by #788