Building with gdrcopy

[jakirkham]

One of the build options in UCX allows using gdrcopy as part of UCT in a loadable module (meaning that it uses dlopen to load this module and handles any errors encountered gracefully). This would be good to include to improve performance in GPU use cases. In theory as gdrcopy is a kernel module, it should be possible to just enable this flag during the build (even if the build is happening somewhere without gdrcopy). The resulting package will only use gdrcopy if it is available in the system, but will skip it otherwise. So should not impact users that do not have gdrcopy. That said, the reality may be different. Will need to investigate how well this works in practice.

[pentschev]

Possibly a naive question, but do we still need a .so file for GDRCopy? My thought is that we need the kernel module to be loaded, and a .so file to be loaded at runtime. For the kernel module, there's nothing we can do with packaging, but for the .so file maybe we can handle it like we do with CUDA. Some version of the driver (kernel module) needs to be loaded, and with that we can run any CUDA version that is supported with that driver version, for this we can have multiple conda package versions, as is the case with cudatoolkit today, no?