CUDA automatically enabled when Build with -Dracon_enable_cuda=ON

[jkbenotmane]

Hi,
are the Parameters for tweaking CUDA defining that racon uses CUDA or is the with -Dracon_enable_cuda=ON build racon automatically using the GPU with the default Parameters? (Usually, I expect some Parameter to specify the cuda device that should be used like e.g. cuda:0). Sorry for the silly question.

[rvaser]

Hi Jasim,
-Dracon_enable_cuda=ON only builds racon with support for CUDA, it will still run on the CPU by default. In order to run it on the GPU, you need to set some (or all) of the following parameters:

    -c, --cudapoa-batches <int>
        default: 0
        number of batches for CUDA accelerated polishing per GPU
    -b, --cuda-banded-alignment
        use banding approximation for polishing on GPU. Only applicable when -c is used.
    --cudaaligner-batches <int>
        default: 0
        number of batches for CUDA accelerated alignment per GPU
    --cudaaligner-band-width <int>
        default: 0
        Band width for cuda alignment. Must be >= 0. Non-zero allows user defined
        band width, whereas 0 implies auto band width determination.

Best regards,
Robert

[jkbenotmane]

Thank you Robert for specifying!
Is there any general Formula for scaling/optimal Parameters for the built in GPU ?

[rvaser]

If I recall correctly, you can divide your GPU memory with 200-250MB to get the number of batches. You can also take some example dataset and try to find best parameters for your GPU.

Best regards,
Robert

[jkbenotmane]

Thank you for the advice, I would like to try but I keep running into this make error:
I think it is similar to this one, but am unable to locate the Bug for racon. (lbcb-sci/raven#35)

-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Using CUDA 11.4 from /usr/local/cuda
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
CMake Warning (dev) at vendor/spoa/vendor/cereal/CMakeLists.txt:2 (project):
  Policy CMP0048 is not set: project() command manages VERSION variables.
  Run "cmake --help-policy CMP0048" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The following variable(s) would be set to empty:

    PROJECT_VERSION
    PROJECT_VERSION_MAJOR
    PROJECT_VERSION_MINOR
    PROJECT_VERSION_PATCH
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found Boost: /home/usr/anaconda3/envs/isoncorrect/lib/cmake/Boost-1.76.0/BoostConfig.cmake (found version "1.76.0") found components: serialization 
-- boost_variant.cpp
-- Found Doxygen: /home/usr/anaconda3/envs/isoncorrect/bin/doxygen (found version "1.9.1 (ab553a81f1725c385ca98109c269b0222389b72a*)") found components: doxygen dot 
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
Setting warning flags
-- Building GenomeWorks libraries as static objects
-- Found CUDA: /usr/local/cuda (found suitable version "11.4", minimum required is "9.0") 
-- Using CUDA 11.4 from /usr/local/cuda
-- Build type: Release
-- Package generator - DEB
-- Found Conda environment in /home/usr/anaconda3/envs/isoncorrect
-- nvcc flags for gwbase:  -lineinfo -lineinfo -use_fast_math -Xcompiler -Wall,-Wno-pedantic -std=c++14
-- nvcc flags for cudapoa:  -lineinfo -lineinfo -use_fast_math -Xcompiler -Wall,-Wno-pedantic -std=c++14 --expt-relaxed-constexpr
-- Using CUDA 11.4 from /usr/local/cuda
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.0;3.5;5.0;5.2;6.0;6.1;7.0;7.0+PTX;7.5;7.5+PTX
-- nvcc flags for cudamapper:  -lineinfo -lineinfo -use_fast_math -Xcompiler -Wall,-Wno-pedantic -lineinfo --expt-extended-lambda -use_fast_math -Xcompiler -Wall,-Wno-pedantic -std=c++14 -gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_70,code=compute_70;-gencode;arch=compute_75,code=compute_75
-- nvcc flags for cudaaligner:  -lineinfo -lineinfo -use_fast_math -Xcompiler -Wall,-Wno-pedantic -std=c++14 --expt-relaxed-constexpr
-- Enabling Doxygen documentation generation
-- Found GTest: /usr/lib/x86_64-linux-gnu/libgtest.a  
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/d/racon/build
(racon) usr@DESKTOP-H68A51G:/mnt/d/racon/build$ make
Scanning dependencies of target gwbase
[  0%] Building CXX object GenomeWorks/common/base/CMakeFiles/gwbase.dir/src/cudautils.cpp.o
In file included from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/fmt.h:21,
                 from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/common.h:28,
                 from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/spdlog.h:12,
                 from /mnt/d/racon/vendor/GenomeWorks/common/base/include/claraparabricks/genomeworks/logging/logging.hpp:99,
                 from /mnt/d/racon/vendor/GenomeWorks/common/base/include/claraparabricks/genomeworks/utils/cudautils.hpp:22,
                 from /mnt/d/racon/vendor/GenomeWorks/common/base/src/cudautils.cpp:17:
/mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/bundled/format.h:3475:55: warning: ISO C++ did not adopt string literal operator templates taking an argument pack of characters [-Wpedantic]
 3475 | FMT_CONSTEXPR internal::udl_formatter<Char, CHARS...> operator""_format() {
      |                                                       ^~~~~~~~
[  0%] Building CXX object GenomeWorks/common/base/CMakeFiles/gwbase.dir/src/logging.cpp.o
In file included from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/fmt.h:21,
                 from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/common.h:28,
                 from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/spdlog.h:12,
                 from /mnt/d/racon/vendor/GenomeWorks/common/base/include/claraparabricks/genomeworks/logging/logging.hpp:99,
                 from /mnt/d/racon/vendor/GenomeWorks/common/base/src/logging.cpp:17:
/mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/bundled/format.h:3475:55: warning: ISO C++ did not adopt string literal operator templates taking an argument pack of characters [-Wpedantic]
 3475 | FMT_CONSTEXPR internal::udl_formatter<Char, CHARS...> operator""_format() {
      |                                                       ^~~~~~~~
[  0%] Building CXX object GenomeWorks/common/base/CMakeFiles/gwbase.dir/src/graph.cpp.o
[  0%] Linking CXX static library ../../../lib/libgwbase.a
[  0%] Built target gwbase
[  0%] Building NVCC (Device) object GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/src/cudaaligner_generated_hirschberg_myers_gpu.cu.o
[  0%] Building NVCC (Device) object GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/src/cudaaligner_generated_ukkonen_gpu.cu.o
[  0%] Building NVCC (Device) object GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/src/cudaaligner_generated_myers_gpu.cu.o
Scanning dependencies of target cudaaligner
[ 11%] Building CXX object GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/src/cudaaligner.cpp.o
In file included from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/fmt.h:21,
                 from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/common.h:28,
                 from /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/spdlog.h:12,
                 from /mnt/d/racon/vendor/GenomeWorks/common/base/include/claraparabricks/genomeworks/logging/logging.hpp:99,
                 from /mnt/d/racon/vendor/GenomeWorks/cudaaligner/src/cudaaligner.cpp:18:
/mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/bundled/format.h:3475:55: error: ISO C++ did not adopt string literal operator templates taking an argument pack of characters [-Werror=pedantic]
 3475 | FMT_CONSTEXPR internal::udl_formatter<Char, CHARS...> operator""_format() {
      |                                                       ^~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/build.make:84: GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/src/cudaaligner.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:4898: GenomeWorks/cudaaligner/CMakeFiles/cudaaligner.dir/all] Error 2
make: *** [Makefile:152: all] Error 2 ```

[rvaser]

Run nano +3475 /mnt/d/racon/vendor/GenomeWorks/3rdparty/spdlog/include/spdlog/fmt/bundled/format.h and delete quotes from operator""_format() (or use any other editor). Recompile and it should work according to the referenced issue.

[jkbenotmane]

Thank you that solved it !