Skip to content

Latest commit

 

History

History
171 lines (125 loc) · 4.68 KB

README.md

File metadata and controls

171 lines (125 loc) · 4.68 KB

Ark2

What is it ?

Provide performance portable Kokkos implementation for multidimensional compressible Magneto Hydrodynamics in 3D.

Dependencies

  • Kokkos with version >= 2.9.00
  • CMake with version >= 3.9

Optional

How to get ark v2 sources

Kokkos sources are included as a git submodule.

To download project "ark2" clone it with option "--recurse-submodules"

git clone --recurse-submodules https://gitlab.erc-atmo.eu/remi.bourgeois/ark-2-mhd

If you performed a regular "git clone", just type

git submodule init
git submodule update

to retrieve kokkos sources.

How to build

A few example of builds. Default values make it compile without MPI, with Kokkos-serial and in Release build type.

Build Serial-Serial (Default)

Create a build directory, configure and make

mkdir build && cd build
cmake ..
cmake --build . -- -j 4

Add variable CXX on the cmake command line to change the compiler (clang++, icpc, pgcc, ....)

Build Serial-OpenMP

Create a build directory, configure and make

mkdir build && cd build
cmake -DKokkos_ENABLE_OPENMP=ON ..
cmake --build . -- -j 4

Add variable CXX on the cmake command line to change the compiler (clang++, icpc, pgcc, ....)

Build Serial-Cuda

Create a build directory, configure and make (for a Maxwell GPU, see Table 4.2 for available architectures)

mkdir build && cd build
export CXX=$PWD/../lib/kokkos/bin/nvcc_wrapper
cmake -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_MAXWELL50=ON ..
cmake --build . -- -j 4

Build MPI-OpenMP

Create a build directory, configure and make

mkdir build && cd build
cmake -DSESSION=MPI_SESSION -DKokkos_ENABLE_OPENMP=ON ..
cmake --build . -- -j 4

Build MPI-Cuda

Please make sure to use a CUDA-aware MPI implementation (OpenMPI or MVAPICH2) built with the proper flags for activating CUDA support.

Create a build directory, configure and make

mkdir build && cd build
export CXX=$PWD/../lib/kokkos/bin/nvcc_wrapper
cmake -DSESSION=MPI_SESSION -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_MAXWELL50=ON ..
cmake --build . -- -j 4

Build PDI

Get the PDI source from https://gitlab.maisondelasimulation.fr/pdidev/pdi We recommande the 1.5.2 release.

Create a build directory, configure and make

cd pdi-1-5-2 && mkdir build && cd build
cmake -DBUILD_HDF5_PARALLEL=OFF -DUSE_yaml=EMBEDDED -DUSE_paraconf=EMBEDDED ..
make install -j

Set the PDI environment by

source path_to/pdi-1.5.2/install/share/pdi/env.bash

How to run

In the build directory just type

./main configuration_file.ini --kokkos-xxx=value

If you add kokkos options (see Table 5.1 for a list of options) from the command line, use the prefixed version (--kokkos-xxx).

Run Serial-OpenMP

./main path/to/configuration_file.ini --kokkos-threads=2

Run MPI-Cuda

To run on 4 GPUs, 1 GPU per MPI task

mpirun -np 4 ./main path/to/configuration_file.ini --kokkos-ndevices=4

Developping with vim and youcomplete plugin

Assuming you are using vim (or neovim) text editor and have installed the youcomplete plugin, you can have semantic autocompletion in a C++ project.

Make sure to have CMake variable CMAKE_EXPORT_COMPILE_COMMANDS set to ON, and symlink the generated file to the top level source directory.

Coding rules

For cmake follow Modern CMake which also cites other sources like coding rules from Effective Modern CMake.

Tips

Verbose mode of nvcc_wrapper

You can set the variable

NVCC_WRAPPER_SHOW_COMMANDS_BEING_RUN=1

when you compile. This prints to the standard output the nvcc command line.

Usage of Kokkos-tools (see Kokkos documentation)

To use one of the tools you have to compile it, which will generate a dynamic library. Before executing the Kokkos application you then have to set the environment variable KOKKOS_PROFILE_LIBRARY to point to the dynamic library e.g. in Bash, in the build directory type:

export KOKKOS_PROFILE_LIBRARY=${PWD}/../lib/kokkos-tools/src/tools/memory-events/kp_memory_events.so

Many of the tools will produce an output file which uses the hostname as well as the process id as part of the filename.