Running FKESSLER on Izumi

Here are the detailed steps to check out the StormSPEED CAM code and create/configure/build/run a FKESSLER analytic test on the NCAR Izumi cluster :

Clone the StormSpeed repository

cd /scratch/cluster/$USER
git clone https://github.com/NCAR/StormSPEED
cd StormSPEED

Check out the StormSPEED CAM code and externals

git checkout stormspeed
./manage_externals/checkout_externals

Setup the Izumi environment for configuring and compiling StormSPEED

For now, this will need to be done for each new terminal session. We are working on making this work out of the box for supported hardware.

module load compiler/intel/20.0.1 lang/python/3.11.5
setenv Kokkos_ROOT "/project/amp/jet/install_izumi/spack/opt/spack/linux-almalinux8-skylake_avx512/gcc-9.3.0/kokkos-3.0.00-f6s53cvwkjph5gbxt7cadkq4psqnph22"
setenv GCC_ROOT "/project/amp/jet/install_izumi/spack/opt/spack/linux-almalinux8-skylake_avx512/gcc-9.3.0/gcc-9.3.0-w7rjyw3sdfhyxaih5huqx7q6tpxhnxmr"
setenv USER_SLIBS "$Kokkos_ROOT/lib64/libkokkoscontainers.so.3.0.0 $Kokkos_ROOT/lib64/libkokkoscore.so.3.0.0 $GCC_ROOT/lib64/libstdc++.so.6.0.28"
setenv LD_LIBRARY_PATH "${LD_LIBRARY_PATH}:${Kokkos_ROOT}/lib64:${GCC_ROOT}/lib64"

Create a Kessler Simple physics test and set the dycore target to theta-l

cd /scratch/cluster/$USER
set CESMDIR=/scratch/cluster/$USER/StormSPEED
$CESMDIR/cime/scripts/create_newcase --compset FKESSLER  --res ne16_g37 --compiler intel --case FKESSLER_StormSPEED.01 --driver mct --walltime 24:00:00 --queue long --run-unsupported

Configure the test

cd FKESSLER_StormSPEED.01
./xmlchange CAM_TARGET=theta-l
./xmlchange STOP_OPTION=ndays,STOP_N=12,RESUBMIT=0
./xmlchange PROJECT=none
./xmlchange JOB_WALLCLOCK_TIME=24:00:00
./xmlchange DOUT_S='FALSE'
./xmlchange BFBFLAG=TRUE
./case.setup

Build the test

./case.build

Run the test

set RUNDIR=`./xmlquery -value RUNDIR`
mkdir -p $RUNDIR/timing/checkpoints
./case.submit