Numerical issues caused by unsafe IntelLLVM flags

[kvrigor]

Background: Since Stages/2023 the MPI compiler backend defaults to IntelLLVM. Classic Intel compilers are still accessible by passing the appropriate option to the MPI compiler:

module load IntelMPI
export CC='mpiicc -cc=icc'
export FC='mpiifort -f90=ifort'
export CXX='mpiicpc -cxx=icpc'

@AGonzalezNicolas ran tests comparing CLM5 and eCLM. Results show non-zero differences for IntelLLVM-compiled eCLM, yet absolute zero errors when classic Intel was used. This brings to question the correctness of the compile flags being passed to IntelLLVM.

While we could get away with sticking with classic Intel compilers, the doomsday warning below urges us to migrate to IntelLLVM soon:

$ mpicc -cc=icc --version
icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated
and will be removed from product release in the second half of 2023. The
Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler
moving forward. Please transition to use this compiler. Use '-diag-disable=10441'
to disable this message.

icc (ICC) 2021.10.0 20230609
Copyright (C) 1985-2023 Intel Corporation.  All rights reserved.

[kvrigor]

As a first step towards a fix, I'd suggest to split Intel and IntelLLVM in SetBuildOptions.cmake:

eCLM/cmake/SetBuildOptions.cmake

Lines 32 to 39 in c9f3348

	elseif(COMPILER STREQUAL "Intel" OR COMPILER STREQUAL "IntelLLVM")
	add_compile_definitions(CPRINTEL)
	set(CMAKE_C_FLAGS "-qno-opt-dynamic-align -std=gnu99 -fp-model precise -qopenmp")
	set(CMAKE_C_FLAGS_DEBUG "-O0 -g")
	set(CMAKE_C_FLAGS_RELEASE "-O2 -debug minimal")
	set(CMAKE_Fortran_FLAGS "-free -qno-opt-dynamic-align -ftz -traceback -convert big_endian -assume byterecl -assume realloc_lhs -fp-model source -qopenmp")
	set(CMAKE_Fortran_FLAGS_DEBUG "-O0 -g -fpe0 -check all")
	set(CMAKE_Fortran_FLAGS_RELEASE "-O2 -debug minimal")

It makes sense to start with the most conservative flags suggested in the Classic ifort to LLVM ifx Porting Guide. COMPILER=IntelLLVM should contain the following:

• add_compile_definitions(CPRINTEL)
• Same CMAKE_C_FLAGS, CMAKE_C_FLAGS_DEBUG, and CMAKE_C_FLAGS_RELEASE as COMPILER=Intel
• Change CMAKE_Fortran_FLAGS:
  • -warn all: Check for compile-time warnings
  • -check all: Enable runtime checks
  • -standard-semantics: Obey Fortran semantics for expressions
  • -fpmodel=precise: Use precise floating-point settings
• Change CMAKE_Fortran_FLAGS_DEBUG:
  • O0: Disable optimization
  • -g -traceback: Create debug symbols and print stacktrace on crash
  • -no-simd: Turn off default OpenMP SIMD
• Change CMAKE_Fortran_FLAGS_RELEASE:
  • O0: Disable optimization

Note: There's also a Porting Guide from Classic C/C++ to LLVM-based C/DPC++. We can check the C flags after we tame the IntelLLVM-based Fortran compiler 😃