New test 'TrilinosInstallTests_find_package_Trilinos' failing in ATS-2 PR builds with failure to find ShyLU_NodeConfig.cmake

[bartlettroscoe]

Description

After the merge of PR #10813, there are now install tests and a test that calls find_package(Trilinos) to see if there are problems with the TrilinosConfig.cmake file. All of those tests were run and passed with PR #10813.

For example, for the last testing iteration for PR #10775 shown here we see the failing test TrilinosInstallTests_find_package_Trilinos here showing the error:

CMake Error at /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/ShyLU/ShyLUConfig.cmake:145 (include):
  include could not find load file:

    /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/ShyLU/../ShyLU_Node/ShyLU_NodeConfig.cmake
Call Stack (most recent call first):
  /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/Trilinos/TrilinosConfig.cmake:114 (include)
  CMakeLists.txt:5 (find_package)

Full test error output: (click to expand)

================================================================================

TEST_0

Configure a dummy project that calls find_package(Trilinos)

Running: "/vscratch1/progenv-20220706/usr/tce/packages/cmake/cmake-3.18.0/bin/cmake" "-DCMAKE_PREFIX_PATH=/vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install" "/vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/Trilinos/packages/TrilinosInstallTests/find_package_Trilinos"

--------------------------------------------------------------------------------

-- Enabled Kokkos devices: CUDA;SERIAL
CMake Warning at /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/Kokkos/KokkosConfig.cmake:213 (MESSAGE):
  The installed Kokkos configuration does not support CXX extensions.
  Forcing -DCMAKE_CXX_EXTENSIONS=Off
Call Stack (most recent call first):
  /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/Amesos2/Amesos2Config.cmake:170 (include)
  /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/MueLu/MueLuConfig.cmake:192 (include)
  /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/TrilinosCouplings/TrilinosCouplingsConfig.cmake:172 (include)
  /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/Trilinos/TrilinosConfig.cmake:114 (include)
  CMakeLists.txt:5 (find_package)


CMake Error at /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/ShyLU/ShyLUConfig.cmake:145 (include):
  include could not find load file:

    /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/ShyLU/../ShyLU_Node/ShyLU_NodeConfig.cmake
Call Stack (most recent call first):
  /vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/install/lib/cmake/Trilinos/TrilinosConfig.cmake:114 (include)
  CMakeLists.txt:5 (find_package)


-- Enabled Kokkos devices: CUDA;SERIAL
Trilinos_FOUND = '1'
Trilinos_SELECTED_PACAKGES_LIST = ''
Trilinos_LIBRARIES = 'PikeBlackBox::pike-blackbox;TrilinosCouplings::trilinoscouplings;PanzerMiniEM::panzer-mini-fe;PanzerAdaptersSTK::panzer-stk;PanzerDiscFE::panzer-disc-fe;PanzerDofMgr::panzer-dof-mgr;PanzerCore::panzer-core;Piro::piro;ROL::rol-pdeopt;ROL::rol;Stokhos::stokhos_muelu;Stokhos::stokhos_muelu_pce_cuda;Stokhos::stokhos_muelu_pce_serial;Stokhos::stokhos_muelu_mp_16_cuda;Stokhos::stokhos_muelu_mp_16_serial;Stokhos::stokhos_ifpack2;Stokhos::stokhos_ifpack2_pce_cuda;Stokhos::stokhos_ifpack2_pce_serial;Stokhos::stokhos_ifpack2_mp_16_cuda;Stokhos::stokhos_ifpack2_mp_16_serial;Stokhos::stokhos_amesos2;Stokhos::stokhos_xpetra;Stokhos::stokhos_xpetra_pce_cuda;Stokhos::stokhos_xpetra_pce_serial;Stokhos::stokhos_xpetra_mp_16_cuda;Stokhos::stokhos_xpetra_mp_16_serial;Stokhos::stokhos_tpetra;Stokhos::stokhos_tpetraext_pce_cuda;Stokhos::stokhos_tpetraext_pce_serial;Stokhos::stokhos_tpetra_pce_cuda;Stokhos::stokhos_tpetra_pce_serial;Stokhos::stokhos_tpetra_sd_pce_cuda;Stokhos::stokhos_tpetra_sd_pce_serial;Stokhos::stokhos_tpetraext_mp_16_cuda;Stokhos::stokhos_tpetraext_mp_16_serial;Stokhos::stokhos_tpetra_mp_16_cuda;Stokhos::stokhos_tpetra_mp_16_serial;Stokhos::stokhos_tpetra_sd_mp_16_cuda;Stokhos::stokhos_tpetra_sd_mp_16_serial;Stokhos::stokhos_sacado;Stokhos::stokhos;Tempus::tempus;Rythmos::rythmos;ShyLU_DDFROSch::shylu_ddfrosch;Zoltan2Sphynx::zoltan2sphynx;MueLu::muelu-adapters;MueLu::muelu-interface;MueLu::muelu;NOX::locatpetra;NOX::locathyra;NOX::locaepetra;NOX::localapack;NOX::loca;NOX::noxepetra;NOX::noxlapack;NOX::nox;Phalanx::phalanx;Krino::krino_parser_lib;Krino::krino_rebalance_utils_lib;Krino::krino_region_lib;Krino::krino_adaptivity_interface_lib;Krino::krino_lib;Krino::krino_master_element_lib;Percept::percept;STKExprEval::stk_expreval;STKSearchUtil::stk_search_util_base;STKUnit_test_utils::stk_unit_test_utils;STKUnit_test_utils::stk_mesh_fixtures;STKBalance::stk_balance_lib;STKTools::stk_tools_lib;STKTools::stk_transfer_utils_lib;STKTransfer::stk_transfer_impl;STKSearch::stk_search;STKIO::stk_io_util;STKIO::stk_io;STKMesh::stk_mesh_base;STKTopology::stk_topology;STKNGP_TEST::stk_ngp_test;STKMath::stk_math;STKCoupling::stk_coupling;STKEmend::stk_emend;STKUtil::stk_util_command_line;STKUtil::stk_util_diag;STKUtil::stk_util_registry;STKUtil::stk_util_env;STKUtil::stk_util_parallel;STKUtil::stk_util_util;Intrepid2::intrepid2;Intrepid::intrepid;Teko::teko;FEI::fei_trilinos;FEI::fei_base;Stratimikos::stratimikos;Stratimikos::stratimikosbelos;Stratimikos::stratimikosamesos2;Stratimikos::stratimikosaztecoo;Stratimikos::stratimikosamesos;Stratimikos::stratimikosml;Stratimikos::stratimikosifpack;Ifpack2::ifpack2-adapters;Ifpack2::ifpack2;Anasazi::anasazitpetra;Anasazi::ModeLaplace;Anasazi::anasaziepetra;Anasazi::anasazi;SEACASAprepro_lib::aprepro_lib;SEACASIoss::io_info_lib;SEACASIoss::Ionit;SEACASIoss::Iotr;SEACASIoss::Iohb;SEACASIoss::Iogs;SEACASIoss::Iotm;SEACASIoss::Iogn;SEACASIoss::Iovs;SEACASIoss::Iopg;SEACASIoss::Ioex;SEACASIoss::Ioss;SEACASNemesis::nemesis;SEACASExodus::exodus;Amesos2::amesos2;Belos::belosxpetra;Belos::belostpetra;Belos::belosepetra;Belos::belos;ML::ml;Ifpack::ifpack;Zoltan2Core::zoltan2;Pamgen::pamgen_extras;Pamgen::pamgen;Amesos::amesos;Galeri::galeri-xpetra;Galeri::galeri-epetra;AztecOO::aztecoo;Isorropia::isorropia;Xpetra::xpetra-sup;Xpetra::xpetra;ThyraTpetraAdapters::thyratpetra;ThyraEpetraExtAdapters::thyraepetraext;ThyraEpetraAdapters::thyraepetra;ThyraCore::thyracore;Domi::domi;TrilinosSS::trilinosss;TpetraCore::tpetraext;TpetraCore::tpetrainout;TpetraCore::tpetra;TpetraCore::tpetraclassic;TpetraTSQR::kokkostsqr;EpetraExt::epetraext;Triutils::triutils;Shards::shards;Zoltan::zoltan;Epetra::epetra;MiniTensor::minitensor;Sacado::sacado;RTOp::rtop;KokkosKernels::kokkoskernels;TeuchosKokkosComm::teuchoskokkoscomm;TeuchosKokkosCompat::teuchoskokkoscompat;TeuchosRemainder::teuchosremainder;TeuchosNumerics::teuchosnumerics;TeuchosComm::teuchoscomm;TeuchosParameterList::teuchosparameterlist;TeuchosParser::teuchosparser;TeuchosCore::teuchoscore;KokkosAlgorithms::kokkosalgorithms;KokkosContainers::kokkoscontainers;KokkosCore::kokkoscore;Gtest::gtest'
-- Configuring incomplete, errors occurred!
See also "/vscratch1/trilinos/jaas/workspace/Trilinos_PR_cuda-10.1.243/pull_request_test/packages/TrilinosInstallTests/CMakeFiles/CMakeOutput.log".

--------------------------------------------------------------------------------

TEST_0: Return code = 1
TEST_0: Pass criteria = Match all REGEX {Trilinos_FOUND = '1'} [PASSED]
TEST_0: Pass criteria = Match all REGEX {-- Configuring done} [FAILED]
TEST_0: Pass criteria = Match all REGEX {-- Generating done} [FAILED]
TEST_0: Pass criteria = ALWAYS_FAIL_ON_NONZERO_RETURN [FAILED]
TEST_0: Result = FAILED

================================================================================

OVERALL FINAL RESULT: TEST FAILED (TrilinosInstallTests_find_package_Trilinos)

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Tasks

• Analyze what is different about builds that show this error ... see below
• Try to disable ShyLU pseudo package in all ATS-2 builds
• ???

[bartlettroscoe]

CC: @srajama1, @trilinos/shylu

NOTE: This issue is undoubtedly related to the fact that the ShyLU Trilinos package is not a real package in that it contains no targets or tests. It is just meant to be a convenience for so that users can configure Trilinos with:

-DTrilinos_ENABLE_ShyLU=ON

This problem is discussed in detail in #9497.

This might be a good time to design and implement the meta-package solution where when someone configures with:

-DTrilinos_ENABLE_ShyLU=ON \
-DTrilinos_ENABLE_TESTS:BOOL=ON \

this will behave the way that someone would expect and where there is not ShyLU package listed in the PackagesList.cmake file. Thinking about this a little bit, this may not be that hard to do. I just need to collect some requirements and then get to work.

[bartlettroscoe]

However, in the short term, it might just be better to comment out the ShyLU package line from the file Trilinos/PackagesList.cmake:

Trilinos/PackagesList.cmake

Line 128 in e061ffc

ShyLU packages/shylu PT

I would rather finish the refactoring needed to complete TriBITSPub/TriBITS#63 before implementing support for meta-packages.

The only negative downside of commenting out the definition of the ShyLU package is that users that are currently enabling:

-DTrilinos_ENABLE_ShyLU=ON \
-DTrilinos_ENABLE_TESTS=ON \

would need to enable:

-DTrilinos_ENABLE_ShyLU_Node=ON \
-DTrilinos_ENABLE_ShyLU_DD=ON \
-DTrilinos_ENABLE_TESTS=ON \

And that would behave exactly the way that one would expect, such as was the case with #9497.

[bartlettroscoe]

However, before I change anything, I want to try to reproduce this error and see what is happening. I also want to understand why this error only occurred in the ATS-2 build on 'vortex' PR-10775-test-ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables-867 and not in any of the other builds that should have the exact same set of enables/disables shown here.

[bartlettroscoe]

The set of initial enables/disables for the ATS-2 build look to be unique from all of the other builds. This seems to be triggering this error (the details are in the expandable section below).

I think the simple solution is to update the GenConfig files to disable the ShyLU package in the ATS-2 build to get around this problem in the short-term.

Analysis of enables/disables of the ATS-2 build compared to other PR builds: (click to expand)

Okay, I can see that the CUDA build on ATS-2 is strange in that it disables a bunch of packages up front as shown here which shows the initial set of enables/disables:

Explicitly enabled packages on input (by user): Tpetra 1

Explicitly enabled SE packages on input (by user): Tpetra STKUtil STKSimd STKTopology STKMesh STKIO STKSearch STKTransfer STKUnit_tests 9

Explicitly disabled packages on input (by user or by default): TriKota PyTrilinos NewPackage 3

Explicitly disabled SE packages on input (by user or by default): ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU SEACASExotec2 TriKota ShyLU_DDCore PanzerExprEval PyTrilinos NewPackage 9

So we see that a bunch of ShyLU_Node and ShyLU_DD subpackages are being disabled. Why?

This results in the final set of enabled and disable packages:

Final set of enabled packages: ... ShyLU_Node ... ShyLU_DD ShyLU ... 55

Final set of enabled SE packages: ... ShyLU_Node ... ShyLU_DDFROSch ShyLU_DD ShyLU ... 103

Final set of non-enabled SE packages: ... ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU ... ShyLU_DDCore ShyLU_DDCommon ... 58

What is strange is that the other ascicgpu CUDA build shown here for the same PR iteration in the same PR shows a different initial set of enables/disables:

Explicitly enabled packages on input (by user): Tpetra 1

Explicitly enabled SE packages on input (by user): Tpetra 1

Explicitly disabled packages on input (by user or by default): Ifpack2 TriKota STK NOX MueLu ShyLU_DD Stokhos Piro PyTrilinos NewPackage TrilinosCouplings 11

Explicitly disabled SE packages on input (by user or by default): ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU SEACASExotec2 Ifpack2 TriKota STK NOX MueLu ShyLU_DDCore ShyLU_DD Stokhos Piro PanzerAdaptersSTK PanzerMiniEM PanzerExprEval PyTrilinos NewPackage TrilinosCouplings 19

Note that only explicitly enabled SE packages in that build are Tpetra but for the ATS-2 builds is Tpetra STKUtil STKSimd STKTopology STKMesh STKIO STKSearch STKTransfer STKUnit_tests. Why are these STK subpackages always being explicitly enabled on input for every PR build on ATS-2?

So, ShyLU subpackages and packages are getting enabled in the ATS-2 build as shown here by the enable/disable logic showing:

Sweep forward enabling all forward library dependent packages because Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON ...

...
-- Setting Trilinos_ENABLE_ShyLU_DDFROSch=ON because Trilinos_ENABLE_Tpetra=ON
...
-- Setting Trilinos_ENABLE_ShyLU_DD=ON because Trilinos_ENABLE_ShyLU_DDFROSch=ON
-- Setting Trilinos_ENABLE_ShyLU=ON because Trilinos_ENABLE_ShyLU_DD=ON
...

However, in the other CUDA build on that ascicgpu machine shown here we see the enable/disable logic showing:

Disabling forward required SE packages and optional intra-package support that have a dependency on disabled SE packages Trilinos_ENABLE_<TRIBITS_PACKAGE>=OFF ...

...
-- Setting Trilinos_ENABLE_ShyLU=OFF because ShyLU has a required library dependence on disabled package ShyLU_DD

[bartlettroscoe]

FYI: I just pushed the branch 10865-ats2-disable-shylu which disables ShyLU for the ATS-2 build. I am going to suggest that it gets directly merged to the PR branch in #10775.

[bartlettroscoe]

With the merge of branch 10865-ats2-disable-shylu into the branch for PR #10775, that test should be fixed in that PR.

[bartlettroscoe]

This is resolved by PR #10872. Putting this in review.

[bartlettroscoe]

CC: @csiefer2

FYI: PR #10872 was merged just 23 hours ago and since then, there have not been any failures. In fact, if you sort this query by build name, you will see this test went from passing to failing in PR builds run after the merge for PRs #10851, #10834, #10827, #10808, #10802, and #10751. That is pretty good evidence that this test is fixed.

Closing as complete.