ATDM Trilinos 'sems-rhel7' configuration broken on new CEE hpwsXYZ and cee-buildXYZ machines #10022
Comments
bartlettroscoe
added
type: bug
|
@trilinos/framework, my understanding is that CEE is pushing everyone to move to these HPWS machines and is eliminating the RWS and EWS machines. I know the current ATDM Trilinos 'sems-rhel7' builds are running on ascicgpu machines but most Trilinos developers can't get access to those machines. (For example, I don't think I currently have access to any ascigpu machines.) I need to have a working set of Trilinos builds in order to test TriBITS work associated with TriBITSPub/TriBITS#367. Otherwise, I just have to do my best to locally test and cross fingers and merge to Trilinos 'develop'. But I would rather run a more comprehensive set of builds before merging future TriBITS changes. |
|
@bartlettroscoe: I don't see this build error showing up in the PrimaryATDM or SecondaryATDM builds. I will raise this github issue during our meeting tomorrow. CC: @jwillenbring, @ZUUL42 |
@e10harvey, I suspect that is because the builds posting to CDash are running on ASCIGPU machines, not HPWS machines. The latter are fairly new. See my comment above. |
|
FYI: As a basis of comparison, I ran the exact same Teuchos 'sems-rhel7' builds with the exact same version of Trilinos on the CEE build machine 'cee-build015' and I got all passing builds and tests (except for 3 failing tests from the CUDA build because this machine does not have a GPU). This shows the problem is with the SEMS modules and/or the ATDM Trilinos env and/or the HPWS machines themselves. Details of Teuchos 'sems-rhel7' builds on 'cee-build015' and results on CDash (click to expand). Now to run the full set of supported builds for Teuchos on the machine 'cee-build015': posted to: with only 3 failing tests in the CUDA build: and all 3 because this machine does not have a GPU and displays the error: as shown in this query: |
|
FYI: The RHEL7 version is newer on the HPWS machine 'hpws055' than on the older CEE machine 'cee-build015'. The details are given in TRILINOSHD-59. It may just be that the SEMS modules are broken for the newer OS. Hopefully the newer SEMS modules build with Spack will work okay. |
|
@fryeguy52: Would you please look into the module issues on the HPWS machine? |
|
FYI: I got a rude reminder this is still broken as documented in #10836 (comment). |
|
And I also hit this as described in #10823 (comment). I don't seem to be remembering that the SEMS modules are broken in the CEE High Performance Work Station (HPWS) machines :-( |
|
It seems this same problem is impacting the new 'ascic0xy' machines as well. See #10999. |
bartlettroscoe
changed the title
|
Well shoot, this same error is occurring on the 'cee-build030' machine as well :-( I am trying to run full Trilinos PR builds as part of testing a CMake upgrade as part of #10355 on the beefy machine 'cee-build030' and I can't successfully load a GenConfig env. Specifically, for the env load scirpt When I source it it produces: And then when I try to configure and build Trilinos, the mpiexec command used by Trilinos produces errors like: While this may be a problem with the SEMS modules, it seems that GenConfig can't figure out of it loaded the env correctly or not. |
|
See #10999 (comment) |
|
This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. |
github-actions
bot
added
the
MARKED_FOR_CLOSURE
|
This issue was closed due to inactivity for 395 days. |
github-actions
bot
added
the
CLOSED_DUE_TO_INACTIVITY
@trilinos/framework
Description
It seems the sems modules are broken on the new CEE 'hpwsXYZ' and 'cee-buildXYZ' machines. And the Trilinos GenConfig scripts don't detect they are broken (see below).
Original Description
It appears that the SEMS RHEL7 modules or perhaps just the ATDM Trilinos 'sems-rhel7' configuration is broken on the new CEE HPWS machines. It seems the builds complete but a lot of tests fail with errors like:
all showing
undefined symbol: ompi_common_verbs_usnic_register_fake_drivers.To demonstrate this, I used a recent version of Trilinos 'develop' from ATDM Trilinos testing day 2021-12-21 55091be as shown by:
that was pretty clean as shown in this query.
I demonstrated the problem by running, on my machine 'hpws055', I ran all of the supported ATDM Trilinos 'sems-rhel7' builds for just the Teuchos test suite as:
Detailed output from ctest-s-local-test-drivers.sh (click to expand)
.
which posted to CDash as shown here:
There are 126 failing tests for each build, 756 failing tests across all of these builds as shown here:
All of these failing tests show the error
undefined symbol: ompi_common_verbs_usnic_register_fake_driversas shown in this query:Internal issues:
The text was updated successfully, but these errors were encountered: