Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panzer: Look into #13717 decomposition issue #13828

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
+360 −2
Open

Panzer: Look into #13717 decomposition issue #13828

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a8d12b2
Panzer :: Start reproducer
reuterb Feb 21, 2025
26a3b04
updating multiblock ic/bc test
rppawlo Feb 25, 2025
b22243d
Panzer: update nox exodus observer to write initial conditions
rppawlo Feb 25, 2025
daf3f8e
Panzer: utility script for seacas
rppawlo Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/panzer/adapters-stk/example/main_driver/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ TRIBITS_COPY_FILES_TO_BINARY_DIR(main_driver_files
periodic_wedge.xml
periodic_wedge.pam
energy-periodic-32bit-limit.xml
energy-ss-tp-multiblock-ic-bc-issue.xml
energy-ss-tp-multiblock-ic-bc-issue.jou
make-mesh.sh
EXEDEPS main_driver
)

Expand Down
40 changes: 40 additions & 0 deletions packages/panzer/adapters-stk/example/main_driver/energy-ss-tp-multiblock-ic-bc-issue.jou
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#{ctype = "tri3"}
#{scheme = "triadvance"}

create surface rectangle width 2 height 1 zplane
split surface 1 across location position -0.05 -2 0 location position -0.05 2 0
merge surface 3 2

set duplicate block elements off
block 1 add surface 3
block 1 name "left"
block 1 element type {ctype}

set duplicate block elements off
block 2 add surface 2
block 2 name "right"
block 2 element type {ctype}

sideset 1 add curve 2 wrt surface 3
sideset 1 name "xmin"
sideset 2 add curve 4 wrt surface 2
sideset 2 name "xmax"
sideset 3 add curve 9 wrt surface 3
sideset 3 add curve 6 wrt surface 2
sideset 3 name "ymin"
sideset 4 add curve 8 wrt surface 3
sideset 4 add curve 7 wrt surface 2
sideset 4 name "ymax"
sideset 5 add curve 5 wrt surface all
sideset 5 name "center"

surface all interval 7
surface all scheme {scheme}
mesh surface all

#{tolower(ctype)}

set exodus netcdf4 on

#export mesh "{ctype}-combined-centers.exo" dimension 2 overwrite
export mesh "energy-ss-tp-multiblock-ic-bc-issue.gen" dimension 2 overwrite
310 changes: 310 additions & 0 deletions packages/panzer/adapters-stk/example/main_driver/energy-ss-tp-multiblock-ic-bc-issue.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
<ParameterList name="Drekar">

<ParameterList name="Mesh">

<Parameter name="Source" type="string" value="Exodus File" />

<ParameterList name="Exodus File">
<Parameter name="File Name" type="string" value="energy-ss-tp-multiblock-ic-bc-issue.gen" />
</ParameterList>

</ParameterList>

<ParameterList name="Initial Conditions">
<ParameterList name="left">
<ParameterList name="TEMPERATURE">
<Parameter name="Value" type="double" value="0.0"/>
</ParameterList>
</ParameterList>
<ParameterList name="right">
<ParameterList name="TEMPERATURE">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
</ParameterList>
</ParameterList>
Comment on lines +13 to +24
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rppawlo So far I've just decomposed the mesh as they instructed and run with inconsistent IC's on the two blocks. That does not seem to give a wonky solution, see attached.
Screenshot 2025-02-21 at 4 16 56 PM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you rerun with 2 or 3 mpi processes? I think 1 looks correct. You may have to check that the initial conditions are printed at the right time. The unit test required some work on IO to be able to reproduce the issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rppawlo This is on 3 mpi processes. I didn't do any of the IO work to visualize the IC; however, I thought we determined that the BC issue should persist to the steady-state solution. I'm not seeing it at the moment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2025-02-26 at 1 13 52 PM

@rppawlo This is on 3 MPI processes with interval 7. This is the converged solution. I don't fully understand what is different between this and the case I ran the other day so need to do some digging.

Copy link
Contributor Author

@reuterb reuterb Feb 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so its your change in the Dirichlet boundary condition which now imposes a consistent BC (with the IC) as opposed to 0.0 on the top and 1.0 on the boundary. Is that what you would expect?


<ParameterList name="Responses">
<ParameterList name="Volume Integral">
<Parameter name="Type" type="string" value="Functional"/>
<Parameter name="Field Name" type="string" value="Volume_Integral"/>
<Parameter name="Element Blocks" type="string" value="left"/>
<Parameter name="Evaluation Types" type="string" value="Residual"/>
</ParameterList>
<ParameterList name="Volume Integral2">
<Parameter name="Type" type="string" value="Functional"/>
<Parameter name="Field Name" type="string" value="Volume_Integral"/>
<Parameter name="Element Blocks" type="string" value="left"/>
<Parameter name="Evaluation Types" type="string" value="Residual"/>
</ParameterList>
</ParameterList>

<ParameterList name="Block ID to Physics ID Mapping">
<Parameter name="left" type="string" value="solid"/>
<Parameter name="right" type="string" value="solid"/>
</ParameterList>

<ParameterList name="Assembly">
<Parameter name="Workset Size" type="int" value="99"/>
<Parameter name="Use Tpetra" type="bool" value="true"/>
</ParameterList>

<ParameterList name="Physics Blocks">

<ParameterList name="solid">

<ParameterList name="EQ 0">
<Parameter name="Type" type="string" value="Energy"/>
<Parameter name="Basis Type" type="string" value="HGrad"/>
<Parameter name="Basis Order" type="int" value="1"/>
<Parameter name="Integration Order" type="int" value="2"/>
<Parameter name="Model ID" type="string" value="fluid model"/>
<Parameter name="Prefix" type="string" value=""/>
</ParameterList>

</ParameterList>

</ParameterList>

<ParameterList name="Closure Models">

<ParameterList name="fluid model">

<ParameterList name="Volume Integral">
</ParameterList>

<ParameterList name="SOURCE_TEMPERATURE">
<Parameter name="Type" type="string" value="Parameter"/>
</ParameterList>
<ParameterList name="Heat Capacity">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
<ParameterList name="Thermal Conductivity">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
<ParameterList name="DENSITY">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
<ParameterList name="HEAT_CAPACITY">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
<ParameterList name="Global Statistics">
<Parameter name="Value" type="string" value="TEMPERATURE,DENSITY"/>
</ParameterList>

</ParameterList>

</ParameterList>

<ParameterList name="User Data">

<ParameterList name="function data one">

</ParameterList>

<ParameterList name="IP Coordinates">
<Parameter name="Integration Order" type="int" value="2"/>
</ParameterList>


</ParameterList>

<ParameterList name="Boundary Conditions">

<ParameterList>
<Parameter name="Type" type="string" value="Dirichlet"/>
<Parameter name="Sideset ID" type="string" value="ymax"/>
<Parameter name="Element Block ID" type="string" value="left"/>
<Parameter name="Equation Set Name" type="string" value="TEMPERATURE"/>
<Parameter name="Strategy" type="string" value="Constant"/>
<ParameterList name="Data">
<Parameter name="Value" type="double" value="0.0"/>
</ParameterList>
</ParameterList>

<ParameterList>
<Parameter name="Type" type="string" value="Dirichlet"/>
<Parameter name="Sideset ID" type="string" value="ymin"/>
<Parameter name="Element Block ID" type="string" value="left"/>
<Parameter name="Equation Set Name" type="string" value="TEMPERATURE"/>
<Parameter name="Strategy" type="string" value="Constant"/>
<ParameterList name="Data">
<Parameter name="Value" type="double" value="0.0"/>
</ParameterList>
</ParameterList>

<ParameterList>
<Parameter name="Type" type="string" value="Dirichlet"/>
<Parameter name="Sideset ID" type="string" value="ymax"/>
<Parameter name="Element Block ID" type="string" value="right"/>
<Parameter name="Equation Set Name" type="string" value="TEMPERATURE"/>
<Parameter name="Strategy" type="string" value="Constant"/>
<ParameterList name="Data">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
</ParameterList>

<ParameterList>
<Parameter name="Type" type="string" value="Dirichlet"/>
<Parameter name="Sideset ID" type="string" value="ymin"/>
<Parameter name="Element Block ID" type="string" value="right"/>
<Parameter name="Equation Set Name" type="string" value="TEMPERATURE"/>
<Parameter name="Strategy" type="string" value="Constant"/>
<ParameterList name="Data">
<Parameter name="Value" type="double" value="1.0"/>
</ParameterList>
</ParameterList>

</ParameterList>

<ParameterList name="Output">
<Parameter name="File Name" type="string" value="energy-ss-tp-multiblock-ic-bc-issue.exo"/>
<!-- This is inefficient: writes to file every residual evaluation: Used for debug, added to exercise capability -->
<ParameterList name="Cell Average Vectors">
<Parameter name="left" type="string" value="GRAD_TEMPERATURE"/>
</ParameterList>
<!-- -->
</ParameterList>

<ParameterList name="Options">
<Parameter name="Write Volume Assembly Graphs" type="bool" value="true"/>
<Parameter name="Volume Assembly Graph Prefix" type="string" value="energy-ss-tp-multiblock-ic-bc-issue"/>
</ParameterList>

<ParameterList name="Active Parameters">
<Parameter name="Number of Parameter Vectors" type="int" value="1"/>
<ParameterList name="Parameter Vector 0">
<Parameter name="Number" type="int" value="1"/>
<Parameter name="Parameter 0" type="string" value="SOURCE_TEMPERATURE"/>
<Parameter name="Initial Value 0" type="double" value="1.0"/>
</ParameterList>
</ParameterList>

<ParameterList name="Solution Control">
<Parameter name="Piro Solver" type="string" value="NOX"/>
<Parameter name="Compute Sensitivities" type="bool" value="0"/>
<Parameter name="Jacobian Operator" type="string" value="Have Jacobian"/>
<ParameterList name="LOCA">
<ParameterList name="Bifurcation"/>
<ParameterList name="Constraints"/>
<ParameterList name="Predictor">
<Parameter name="Method" type="string" value="Constant"/>
</ParameterList>
<ParameterList name="Stepper">
<Parameter name="Continuation Method" type="string" value="Natural"/>
<Parameter name="Initial Value" type="double" value="1.0"/>
<Parameter name="Continuation Parameter" type="string" value="Parameter 0"/>
<Parameter name="Max Steps" type="int" value="6"/>
<Parameter name="Max Value" type="double" value="12.25"/>
<Parameter name="Min Value" type="double" value="0.5"/>
<Parameter name="Compute Eigenvalues" type="bool" value="1"/>
<ParameterList name="Eigensolver">
<Parameter name="Method" type="string" value="Anasazi"/>
<Parameter name="Operator" type="string" value="Shift-Invert"/>
<Parameter name="Num Blocks" type="int" value="3"/>
<Parameter name="Num Eigenvalues" type="int" value="1"/>
<Parameter name="Block Size" type="int" value="1"/>
<Parameter name="Maximum Restarts" type="int" value="0"/>
<Parameter name="Shift" type="double" value="1.0"/>
</ParameterList>
</ParameterList>
<ParameterList name="Step Size">
<Parameter name="Initial Step Size" type="double" value="0.5"/>
<Parameter name="Aggressiveness" type="double" value="2.0"/>
</ParameterList>
</ParameterList>
<ParameterList name="NOX">
<ParameterList name="Direction">
<Parameter name="Method" type="string" value="Newton"/>
<ParameterList name="Newton">
<Parameter name="Forcing Term Method" type="string" value="Constant"/>
<Parameter name="Rescue Bad Newton Solve" type="bool" value="1"/>
<ParameterList name="Stratimikos Linear Solver">
<ParameterList name="NOX Stratimikos Options">
</ParameterList>
<ParameterList name="Stratimikos">
<Parameter name="Linear Solver Type" type="string" value="Belos"/>
<Parameter name="Preconditioner Type" type="string" value="Ifpack2"/>
<ParameterList name="Linear Solver Types">
<ParameterList name="Belos">
<Parameter name="Solver Type" type="string" value="Pseudo Block GMRES"/>
<ParameterList name="Solver Types">
<ParameterList name="Pseudo Block GMRES">
<Parameter name="Convergence Tolerance" type="double" value="1e-5"/>
<Parameter name="Output Frequency" type="int" value="10"/>
<Parameter name="Output Style" type="int" value="1"/>
<Parameter name="Verbosity" type="int" value="33"/>
<Parameter name="Maximum Iterations" type="int" value="100"/>
<Parameter name="Block Size" type="int" value="1"/>
<Parameter name="Num Blocks" type="int" value="20"/>
<Parameter name="Flexible Gmres" type="bool" value="0"/>
</ParameterList>
</ParameterList>
</ParameterList>
</ParameterList>
<ParameterList name="Preconditioner Types">
<ParameterList name="Ifpack2">
<Parameter name="Overlap" type="int" value="1"/>
<Parameter name="Prec Type" type="string" value="ILUT"/>
<ParameterList name="Ifpack2 Settings">
<Parameter name="fact: drop tolerance" type="double" value="0"/>
<Parameter name="fact: ilut level-of-fill" type="double" value="1"/>
<Parameter name="fact: level-of-fill" type="int" value="1"/>
</ParameterList>
</ParameterList>
</ParameterList>
</ParameterList>
</ParameterList>
</ParameterList>
</ParameterList>
<ParameterList name="Line Search">
<ParameterList name="Full Step">
<Parameter name="Full Step" type="double" value="1"/>
</ParameterList>
<Parameter name="Method" type="string" value="Full Step"/>
</ParameterList>
<Parameter name="Nonlinear Solver" type="string" value="Line Search Based"/>
<ParameterList name="Printing">
<Parameter name="Output Precision" type="int" value="3"/>
<Parameter name="Output Processor" type="int" value="0"/>
<ParameterList name="Output Information">
<Parameter name="Error" type="bool" value="1"/>
<Parameter name="Warning" type="bool" value="1"/>
<Parameter name="Outer Iteration" type="bool" value="1"/>
<Parameter name="Parameters" type="bool" value="1"/>
<Parameter name="Details" type="bool" value="1"/>
<Parameter name="Linear Solver Details" type="bool" value="1"/>
<Parameter name="Stepper Iteration" type="bool" value="1"/>
<Parameter name="Stepper Details" type="bool" value="1"/>
<Parameter name="Stepper Parameters" type="bool" value="1"/>
</ParameterList>
</ParameterList>
<ParameterList name="Solver Options">
<Parameter name="Status Test Check Type" type="string" value="Minimal"/>
</ParameterList>
<ParameterList name="Status Tests">
<Parameter name="Test Type" type="string" value="Combo"/>
<Parameter name="Combo Type" type="string" value="OR"/>
<Parameter name="Number of Tests" type="int" value="2"/>
<ParameterList name="Test 0">
<Parameter name="Test Type" type="string" value="Combo"/>
<Parameter name="Combo Type" type="string" value="AND"/>
<Parameter name="Number of Tests" type="int" value="2"/>
<ParameterList name="Test 0">
<Parameter name="Test Type" type="string" value="NormF"/>
<Parameter name="Tolerance" type="double" value="1.0e-8"/>
</ParameterList>
<ParameterList name="Test 1">
<Parameter name="Test Type" type="string" value="RelativeNormF"/>
<Parameter name="Tolerance" type="double" value="1.0e-4"/>
</ParameterList>
</ParameterList>
<ParameterList name="Test 1">
<Parameter name="Test Type" type="string" value="MaxIters"/>
<Parameter name="Maximum Iterations" type="int" value="10"/>
</ParameterList>
</ParameterList>
</ParameterList>
</ParameterList>


</ParameterList>
2 changes: 2 additions & 0 deletions packages/panzer/adapters-stk/example/main_driver/make-mesh.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
cubit -nogui -nographics -nojournal -batch test.jou
decomp --processors 3 --rcb tri3-combined-centers.exo
3 changes: 3 additions & 0 deletions packages/panzer/adapters-stk/example/main_driver/set_seacas_decomp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Allows panzer to read an exodus mesh that has not been sliced. It
# will be sliced at runtime. Source this to avoid needin nem_slice.
export IOSS_PROPERTIES="DECOMPOSITION_METHOD=rib"
4 changes: 2 additions & 2 deletions packages/panzer/adapters-stk/example/main_driver/user_app_NOXObserver_WriteToExodus.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ namespace user_app {

void runPreSolve(const NOX::Solver::Generic& /* solver */)
{

m_mesh->writeToExodus(0.0);
}

void runPostSolve(const NOX::Solver::Generic& solver)
Expand Down Expand Up @@ -104,7 +104,7 @@ namespace user_app {
m_response_library->evaluate<panzer::Traits::Residual>(ae_inargs);

// write to disk
m_mesh->writeToExodus(0.0);
m_mesh->writeToExodus(1.0);
}

protected:
Expand Down
Loading