-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathinstall.txt
249 lines (198 loc) · 10.1 KB
/
install.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
----------------------------------------------------------------------
To do a spack install and test for a machine <machine> at site <site>:
1. Set up a top-level "~/gits/" directory on <machine>, and do all the git clones into it.
You can really put them wherever you want, but the remainder of these
instructions assume everything goes into ~/gits/
2. Set up the repositories:
cd ~/gits
git clone https://github.com/jmellorcrummey/spack-configs
That's where this file lives, so you must have already done that!
# Clean up any previous spack to circumvent any spack caching
/bin/rm -rf ~/gits/spack
git clone https://github.com/spack/spack
/bin/rm -rf ~/.spack
You may add a -b <branch> to use something other than the base spack.
FYI:
The hpctoolkit sources may not be needed, but can be gotten by:
git clone https://gitlab.com/hpctoolkit/hpctoolkit
See build.txt for more information
The minitest sources may not be needed, but can be gotten by
git clone https://gitlab.com/hpctoolkit/minitest.git
See the README.md file in that repository for more information
3. Set up your shell to use spack and the spacklink setup command:
for bash:
vi ~/.bashrc #Add the following:
umask 002 (or 022 to make everything NOT group-writable)
export SPACK_ROOT=~/gits/spack
export PATH=${SPACK_ROOT}/bin:~gits/spack-configs/bin:${PATH}
source ${SPACK_ROOT}/share/spack/setup-env.sh
export <PROXIES, as needed for spack usage>
source ~/.bashrc
4. Tell spack to use the right gcc:
First:
which gcc; gcc --version
Then:
module avail gcc ( on some systems use module avail gnu )
If a module version is later than the one on your path,
module load gcc.xxx -- to put that later version on your path
Run "spack compiler find" to tell spack about those compilers.
If compiler version is 5.x or later, it's good enough to install hpctoolkit,
and you are done with this step.
If it is not:
Run "spack install [email protected]" (or some other version 5.x or later)
"module load <the gcc you've just installed>"
Run "spack compiler find" again to tell spack about it
Run "spack install [email protected]" again, to have it built with itself,
so all future installs will be in the same directory (the compiler
version is part of the full path name)..
5. Set up the spack install environment:
First, go to the spack-configs repo, and prepare the appropriate files
for installing at <site> on <machine>:
<site>/<machine>/<machine>.config.yaml
<site>/<machine>/<machine>.modules.yaml
<site>/<machine>/<machine>.packages.yaml
The second "<machine>." preface to the files is to make it easy to
ensure the links are for the right machine.
Those files may need to be tweaked for the particular installation, which will be done in step 6.
If <site> or <machine> is not yet there, create the directories, and copy the three files
from the directory for a similar machine. Besure to set the <machine>.*.yaml names
for the right machine.
Run "spacklink <$SPACK_ROOT> <site> <machine>"
The spacklink script is in the bin directory in the spack-configs repository
That script does the following:
cd $SPACK_ROOT/etc/spack
rm *.yaml
ln -s .../spack-configs/<site>/<machine>/<machine>.config.yaml config.yaml
ln -s .../spack-configs/<site>/<machine>/<machine>.modules.yaml modules.yaml
ln -s .../spack-configs/<site>/<machine>/<machine>.packages.yaml packages.yaml
Note that if any of those files are not present, no link will be made, and the spack
default will be used.
VERIFY that the machine you are on matches <machine> in the directory and
the in file names of the symlinks.
The <machine>.modules.yaml file describes the modules to be built.
There are three types of modules, lmod, tcl, and dotkit: the module files in
the various subdirectories build all three.
dotkit is now obsolete.
The modules built in this step have long, human-hostile names; more
user-friendly modules are constructed below.
6. Configure the installation
The installation configuration is controlled by the three .yaml files referred to above.
The syntax of a .yaml file is quite precise; the indentation of any line in is a multiple
of two spaces, the ordering and indentation of the lines specify a structure.
The <machine>.config.yaml says where to install its packages and modules
It seems best to set a top-level directory, <INSTALL> and set up as follows:
config:
install_tree: <INSTALL>/packages
module_roots:
lmod: <INSTALL>/modules/lmod
tcl: <INSTALL>/modules/tcl
The rest of the file will probably not need configuration
The top-level directory in spack-configs contains a file hpctoolkit.fixed.packages.yaml
It contains the packages.yaml file from the current hpctoolkit git repo, with
the changes needed to install the latest release.
For 2021.5.15, it sets dyninst@master and [email protected]
The <machine>.packages.yaml file describes the dependencies for the install.
If you do not have a <machine>.packages.yaml file, the spack installation
will build all necessary prerequisites.
There is a subtle difference between having a packages.yaml file specifying all
the prerequisites and not specifying things, but letting spack build them.
If you specify the prerequisites, you will get the named versions; if not,
spack may build a different version. It is best to start with the packages.yaml
file that is in the hpctoolkit repository as .../spack/packages.yaml, or the copy
in the top-level directory which has fixes for the latest release, and make
changes against it.
For many systems, you will need to customize the packages.yaml file, editing it via its symlink,
for the specific system, editing various entries:
a. Run module avail mpi, and find a suitable MPI to use., or install one
Edit the entry for openmpi to refer to it. It is importants to set this
parameter, since the batch systems are aware of MPI and know how to use it.
Note that the module name is case-sensitive.
This is crucial on systems with a head node and compute nodes, since most
such systems have specific interactions with job submission and the MPI is use.
b. Find the installed CUDA on the system, and define it in the packages.yaml file
This is essential for GPU support, since the specific CUDA for HPCT must match
the one with the kernel drivers installed.
c. If you are installing on x86, set the architecture to use, following the "packages" line:
packages:
all:
target: ['x86_64','ppc64','ppc64le','aarch64']
That tells spack to compile for a generic x86/ppc/arm machine, rather than the specific machine
on which you are doing the install.
If you want, you can also:
a. Find a suitable python on the system or in a module, or install one
Edit the entry for python to refer to it
b. Find a suitable cmake on the system or in a module, or install one
Edit the entry for cmake to refer to it
c. Find a suitable perl on the system or in a module, or install one
Edit the entry for perl to refer to it
If you don't do those, the spack install will build them anyway, and that will take more install time.
8. Do the install of hpctoolkit
Decide the specific version and options you want to build.
hpctoolkit@<version> <options> <compiler>
<version>
the default is the latest release @2021.03.01
@develop -- for current bits
<options>
+mpi
+cuda
+papi
+all-static
Multiple options can be concatenated
<compiler>
%gcc@<version> -- only needed if the spack default is not appropriate
You can see what will get built with:
spack spec hpctoolkit@<version> <options> <compiler>
Then execute the spack install command:
<cmd-to-run> spack [-k] install hpctoolkit@<version><options><compiler>
The -k turns off security checking, and may be needed on systems requiring network proxies.
On non-compute node machines, <cmd-to-run> is not needed.
At LLNL:
On rzansel and lassen, <cmd-to-run> = "lalloc 1"
On rzmanta and ray, <cmd-to-run> = "lalloc 1"
On rzgenie and quartz, <cmd-to-run> = "salloc -N 1 -ppdebug"
At LANL:
On kodiak, <cmd-to-run> = "salloc -N 1 --qos=interactive"
At Sandia:
On vortex, <cmd-to-run> = "lalloc 1"
On eclipse, <cmd-to-run> = "salloc -N1"
On stria, <cmd-to-run> = "salloc -N1"
On doom, <cmd-to-run> = "salloc -N1"
At ANL:
At NERSC:
9. Install the hpcviewer, where appropriate
spack install hpcviewer@<version> <option>
<version>
the default is the latest release, @2021.03
For consistency the module name for it should be made to match the hpctoolkit release, @2021.03.01,
rather than match the installed name.
On all supported machines <option> is not needed
10. Construct a human-readable module to be used by others
These files can live in a directory,
<path-to-modulefiles-directory>/{hpctoolkit,hpcviewer}/
that is accessible to others.
The users would run:
module use <path-to-modulefiles-directory>
module avail hpc
to see what versions are available
To construct lmod modules,
cd <INSTALL>/modules/lmod
find . -name hpctoolkit
It will be a directory; cd to it
It will have a <version>.lua file corresponding to each of the hpctoolkit
installs that you ran.
cp <version>.lua <path-to-modulefiles-directory>/hpctoolkit/<version>.lua
The <path-to-modulefiles-directory> may also have a file named "default", which
should be a symlink to the version that is the default.
It also may have a file named "rolling-release", which should be a symlink to
the latest version installed.
Update those two links, as appropriate.
Follow the same instructions for hpcviewer.
If desired, you can merge the the "prepend-path PATH" from the hpcviewer module
into the hpctoolkit module, so that loading it will also load the viewer.
11. Verify the installation:
module use <path-to-modulefiles-directory>
module avail hpc
For each hpctoolkit <version>:
module load <version>
hpcrun --version
Make sure they correspond to the installation as expected