Skip to content

Commit

Permalink
remove internal spack package repo (#180)
Browse files Browse the repository at this point in the history 
Replace stackinator spack package repo with with system and site repos

The motivation is to remove CSCS-specific configuration from the stackinator tool, and allow for more fine-grained system-specific and site-specific package definitions..

The CSCS configuration has moved to https://github.com/eth-cscs/alps-cluster-config

Fixes #179
  • Loading branch information
@bcumming
bcumming authored Apr 3, 2024
1 parent d298eb4 commit 62f0fa6
Show file tree
Hide file tree
Showing 44 changed files with 194 additions and 4,918 deletions.
4 changes: 2 additions & 2 deletions .flake8
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[flake8]
max-line-length = 88
exclude = stackinator/repo/packages/, unittests/recipes/with-repo/repo/packages/, external/
max-line-length = 120
exclude = unittests/recipes/with-repo/repo/packages/, external/
extend-ignore = E203
2 changes: 1 addition & 1 deletion .github/workflows/lint.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
python -m pip install black flake8 isort mypy
- name: Black
run: |
black --check --verbose --exclude stackinator/repo stackinator unittests
black --check --verbose stackinator unittests
- name: isort
run: |
isort --check --skip stackinator/repo --diff .
Expand Down
1 change: 0 additions & 1 deletion MANIFEST.in
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
graft stackinator/etc
graft stackinator/repo
graft stackinator/schema
graft stackinator/share
graft stackinator/templates
41 changes: 40 additions & 1 deletion docs/cluster-config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ A cluster configuration is a directory with the following structure:
/path/to/cluster/configuration
├─ compilers.yaml # system compiler
├─ packages.yaml # external system packages
└─ concretiser.yaml
├─ concretiser.yaml
└─ repos.yaml # optional reference to additional site packages
```

The configuration is provided during the [configuration](configuring.md) step with the `--system/-s` flag.
Expand All @@ -31,3 +32,41 @@ If there are additional system packages that you want to use in a recipe, consid
* the more dependencies, the more potential that software stacks will have to be rebuilt when the system is updated, and the more potential there are for breaking changes;
* the external packages are part of the Spack upstream configuration generated with the Stack - you might be constraining the choices of downstream users.

## Site and System Configurations

The `repo.yaml` configuration can be used to provide a list of additional Spack package repositories to use on the target system.

These are applied automatically to every recipe built on the target cluster.

To provide site wide defaults, links to additional package repositories can be provdided in the the cluster definition.
For example, the following definition would link to a set of site-wide package definitions

```yaml
repos:
- ../site/repo
```
The paths are always interpretted as relative to the system configuration.
This is designed to make it encourage putting cluster definitions and the site description in the same git repository.
```
/path/to/cluster-configs
├─ my_cluster
│ ├─ compilers.yaml
│ ├─ packages.yaml
│ ├─ concretiser.yaml
│ └─ repos.yaml # refers to ../site/repo
└─ site
└─ repo # the site wide repo
└─ packages

## Package Precedence

If custom package definitions are provided for the same package in more than one location, Stackinator has to choose which definition to use.

There following precedence is applied, in descending order of precidence:
* packages defined in the (optional) `repo` path in the [recipe](recipes.md#custom-spack-packages)
* packages defined in the (optional) site repo(s) defined in the `repo/repos.yaml` file of cluster configuration (documented here)
* packages provided by Spack (in the `var/spack/repos/builtin` path)

As of Stackinator v4, the definitions of some custom repositories (mainly CSCS' custom cray-mpich and its dependencies) was removed from Stackinator, and moved to the the site configuration
35 changes: 22 additions & 13 deletions docs/recipes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -161,18 +161,23 @@ rocm-env:
# ...
```

As new versions of cray-mpich are released with CPE, they are added to Stackinator.
The following versions of cray-mpich are currently provided:
!!! alps

| cray-mpich | CPE | notes |
| :------------ | :-------- | :----------------------- |
| 8.1.25 | 23.03 | released 2023-02-26 **default** |
| 8.1.24 | 23.02 | released 2023-01-19 |
| 8.1.23 | 22.12 | released 2022-11-29 |
| 8.1.21.1 | 22.11 | released 2022-10-25 |
| 8.1.18.4 | 22.08 | released 2022-07-21 |
As new versions of cray-mpich are released with CPE, they are provided on Alps vClusters, via the Spack package repo in the [CSCS cluster configuration repo](https://github.com/eth-cscs/alps-cluster-config/tree/master/site/repo).
The following versions of cray-mpich are currently provided:

| cray-mpich | CPE | notes |
| :------------ | :-------- | :----------------------- |
| 8.1.29 | 24.03 | pre-release |
| 8.1.28 | 23.12 | released 2023-12 **default** |
| 8.1.27 | 23.09 | released 2023-09 |
| 8.1.26 | 23.06 | released 2023-06 |
| 8.1.25 | 23.03 | released 2023-02-26 |
| 8.1.24 | 23.02 | released 2023-01-19 |
| 8.1.23 | 22.12 | released 2022-11-29 |
| 8.1.21.1 | 22.11 | released 2022-10-25 |
| 8.1.18.4 | 22.08 | released 2022-07-21 |

!!! alps
All versions of cray-mpich in the table have been validated on Alps vClusters with Slingshot 11 and libfabric 1.15.2.

!!! note
Expand Down Expand Up @@ -316,12 +321,16 @@ repo
└─ package.py
```

Additional custom packages can be provided as part of the cluster configuration, as well as additional site packages.
These packages are all optional, and will be installed together in a single Spack package repository that is made available to downstream users of the generated uenv stack.
See the documentation for [cluster configuration](cluster-config.md) for more detail.

Stackinator internally provides its own package repository with a custom package for `cray-mpich` package, which it puts in the `alps` namespace.
The `alps` repository is installed alongside the packages, and is automatically available to all Spack users that use the Spack stack as an upstream.
!!! alps
All packages are installed under a single spack package repository called `alps`.
The CSCS configurations in [github.com/eth-cscs/alps-cluster-config](https://github.com/eth-cscs/alps-cluster-config) provides a site configuration that defines cray-mpich, its dependencies, and the most up to date versions of cuda, nvhpc etc to all clusters on Alps.

!!! warning
Unlike Spack package repositories, any `repos.yaml` file in the `repo` path will be ignored and a warning will be issued.
Unlike Spack package repositories, any `repos.yaml` file in the `repo` path will be ignored.
This is because the provided packages are added to the `alps` namespace.

## Post install configuration
Expand Down
7 changes: 5 additions & 2 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[tool.black]
line-length = 120

[tool.isort]
profile = "black"
skip = ["external/"]
profile = "black"
skip = ["external/"]
137 changes: 82 additions & 55 deletions stackinator/builder.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
import jinja2
import yaml

from . import VERSION, cache, root_logger
from . import VERSION, cache, root_logger, spack_util


def install(src, dst, *, ignore=None, symlinks=False):
Expand Down Expand Up @@ -197,7 +197,7 @@ def generate(self, recipe):
self._logger.debug(capture.stdout.decode("utf-8"))

if capture.returncode != 0:
self._logger.debug(f'error cloning the repository {spack["repo"]}')
self._logger.error(f'error cloning the repository {spack["repo"]}')
capture.check_returncode()

# Check out a branch or commit if one was specified
Expand All @@ -212,9 +212,7 @@ def generate(self, recipe):
self._logger.debug(capture.stdout.decode("utf-8"))

if capture.returncode != 0:
self._logger.debug(
f'unable to change to the requested commit {spack["commit"]}'
)
self._logger.debug(f'unable to change to the requested commit {spack["commit"]}')
capture.check_returncode()

# load the jinja templating environment
Expand Down Expand Up @@ -244,11 +242,7 @@ def generate(self, recipe):

make_user_template = jinja_env.get_template("Make.user")
with (self.path / "Make.user").open("w") as f:
f.write(
make_user_template.render(
build_path=self.path, store=recipe.mount, verbose=False
)
)
f.write(make_user_template.render(build_path=self.path, store=recipe.mount, verbose=False))
f.write("\n")

etc_path = self.root / "etc"
Expand All @@ -267,9 +261,7 @@ def generate(self, recipe):
post_hook = recipe.post_install_hook
if post_hook is not None:
self._logger.debug("installing post-install-hook script")
jinja_recipe_env = jinja2.Environment(
loader=jinja2.FileSystemLoader(recipe.path)
)
jinja_recipe_env = jinja2.Environment(loader=jinja2.FileSystemLoader(recipe.path))
post_hook_template = jinja_recipe_env.get_template("post-install")
post_hook_destination = store_path / "post-install-hook"

Expand All @@ -286,9 +278,7 @@ def generate(self, recipe):
pre_hook = recipe.pre_install_hook
if pre_hook is not None:
self._logger.debug("installing pre-install-hook script")
jinja_recipe_env = jinja2.Environment(
loader=jinja2.FileSystemLoader(recipe.path)
)
jinja_recipe_env = jinja2.Environment(loader=jinja2.FileSystemLoader(recipe.path))
pre_hook_template = jinja_recipe_env.get_template("pre-install")
pre_hook_destination = store_path / "pre-install-hook"

Expand All @@ -315,9 +305,7 @@ def generate(self, recipe):
"mirrors.yaml have been removed from cluster configurations,"
" use the --cache option on stack-config instead."
)
raise RuntimeError(
"Unsupported mirrors.yaml file in cluster configuration."
)
raise RuntimeError("Unsupported mirrors.yaml file in cluster configuration.")

# construct full file path
src = system_config_path / f_config.name
Expand Down Expand Up @@ -348,46 +336,91 @@ def generate(self, recipe):
with packages_path.open("w") as fid:
fid.write(packages_yaml)

# Configure the CSCS custom spack environments.
# Step 1: copy the CSCS repo to store_path where, it will be used to
# Add custom spack package recipes, configured via Spack repos.
# Step 1: copy Spack repos to store_path where they will be used to
# build the stack, and then be part of the upstream provided
# to users of the stack.
repo_src = self.root / "repo"
#
# Packages in the recipe are prioritised over cluster specific packages,
# etc. The order of preference from highest to lowest is:
#
# 3. recipe/repo
# 2. cluster-config/repos.yaml
# - if the repos.yaml file exists it will contain a list of relative paths
# to search for package
# 1. spack/var/spack/repos/builtin

# Build a list of repos with packages to install.
repos = []

# check for a repo in the recipe
if recipe.spack_repo is not None:
self._logger.debug(f"adding recipe spack package repo: {recipe.spack_repo}")
repos.append(recipe.spack_repo)

# look for repos.yaml file in the system configuration
repo_yaml = system_config_path / "repos.yaml"
if repo_yaml.exists() and repo_yaml.is_file():
# open repos.yaml file and reat the list of repos
with repo_yaml.open() as fid:
raw = yaml.load(fid, Loader=yaml.Loader)
P = raw["repos"]

self._logger.debug(f"the system configuration has a repo file {repo_yaml} refers to {P}")

# test each path
for rel_path in P:
repo_path = (system_config_path / rel_path).resolve()
if spack_util.is_repo(repo_path):
repos.append(repo_path)
self._logger.debug(f"adding site spack package repo: {repo_path}")
else:
self._logger.error(f"{repo_path} from {repo_yaml} is not a spack package repository")
raise RuntimeError("invalid system-provided package repository")

self._logger.debug(f"full list of spack package repo: {repos}")

# Delete the store/repo path, if it already exists.
# Do this so that incremental builds (though not officially supported) won't break if a repo is updated.
repo_dst = store_path / "repo"
self._logger.debug(f"creating the stack spack prepo in {repo_dst}")
if repo_dst.exists():
self._logger.debug(f"{repo_dst} exists ... deleting")
shutil.rmtree(repo_dst)

install(repo_src, repo_dst)
# Iterate over the source repositories copying their contents to the consolidated repo in the uenv.
# Do overwrite packages that have been copied from an earlier source repo, enforcing a descending
# order of precidence.
if len(repos) > 0:
pkg_dst = repo_dst / "packages"
pkg_dst.mkdir(mode=0o755, parents=True)
self._logger.debug(f"created the repo packages path {pkg_dst}")
for repo_src in repos:
self._logger.debug(f"installing repo {repo_src}")
packages_path = repo_src / "packages"
for pkg_path in packages_path.iterdir():
dst = pkg_dst / pkg_path.name
if pkg_path.is_dir() and not dst.exists():
self._logger.debug(f" installing package {pkg_path} to {pkg_dst}")
install(pkg_path, dst)
elif dst.exists():
self._logger.debug(f" NOT installing package {pkg_path}")
# create the repo.yaml file that configures the repo.
with (repo_dst / "repo.yaml").open("w") as f:
f.write(
"""\
repo:
namespace: alps
"""
)

# Step 2: Create a repos.yaml file in build_path/config
# Create a repos.yaml file in build_path/config
repos_yaml_template = jinja_env.get_template("repos.yaml")
with (config_path / "repos.yaml").open("w") as f:
repo_path = recipe.mount / "repo"
f.write(
repos_yaml_template.render(
repo_path=repo_path.as_posix(), verbose=False
)
)
f.write(repos_yaml_template.render(repo_path=repo_path.as_posix(), verbose=False))
f.write("\n")

# Add user-defined repo to internal repo
user_repo_path = recipe.path / "repo"
if user_repo_path.exists() and user_repo_path.is_dir():
user_repo_yaml = user_repo_path / "repo.yaml"
if user_repo_yaml.exists():
self._logger.warning(f"Found 'repo.yaml' file in {user_repo_path}")
self._logger.warning(
"'repo.yaml' is ignored, packages are added to the 'alps' repo"
)

# Copy user-provided recipes into repo
user_repo_packages = user_repo_path / "packages"
for user_recipe_dir in user_repo_packages.iterdir():
if user_recipe_dir.is_dir(): # iterdir() yelds files too
install(
user_recipe_dir, repo_dst / "packages" / user_recipe_dir.name
)

# Generate the makefile and spack.yaml files that describe the compilers
compiler_files = recipe.compiler_files
compiler_path = self.path / "compilers"
Expand Down Expand Up @@ -446,19 +479,13 @@ def generate(self, recipe):
# write a json file with basic meta data
with (meta_path / "configure.json").open("w") as f:
# default serialisation is str to serialise the pathlib.PosixPath
f.write(
json.dumps(
self.configuration_meta, sort_keys=True, indent=2, default=str
)
)
f.write(json.dumps(self.configuration_meta, sort_keys=True, indent=2, default=str))
f.write("\n")

# write a json file with the environment view meta data
with (meta_path / "env.json").open("w") as f:
# default serialisation is str to serialise the pathlib.PosixPath
f.write(
json.dumps(self.environment_meta, sort_keys=True, indent=2, default=str)
)
f.write(json.dumps(self.environment_meta, sort_keys=True, indent=2, default=str))
f.write("\n")

# copy the recipe to a recipe subdirectory of the meta path
Expand Down
8 changes: 2 additions & 6 deletions stackinator/etc/add-compiler-links.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,7 @@ def has_prefix(path, prefix):

paths = []
for c in compilers:
local_paths = set(
[os.path.dirname(v) for k, v in c["paths"].items() if v is not None]
)
local_paths = set([os.path.dirname(v) for k, v in c["paths"].items() if v is not None])
paths += local_paths
print(f'adding compiler {c["spec"]} -> {[p for p in local_paths]}')

Expand All @@ -96,9 +94,7 @@ def has_prefix(path, prefix):

# parse PATH to remove references to the build directory
if export["variable"] == "PATH":
paths = [
p for p in export["paths"] if not has_prefix(p, args.build_path)
]
paths = [p for p in export["paths"] if not has_prefix(p, args.build_path)]
lines.append(f"export PATH={':'.join(paths)};\n")

# drop the SPACK_ENV variable
Expand Down
Loading

0 comments on commit 62f0fa6

Please sign in to comment.