From 5754c087ca73b1d9ba93e6b2ee7a090881fb945c Mon Sep 17 00:00:00 2001 From: Jan Weldert <32642322+JanWeldert@users.noreply.github.com> Date: Tue, 29 Oct 2024 15:12:18 +0100 Subject: [PATCH 01/20] First step of cleaning up INSTALL.md The installation file is very large and a bit confusing. We need to clean up a little. I started by clarifying the difference between the two presented methods and expanding the anaconda version. --- INSTALL.md | 132 ++++++++++++++++++++++++----------------------------- 1 file changed, 60 insertions(+), 72 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 90454b27e..e86563dbf 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,76 +1,46 @@ ## Installation Guide -### **Instruction to install PISA on your local machine** +### **Instruction to install PISA using Anaconda or Miniconda** 1. Install the essential dependencies - - 1. Anaconda (https://www.anaconda.com/) - ```bash - wget -nc https://repo.anaconda.com/archive/Anaconda3-2023.07-2-Linux-x86_64.sh - - bash Anaconda3-2023.07-2-Linux-x86_64.sh - ``` - 2. pip (Python package installer) - ```bash - conda install pip - ``` - 3. git - ```bash - sudo apt install git - ``` - - Other required libraries (listed below) will be installed automatically during the setup (listed in `setup.py`). - - - fast-histogram >= 0.10 - - h5py - - iminuit >= 2 - - line_profiler - - llvmlite - - matplotlib >= 3.0 - - nlopt - - numba >= 0.53 - - numpy >=1.17, < 1.23 - - pandas - - pint <=0.19 - - py-cpuinfo - - pyarrow - - scikit-learn <= 1.1.2 - - scipy >= 1.6 - - simplejson == 3.18.4 - - sympy - - tables - - tabulate - - tqdm - - uncertainties - - kiwisolver >= 1.3.1 - - cycler >= 0.10 - - packaging >= 20.0 - - fonttools >= 4.22.0 - - python-dateutil >= 2.7 - - pillow >= 8 - - contourpy >= 1.0.1 - - pyparsing >= 2.3.1 - - threadpoolctl >= 2.0.0 - - joblib >= 1.0.0 - - numexpr - - pytz >= 2020.1 - - tzdata >= 2022.1 - - mpmath >= 0.19 - - blosc2 >= 2.3.0 - - future - - ndindex >= 1.4 - - msgpack - - six >= 1.5 -2. Create conda environment/ virtual environment for the installation of PISA. Assume that the name of the environment is `pisa_env` you can choose your preferred version of python >= 3.10 + + Choose either Anaconda (~4.4 GB) or Miniconda (~480 MB) + + Anaconda (https://docs.anaconda.com/anaconda/) + ```bash + mkdir -p PATH_TO_ANACONDA/anaconda3 + wget https://repo.anaconda.com/archive/Anaconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/anaconda3/anaconda.sh + bash PATH_TO_ANACONDA/anaconda3/anaconda.sh -b -u -p PATH_TO_ANACONDA/anaconda3 + rm PATH_TO_ANACONDA/anaconda3/anaconda.sh + ``` + You can view a list of available installer versions at https://repo.anaconda.com/archive/. + + Miniconda (https://docs.anaconda.com/miniconda/) + ```bash + mkdir -p PATH_TO_ANACONDA/miniconda3 + wget https://repo.anaconda.com/miniconda/Miniconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/miniconda3/miniconda.sh + bash PATH_TO_ANACONDA/miniconda3/miniconda.sh -b -u -p PATH_TO_ANACONDA/miniconda3 + rm PATH_TO_ANACONDA/miniconda3/miniconda.sh + ``` + You can view a list of available installer versions at https://repo.anaconda.com/miniconda/. + + **If you install on the cobalts, rather use /data/user/YOURNAME/ than $HOME/ for PATH_TO_ANACONDA** + + Note that you will also need pip and git. They are already installed on the cobalts, ask your local system administrator if they are not in your local machine. + + Other required libraries will be installed automatically during the setup (listed in https://github.com/icecube/pisa/blob/master/setup.py). + +2. Create an environment for the installation of PISA (after activating anaconda). You can choose your preferred version of python >= 3.10. E.g. use: ```bash -conda create -n pisa_env python=3.10 +source PATH_TO_ANACONDA/miniconda3/bin/activate +conda create -n NAME_OF_YOUR_PISA_ENV python=3.10 ``` -3. Activate the newly created environment (You can see the name of your environment in the bash shell after the activation) +3. Activate the newly created environment ```bash -conda activate pisa_env +conda activate NAME_OF_YOUR_PISA_ENV ``` 4. Clone the PISA repository from github (https://github.com/icecube/pisa.git) @@ -82,22 +52,40 @@ git clone https://github.com/icecube/pisa.git 5. Install PISA with the following command ```bash -pip install -e pisa +pip install -e $PISA[develop] -vvv ``` -Arguments: - -* `pisa ` directory of source code -* `-e` (editable) Installs from source located at `$PISA` (Installed directory - pisa) and allows for changes to the source code within to be immediately propagated to your Python installation. -* `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv` +Explanation: +* First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). +* `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. +Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). * `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. +* `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. +* If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. + +#### Notes: +* You can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. + + +#### Reinstall PISA + +Sometimes a change within PISA requires re-installation (particularly if a compiled module changes, the below forces re-compilation). + +```bash +pip install -e $PISA[develop] --force-reinstall -vvv +``` + +Note that if files change names or locations, though, the above can still not be enough. +In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). +them. ### **Test your installation by running a simple script:** -- Activate your python environment (`pisa_env`) +- Activate your python environment ```bash - conda activate pisa_env + source PATH_TO_ANACONDA/miniconda3/bin/activate + conda activate NAME_OF_YOUR_PISA_ENV ``` - Start python interpretator in terminal @@ -137,7 +125,7 @@ Arguments: outputs['nutau_cc'].plot(ax=axes[2], cmap='RdYlBu_r', vmin=0, vmax=1); ``` -### Instructions to install PISA on Madison working group servers cobalts (current guide from August 2023): +### Instructions to install PISA using cvmfs and virtual environment: Assuming you already are in the directory where you want to store fridge/pisa source files and the python virtualenv and build pisa. You also need access to github through your account. From 23f68119196a8f9d2e30f32d1c726125b744edce Mon Sep 17 00:00:00 2001 From: Jan Weldert <32642322+JanWeldert@users.noreply.github.com> Date: Wed, 30 Oct 2024 11:58:41 +0100 Subject: [PATCH 02/20] More changes to INSTALL.md --- INSTALL.md | 227 ++++++++++------------------------------------------- 1 file changed, 43 insertions(+), 184 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index e86563dbf..102536819 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,12 +1,12 @@ -## Installation Guide +# Installation Guide -### **Instruction to install PISA using Anaconda or Miniconda** +## Instruction to install PISA using Anaconda or Miniconda 1. Install the essential dependencies Choose either Anaconda (~4.4 GB) or Miniconda (~480 MB) - Anaconda (https://docs.anaconda.com/anaconda/) + Anaconda (https://docs.anaconda.com/anaconda/), e.g. ```bash mkdir -p PATH_TO_ANACONDA/anaconda3 wget https://repo.anaconda.com/archive/Anaconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/anaconda3/anaconda.sh @@ -14,8 +14,8 @@ rm PATH_TO_ANACONDA/anaconda3/anaconda.sh ``` You can view a list of available installer versions at https://repo.anaconda.com/archive/. - - Miniconda (https://docs.anaconda.com/miniconda/) + + Miniconda (https://docs.anaconda.com/miniconda/), e.g. ```bash mkdir -p PATH_TO_ANACONDA/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/miniconda3/miniconda.sh @@ -23,7 +23,7 @@ rm PATH_TO_ANACONDA/miniconda3/miniconda.sh ``` You can view a list of available installer versions at https://repo.anaconda.com/miniconda/. - + **If you install on the cobalts, rather use /data/user/YOURNAME/ than $HOME/ for PATH_TO_ANACONDA** Note that you will also need pip and git. They are already installed on the cobalts, ask your local system administrator if they are not in your local machine. @@ -43,10 +43,19 @@ conda create -n NAME_OF_YOUR_PISA_ENV python=3.10 conda activate NAME_OF_YOUR_PISA_ENV ``` -4. Clone the PISA repository from github (https://github.com/icecube/pisa.git) +4. Clone the PISA repository from github (https://github.com/icecube/pisa.git). You can also create your own fork first. + +In your terminal, define a directory for PISA sourcecode to live in. +```bash +export PISA="PATH_WHERE_PISA_SHOULD_LIVE/pisa +``` +Add this line to your ~/.bashrc file so you can refer to the `$PISA` variable without doing this everytime. +PATH_WHERE_PISA_SHOULD_LIVE could for example be the same as PATH_TO_ANACONDA. +The clone the source code ```bash -git clone https://github.com/icecube/pisa.git +mkdir -p $PISA +git clone https://github.com/icecube/pisa.git $PISA ``` 5. Install PISA with the following command @@ -56,18 +65,18 @@ pip install -e $PISA[develop] -vvv ``` Explanation: -* First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). -* `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. -Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). -* `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. -* `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. -* If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. + * First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). + * `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. + Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). + * `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. + * `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. + * If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. #### Notes: -* You can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. + * You can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. -#### Reinstall PISA +### Reinstall PISA Sometimes a change within PISA requires re-installation (particularly if a compiled module changes, the below forces re-compilation). @@ -79,53 +88,34 @@ Note that if files change names or locations, though, the above can still not be In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). them. -### **Test your installation by running a simple script:** -- Activate your python environment +### Test PISA + + First activate your python environment (if not already active) ```bash source PATH_TO_ANACONDA/miniconda3/bin/activate conda activate NAME_OF_YOUR_PISA_ENV ``` -- Start python interpretator in terminal - ```bash - python - ``` -- Import `Pipline` modelue and `matplotlib.pyplot` +#### Unit Tests - ```python - from pisa.core import Pipeline - import matplotlib.pyplot as plt - ``` - ``` - << PISA is running in single precision (FP32) mode; numba is running on CPU (single core) >> - ``` -- Load an example pipeline +Throughout the codebase there are `test_*.py` files and `test_*` functions within various `*.py` files that represent unit tests. +Unit tests are designed to ensure that the basic mechanisms of objects' functionality work. - ```python - template_maker = Pipeline("settings/pipeline/osc_example.cfg") - ``` -- Run the pipeline +These are all run, plus additional tests (takes about 15-20 minutes on a laptop) with the command +```bash +$PISA/pisa_tests/test_command_lines.sh +``` - ```python - template_maker.run() - ``` -- Get the oscillation probabilities as a map +#### Running a simple script - ```python - outputs = template_maker.data.get_mapset('prob_mu') - ``` -- Plot the oscillogram +```bash +$PISA/pisa/core/pipeline.py --pipeline settings/pipeline/example.cfg --outdir /tmp/pipeline_output --intermediate --pdf -v +``` - ```python - fig, axes = plt.subplots(figsize=(18, 5), ncols=3) - outputs['nue_cc'].plot(ax=axes[0], cmap='RdYlBu_r', vmin=0, vmax=1); - outputs['numu_cc'].plot(ax=axes[1], cmap='RdYlBu_r', vmin=0, vmax=1); - outputs['nutau_cc'].plot(ax=axes[2], cmap='RdYlBu_r', vmin=0, vmax=1); - ``` -### Instructions to install PISA using cvmfs and virtual environment: +## Instructions to install PISA using cvmfs and virtual environment Assuming you already are in the directory where you want to store fridge/pisa source files and the python virtualenv and build pisa. You also need access to github through your account. @@ -204,46 +194,7 @@ export HDF5_USE_FILE_LOCKING='FALSE' `pip install PACKAGE` (for example `jupyter`) - -### Quickstart (old guide) - -_Note that terminal commands below are intended for the bash shell. You'll have to translate if you use a different shell._ - -1. Obtain a github user ID if you don’t have one already
- https://github.com - * Sign up for Github education pack for many features for free, too
- https://education.github.com/pack -1. Fork PISA on github so you have your own copy to work from
- https://github.com/IceCubeOpenSource/pisa#fork-destination-box -1. _(optional)_ Set up shared-key ssh access to github so you don’t have to enter passwords
- https://help.github.com/articles/connecting-to-github-with-ssh -1. In your terminal, define a directory for PISA sourcecode to live in. For example:
- `export PISA="~/src/pisa"` - * Add this line to your ~/.bashrc file so you can refer to the `$PISA` variable without doing this everytime. -1. Create the directory
- `mkdir -p $PISA` -1. Clone the PISA repo to your local computer (at command line) - * If you set up shared-key auth above
- `git clone git@github.com:/pisa.git $PISA` - * Otherwise
- `git clone https://github.com//pisa.git $PISA` -1. Install the ***Python 3.7 / 64 bit*** Anaconda or Miniconda python distribution for either Mac or Linux (as your user, _not_ as root), if you don’t have it already - * Anaconda (full-featured Python distribution, ~500 MB)
- https://www.anaconda.com/download - * Miniconda (just the essentials, ~40 MB)
- https://conda.io/miniconda.html -1. Create a new conda environment, ideally with a python version compatible with the python requirements below (cf. conda's getting started guide). -1. Active your new conda environment. -1. Install PISA including optional packages and development tools (`develop`), if desired
- `pip install -e $PISA[develop] -vvv` -1. Run a quick test: generate templates in the staged mode
-`$PISA/pisa/core/pipeline.py --pipeline settings/pipeline/example.cfg --outdir /tmp/pipeline_output --intermediate --pdf -v` - -See [github.com/IceCubeOpenSource/pisa/wiki/installation_specific_examples](https://github.com/IceCubeOpenSource/pisa/wiki/installation_specific_examples) for users' recipes for installing PISA under various circumstances. -Please add notes there and/or add your own recipe if your encounter a unique installation issue. -Also, for instructions on running PISA on Open Science Grid (OSG) nodes, see [github.com/IceCubeOpenSource/pisa/wiki/Running-PISA-on-GRID-nodes-with-access-to-CVMFS](https://github.com/jllanfranchi/pisa/wiki/Running-PISA-on-GRID-nodes-with-access-to-CVMFS) - -The following sections delve into deeper detail on installing PISA. +## Additional information ### Python Distributions @@ -253,8 +204,7 @@ Although the selection of maintained packages is smaller than if you use the `pi The other advantage to these distributions is that they easily install without system administrative privileges (and install in a user directory) and come with the non-Python binary libraries upon which many Python modules rely, making them ideal for setup on e.g. clusters. -* **Note**: Make sure that your `PATH` variable points to e.g. `/bin` and *not* your system Python directory. To check this, type: `echo $PATH`; to udpate it, add `export PATH=/bin:$PATH` to your .bashrc file. -* Python 3.7.x can also be found from the Python website [python.org/downloads](https://www.python.org/downloads/) or pre-packaged for almost any OS. +**Note**: Make sure that your `PATH` variable points to e.g. `/bin` and *not* your system Python directory. To check this, type: `echo $PATH`; to udpate it, add `export PATH=/bin:$PATH` to your .bashrc file. ### Required Dependencies @@ -275,32 +225,7 @@ Also note that Python, HDF5, and pip support come pre-packaged or as `conda`-ins `sudo apt install libhdf5-10` * [llvm](http://llvm.org) Compiler needed by Numba. This is automatically installed in Anaconda alongside `numba`. * Anaconda
- `conda install numba=0.45.1` - -Required Python modules that are installed automatically when you use the `pip` command detailed later: -* [decorator](https://pypi.python.org/pypi/decorator) -* [h5py](http://www.h5py.org) -* [iminuit](): Python interface to the MINUIT2 C++ package, used for proper covariance matrices during minimization -* [kde](https://github.com/IceCubeOpenSource/kde) - * You can install the `kde` module manually if it fails to install automatically: - * Including CUDA support:
- `pip install git+https://github.com/icecubeopensource/kde.git#egg=kde[cuda]` - * Without CUDA support:
- `pip install git+https://github.com/icecubeopensource/kde.git#egg=kde` -* [line_profiler](https://pypi.python.org/pypi/line_profiler): detailed profiling output
- * if automatic pip installation of line_profiler fails, you may want to try `conda install line_profiler` if you are using anaconda -* [matplotlib>=3.0](http://matplotlib.org) >= 3.0 required -* [numba==0.53.1](http://numba.pydata.org) Just-in-time compilation of decorated Python functions to native machine code via LLVM. This package is required to use PISA pi; also in cake it can accelerate certain routines significantly. If not using Anaconda to install, you must have LLVM installed already on your system (see above). -* [numpy](http://www.numpy.org) version >= 1.17 required -* [pint>=0.8.1](https://pint.readthedocs.org) >= 0.8.1 required - * if automatic pip installation of pint fails, you may want to try `conda install pint` if you are using anaconda -* [scipy](http://www.scipy.org) version >= 0.17 required -* [setuptools](https://setuptools.readthedocs.io) version >= 18.5 required -* [simplejson](https://github.com/simplejson/simplejson) version >= 3.2.0 required -* [tables](http://www.pytables.org) -* [uncertainties](https://pythonhosted.org/uncertainties) -* [py-cpuinfo](https://pypi.org/project/py-cpuinfo) retrieve detailed CPU and architecture info for documenting in tests / obtaining results -* [sympy](https://www.sympy.org/en/) Used for testing `nsi_params.py` + `conda install numba` ### Optional Dependencies @@ -371,35 +296,6 @@ Note that it is also discouraged, but you _can_ install PISA as an unprivileged This is not quite as clean as a virtual environment, and the issue with coflicting package dependency versions remains. -### Install PISA - -```bash -pip install -e $PISA[develop] -vvv -``` -Explanation: -* First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). -* `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. -Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). -* `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. -* `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. -* If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. - -__Notes:__ -* You can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. - - -### Reinstall PISA - -Sometimes a change within PISA requires re-installation (particularly if a compiled module changes, the below forces re-compilation). - -```bash -pip install -e $PISA[develop] --force-reinstall -vvv -``` - -Note that if files change names or locations, though, the above can still not be enough. -In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). - - ### Compile the documentation To compile a new version of the documentation to html (pdf and other formats are available by replacing `html` with `pdf`, etc.): @@ -411,40 +307,3 @@ In case code structure has changed, rebuild the apidoc by executing ```bash cd $PISA/docs && make html ``` - - -### Test PISA - -#### Unit Tests - -Throughout the codebase there are `test_*.py` files and `test_*` functions within various `*.py` files that represent unit tests. -Unit tests are designed to ensure that the basic mechanisms of objects' functionality work. - -These are all run, plus additional tests (takes about 15-20 minutes on a laptop) with the command -```bash -$PISA/pisa_tests/test_command_lines.sh -``` - -### Run a basic analysis - -To make sure that an analysis can be run, try running an Asimov analysis of neutrino mass ordering (NMO) with the following (this takes about one minute on a laptop; note, though, that the result is not terribly accurate due to the use of coarse binning and low Monte Carlo statistics): -```bash -export PISA_FTYPE=fp64 -$PISA/pisa/analysis/hypo_testing.py --logdir /tmp/nmo_test analysis \ - --h0-pipeline settings/pipeline/example.cfg \ - --h0-param-selections="ih" \ - --h1-param-selections="nh" \ - --data-param-selections="nh" \ - --data-is-mc \ - --min-method slsqp \ - --metric=chi2 \ - --pprint -v -``` - -The above command sets the null hypothesis (h0) to be the inverted hierarchy (ih) and the hypothesis to be tested (h1) to the normal hierarchy (nh). -Meanwhile, the Asimov dataset is derived from the normal hierarchy. - -The significance for distinguishing NH from IH in this case (with the crude but fast settings specified) is shown by typing the follwoing command (which should output something close to 4.3): -```bash -hypo_testing_postprocess.py --asimov --detector "pingu_v39" --dir /tmp/nmo_test/hypo* -``` From d77f3a4521cbb4490dee14fbb4a5a5cb2464ee09 Mon Sep 17 00:00:00 2001 From: Jan Weldert <32642322+JanWeldert@users.noreply.github.com> Date: Thu, 31 Oct 2024 11:21:59 +0100 Subject: [PATCH 03/20] Update INSTALL.md --- INSTALL.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 102536819..6cc96e026 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -2,11 +2,13 @@ ## Instruction to install PISA using Anaconda or Miniconda +The following commands are intended for execution in a terminal. If you encounter any problems with the installation, please file an issue and/or ask on slack. + 1. Install the essential dependencies - Choose either Anaconda (~4.4 GB) or Miniconda (~480 MB) + Choose either Anaconda (~4.4 GB) OR Miniconda (~480 MB) - Anaconda (https://docs.anaconda.com/anaconda/), e.g. + **Anaconda** (https://docs.anaconda.com/anaconda/), e.g. ```bash mkdir -p PATH_TO_ANACONDA/anaconda3 wget https://repo.anaconda.com/archive/Anaconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/anaconda3/anaconda.sh @@ -15,7 +17,7 @@ ``` You can view a list of available installer versions at https://repo.anaconda.com/archive/. - Miniconda (https://docs.anaconda.com/miniconda/), e.g. + **Miniconda** (https://docs.anaconda.com/miniconda/), e.g. ```bash mkdir -p PATH_TO_ANACONDA/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/miniconda3/miniconda.sh @@ -26,9 +28,9 @@ **If you install on the cobalts, rather use /data/user/YOURNAME/ than $HOME/ for PATH_TO_ANACONDA** - Note that you will also need pip and git. They are already installed on the cobalts, ask your local system administrator if they are not in your local machine. + **Note** that you will also need git. It is already installed on the cobalts, ask your local system administrator if it is not on your local machine. The other non-python requirements are listed [here](https://github.com/icecube/pisa/blob/master/INSTALL.md#required-dependencies) and should already come with the conda environment. - Other required libraries will be installed automatically during the setup (listed in https://github.com/icecube/pisa/blob/master/setup.py). + Other required libraries will be installed automatically during the setup (listed in https://github.com/icecube/pisa/blob/master/setup.py). If something is missing, you can install it via pip afterwards. 2. Create an environment for the installation of PISA (after activating anaconda). You can choose your preferred version of python >= 3.10. E.g. use: @@ -43,13 +45,13 @@ conda create -n NAME_OF_YOUR_PISA_ENV python=3.10 conda activate NAME_OF_YOUR_PISA_ENV ``` -4. Clone the PISA repository from github (https://github.com/icecube/pisa.git). You can also create your own fork first. +4. Clone the PISA repository from github (https://github.com/icecube/pisa.git). You can also create your own fork first. For more information on how to obtain the pisa source code see [obtain-pisa-sourcecode](https://github.com/icecube/pisa/blob/master/INSTALL.md#obtain-pisa-sourcecode) -In your terminal, define a directory for PISA sourcecode to live in. +Define a directory for PISA sourcecode to live in. ```bash export PISA="PATH_WHERE_PISA_SHOULD_LIVE/pisa ``` -Add this line to your ~/.bashrc file so you can refer to the `$PISA` variable without doing this everytime. +Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this everytime. PATH_WHERE_PISA_SHOULD_LIVE could for example be the same as PATH_TO_ANACONDA. The clone the source code @@ -64,7 +66,7 @@ git clone https://github.com/icecube/pisa.git $PISA pip install -e $PISA[develop] -vvv ``` -Explanation: +**Explanation:** * First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). * `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). @@ -72,8 +74,7 @@ Explanation: * `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. * If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. -#### Notes: - * You can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. +**Note** that you can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. ### Reinstall PISA @@ -258,7 +259,6 @@ Optional dependencies. Some of these must be installed manually prior to install #### Develop PISA: Fork then clone If you wish to modify PISA and contribute your code changes back to the PISA project (*highly recommended!*), fork `IceCubeOpenSource/pisa` from Github. -*(How to work with the `cake` branch of PISA will be detailed below.)* Forking creates your own version of PISA within your Github account. You can freely create your own *branch*, modify the code, and then *add* and *commit* changes to that branch within your fork of PISA. From 0b14e82818ca7bbba42fcc0d8c19b5ca946877e4 Mon Sep 17 00:00:00 2001 From: Jan Weldert <32642322+JanWeldert@users.noreply.github.com> Date: Tue, 5 Nov 2024 13:49:22 +0100 Subject: [PATCH 04/20] Update test script and add some clarifications --- INSTALL.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 6cc96e026..d0cc3d5c1 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,8 +1,9 @@ # Installation Guide + ## Instruction to install PISA using Anaconda or Miniconda -The following commands are intended for execution in a terminal. If you encounter any problems with the installation, please file an issue and/or ask on slack. +The following commands are intended for execution in a terminal. If you encounter any problems with the installation, please create an issue and/or ask on the IceCube slack (if possible). 1. Install the essential dependencies @@ -32,7 +33,7 @@ The following commands are intended for execution in a terminal. If you encounte Other required libraries will be installed automatically during the setup (listed in https://github.com/icecube/pisa/blob/master/setup.py). If something is missing, you can install it via pip afterwards. -2. Create an environment for the installation of PISA (after activating anaconda). You can choose your preferred version of python >= 3.10. E.g. use: +2. Create an environment for the installation of PISA (after activating anaconda). You can choose your preferred version of python >= 3.10. E.g. for miniconda with python 3.10 use: ```bash source PATH_TO_ANACONDA/miniconda3/bin/activate @@ -85,20 +86,28 @@ Sometimes a change within PISA requires re-installation (particularly if a compi pip install -e $PISA[develop] --force-reinstall -vvv ``` -Note that if files change names or locations, though, the above can still not be enough. +**Note** that if files change names or locations, though, the above can still not be enough. In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). them. ### Test PISA - First activate your python environment (if not already active) + First activate your python environment (if not already active). Assuming you used Miniconda, you can do that with: ```bash source PATH_TO_ANACONDA/miniconda3/bin/activate conda activate NAME_OF_YOUR_PISA_ENV ``` +#### Running a simple script + +Running the following command should create a folder in `/tmp` containing a pdf with output maps for different neutrino types and interactions. + +```bash +$PISA/pisa/core/distribution_maker.py --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir /tmp/pipeline_output --pdf +``` + #### Unit Tests Throughout the codebase there are `test_*.py` files and `test_*` functions within various `*.py` files that represent unit tests. @@ -109,12 +118,6 @@ These are all run, plus additional tests (takes about 15-20 minutes on a laptop) $PISA/pisa_tests/test_command_lines.sh ``` -#### Running a simple script - -```bash -$PISA/pisa/core/pipeline.py --pipeline settings/pipeline/example.cfg --outdir /tmp/pipeline_output --intermediate --pdf -v -``` - ## Instructions to install PISA using cvmfs and virtual environment From 9c7e2f8db40d0e0807d51dc243a6f37076dca521 Mon Sep 17 00:00:00 2001 From: Jan Weldert <32642322+JanWeldert@users.noreply.github.com> Date: Thu, 7 Nov 2024 14:06:10 +0100 Subject: [PATCH 05/20] Typo --- INSTALL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/INSTALL.md b/INSTALL.md index d0cc3d5c1..140a1a9bc 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -55,7 +55,7 @@ export PISA="PATH_WHERE_PISA_SHOULD_LIVE/pisa Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this everytime. PATH_WHERE_PISA_SHOULD_LIVE could for example be the same as PATH_TO_ANACONDA. -The clone the source code +Then clone the source code ```bash mkdir -p $PISA git clone https://github.com/icecube/pisa.git $PISA From d04604100703d9bf8c96d18863add362887e03b0 Mon Sep 17 00:00:00 2001 From: Jan Weldert <32642322+JanWeldert@users.noreply.github.com> Date: Mon, 11 Nov 2024 15:30:43 +0100 Subject: [PATCH 06/20] Last fine tuning --- INSTALL.md | 65 +++++++++++++++++++++++++----------------------------- 1 file changed, 30 insertions(+), 35 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 140a1a9bc..7496eb5fa 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -5,27 +5,27 @@ The following commands are intended for execution in a terminal. If you encounter any problems with the installation, please create an issue and/or ask on the IceCube slack (if possible). -1. Install the essential dependencies +**1.** Install the essential dependencies Choose either Anaconda (~4.4 GB) OR Miniconda (~480 MB) **Anaconda** (https://docs.anaconda.com/anaconda/), e.g. ```bash mkdir -p PATH_TO_ANACONDA/anaconda3 - wget https://repo.anaconda.com/archive/Anaconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/anaconda3/anaconda.sh + wget https://repo.anaconda.com/archive/Anaconda3-INSTALLER_VERSION-Linux-x86_64.sh -O PATH_TO_ANACONDA/anaconda3/anaconda.sh bash PATH_TO_ANACONDA/anaconda3/anaconda.sh -b -u -p PATH_TO_ANACONDA/anaconda3 rm PATH_TO_ANACONDA/anaconda3/anaconda.sh ``` - You can view a list of available installer versions at https://repo.anaconda.com/archive/. + You can find a list of available INSTALLER_VERSIONs at https://repo.anaconda.com/archive/. **Miniconda** (https://docs.anaconda.com/miniconda/), e.g. ```bash mkdir -p PATH_TO_ANACONDA/miniconda3 - wget https://repo.anaconda.com/miniconda/Miniconda3--Linux-x86_64.sh -O PATH_TO_ANACONDA/miniconda3/miniconda.sh + wget https://repo.anaconda.com/miniconda/Miniconda3-INSTALLER_VERSION-Linux-x86_64.sh -O PATH_TO_ANACONDA/miniconda3/miniconda.sh bash PATH_TO_ANACONDA/miniconda3/miniconda.sh -b -u -p PATH_TO_ANACONDA/miniconda3 rm PATH_TO_ANACONDA/miniconda3/miniconda.sh ``` - You can view a list of available installer versions at https://repo.anaconda.com/miniconda/. + You can find a list of available INSTALLER_VERSIONs at https://repo.anaconda.com/miniconda/. **If you install on the cobalts, rather use /data/user/YOURNAME/ than $HOME/ for PATH_TO_ANACONDA** @@ -33,39 +33,39 @@ The following commands are intended for execution in a terminal. If you encounte Other required libraries will be installed automatically during the setup (listed in https://github.com/icecube/pisa/blob/master/setup.py). If something is missing, you can install it via pip afterwards. -2. Create an environment for the installation of PISA (after activating anaconda). You can choose your preferred version of python >= 3.10. E.g. for miniconda with python 3.10 use: +**2.** Create an environment for the installation of PISA (after activating anaconda). E.g. for miniconda with python 3.10 use: -```bash -source PATH_TO_ANACONDA/miniconda3/bin/activate -conda create -n NAME_OF_YOUR_PISA_ENV python=3.10 -``` + ```bash + source PATH_TO_ANACONDA/miniconda3/bin/activate + conda create -n NAME_OF_YOUR_PISA_ENV python=3.10 + ``` -3. Activate the newly created environment +**3.** Activate the newly created environment -```bash -conda activate NAME_OF_YOUR_PISA_ENV -``` + ```bash + conda activate NAME_OF_YOUR_PISA_ENV + ``` -4. Clone the PISA repository from github (https://github.com/icecube/pisa.git). You can also create your own fork first. For more information on how to obtain the pisa source code see [obtain-pisa-sourcecode](https://github.com/icecube/pisa/blob/master/INSTALL.md#obtain-pisa-sourcecode) +**4.** Clone the PISA repository from github (https://github.com/icecube/pisa.git). You can also create your own fork first. For more information on how to obtain the pisa source code see [obtain-pisa-sourcecode](https://github.com/icecube/pisa/blob/master/INSTALL.md#obtain-pisa-sourcecode) -Define a directory for PISA sourcecode to live in. -```bash -export PISA="PATH_WHERE_PISA_SHOULD_LIVE/pisa -``` -Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this everytime. -PATH_WHERE_PISA_SHOULD_LIVE could for example be the same as PATH_TO_ANACONDA. + Define a directory for PISA sourcecode to live in. + ```bash + export PISA="PATH_WHERE_PISA_SHOULD_LIVE/pisa" + ``` + Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this everytime. + PATH_WHERE_PISA_SHOULD_LIVE could for example be the same as PATH_TO_ANACONDA. -Then clone the source code -```bash -mkdir -p $PISA -git clone https://github.com/icecube/pisa.git $PISA -``` + Then clone the source code + ```bash + mkdir -p $PISA + git clone https://github.com/icecube/pisa.git $PISA + ``` -5. Install PISA with the following command +**5.** Install PISA with the following command -```bash -pip install -e $PISA[develop] -vvv -``` + ```bash + pip install -e $PISA[develop] -vvv + ``` **Explanation:** * First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). @@ -113,11 +113,6 @@ $PISA/pisa/core/distribution_maker.py --pipeline settings/pipeline/IceCube_3y_ne Throughout the codebase there are `test_*.py` files and `test_*` functions within various `*.py` files that represent unit tests. Unit tests are designed to ensure that the basic mechanisms of objects' functionality work. -These are all run, plus additional tests (takes about 15-20 minutes on a laptop) with the command -```bash -$PISA/pisa_tests/test_command_lines.sh -``` - ## Instructions to install PISA using cvmfs and virtual environment From 1021f6264b99c078059e4fe0eccb42426a5d45de Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Thu, 14 Nov 2024 18:00:52 +0100 Subject: [PATCH 07/20] quickstart guide with miniforge to the top for now --- INSTALL.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/INSTALL.md b/INSTALL.md index 7496eb5fa..7c777d1d1 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,5 +1,45 @@ # Installation Guide +## Quickstart + +_Note that terminal commands below are intended for the bash shell. You'll have to translate if you use a different shell._ + +1. _(optional)_ Obtain a github user ID if you don’t have one already. (Otherwise, you will only have read access.)
+ https://github.com + * Sign up for Github education pack for many features for free, too
+ https://education.github.com/pack +1. _(optional)_ Fork PISA on github so you have your own copy to work from
+ https://github.com/icecube/pisa/fork +1. _(optional)_ Set up passwordless ssh access to github
+ https://help.github.com/articles/connecting-to-github-with-ssh +1. In your terminal, define a directory for PISA source code to live in. For example:
+ `export PISA="~/src/pisa"` + * Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this every time. +1. Create the directory
+ `mkdir -p $PISA` +1. Clone the PISA repository to your local computer (on the command line) + * If you forked PISA above
+ * If you set up ssh authentication above
+ `git clone git@github.com:/pisa.git $PISA` + * Otherwise
+ `git clone https://github.com//pisa.git $PISA` + * Otherwise
+ * If you set up ssh authentication above
+ `git clone git@github.com:icecube/pisa.git $PISA` + * Otherwise
+ `git clone https://github.com/icecube/pisa.git $PISA` +1. Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root), if you don’t have it already + https://github.com/conda-forge/miniforge/releases/latest +1. Create a new conda environment, with a python version compatible with the python requirements below. +1. Active your new conda environment. +1. Install PISA including optional packages and development tools (`develop`), if desired
+ `pip install -e $PISA[develop] -vvv` +1. Run a quick test
+ ```bash + pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir /tmp/pipeline_output --pdf + ``` + This command should have created the folder `/tmp/pipeline_output` containing a pdf with output maps for different neutrino types and interactions. + ## Instruction to install PISA using Anaconda or Miniconda From 5966494ca6291ffd28cb971f3d18204e3258d5d6 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Thu, 14 Nov 2024 19:11:36 +0100 Subject: [PATCH 08/20] Drop some baggage that potentially still needs overhaul --- INSTALL.md | 312 +++++------------------------------------------------ 1 file changed, 28 insertions(+), 284 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 7c777d1d1..6b484ab15 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,6 +1,6 @@ # Installation Guide -## Quickstart +## Quick start _Note that terminal commands below are intended for the bash shell. You'll have to translate if you use a different shell._ @@ -13,7 +13,7 @@ _Note that terminal commands below are intended for the bash shell. You'll have 1. _(optional)_ Set up passwordless ssh access to github
https://help.github.com/articles/connecting-to-github-with-ssh 1. In your terminal, define a directory for PISA source code to live in. For example:
- `export PISA="~/src/pisa"` + `export PISA=~/src/pisa` * Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this every time. 1. Create the directory
`mkdir -p $PISA` @@ -23,102 +23,49 @@ _Note that terminal commands below are intended for the bash shell. You'll have `git clone git@github.com:/pisa.git $PISA` * Otherwise
`git clone https://github.com//pisa.git $PISA` - * Otherwise
+ * If you didn't fork PISA
* If you set up ssh authentication above
`git clone git@github.com:icecube/pisa.git $PISA` * Otherwise
`git clone https://github.com/icecube/pisa.git $PISA` 1. Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root), if you don’t have it already - https://github.com/conda-forge/miniforge/releases/latest -1. Create a new conda environment, with a python version compatible with the python requirements below. -1. Active your new conda environment. -1. Install PISA including optional packages and development tools (`develop`), if desired
- `pip install -e $PISA[develop] -vvv` + https://conda-forge.org/download/ +1. _(optional)_ If you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end +1. Create and activate a new conda environment, with a python version compatible with the python requirements below. We suggest using mamba as a drop-in replacement for conda, for example
+ ```bash + mamba create -n python=3.10 + mamba activate + ``` +1. Install PISA + * either with default packages only and without development tools
+ `pip install -e $PISA -vvv` + * or, if desired, including optional packages and development tools
+ `pip install -e $PISA[develop] -vvv` 1. Run a quick test
```bash - pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir /tmp/pipeline_output --pdf + pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf ``` - This command should have created the folder `/tmp/pipeline_output` containing a pdf with output maps for different neutrino types and interactions. + This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. -## Instruction to install PISA using Anaconda or Miniconda -The following commands are intended for execution in a terminal. If you encounter any problems with the installation, please create an issue and/or ask on the IceCube slack (if possible). + -**1.** Install the essential dependencies - - Choose either Anaconda (~4.4 GB) OR Miniconda (~480 MB) - **Anaconda** (https://docs.anaconda.com/anaconda/), e.g. - ```bash - mkdir -p PATH_TO_ANACONDA/anaconda3 - wget https://repo.anaconda.com/archive/Anaconda3-INSTALLER_VERSION-Linux-x86_64.sh -O PATH_TO_ANACONDA/anaconda3/anaconda.sh - bash PATH_TO_ANACONDA/anaconda3/anaconda.sh -b -u -p PATH_TO_ANACONDA/anaconda3 - rm PATH_TO_ANACONDA/anaconda3/anaconda.sh - ``` - You can find a list of available INSTALLER_VERSIONs at https://repo.anaconda.com/archive/. - - **Miniconda** (https://docs.anaconda.com/miniconda/), e.g. - ```bash - mkdir -p PATH_TO_ANACONDA/miniconda3 - wget https://repo.anaconda.com/miniconda/Miniconda3-INSTALLER_VERSION-Linux-x86_64.sh -O PATH_TO_ANACONDA/miniconda3/miniconda.sh - bash PATH_TO_ANACONDA/miniconda3/miniconda.sh -b -u -p PATH_TO_ANACONDA/miniconda3 - rm PATH_TO_ANACONDA/miniconda3/miniconda.sh - ``` - You can find a list of available INSTALLER_VERSIONs at https://repo.anaconda.com/miniconda/. - - **If you install on the cobalts, rather use /data/user/YOURNAME/ than $HOME/ for PATH_TO_ANACONDA** - - **Note** that you will also need git. It is already installed on the cobalts, ask your local system administrator if it is not on your local machine. The other non-python requirements are listed [here](https://github.com/icecube/pisa/blob/master/INSTALL.md#required-dependencies) and should already come with the conda environment. - - Other required libraries will be installed automatically during the setup (listed in https://github.com/icecube/pisa/blob/master/setup.py). If something is missing, you can install it via pip afterwards. - -**2.** Create an environment for the installation of PISA (after activating anaconda). E.g. for miniconda with python 3.10 use: - - ```bash - source PATH_TO_ANACONDA/miniconda3/bin/activate - conda create -n NAME_OF_YOUR_PISA_ENV python=3.10 - ``` - -**3.** Activate the newly created environment - - ```bash - conda activate NAME_OF_YOUR_PISA_ENV - ``` - -**4.** Clone the PISA repository from github (https://github.com/icecube/pisa.git). You can also create your own fork first. For more information on how to obtain the pisa source code see [obtain-pisa-sourcecode](https://github.com/icecube/pisa/blob/master/INSTALL.md#obtain-pisa-sourcecode) - - Define a directory for PISA sourcecode to live in. - ```bash - export PISA="PATH_WHERE_PISA_SHOULD_LIVE/pisa" - ``` - Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this everytime. - PATH_WHERE_PISA_SHOULD_LIVE could for example be the same as PATH_TO_ANACONDA. - - Then clone the source code - ```bash - mkdir -p $PISA - git clone https://github.com/icecube/pisa.git $PISA - ``` - -**5.** Install PISA with the following command - - ```bash - pip install -e $PISA[develop] -vvv - ``` +## Additional information -**Explanation:** - * First, note that this is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). - * `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. +### Installation +* First, note that the installation is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). +* `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). - * `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. - * `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. - * If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. +* `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. +* `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. +* If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. **Note** that you can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. -### Reinstall PISA +### Re-installation Sometimes a change within PISA requires re-installation (particularly if a compiled module changes, the below forces re-compilation). @@ -128,215 +75,11 @@ pip install -e $PISA[develop] --force-reinstall -vvv **Note** that if files change names or locations, though, the above can still not be enough. In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). -them. - - -### Test PISA - - First activate your python environment (if not already active). Assuming you used Miniconda, you can do that with: - - ```bash - source PATH_TO_ANACONDA/miniconda3/bin/activate - conda activate NAME_OF_YOUR_PISA_ENV - ``` - -#### Running a simple script - -Running the following command should create a folder in `/tmp` containing a pdf with output maps for different neutrino types and interactions. - -```bash -$PISA/pisa/core/distribution_maker.py --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir /tmp/pipeline_output --pdf -``` - -#### Unit Tests - -Throughout the codebase there are `test_*.py` files and `test_*` functions within various `*.py` files that represent unit tests. -Unit tests are designed to ensure that the basic mechanisms of objects' functionality work. - - -## Instructions to install PISA using cvmfs and virtual environment - -Assuming you already are in the directory where you want to store fridge/pisa source files and the python virtualenv and build pisa. You also need access to github through your account. - - -#### Clone into the fridge and pisa (ideally your own fork): - -``` -git clone git@github.com:icecube/wg-oscillations-fridge.git ./fridge - -git clone git@github.com:USERNAME/pisa.git ./pisa -``` - - -#### Load cvmfs environment: - -``` -unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` -``` - -`which python` should now point to `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python` - -**Note:** It's not tested whether this works with a cvmfs version newer than `py3-v4.2.1`. - - -#### Create virtual environment: - -``` -python -m venv ./YOUR_PISA_PY3_VENV -``` - - -#### Start the virtual environment and install pisa from source: - -``` -source ./YOUR_PISA_PY3_VENV/bin/activate -``` - -(should now show that you are in the environment) - -``` -pip install -e pisa -``` - - -#### Modify your environment by adding lines to `./YOUR_PISA_PY3_VENV/bin/activate` - -Every time you want to use pisa now, you need to activate your virtual environment by running `source ./YOUR_PISA_PY3_VENV/bin/activate`. In order to set some useful environment variables you might want to add the following lines (or more if needed) to the end of the `./YOUR_PISA_PY3_VENV/bin/activate` script: - -``` -# PISA source -export PISA="/data/user/USERNAME/PATH_TO_PISA" - -# set some custom environment variables and load fridge -export PISA_RESOURCES="/data/user/USERNAME/PATH_TO_FRIDGE/analysis/common" -export PISA_RESOURCES=$PISA_RESOURCES:"/data/user/USERNAME/PATH_TO_FRIDGE/analysis" - -source "/data/user/USERNAME/PATH_TO_FRIDGE/setup_fridge.sh" - -# Add this project to the python path -export PYTHONPATH=$FRIDGE_DIR:$PYTHONPATH - -# Madison -export PISA_RESOURCES=/data/ana/LE:$PISA_RESOURCES -export PISA_RESOURCES=/data/ana/BSM/HNL/:$PISA_RESOURCES - -export PISA_RESOURCES=$FRIDGE_DIR:$FRIDGE_DIR/analysis:$FRIDGE_DIR/analysis/YOUR_ANALYSIS/settings:$FRIDGE_DIR/analysis/YOUR_ANALYSIS/analysis:$FRIDGE_DIR/analysis/common:$PISA_RESOURCES - -export PISA_CACHE=~/cache/ -export PISA_FTYPE=fp64 -export HDF5_USE_FILE_LOCKING='FALSE' -``` - - -#### Install any additional packages that you might want - -`pip install PACKAGE` (for example `jupyter`) - - -## Additional information - - -### Python Distributions - -Obtaining Python and Python packages, and handling interdependencies in those packages tends to be easiest if you use a Python distribution, such as [Anaconda](https://www.continuum.io/downloads) or [Canopy](https://www.enthought.com/products/canopy). -Although the selection of maintained packages is smaller than if you use the `pip` command to obtain packages from the Python Package Index (PyPi), you can still use `pip` with these distributions. - -The other advantage to these distributions is that they easily install without system administrative privileges (and install in a user directory) and come with the non-Python binary libraries upon which many Python modules rely, making them ideal for setup on e.g. clusters. - -**Note**: Make sure that your `PATH` variable points to e.g. `/bin` and *not* your system Python directory. To check this, type: `echo $PATH`; to udpate it, add `export PATH=/bin:$PATH` to your .bashrc file. - - -### Required Dependencies - -To install PISA, you'll need to have the following non-python requirements. -Note that these are not installed automatically, and you must install them yourself prior to installing PISA. -Also note that Python, HDF5, and pip support come pre-packaged or as `conda`-installable packages in the Anaconda Python distribution. -* [python](http://www.python.org) — version 3.x - * Anaconda: built in -* [pip](https://pip.pypa.io) version >= 1.8 required - * Anaconda:
- `conda install pip` -* [git](https://git-scm.com) - * In Ubuntu,
- `sudo apt install git` -* [hdf5](http://www.hdfgroup.org/HDF5) — install with `--enable-cxx` option - * In Ubuntu,
- `sudo apt install libhdf5-10` -* [llvm](http://llvm.org) Compiler needed by Numba. This is automatically installed in Anaconda alongside `numba`. - * Anaconda
- `conda install numba` - - -### Optional Dependencies - -Optional dependencies. Some of these must be installed manually prior to installing PISA, and some will be installed automatically by pip, and this seems to vary from system to system. Therefore you can first try to run the installation, and just install whatever pip says it needed, or just use apt, pip, and/or conda to install the below before running the PISA installation. - -* [LeptonWeighter](https://github.com/icecube/leptonweighter) Required for the `data.licloader_weighter` service. -* [MCEq](http://github.com/afedynitch/MCEq) Required for `flux.mceq` service. -* [daemonflux](https://github.com/mceq-project/daemonflux) Recuired for `flux.daemon_flux` service. -* [nuSQuiDS](https://github.com/arguelles/nuSQuIDS) Required for `osc.nusquids` service. -* [pandas](https://pandas.pydata.org/) Required for datarelease (csv) stages. -* [OpenMP](http://www.openmp.org) Intra-process parallelization to accelerate code on on multi-core/multi-CPU computers. - * Available from your compiler: gcc supports OpenMP 4.0 and Clang >= 3.8.0 supports OpenMP 3.1. Either version of OpenMP should work, but Clang has yet to be tested for its OpenMP support. -* [Photospline](https://github.com/icecube/photospline) Required for the `flux.airs` service. -* [Pylint](http://www.pylint.org): Static code checker and style analyzer for Python code. Note that our (more or less enforced) coding conventions are codified in the pylintrc file in PISA, which will automatically be found and used by Pylint when running on code within a PISA package.
- * Installed alongside PISA if you specify option `['develop']` to `pip` -* [recommonmark](http://recommonmark.readthedocs.io/en/latest/) Translator to allow markdown docs/docstrings to be used; plugin for Sphinx. (Required to compile PISA's documentation.) - * Installed alongside PISA if you specify option `['develop']` to `pip` -* [ROOT >= 6.12.04 with PyROOT](https://root.cern.ch) Necessary for `xsec.genie`, `unfold.roounfold` and `absorption.pi_earth_absorption` services, and to read ROOT cross section files in the `crossSections` utils module. Due to a bug in ROOT's python support (documented here https://github.com/IceCubeOpenSource/pisa/issues/430), you need at least version 6.12.04 of ROOT. -* [Sphinx](http://www.sphinx-doc.org/en/stable/) version >= 1.3 - * Installed alongside PISA if you specify option `['develop']` to `pip` -* [versioneer](https://github.com/warner/python-versioneer) Automatically get versions from git and make these embeddable and usable in code. Note that the install process is unique since it first places `versioneer.py` in the PISA root directory, and then updates source files within the repository to provide static and dynamic version info. - * Installed alongside PISA if you specify option `['develop']` to `pip` -* [black](https://github.com/ambv/black) Format your Python code, _automatically_, with typically very nice results! - * Note this only works in Python3 - - -### Obtain PISA sourcecode - -#### Develop PISA: Fork then clone - -If you wish to modify PISA and contribute your code changes back to the PISA project (*highly recommended!*), fork `IceCubeOpenSource/pisa` from Github. - -Forking creates your own version of PISA within your Github account. -You can freely create your own *branch*, modify the code, and then *add* and *commit* changes to that branch within your fork of PISA. -When you want to share your changes with `IceCubeOpenSource/pisa`, you can then submit a *pull request* to `IceCubeOpenSource/pisa` which can be merged by the PISA administrator (after the code is reviewed and tested, of course). - -* Navigate to the [PISA github page](https://github.com/IceCubeOpenSource/pisa) and fork the repository by clicking on the ![fork](images/ForkButton.png) button. -* Clone the repository into the `$PISA` directory via one of the following commands (`` is your Github username): - * either SSH access to repo:
-`git clone git@github.com:/pisa.git $PISA -` - * or HTTPS access to repo:
-`git clone https://github.com//pisa.git $PISA` - - -#### Using but not developing PISA: Just clone - -If you just wish to pull changes from github (and not submit any changes back), you can just clone the sourcecode without creating a fork of the project. - -* Clone the repository into the `$PISA` directory via one of the following commands: - * either SSH access to repo:
-`git clone git@github.com:IceCubeOpenSource/pisa.git $PISA` - * or HTTPS access to repo:
-`git clone https://github.com/IceCubeOpenSource/pisa.git $PISA` - - -### Ensure a clean install using virtualenv or conda env - -It is absolutely discouraged to install PISA as a root (privileged) user. -PISA is not vetted for security vulnerabilities, so should always be installed and run as a regular (unprivileged) user. - -It is suggested (but not required) that you install PISA within a virtual environment (or in a conda env if you're using Anaconda or Miniconda Python distributions). -This minimizes cross-contamination by PISA of a system-wide (or other) Python installation with conflicting required package versions, guarantees that you can install PISA as an unprivileged user, guarantees that PISA's dependencies are met, and allows for multiple versions of PISA to be installed simultaneously (each in a different virtualenv / conda env). - -Note that it is also discouraged, but you _can_ install PISA as an unprivileged user using your system-wide Python install with the `--user` option to `pip`. -This is not quite as clean as a virtual environment, and the issue with coflicting package dependency versions remains. ### Compile the documentation -To compile a new version of the documentation to html (pdf and other formats are available by replacing `html` with `pdf`, etc.): +In case you installed the optional "develop" dependencies: compile a new version of the documentation to html via ```bash cd $PISA && sphinx-apidoc -f -o docs/source pisa ``` @@ -345,3 +88,4 @@ In case code structure has changed, rebuild the apidoc by executing ```bash cd $PISA/docs && make html ``` +(Run `make help` to check which other documentation formats are available.) From 2463c701386d2430601ceca12ac92a3a2bb9ce51 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Fri, 15 Nov 2024 16:04:52 +0100 Subject: [PATCH 09/20] Expand INSTALL.md structure --- INSTALL.md | 149 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 111 insertions(+), 38 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 6b484ab15..0dd7f72e0 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,56 +1,129 @@ # Installation Guide +_Note that all terminal commands below are intended for the bash shell. You'll have to translate if you use a different shell._ ## Quick start +This guide will enable you to _use_ PISA within about five minutes. If you are more interested in contributing to PISA's development, please refer to the [advanced installation guide](#advanced-installation-guide) instead. + +1. Install [git](https://git-scm.com) if you don't have it already + * In Ubuntu,
+ ```bash + sudo apt install git + ``` +1. In your terminal, define a directory for PISA source code to live in, and create the directory. For example:
+ ```bash + export PISA=~/src/pisa + mkdir -p $PISA + ``` +1. Clone the PISA repository to your local computer
+ ```bash + git clone https://github.com/icecube/pisa.git $PISA + ``` +1. Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root)
+ https://conda-forge.org/download/
+ * In case you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end. +1. Create and activate a new conda environment, with a python version compatible with the python requirements below. Using mamba as a drop-in replacement for conda:
+ ```bash + mamba create -n python=3.10 + mamba activate + ``` +1. Install PISA with default packages only and without development tools
+ ```bash + pip install -e $PISA -vvv + ``` +1. Run a quick test
+ ```bash + pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf + ``` + This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. + +## Advanced installation guide + +### Preparation + +To ensure that you can contribute to PISA's development, first obtain a GitHub user ID if you don’t have one already, and optionally sign up for GitHub education pack for many features for free, too:
+https://education.github.com/pack + +Fork PISA on GitHub so you have your own copy of the repository to work on, from which you can create pull requests:
+https://github.com/icecube/pisa/fork + +If you like, set up passwordless ssh access to github:
+https://help.github.com/articles/connecting-to-github-with-ssh + +In your terminal, define a directory for PISA source code to live in, e.g.,
+```bash +export PISA=~/src/pisa +``` + +Also add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this every time. + +Create the above directory:
+```bash +mkdir -p $PISA +``` + +Install [git](https://git-scm.com) if you don't have it already. On, e.g., Ubuntu: `sudo apt install git`. -_Note that terminal commands below are intended for the bash shell. You'll have to translate if you use a different shell._ - -1. _(optional)_ Obtain a github user ID if you don’t have one already. (Otherwise, you will only have read access.)
- https://github.com - * Sign up for Github education pack for many features for free, too
- https://education.github.com/pack -1. _(optional)_ Fork PISA on github so you have your own copy to work from
- https://github.com/icecube/pisa/fork -1. _(optional)_ Set up passwordless ssh access to github
- https://help.github.com/articles/connecting-to-github-with-ssh -1. In your terminal, define a directory for PISA source code to live in. For example:
- `export PISA=~/src/pisa` - * Add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this every time. -1. Create the directory
- `mkdir -p $PISA` -1. Clone the PISA repository to your local computer (on the command line) - * If you forked PISA above
- * If you set up ssh authentication above
- `git clone git@github.com:/pisa.git $PISA` - * Otherwise
- `git clone https://github.com//pisa.git $PISA` - * If you didn't fork PISA
- * If you set up ssh authentication above
- `git clone git@github.com:icecube/pisa.git $PISA` - * Otherwise
- `git clone https://github.com/icecube/pisa.git $PISA` -1. Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root), if you don’t have it already - https://conda-forge.org/download/ +Next, clone the PISA repository to your local computer. On the command line, + * if you set up ssh authentication above
+ ```bash + git clone git@github.com:/pisa.git $PISA + ``` + * otherwise
+ ```bash + git clone https://github.com//pisa.git $PISA + ``` + +Below we describe two different sets of pre-installation steps:
+ +The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the Miniforge distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
+ +The [second (alternative)](#alternative-cvmfs-and-virtualenv) assumes you have access to IceCube's cvmfs repository and would like to use one of its Python and software distributions. Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2). + +### Default: Miniforge distribution + +Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root) from https://conda-forge.org/download/. 1. _(optional)_ If you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end 1. Create and activate a new conda environment, with a python version compatible with the python requirements below. We suggest using mamba as a drop-in replacement for conda, for example
```bash mamba create -n python=3.10 mamba activate ``` -1. Install PISA - * either with default packages only and without development tools
- `pip install -e $PISA -vvv` - * or, if desired, including optional packages and development tools
- `pip install -e $PISA[develop] -vvv` -1. Run a quick test
- ```bash - pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf - ``` - This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. +### Alternative: CVMFS and virtualenv +Switch to the directory where you want to install PISA and create a virtual python environment (`virtualenv`).
+Load the CVMFS environment:
+```bash +unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` +``` +On the cobalt machines of the IceCube collaboration, make sure that `which python` now outputs `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python`. +Create the virtual environment:
+```bash +python -m venv ./ +``` + +Load the virtual environment:
+```bash +source .//bin/activate +``` +The shell should now indicate that you are in the environment. +### Final step: install and test PISA +You can now proceed to install PISA either with default packages only and without development tools
+```bash +pip install -e $PISA -vvv +``` +or, if desired, including optional packages and development tools
+```bash +pip install -e $PISA[develop] -vvv +``` +If the installation went smoothly, you are now ready to run a quick test
+```bash +pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf +``` +This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. ## Additional information From ca11cae9c3e8752b2f012dd002cd36efc1906bfb Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Fri, 15 Nov 2024 17:00:14 +0100 Subject: [PATCH 10/20] Some reformatting of INSTALL.md and expand with command details --- INSTALL.md | 132 +++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 92 insertions(+), 40 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 0dd7f72e0..d93892e49 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -5,10 +5,13 @@ _Note that all terminal commands below are intended for the bash shell. You'll h This guide will enable you to _use_ PISA within about five minutes. If you are more interested in contributing to PISA's development, please refer to the [advanced installation guide](#advanced-installation-guide) instead. 1. Install [git](https://git-scm.com) if you don't have it already - * In Ubuntu,
- ```bash - sudo apt install git - ``` +
+ in Ubuntu + + ```bash + sudo apt install git + ``` +
1. In your terminal, define a directory for PISA source code to live in, and create the directory. For example:
```bash export PISA=~/src/pisa @@ -18,13 +21,13 @@ This guide will enable you to _use_ PISA within about five minutes. If you are m ```bash git clone https://github.com/icecube/pisa.git $PISA ``` -1. Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root)
+1. Install the latest Miniforge Python distribution for either Mac or Linux (as your user, _not_ as root)
https://conda-forge.org/download/
* In case you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end. -1. Create and activate a new conda environment, with a python version compatible with the python requirements below. Using mamba as a drop-in replacement for conda:
+1. Create and activate a new conda environment, with a Python version compatible with the Python requirements below. Using mamba as a drop-in replacement for conda:
```bash - mamba create -n python=3.10 - mamba activate + mamba create -n python=3.10 + mamba activate ``` 1. Install PISA with default packages only and without development tools
```bash @@ -32,9 +35,9 @@ This guide will enable you to _use_ PISA within about five minutes. If you are m ``` 1. Run a quick test
```bash - pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf + pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf ``` - This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. + This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. ## Advanced installation guide @@ -61,19 +64,36 @@ Create the above directory:
mkdir -p $PISA ``` -Install [git](https://git-scm.com) if you don't have it already. On, e.g., Ubuntu: `sudo apt install git`. +Install [git](https://git-scm.com) if you don't have it already. +
+ in Ubuntu + + ```bash + sudo apt install git + ``` +
+ + Next, clone the PISA repository to your local computer. On the command line, - * if you set up ssh authentication above
- ```bash - git clone git@github.com:/pisa.git $PISA - ``` - * otherwise
- ```bash - git clone https://github.com//pisa.git $PISA - ``` +
+ with ssh authentication + + ```bash + git clone git@github.com:/pisa.git $PISA + ``` +
+ +
+ without + + ```bash + git clone https://github.com//pisa.git $PISA + ``` +
+ -Below we describe two different sets of pre-installation steps:
+Below we describe two different ways of setting up the PISA Python environment:
The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the Miniforge distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
@@ -81,17 +101,38 @@ The [second (alternative)](#alternative-cvmfs-and-virtualenv) assumes you have a ### Default: Miniforge distribution -Install the latest Miniforge python distribution for either Mac or Linux (as your user, _not_ as root) from https://conda-forge.org/download/. -1. _(optional)_ If you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end -1. Create and activate a new conda environment, with a python version compatible with the python requirements below. We suggest using mamba as a drop-in replacement for conda, for example
- ```bash - mamba create -n python=3.10 - mamba activate - ``` +Install the latest Miniforge Python distribution for either Mac or Linux (as your user, _not_ as root) from https://conda-forge.org/download/. +
+ command suggestions + + ```bash + mkdir -p /miniforge3 + wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname}-$(uname -m).sh -O /miniforge3/miniforge.sh + bash /miniforge3/miniforge.sh -b -u -p /miniforge3 + rm /miniforge3/miniforge.sh + ``` + If you are installing on the cobalt machines of the IceCube collaboration, consider using `/data/user/` instead of `$HOME` for ``. +
+ +
+ if you declined to update your shell profile to automatically initialize conda + + ```bash + eval "$(/miniforge3/bin/conda shell.bash hook)" + ``` + will activate the base environment as prompted at the end of the Miniforge installation script + +
+ +Create and activate a new conda environment, with a Python version compatible with the Python requirements below. We suggest using mamba as a drop-in replacement for conda:
+ ```bash + mamba create -n python=3.10 + mamba activate + ``` ### Alternative: CVMFS and virtualenv -Switch to the directory where you want to install PISA and create a virtual python environment (`virtualenv`).
+Switch to the directory where you want to install PISA and create a virtual Python environment (`virtualenv`).
Load the CVMFS environment:
```bash unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` @@ -100,30 +141,41 @@ On the cobalt machines of the IceCube collaboration, make sure that `which pytho Create the virtual environment:
```bash -python -m venv ./ +python -m venv ./ ``` Load the virtual environment:
```bash -source .//bin/activate +source .//bin/activate ``` The shell should now indicate that you are in the environment. ### Final step: install and test PISA -You can now proceed to install PISA either with default packages only and without development tools
-```bash -pip install -e $PISA -vvv -``` -or, if desired, including optional packages and development tools
-```bash -pip install -e $PISA[develop] -vvv -``` +You can now proceed to install PISA, either + +
+ with default packages only and without development tools + + ```bash + pip install -e $PISA -vvv + ``` +
+ +or, if desired, + +
+ including optional packages and development tools + + ```bash + pip install -e $PISA[develop] -vvv + ``` +
If the installation went smoothly, you are now ready to run a quick test
```bash -pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf +pisa-distribution_maker --pipeline settings/pipeline/IceCube_3y_neutrinos.cfg --outdir --pdf ``` -This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. +This command should have created the folder `` containing a pdf with output maps for different neutrino types and interactions. ## Additional information From 0a5a55f14c04fccc6fe24ee3e1a860d267ffbd65 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 16 Nov 2024 15:44:40 +0100 Subject: [PATCH 11/20] minor --- INSTALL.md | 52 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index d93892e49..bdad9d2f8 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -95,9 +95,16 @@ Next, clone the PISA repository to your local computer. On the command line, Below we describe two different ways of setting up the PISA Python environment:
-The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the Miniforge distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
+The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the [Miniforge](https://conda-forge.org/docs/user/introduction/) distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
+ +The [second (alternative)](#alternative-cvmfs-and-virtualenv) assumes you have access to IceCube's CernVM-FS (CVMFS) repository and would like to use one of its Python and software distributions. Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2). + +
+ In case you are installing one of IceCube's Cobalt nodes + + Consider using `/data/user/` instead of e.g. `$HOME` as installation location below. +
-The [second (alternative)](#alternative-cvmfs-and-virtualenv) assumes you have access to IceCube's cvmfs repository and would like to use one of its Python and software distributions. Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2). ### Default: Miniforge distribution @@ -107,48 +114,49 @@ Install the latest Miniforge Python distribution for either Mac or Linux (as you ```bash mkdir -p /miniforge3 - wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname}-$(uname -m).sh -O /miniforge3/miniforge.sh - bash /miniforge3/miniforge.sh -b -u -p /miniforge3 - rm /miniforge3/miniforge.sh + wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" + bash "Miniforge3-$(uname)-$(uname -m).sh" -p /miniforge3 -u + rm "Miniforge3-$(uname)-$(uname -m).sh" ``` - If you are installing on the cobalt machines of the IceCube collaboration, consider using `/data/user/` instead of `$HOME` for ``. - -
- if you declined to update your shell profile to automatically initialize conda - + **Notes:** + * To perform SHA-256 checksum verification of the Miniforge installer, download the installer (`.sh`) for your platform whose name contains the release version and the corresponding `.sha256` checksum file from https://github.com/conda-forge/miniforge/releases/latest, then execute ```sha256sum -c "Miniforge3--$(uname)-$(uname -m).sh.sha256"```. + * You can decline having your shell profile updated to automatically initialize conda. In this case ```bash eval "$(/miniforge3/bin/conda shell.bash hook)" ``` - will activate the base environment as prompted at the end of the Miniforge installation script - + will activate the base environment as prompted at the end of the Miniforge installation script. Doing so is required to proceed with this installation and whenever PISA is used again. The successful activation is indicated by the shell prompt `(base)`. An overview of the packages in the base environment can be gained via `mamba/conda list`.
-Create and activate a new conda environment, with a Python version compatible with the Python requirements below. We suggest using mamba as a drop-in replacement for conda:
+It is recommended to keep the base environment stable. Therefore, create and activate a new conda environment, with a Python version compatible with the Python requirements below.
```bash - mamba create -n python=3.10 - mamba activate + conda create -n python=3.10 + conda activate ``` +A shell prompt with `` name in parentheses should now confirm the successful activation. ### Alternative: CVMFS and virtualenv -Switch to the directory where you want to install PISA and create a virtual Python environment (`virtualenv`).
Load the CVMFS environment:
```bash unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` ``` -On the cobalt machines of the IceCube collaboration, make sure that `which python` now outputs `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python`. +
+ On one of IceCube's Cobalt nodes + + Verify that `which python` outputs `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python`. +
-Create the virtual environment:
+After switching to the directory where you want to install PISA, create the virtual environment:
```bash -python -m venv ./ +python -m venv ./ ``` -Load the virtual environment:
+Activate the virtual environment:
```bash -source .//bin/activate +source .//bin/activate ``` -The shell should now indicate that you are in the environment. +A shell prompt with the virtual environment's directory name in parentheses should now confirm the successful activation. ### Final step: install and test PISA You can now proceed to install PISA, either From 80edb66c5175720aa3961000b588357d530c2964 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 16 Nov 2024 19:28:28 +0100 Subject: [PATCH 12/20] Restructure for default git install into conda env and update required dependencies --- INSTALL.md | 106 ++++++++++++++++++++++++++++------------------------- 1 file changed, 56 insertions(+), 50 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index bdad9d2f8..899bef5b6 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -4,15 +4,19 @@ _Note that all terminal commands below are intended for the bash shell. You'll h ## Quick start This guide will enable you to _use_ PISA within about five minutes. If you are more interested in contributing to PISA's development, please refer to the [advanced installation guide](#advanced-installation-guide) instead. -1. Install [git](https://git-scm.com) if you don't have it already -
- in Ubuntu - +1. Install the latest Miniforge Python distribution for either Mac or Linux (as your user, _not_ as root)
+ https://conda-forge.org/download/
+ * In case you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end. +1. In your terminal, create and activate a new conda environment, with a Python version compatible with the Python requirements below
+ ```bash + conda create -n python=3.10 + conda activate + ``` +1. If your system doesn't already have it, install [git](https://git-scm.com) into this environment. (We use `mamba` as a drop-in replacement for the `conda` package manager.) ```bash - sudo apt install git + mamba install git ``` -
-1. In your terminal, define a directory for PISA source code to live in, and create the directory. For example:
+1. Define a directory for PISA source code to live in, and create the directory. For example:
```bash export PISA=~/src/pisa mkdir -p $PISA @@ -21,14 +25,6 @@ This guide will enable you to _use_ PISA within about five minutes. If you are m ```bash git clone https://github.com/icecube/pisa.git $PISA ``` -1. Install the latest Miniforge Python distribution for either Mac or Linux (as your user, _not_ as root)
- https://conda-forge.org/download/
- * In case you declined to update your shell profile to automatically initialize conda, activate the base environment as prompted at the end. -1. Create and activate a new conda environment, with a Python version compatible with the Python requirements below. Using mamba as a drop-in replacement for conda:
- ```bash - mamba create -n python=3.10 - mamba activate - ``` 1. Install PISA with default packages only and without development tools
```bash pip install -e $PISA -vvv @@ -43,8 +39,11 @@ This guide will enable you to _use_ PISA within about five minutes. If you are m ### Preparation -To ensure that you can contribute to PISA's development, first obtain a GitHub user ID if you don’t have one already, and optionally sign up for GitHub education pack for many features for free, too:
-https://education.github.com/pack +To ensure that you can contribute to PISA's development, first obtain a GitHub user ID if you don’t have one already.
+
+ optional sign up for GitHub education pack for many features for free + https://education.github.com/pack +
Fork PISA on GitHub so you have your own copy of the repository to work on, from which you can create pull requests:
https://github.com/icecube/pisa/fork @@ -64,35 +63,6 @@ Create the above directory:
mkdir -p $PISA ``` -Install [git](https://git-scm.com) if you don't have it already. -
- in Ubuntu - - ```bash - sudo apt install git - ``` -
- - - -Next, clone the PISA repository to your local computer. On the command line, -
- with ssh authentication - - ```bash - git clone git@github.com:/pisa.git $PISA - ``` -
- -
- without - - ```bash - git clone https://github.com//pisa.git $PISA - ``` -
- - Below we describe two different ways of setting up the PISA Python environment:
The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the [Miniforge](https://conda-forge.org/docs/user/introduction/) distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
@@ -100,7 +70,7 @@ The [first (default)](#default-miniforge-distribution) obtains Python and Python The [second (alternative)](#alternative-cvmfs-and-virtualenv) assumes you have access to IceCube's CernVM-FS (CVMFS) repository and would like to use one of its Python and software distributions. Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2).
- In case you are installing one of IceCube's Cobalt nodes + in case of installation one of IceCube's Cobalt nodes Consider using `/data/user/` instead of e.g. `$HOME` as installation location below.
@@ -128,7 +98,7 @@ Install the latest Miniforge Python distribution for either Mac or Linux (as you will activate the base environment as prompted at the end of the Miniforge installation script. Doing so is required to proceed with this installation and whenever PISA is used again. The successful activation is indicated by the shell prompt `(base)`. An overview of the packages in the base environment can be gained via `mamba/conda list`. -It is recommended to keep the base environment stable. Therefore, create and activate a new conda environment, with a Python version compatible with the Python requirements below.
+It is recommended to keep the base environment stable. Therefore, create and activate a new conda environment, with a Python version compatible with the Python requirements below:
```bash conda create -n python=3.10 conda activate @@ -142,7 +112,7 @@ Load the CVMFS environment:
unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` ```
- On one of IceCube's Cobalt nodes + on one of IceCube's Cobalt nodes Verify that `which python` outputs `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python`.
@@ -158,7 +128,29 @@ source .//bin/activate ``` A shell prompt with the virtual environment's directory name in parentheses should now confirm the successful activation. -### Final step: install and test PISA +### Common final steps: clone, install and test PISA + +Install [git](https://git-scm.com) if you [don't have it](#required-dependencies) already. + +Next, clone the PISA repository to your local computer. On the command line, +
+ with ssh authentication + + ```bash + git clone git@github.com:/pisa.git $PISA + ``` +
+ +
+ without + + ```bash + git clone https://github.com//pisa.git $PISA + ``` + See https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls if you have issues authenticating in this case. +
+ + You can now proceed to install PISA, either
@@ -187,6 +179,20 @@ This command should have created the folder `` containing a pdf wit ## Additional information +### Required Dependencies + +With the exception of `Python` itself (and possibly `git`), the installation methods outlined above should not demand the _manual_ prior installation of any Python or non-Python requirements for PISA. +Support for all of these comes pre-packaged or as `conda`/`mamba`-installable packages in the Miniforge Python distribution. +* [python](http://www.python.org) — version >= 3.6 and <= 3.10 required + * Miniforge & CVMFS: built in +* [pip](https://pip.pypa.io) version >= 1.8 and <= 25 required + * Miniforge & CVMFS: built in +* [git](https://git-scm.com) + * Miniforge: `mamba install git` + * or system wide, e.g. in Ubuntu
+ `sudo apt install git` + * it is already installed on IceCube's Cobalt nodes + ### Installation * First, note that the installation is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). * `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. From eb0e606b4471f46f2ce5c81808e0ae72711a7b90 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 16 Nov 2024 19:58:09 +0100 Subject: [PATCH 13/20] Clean up optional dependencies (completeness not guaranteed) and fix min. py version --- INSTALL.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/INSTALL.md b/INSTALL.md index 899bef5b6..25053540e 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -183,7 +183,7 @@ This command should have created the folder `` containing a pdf wit With the exception of `Python` itself (and possibly `git`), the installation methods outlined above should not demand the _manual_ prior installation of any Python or non-Python requirements for PISA. Support for all of these comes pre-packaged or as `conda`/`mamba`-installable packages in the Miniforge Python distribution. -* [python](http://www.python.org) — version >= 3.6 and <= 3.10 required +* [python](http://www.python.org) — version >= 3.8 and <= 3.10 required * Miniforge & CVMFS: built in * [pip](https://pip.pypa.io) version >= 1.8 and <= 25 required * Miniforge & CVMFS: built in @@ -215,6 +215,27 @@ pip install -e $PISA[develop] --force-reinstall -vvv **Note** that if files change names or locations, though, the above can still not be enough. In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). +### Optional Dependencies + +Some of the following optional dependencies must be installed manually prior to installing PISA, and some will be installed automatically by pip, and this seems to vary from system to system. Therefore you can first try to run the installation, and just install whatever pip says it needed, or just use apt, pip, or conda/mamba to install the below before running the PISA installation. + +* [LeptonWeighter](https://github.com/icecube/leptonweighter) Required for `data.licloader_weighter` service. +* [MCEq](https://github.com/afedynitch/MCEq) Required for `flux.mceq` service. +* [nuSQuiDS](https://github.com/arguelles/nuSQuIDS) Required for `osc.nusquids` service. +* [OpenMP](https://openmp.org) Intra-process parallelization to accelerate code on on multi-core/multi-CPU computers. + * Available from your compiler: gcc supports OpenMP 4.0 and Clang >= 3.8.0 supports OpenMP 3.1. Either version of OpenMP should work, but Clang has yet to be tested for its OpenMP support. +* [Photospline](https://github.com/icecube/photospline) Required for `flux.airs` service. +* [Pylint](https://pylint.org): Static code checker and style analyzer for Python code. Note that our (more or less enforced) coding conventions are codified in the pylintrc file in PISA, which will automatically be found and used by Pylint when running on code within a PISA package.
+ * Installed alongside PISA if you specify option `['develop']` to `pip` +* [recommonmark](http://recommonmark.readthedocs.io/en/latest/) Translator to allow markdown docs/docstrings to be used; plugin for Sphinx. (Required to compile PISA's documentation.) + * Installed alongside PISA if you specify option `['develop']` to `pip` +* [ROOT >= 6.12.04 with PyROOT](https://root.cern.ch) Required for `absorption.earth_absorption` service, and to read ROOT cross section files in the `crossSections` utils module. Due to a bug in ROOT's Python support (documented here https://github.com/IceCubeOpenSource/pisa/issues/430), you need at least version 6.12.04 of ROOT. +* [Sphinx](https://www.sphinx-doc.org) version >= 1.3 + * Installed alongside PISA if you specify option `['develop']` to `pip` +* [versioneer](https://github.com/python-versioneer/python-versioneer) Automatically get versions from git and make these embeddable and usable in code. Note that the install process is unique since it first places `versioneer.py` in the PISA root directory, and then updates source files within the repository to provide static and dynamic version info. + * Installed alongside PISA if you specify option `['develop']` to `pip` +* [black](https://github.com/psf/black) Format your Python code, _automatically_, with typically very nice results! + ### Compile the documentation From 92e184d8829ad817ac457fd94dc6fbc7ebef9bdd Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 16 Nov 2024 20:20:03 +0100 Subject: [PATCH 14/20] Bring back notes on clean install + few more optional-dependencies mods. --- INSTALL.md | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 25053540e..34f82af6d 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -220,31 +220,45 @@ In this case, the old files have to be removed manually (along with any associat Some of the following optional dependencies must be installed manually prior to installing PISA, and some will be installed automatically by pip, and this seems to vary from system to system. Therefore you can first try to run the installation, and just install whatever pip says it needed, or just use apt, pip, or conda/mamba to install the below before running the PISA installation. * [LeptonWeighter](https://github.com/icecube/leptonweighter) Required for `data.licloader_weighter` service. -* [MCEq](https://github.com/afedynitch/MCEq) Required for `flux.mceq` service. +* [MCEq](https://github.com/afedynitch/MCEq) Required for `create_barr_sys_tables_mceq.py` script. * [nuSQuiDS](https://github.com/arguelles/nuSQuIDS) Required for `osc.nusquids` service. * [OpenMP](https://openmp.org) Intra-process parallelization to accelerate code on on multi-core/multi-CPU computers. * Available from your compiler: gcc supports OpenMP 4.0 and Clang >= 3.8.0 supports OpenMP 3.1. Either version of OpenMP should work, but Clang has yet to be tested for its OpenMP support. * [Photospline](https://github.com/icecube/photospline) Required for `flux.airs` service. -* [Pylint](https://pylint.org): Static code checker and style analyzer for Python code. Note that our (more or less enforced) coding conventions are codified in the pylintrc file in PISA, which will automatically be found and used by Pylint when running on code within a PISA package.
+* [Pylint](https://pylint.org) Static code checker and style analyzer for Python code. Note that our (more or less enforced) coding conventions are codified in the pylintrc file in PISA, which will automatically be found and used by Pylint when running on code within a PISA package. + * Installed alongside PISA if you specify option `['develop']` to `pip` +* [Pytest](https://docs.pytest.org/) Python testing framework. Used by a couple unit tests. * Installed alongside PISA if you specify option `['develop']` to `pip` * [recommonmark](http://recommonmark.readthedocs.io/en/latest/) Translator to allow markdown docs/docstrings to be used; plugin for Sphinx. (Required to compile PISA's documentation.) * Installed alongside PISA if you specify option `['develop']` to `pip` * [ROOT >= 6.12.04 with PyROOT](https://root.cern.ch) Required for `absorption.earth_absorption` service, and to read ROOT cross section files in the `crossSections` utils module. Due to a bug in ROOT's Python support (documented here https://github.com/IceCubeOpenSource/pisa/issues/430), you need at least version 6.12.04 of ROOT. -* [Sphinx](https://www.sphinx-doc.org) version >= 1.3 +* [Sphinx >= 1.3](https://www.sphinx-doc.org) + * Installed alongside PISA if you specify option `['develop']` to `pip` +* [Read the Docs Sphinx Theme](https://github.com/readthedocs/sphinx_rtd_theme) * Installed alongside PISA if you specify option `['develop']` to `pip` * [versioneer](https://github.com/python-versioneer/python-versioneer) Automatically get versions from git and make these embeddable and usable in code. Note that the install process is unique since it first places `versioneer.py` in the PISA root directory, and then updates source files within the repository to provide static and dynamic version info. * Installed alongside PISA if you specify option `['develop']` to `pip` * [black](https://github.com/psf/black) Format your Python code, _automatically_, with typically very nice results! +### Ensure a clean install using virtualenv or conda env + +It is absolutely discouraged to install PISA as a root (privileged) user. +PISA is not vetted for security vulnerabilities, so should always be installed and run as a regular (unprivileged) user. + +It is suggested (but not required) that you install PISA within a virtual environment (or in a conda env if you're using Anaconda, Miniconda, or Miniforge Python distributions). +This minimizes cross-contamination by PISA of a system-wide (or other) Python installation with conflicting required package versions, guarantees that you can install PISA as an unprivileged user, that PISA's dependencies are met, and allows for multiple versions of PISA to be installed simultaneously (each in a different virtualenv / conda env). + +Note that it is also discouraged, but you _can_ install PISA as an unprivileged user using your system-wide Python install with the `--user` option to `pip`. +This is not quite as clean as a virtual environment, and the issue with coflicting package dependency versions remains. ### Compile the documentation -In case you installed the optional "develop" dependencies: compile a new version of the documentation to html via +In case you installed the optional "develop" dependencies, you can compile a (new) version of the documentation to html via ```bash cd $PISA && sphinx-apidoc -f -o docs/source pisa ``` -In case code structure has changed, rebuild the apidoc by executing +In case code structure has changed, rebuild the API documentation by executing ```bash cd $PISA/docs && make html ``` From 2ab3b470d8e0ef659071388804a6baf7512d1e1d Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 16 Nov 2024 20:53:30 +0100 Subject: [PATCH 15/20] Mention location of list of required python modules --- INSTALL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/INSTALL.md b/INSTALL.md index 34f82af6d..d1886d0f5 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -193,6 +193,8 @@ Support for all of these comes pre-packaged or as `conda`/`mamba`-installable pa `sudo apt install git` * it is already installed on IceCube's Cobalt nodes +Required Python modules whose installation is taken care of by pip are specified in [setup.py](setup.py). + ### Installation * First, note that the installation is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). * `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. From a5108690cdf9a34deec9fc90e94e9b266e969130 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 16 Nov 2024 23:39:26 +0100 Subject: [PATCH 16/20] stop mixing up venv and virtualenv and comment on venv location --- INSTALL.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index d1886d0f5..86c42bec8 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -67,7 +67,7 @@ Below we describe two different ways of setting up the PISA Python environment:< The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the [Miniforge](https://conda-forge.org/docs/user/introduction/) distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
-The [second (alternative)](#alternative-cvmfs-and-virtualenv) assumes you have access to IceCube's CernVM-FS (CVMFS) repository and would like to use one of its Python and software distributions. Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2). +The [second (alternative)](#alternative-cvmfs-and-venv) assumes you have access to IceCube's CernVM-FS (CVMFS) repository and would like to use one of the Python installations it provides as the "base" of a [venv](https://docs.python.org/3/library/venv.html). Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2).
in case of installation one of IceCube's Cobalt nodes @@ -105,7 +105,7 @@ It is recommended to keep the base environment stable. Therefore, create and act ``` A shell prompt with `` name in parentheses should now confirm the successful activation. -### Alternative: CVMFS and virtualenv +### Alternative: CVMFS and venv Load the CVMFS environment:
```bash @@ -117,16 +117,16 @@ unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` Verify that `which python` outputs `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python`.
-After switching to the directory where you want to install PISA, create the virtual environment:
+Create a virtual environment folder into which all Python packages will be installed, which can be placed wherever you please. (Note that there are [disadvantages](https://pybit.es/articles/a-better-place-to-put-your-python-virtual-environments/) to putting it in your local PISA repository's top-level directory `$PISA`.) ```bash -python -m venv ./ +python -m venv /PATH/TO/ ``` Activate the virtual environment:
```bash -source .//bin/activate +source /PATH/TO//bin/activate ``` -A shell prompt with the virtual environment's directory name in parentheses should now confirm the successful activation. +A shell prompt with the virtual environment's name in parentheses should now confirm the successful activation. ### Common final steps: clone, install and test PISA @@ -242,16 +242,16 @@ Some of the following optional dependencies must be installed manually prior to * Installed alongside PISA if you specify option `['develop']` to `pip` * [black](https://github.com/psf/black) Format your Python code, _automatically_, with typically very nice results! -### Ensure a clean install using virtualenv or conda env +### Ensure a clean install using venv or conda env It is absolutely discouraged to install PISA as a root (privileged) user. PISA is not vetted for security vulnerabilities, so should always be installed and run as a regular (unprivileged) user. It is suggested (but not required) that you install PISA within a virtual environment (or in a conda env if you're using Anaconda, Miniconda, or Miniforge Python distributions). -This minimizes cross-contamination by PISA of a system-wide (or other) Python installation with conflicting required package versions, guarantees that you can install PISA as an unprivileged user, that PISA's dependencies are met, and allows for multiple versions of PISA to be installed simultaneously (each in a different virtualenv / conda env). +This minimizes cross-contamination by PISA of a system-wide (or other) Python installation with conflicting required package versions, guarantees that you can install PISA as an unprivileged user, that PISA's dependencies are met, and allows for multiple versions of PISA to be installed simultaneously (each in a different venv / conda env). Note that it is also discouraged, but you _can_ install PISA as an unprivileged user using your system-wide Python install with the `--user` option to `pip`. -This is not quite as clean as a virtual environment, and the issue with coflicting package dependency versions remains. +This is not quite as clean as a virtual environment, and the issue with conflicting package dependency versions remains. ### Compile the documentation From 66d63d1e8dcf28df1ac6431e75d7e2f99c8a992b Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sun, 17 Nov 2024 12:56:17 +0100 Subject: [PATCH 17/20] More optional deps --- INSTALL.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 86c42bec8..cce4e21c4 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -48,7 +48,7 @@ To ensure that you can contribute to PISA's development, first obtain a GitHub u Fork PISA on GitHub so you have your own copy of the repository to work on, from which you can create pull requests:
https://github.com/icecube/pisa/fork -If you like, set up passwordless ssh access to github:
+If you like, set up passwordless ssh access to GitHub:
https://help.github.com/articles/connecting-to-github-with-ssh In your terminal, define a directory for PISA source code to live in, e.g.,
@@ -84,7 +84,7 @@ Install the latest Miniforge Python distribution for either Mac or Linux (as you ```bash mkdir -p /miniforge3 - wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" + wget "https://.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" bash "Miniforge3-$(uname)-$(uname -m).sh" -p /miniforge3 -u rm "Miniforge3-$(uname)-$(uname -m).sh" ``` @@ -221,6 +221,8 @@ In this case, the old files have to be removed manually (along with any associat Some of the following optional dependencies must be installed manually prior to installing PISA, and some will be installed automatically by pip, and this seems to vary from system to system. Therefore you can first try to run the installation, and just install whatever pip says it needed, or just use apt, pip, or conda/mamba to install the below before running the PISA installation. +* [emcee](https://github.com/dfm/emcee) Required for MCMC sampling functionality in the `llh_client`& `llh_server` utils modules and the `analysis` module. +* [GLoBES wrapper](https://github.com/atrettin/GLoBES_wrapper) Required for `osc.globes` service. * [LeptonWeighter](https://github.com/icecube/leptonweighter) Required for `data.licloader_weighter` service. * [MCEq](https://github.com/afedynitch/MCEq) Required for `create_barr_sys_tables_mceq.py` script. * [nuSQuiDS](https://github.com/arguelles/nuSQuIDS) Required for `osc.nusquids` service. From 9727286f350bbcf8728907852a3f912c93746277 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Fri, 22 Nov 2024 15:46:09 +0100 Subject: [PATCH 18/20] Minor tweaks --- INSTALL.md | 107 ++++++++++++++++++++++++----------------------------- 1 file changed, 48 insertions(+), 59 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index cce4e21c4..435dd0654 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -16,9 +16,9 @@ This guide will enable you to _use_ PISA within about five minutes. If you are m ```bash mamba install git ``` -1. Define a directory for PISA source code to live in, and create the directory. For example:
+1. Define a directory for the PISA source code (~ 500 MB) to live in, and create the directory. For example:
```bash - export PISA=~/src/pisa + export PISA=/pisa mkdir -p $PISA ``` 1. Clone the PISA repository to your local computer
@@ -53,9 +53,14 @@ https://help.github.com/articles/connecting-to-github-with-ssh In your terminal, define a directory for PISA source code to live in, e.g.,
```bash -export PISA=~/src/pisa +export PISA=/pisa ``` +
+ suggested paths + + Since the PISA source occupies several hundreds of MB of drive space: on one of IceCube's Cobalt nodes, consider setting `` to `/data/user/` instead of e.g. `$HOME`. +
Also add this line to your `~/.bashrc` file so you can refer to the `$PISA` variable without doing this every time. Create the above directory:
@@ -69,12 +74,8 @@ The [first (default)](#default-miniforge-distribution) obtains Python and Python The [second (alternative)](#alternative-cvmfs-and-venv) assumes you have access to IceCube's CernVM-FS (CVMFS) repository and would like to use one of the Python installations it provides as the "base" of a [venv](https://docs.python.org/3/library/venv.html). Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2). -
- in case of installation one of IceCube's Cobalt nodes - - Consider using `/data/user/` instead of e.g. `$HOME` as installation location below. -
- +> [!NOTE] +> Miniforge or the venv virtual environment folder into which all Python packages will be installed can be placed wherever you please. (Note that there are [disadvantages](https://pybit.es/articles/a-better-place-to-put-your-python-virtual-environments/) to putting either in your local PISA repository's top-level directory `$PISA`.) However, on one of IceCube's Cobalt nodes, again consider `/data/user/` instead of e.g. `$HOME`. ### Default: Miniforge distribution @@ -117,7 +118,6 @@ unset OS_ARCH; eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh` Verify that `which python` outputs `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/bin/python`.
-Create a virtual environment folder into which all Python packages will be installed, which can be placed wherever you please. (Note that there are [disadvantages](https://pybit.es/articles/a-better-place-to-put-your-python-virtual-environments/) to putting it in your local PISA repository's top-level directory `$PISA`.) ```bash python -m venv /PATH/TO/ ``` @@ -153,23 +153,23 @@ Next, clone the PISA repository to your local computer. On the command line, You can now proceed to install PISA, either -
- with default packages only and without development tools - - ```bash - pip install -e $PISA -vvv - ``` -
+with default packages only and without development tools +```bash +pip install -e $PISA -vvv +``` +or, if desired, including optional packages and development tools +```bash +pip install -e $PISA[develop] -vvv +``` -or, if desired, - -
- including optional packages and development tools - - ```bash - pip install -e $PISA[develop] -vvv - ``` -
+> [!NOTE] +> * The installation is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). +> * `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. +> Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). +> * `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. +> * `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. +> * If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. +> You can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to re-install PISA when you need these to recompile. If the installation went smoothly, you are now ready to run a quick test
```bash @@ -179,6 +179,28 @@ This command should have created the folder `` containing a pdf wit ## Additional information +### Ensure a clean install using venv or conda env + +It is absolutely discouraged to install PISA as a root (privileged) user. +PISA is not vetted for security vulnerabilities, so should always be installed and run as a regular (unprivileged) user. + +It is suggested (but not required) that you install PISA within a virtual environment (or in a conda env if you're using Anaconda, Miniconda, or Miniforge Python distributions). +This minimizes cross-contamination by PISA of a system-wide (or other) Python installation with conflicting required package versions, guarantees that you can install PISA as an unprivileged user, that PISA's dependencies are met, and allows for multiple versions of PISA to be installed simultaneously (each in a different venv / conda env). + +Note that it is also discouraged, but you _can_ install PISA as an unprivileged user using your system-wide Python install with the `--user` option to `pip`. +This is not quite as clean as a virtual environment, and the issue with conflicting package dependency versions remains. + +### Re-installation + +Sometimes a change within PISA requires re-installation (particularly if a compiled module changes, the below forces re-compilation). + +```bash +pip install -e $PISA[develop] --force-reinstall -vvv +``` + +**Note** that if files change names or locations, though, the above can still not be enough. +In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). + ### Required Dependencies With the exception of `Python` itself (and possibly `git`), the installation methods outlined above should not demand the _manual_ prior installation of any Python or non-Python requirements for PISA. @@ -195,28 +217,6 @@ Support for all of these comes pre-packaged or as `conda`/`mamba`-installable pa Required Python modules whose installation is taken care of by pip are specified in [setup.py](setup.py). -### Installation -* First, note that the installation is ***not run as administrator***. It is discouraged to do so (and has not been tested this way). -* `-e $PISA` (or equivalently, `--editable $PISA`): Installs from source located at `$PISA` and allows for changes to the source code within to be immediately propagated to your Python installation. - Within the Python library tree, all files under `pisa` are links to your source code, so changes within your source are seen directly by the Python installation. Note that major changes to your source code (file names or directory structure changing) will require re-installation, though, for the links to be updated (see below for the command for re-installing). -* `[develop]` Specify optional dependency groups. You can omit any or all of these if your system does not support them or if you do not need them. -* `-vvv` Be maximally verbose during the install. You'll see lots of messages, including warnings that are irrelevant, but if your installation fails, it's easiest to debug if you use `-vvv`. -* If a specific compiler is set by the `CC` environment variable (`export CC=`), it will be used; otherwise, the `cc` command will be run on the system for compiling C-code. - -**Note** that you can work with your installation using the usual git commands (pull, push, etc.). However, these ***won't recompile*** any of the extension (i.e. pyx, _C/C++_) libraries. See below for how to reinstall PISA when you need these to recompile. - - -### Re-installation - -Sometimes a change within PISA requires re-installation (particularly if a compiled module changes, the below forces re-compilation). - -```bash -pip install -e $PISA[develop] --force-reinstall -vvv -``` - -**Note** that if files change names or locations, though, the above can still not be enough. -In this case, the old files have to be removed manually (along with any associated `.pyc` files, as Python will use these even if the `.py` files have been removed). - ### Optional Dependencies Some of the following optional dependencies must be installed manually prior to installing PISA, and some will be installed automatically by pip, and this seems to vary from system to system. Therefore you can first try to run the installation, and just install whatever pip says it needed, or just use apt, pip, or conda/mamba to install the below before running the PISA installation. @@ -244,17 +244,6 @@ Some of the following optional dependencies must be installed manually prior to * Installed alongside PISA if you specify option `['develop']` to `pip` * [black](https://github.com/psf/black) Format your Python code, _automatically_, with typically very nice results! -### Ensure a clean install using venv or conda env - -It is absolutely discouraged to install PISA as a root (privileged) user. -PISA is not vetted for security vulnerabilities, so should always be installed and run as a regular (unprivileged) user. - -It is suggested (but not required) that you install PISA within a virtual environment (or in a conda env if you're using Anaconda, Miniconda, or Miniforge Python distributions). -This minimizes cross-contamination by PISA of a system-wide (or other) Python installation with conflicting required package versions, guarantees that you can install PISA as an unprivileged user, that PISA's dependencies are met, and allows for multiple versions of PISA to be installed simultaneously (each in a different venv / conda env). - -Note that it is also discouraged, but you _can_ install PISA as an unprivileged user using your system-wide Python install with the `--user` option to `pip`. -This is not quite as clean as a virtual environment, and the issue with conflicting package dependency versions remains. - ### Compile the documentation In case you installed the optional "develop" dependencies, you can compile a (new) version of the documentation to html via From 846f2c76ae558a4a2c410152e15151f1c905e030 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Sat, 23 Nov 2024 18:28:02 +0100 Subject: [PATCH 19/20] small main readme fixes and tweaks (todo: update notebook used to produce readme by generate_readme.sh) --- README.md | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 9d82a43bb..04f5110c6 100755 --- a/README.md +++ b/README.md @@ -13,14 +13,14 @@ [Conventions](pisa/general_conventions.md) | [License](LICENSE) -PISA is a software written to analyze the results (or expected results) of an experiment based on Monte Carlo simulation. +PISA is a software written to analyze the results (or expected results) of an experiment based on Monte Carlo (MC) simulation. -In particular, PISA was written by and for the IceCube Collaboration for analyses employing the [IceCube Neutrino Observatory](https://icecube.wisc.edu/), including the [DeepCore](https://arxiv.org/abs/1109.6096) and the planned [Upgrade]([https://arxiv.org/abs/2307.15295](https://arxiv.org/pdf/1908.09441.pdf)) low-energy in-fill arrays. +In particular, PISA was written by and for the IceCube Collaboration for analyses employing the [IceCube Neutrino Observatory](https://icecube.wisc.edu/), including the [DeepCore](https://arxiv.org/abs/1109.6096) and the planned [Upgrade](https://arxiv.org/abs/2307.15295) low-energy in-fill arrays. > [!NOTE] > However, any experiment can make use of PISA for analyzing expected and actual results. -PISA was originally developed to cope with low-statistics Monte Carlo (MC) by using parameterizations of the MC and operate on histograms of the data rather than directly reweighting the MC. However, PISA's methods apply equally well to high-MC situations, and PISA also performs traditional reweighted-MC analysis as well. +PISA was originally developed to cope with low-statistics MC by using parameterizations of the MC and operate on histograms of the data rather than directly reweighting the MC. However, PISA's methods apply equally well to high-MC situations, and PISA also performs traditional reweighted-MC analysis as well. If you use PISA, please cite our publication ([e-Print available here](https://arxiv.org/abs/1803.05390)): ``` @@ -36,13 +36,7 @@ Published in: Nucl.Instrum.Meth.A 977 (2020) 164332 ## Installation -```shell -git clone git@github.com:icecube/pisa.git -cd pisa -pip install -e . -``` - -For detailed installation instructions and common issues see [Installation](INSTALL.md) +For quick-start and detailed installation instructions see [Installation](INSTALL.md). ## Minimal Example @@ -64,14 +58,14 @@ Instantiate a `Pipeline` or multiple pipelines in a `DistributionMaker` using PI template_maker = Pipeline("settings/pipeline/osc_example.cfg") ``` -Run the pipleine with nominal settings +Run the pipeline with nominal settings ```python template_maker.run() ``` -Get the oscillation probabilities +Get the oscillation probabilities $P_{\nu_\mu\to\nu_\beta}$ ```python From 0de26be42e80ac807c376118fec7021e08c731c9 Mon Sep 17 00:00:00 2001 From: T Ehrhardt Date: Mon, 25 Nov 2024 12:56:45 +0100 Subject: [PATCH 20/20] move note and fix wget url --- INSTALL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 435dd0654..429e4e4b7 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -70,13 +70,13 @@ mkdir -p $PISA Below we describe two different ways of setting up the PISA Python environment:
+> [!NOTE] +> Miniforge or the venv virtual environment folder into which all Python packages will be installed can be placed wherever you please. (Note that there are [disadvantages](https://pybit.es/articles/a-better-place-to-put-your-python-virtual-environments/) to putting either in your local PISA repository's top-level directory `$PISA`.) However, on one of IceCube's Cobalt nodes, again consider `/data/user/` instead of e.g. `$HOME`. + The [first (default)](#default-miniforge-distribution) obtains Python and Python packages, as well as any non-Python binary libraries upon which many Python libraries rely, from the [Miniforge](https://conda-forge.org/docs/user/introduction/) distribution. This makes it ideal for setup on e.g. clusters, but also works well for your personal computer.
The [second (alternative)](#alternative-cvmfs-and-venv) assumes you have access to IceCube's CernVM-FS (CVMFS) repository and would like to use one of the Python installations it provides as the "base" of a [venv](https://docs.python.org/3/library/venv.html). Our instructions have only been tested for the [`py3-v4.2.1` distribution](https://docs.icecube.aq/icetray/main/info/cvmfs.html#py3-v4-2). -> [!NOTE] -> Miniforge or the venv virtual environment folder into which all Python packages will be installed can be placed wherever you please. (Note that there are [disadvantages](https://pybit.es/articles/a-better-place-to-put-your-python-virtual-environments/) to putting either in your local PISA repository's top-level directory `$PISA`.) However, on one of IceCube's Cobalt nodes, again consider `/data/user/` instead of e.g. `$HOME`. - ### Default: Miniforge distribution Install the latest Miniforge Python distribution for either Mac or Linux (as your user, _not_ as root) from https://conda-forge.org/download/. @@ -85,7 +85,7 @@ Install the latest Miniforge Python distribution for either Mac or Linux (as you ```bash mkdir -p /miniforge3 - wget "https://.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" + wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" bash "Miniforge3-$(uname)-$(uname -m).sh" -p /miniforge3 -u rm "Miniforge3-$(uname)-$(uname -m).sh" ```