Skip to content

Commit

Permalink
Merge pull request #311 from ashwinvis/fix-packaging-2024
Browse files Browse the repository at this point in the history
Fix packaging episode
  • Loading branch information
bast authored Nov 6, 2024
2 parents 2846bc1 + c278949 commit 8fe9a52
Show file tree
Hide file tree
Showing 2 changed files with 94 additions and 28 deletions.
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,7 @@ help:
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

# Live reload site documents for local development
livehtml:
sphinx-autobuild "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
118 changes: 90 additions & 28 deletions content/packaging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -140,15 +140,17 @@ Sharing packages via PyPI
Once we are able to pip-install the example package locally, we are ready for
upload.

We exercise by uploading to `test-PyPI <https://test.pypi.org/>`__, not the
We exercise by uploading to test-PyPI_, not the
real `PyPI <https://pypi.org/>`__, so that if we mess things up, nothing bad
happens.

We need two more things:

- We will do this using `Twine <https://twine.readthedocs.io/>`__ so you need
to pip install that, too.
- You need an account on `test-PyPI <https://test.pypi.org/>`__.
- You need an account on test-PyPI_

.. _test-PyPI: https://test.pypi.org/

.. highlight:: console

Expand All @@ -165,13 +167,64 @@ And use twine to upload the distribution files to test-PyPI::
$ twine upload -r testpypi dist/*

Uploading distributions to https://test.pypi.org/legacy/
Enter your username:
Enter your password:
Enter your API token:


.. _Create API token: https://test.pypi.org/manage/account/token/

.. note::

To generate an API token, proceed to the `Create API token`_ page in test-PyPI.
You will be prompted for your password.

.. solution:: The long-version for finding the *Create API token* page

1. Log on to test-PyPI_ at https://test.pypi.org
2. In the top-right corner, click on the drop-down menu and click **Account settings** or
follow this `link <https://test.pypi.org/manage/account/#api-tokens>`__.
3. Scroll down to the section **API tokens** and click the button **Add API token**,
which opens up the
`Create API token`_ page.


#. Under **Token name** write something memorable.
It should remind you the *purpose* such as
or the *name of the computer* so you can delete it later.
#. Under **Scope** select ``Entire account (all projects)``.
#. Click on **Create token**.
#. Click on **Copy token** once a long string which starts
with ``pypi-`` is generated.

Once this is done, create yet another virtual environment and try to install from test-PyPI (adapt "myname")::
Paste that token back into terminal where ``twine upload ...`` is running and press ENTER.

$ pip install -i https://test.pypi.org/simple/ calculator-myname
Once this is done, create yet another virtual environment and try to install from test-PyPI (adapt ``myname``).

.. tabs::

.. tab:: Linux / macOS

.. code-block:: console
:emphasize-lines: 4-7
$ python3 -m venv venv-calculator
$ source venv-calculator/bin/activate
$ which python
$ python3 -m pip install \
-i https://test.pypi.org/simple/ \
--extra-index-url https://pypi.org/simple/ \
calculator-myname
$ deactivate
.. tab:: Windows

.. code-block:: console
:emphasize-lines: 4
$ python3 -m venv venv-calculator
$ venv-calculator\Scripts\activate
$ where python
$ python3 -m pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ calculator-myname
$ deactivate
Tools that simplify sharing via PyPI
------------------------------------
Expand All @@ -188,6 +241,8 @@ There are at least two tools which try to make the packaging and PyPI interactio
- `Poetry <https://python-poetry.org/>`__
- `Flit <https://flit.pypa.io/>`__

If you upload packages to PyPI or test PyPI often you can create an API token and
`save it in the .pypirc file <https://packaging.python.org/en/latest/specifications/pypirc/#common-configurations>`__.

Building a conda package and share it
-------------------------------------
Expand All @@ -199,42 +254,44 @@ Building a conda package and share it

.. callout:: Prerequisites

To create a conda package, `conda-build` package is required. You may install it with **Anaconda Navigator** or from the command line::
To generate a conda build recipe, the package ``grayskull`` and
to build it, the package ``conda-build`` are required.
You may install these with **Anaconda Navigator** or from the command line::

$ conda install conda-build
$ conda install -n base grayskull conda-build


The simplest way for creating a conda package for your python script is to
first publish it in `PyPI <https://pypi.org/>`__ following the steps explained
above.


Building a python package with conda skeleton pypi
***************************************************
Building a python package with grayskull and conda-build
********************************************************

Once build, the conda package can be installed locally. For this example, we
will use `runtest <https://pypi.org/project/runtest/>`__. `runtest
<https://github.com/bast/runtest>`__ is a numerically tolerant end-to-end test
library for research software.

1. Create pypi skeleton::
1. Generate the *recipe* by executing (``grayskull`` or ``conda grayskull``)::

$ conda skeleton pypi runtest
$ conda grayskull pypi runtest

The command above will create a new folder called `runtest` containing a file `meta.yaml`, the conda recipe for `runtest`.
The command above will create a new folder called `runtest` containing a file `meta.yaml`,
the conda recipe for building the `runtest` package.

2. Edit `meta.yaml` and update requirements:
2. View the contents of `meta.yaml` and ensure requirements :

.. code-block:: yaml
requirements:
host:
- pip
- python
- flit
- flit-core >=2,<4
- pip
run:
- python
- flit
In the requirements above, we specified what is required for the `host <https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#host>`__ and for `running <https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#run>`__ the package.

Expand All @@ -248,7 +305,7 @@ library for research software.

Your package is now ready to be build with conda::

$ conda-build runtest
$ conda build runtest


.. callout:: Conda package location
Expand All @@ -257,11 +314,15 @@ library for research software.

.. code-block:: none
~/anaconda3/conda-bld/win-64/runtest-2.2.1-py38_0.tar.bz2
/home/username/miniforge3/conda-bld/noarch/runtest-2.3.4-py_0.tar.bz2
The prefix `~/anaconda3/` may be different on your machine and depending on your operating system (Linux, Mac-OSX or Windows) the sub-folder `win-64` differs too (for instance `linux-64` on Linux machines).
The prefix ``/home/username/miniforge3/`` may be different on your machine.
depending on your operating system (Linux, Mac-OSX or Windows). The sub-folder is named ``noarch`` since
it is a pure-python package and the recipe indicates the same.

The conda package we have created is specific to your platform (here `win-64`). It can be converted to other platforms using `conda convert <https://docs.conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html#converting-a-package-for-use-on-all-platforms>`__.
If package contained compiled code then the sub-folder would have been named ``win-64`` or ``linux-64``.
It could then be converted to other platforms using
`conda convert <https://docs.conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html#converting-a-package-for-use-on-all-platforms>`__.

4. Check within new environment

Expand All @@ -286,7 +347,10 @@ library for research software.
.. callout:: Building a conda package from scratch

It is possible to build a conda package from scratch without using conda skeleton. We recommend you to check the `conda-build documentation <https://docs.conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html>`__ for more information.
It is possible to build a conda package from scratch without using conda grayskull.
We recommend you to check the
`conda-build documentation <https://docs.conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html>`__
for more information.

To be able to share and install your local conda package anywhere (on other platforms), you would need to upload it to a `conda channel <https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html>`__ (see below).

Expand All @@ -295,15 +359,13 @@ To be able to share and install your local conda package anywhere (on other plat
Publishing a python package
***************************

- Upload your package to *Anaconda.org*: see instructions `here
<https://docs.conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs-skeleton.html#optional-uploading-packages-to-anaconda-org>`__.
Please note that you will have to create an account on Anaconda.

- Upload your package to `conda-forge <https://conda-forge.org/>`__:
conda-forge is a conda channel: it contains community-led collection of
recipes, build infrastructure and distributions for the conda package
manager. Anyone can public conda packages to conda-forge if certain
`guidelines <https://conda-forge.org/docs/>`__ are respected.
manager. Anyone can
`publish conda packages to conda-forge <https://conda-forge.org/docs/maintainer/adding_pkgs/>`__
if certain
`guidelines <https://conda-forge.org/docs/maintainer/guidelines/>`__ are respected.

- Upload your package to `bioconda <https://bioconda.github.io/>`_: bioconda is
a very popular channel for the conda package manager specializing in
Expand Down

0 comments on commit 8fe9a52

Please sign in to comment.