DIST/DEV: Improve method of building packages in development

[asoplata]

For our eventually-automated package building, George recommended that we use a modern python package manager like poetry, uv, or pdm. Note that is has NO impact on the user’s install, and instead is only for making things more convenient for us developers. See details below.

Note that this is related to, but not the same as, #950 . That said, we can make the changes within this current Issue without requiring the changes in #950 .

Subtasks

• Investigate uv, pdm, poetry, and others
• Test building a simple package with whatever we choose
• Incorporate the builder package into Github Actions that automate the release process (George’s meeting notes below are especially helpful for this)
• Setup “Trusted Publishers” for better security (no personal Pypi API tokens, etc.)

Notes

• George specifically provided very useful information to Dylan and Austin about this in this meeting: https://www.notion.so/jonescompneurolab/HNN-Core-Improvements-Meeting-with-George-Dang-abc4a36e4c844b38ba81da39122ab35a
• Notably, poetry is popular and has been around the longest, BUT uses a nonstandard pyproject.toml (pdm's documentation points this out here https://github.com/pdm-project/pdm?tab=readme-ov-file#poetry ). Austin thinks we should avoid it for this reason alone, and is particularly in favor of uv.
• This has many benefits:
  • Improved speed when installing locally (who cares)
  • …improved speed when installing which is also relevant for our CI runners! As of 2024-11-21, each testing CI run uses, very loosely, around 20% of its runtime just installing hnn-core. This may be significantly reduced if we use something like uv .
  • These can also be used to manage your Python versions AND virtual environments, reducing the need to rely on an external tool like conda or pyenv etc. For a new developer, we may be able to get their environment setup in very few commands (2?).
  • Enables use of lockfiles and similar methods at controlling consistency between developer installs. This way, all developers can both
    1. trust that they are using the same versions of dependencies as other developers by default, but also
    2. test against an explicit version change in a dependency if there is an incompatibility, and use this same method to communicate the dependency version change itself to other devs
  • To be clear and verbose, this would also replace our use of dependencies for the building itself, such as replacing twine https://github.com/astral-sh/uv?tab=readme-ov-file#highlights
  • We can merge our independent doc building pipeline using the same tooling we use to create our actually packaged code. We would no longer need 2 different Makefiles that are managed differently.

[jasmainak]

I have used poetry in the past and did not like it :) The builds just take too long and get hung, but maybe it has improved. Again, same message ... look around what other repos in the python scientific ecosystem are doing and copy !