Added documentation and review suggestions

- Add dependencies and entries points to conda meta.yaml
- Add usage documentation
- Abort adding metadata in checkout if branch does not have config.yaml, add error message and test
- Extend payu branch to list if branch has a config file
- Extend payu branch to list remote branches, added —remote flag to cmd, added test
- Move chdir into a try-finally block when cloning.
- Tidy test_branch.py, added functions for common tests, add tests for previous uuid

conda/meta.yaml

@@ -12,6 +12,9 @@ build:
         - payu-collate = payu.subcommands.collate_cmd:runscript
         - payu-profile = payu.subcommands.profile_cmd:runscript
         - payu-sync = payu.subcommands.sync_cmd:runscript
+        - payu-branch = payu.subcommands.branch_cmd:runscript
+        - payu-clone = payu.subcommands.clone_cmd:runscript
+        - payu-checkout = payu.subcommands.checkout_cmd:runscript
 
 source:
     git_url: ../
@@ -33,6 +36,9 @@ requirements:
         # extra for the pypi package
         - pyOpenSSL >=0.14
         - cryptography>=1.3.4
+        - shortuuid >= 1.0.11
+        - GitPython >= 3.1.40
+        - ruamel.yaml >= 0.18.5
 
 test:
     imports:

docs/source/usage.rst

@@ -21,6 +21,49 @@ simultaneously that can share common executables and input data. It also
 allows the flexibility to have the relatively small control directories
 in a location that is continuously backed up.
 
+Experiment names and metadata
+-----------------------------
+
+The *laboratory* contains the following subdirectories:
+* ``work``, which is where the model is run. This contains a temporary directory
+for each experiment which is removed after a successful run.
+* ``archive``, which is where the output of completed runs is stored for each 
+experiment. 
+
+An experiment name is used identify the experiment inside the ``work`` and 
+``archive`` sub-directories. This is stored in a metadata file, 
+``metadata.yaml``, in the *control directory*. 
+The experiment name and a UUID, to uniquely identify the experiment, 
+is set in ``metadata.yaml`` when:
+* Using payu to clone a pre-existing git_ repository of the *control directory*
+* Using payu to create and checkout a new git branch in the *control directory*
+* Or, it is set automatically when setting up an experiment run if there is 
+not a pre-existing metadata file.
+
+The experiment name historically would default to the name of the *control 
+directory* or the configured ``experiment`` value (see :ref:`config`). This is 
+still supported for experiments with a pre-existing archived outputs. To support 
+git branches and ensure uniqueness in shared archives, the branch name and 
+a short version of the experiment UUID are added to new experiment names. 
+For example, given a control directory named ``my_expt`` and a UUID of 
+``9fAsTc4sNYsH2ZBQGYK9TG``, the experiment name would be:
+* ``my_expt-perturb-9fAsT`` - if running an experiment on a branch named 
+``perturb``.
+* ``my_expt-9fAsT`` - if the control directory was not a git repository or 
+running experiments from the ``main`` or ``master`` branch.
+* or ``my_expt`` - if running an older experiment that has a pre-existing 
+archive.
+
+Using a git repository for the experiment
+-----------------------------------------
+
+It is recommended to use version control using git_ for the payu 
+*control directory*. This allows the experiment to be easily copied via 
+cloning. There is inbuilt support in payu for an experiment runlog which 
+tracks changes to files between experiment runs. There are payu commands 
+for creating and moving between git branches so multiple related experiments 
+can be run from the same control directory.
+
 Setting up the laboratory
 =========================
 
@@ -85,22 +128,45 @@ Populate laboratory directories
 
    You will want a unique name for each input directory.
 
+
 Clone experiment
------------------
+----------------
 
-The payu control directory is maintained under version control using 
-git_ so existing experiments can be cloned. This is the best way to copy
-an experiment as it guarantees that only the required files are copied
-to a new control directory, and maintains a link to the original 
-experiment through the shared git history.
+This is the best way to copy an experiment as it guarantees that only the 
+required files are copied to a new control directory, and maintains a link 
+to the original experiment through the shared git history. To clone the 
+repository, you can use ``git clone`` or ``payu clone`` which is a wrapper 
+around ``git clone`` which additionally creates or updates the metadata file.
 
 For example::
     
       mkdir -p ${HOME}/${MODEL}
       cd ${HOME}/${MODEL}
-      git clone https://github.com/payu-org/mom-example.git my_expt
+      payu clone ${REPOSITORY} my_expt
       cd my_expt
 
+Where ``${REPOSITORY}`` is the git URL or path of the repository to clone from, 
+for example, https://github.com/payu-org/mom-example.git.
+
+To clone and checkout an existing git branch, use the ``--branch`` flag and 
+specify the branch name::
+
+      payu clone --branch ${EXISTING_BRANCH} ${REPOSITORY} my_expt
+
+To create and checkout a new git branch use ``--new-branch`` and specify a 
+new branch name:
+
+      payu clone --new-branch ${NEW_BRANCH} ${REPOSITORY} my_expt
+
+To see more configuration options for ``payu clone``, 
+run:: 
+
+      payu clone --help
+
+Alternatively to creating and checking out branches in ``payu clone``, 
+``payu checkout`` can be used instead (see :ref:`Switching between 
+related experiments`). 
+
 
 Create experiment
 -----------------
@@ -305,3 +371,49 @@ at a later date. To sync all restarts including the latest restarts, use the
 ``--sync-restarts`` flag::
 
    payu sync  --sync-restarts
+
+
+Switching between related experiments
+=====================================
+
+To be able to run related experiments from the same control directory 
+using git branches, you can use ``payu checkout`` which is a wrapper around 
+``git checkout``. Creating new branches will generate a new UUID and 
+branch-UUID-aware experiment name in the metadata file. 
+Switching branches will change ``work`` and ``archive`` symlinks in the control 
+directory to point to directories in *laboratory* if they exist.
+
+To create a git branch for a new experiment, use the ``-b`` flag. 
+For example, to create and checkout a new branch called ``perturb1``, run::
+
+      payu checkout -b perturb1
+
+To branch a new experiment from an existing branch, specify the branch name 
+or a commit hash after the new branch name. For example, 
+the following creates a new experiment branch called ``perturb2`` 
+that starts from ``perturb1``:: 
+
+      payu checkout -b perturb2 perturb1
+
+To specify a restart path to start from using the ``--restart``/ ``-r`` flag, 
+for example::
+
+      payu checkout -b perturb --restart path/to/restart
+
+Note: This can also be achieved by configuring ``restart`` (see :ref:`config`).
+
+To checkout an existing branch and experiment. For example, 
+the following checks out the ``perturb1`` branch:: 
+
+      payu checkout perturb1
+
+To see more ``payu checkout`` options, run::
+
+      payu checkout --help
+
+For more information on git branches that exist in the control directory 
+repository, run::
+
+      payu branch # Display local branches UUIDs
+      payu branch --verbose # Display local branches metadata 
+      payu branch --remote # Display remote branches UUIDs