Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation #56

Merged
merged 28 commits into from
Jun 26, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
38ffa75
Restructure sphinx documentation creation
giffels May 9, 2019
cb8cb93
Add TARDIS logo in png format
giffels May 9, 2019
437f450
Enable readthedocs creation
giffels May 9, 2019
b8d3cf3
Add batch system adapter documentation
giffels May 28, 2019
2c5d467
Add contribute, site and plugin documentation
giffels May 29, 2019
be8f671
Adding site and plugins sections. Added documentation for FakeBatchSy…
giffels Jun 3, 2019
999a1f9
Change year to 2019
giffels Jun 5, 2019
15c55ce
Adding documentation for HTCondor batch system adapter
giffels Jun 5, 2019
ce2530a
Add documentation for telegraf monitoring plugin
giffels Jun 7, 2019
cadfdb1
Move to cobald-tardis readthedocs.org
giffels Jun 18, 2019
65adf93
Add Python Version badge
giffels Jun 21, 2019
9dcff3a
Fix htcondor utilisation/allocation definition
giffels Jun 24, 2019
da12055
Add autogen documentation for fakesite and simulators
giffels Jun 24, 2019
40d7d58
Change project to cobald-tardis and adjust release version
giffels Jun 24, 2019
f4c5588
Bump version to 0.2.0
giffels Jun 26, 2019
d67247f
Add documentation for SQlite and plugins in general
giffels Jun 26, 2019
143ca36
Satisfy yamllinter
giffels Jun 26, 2019
b0c2a73
Add in progress to contribute docu
giffels Jun 26, 2019
23e548f
Add documentation for the site adapter
giffels Jun 26, 2019
8467d8d
Add reference point to batchsytem adapter docu
giffels Jun 26, 2019
bf43a26
Add doucmentation for executors
giffels Jun 26, 2019
125b223
Update docs/source/adapters/batchsystem.rst
giffels Jun 26, 2019
386fad5
Update tardis/adapters/batchsystems/fakebatchsystem.py
giffels Jun 26, 2019
82e5714
Update tardis/plugins/sqliteregistry.py
giffels Jun 26, 2019
90892d0
Update tardis/adapters/batchsystems/htcondor.py
giffels Jun 26, 2019
77a7776
Replace optionality with requirement
giffels Jun 26, 2019
1dbc0e9
Fix whitespace
giffels Jun 26, 2019
16cc520
Update docs/source/plugins/plugins.rst
giffels Jun 26, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
python:
version: 3
setup_py_install: true
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[![Build Status](https://travis-ci.org/MatterMiners/tardis.svg?branch=master)](https://travis-ci.org/MatterMiners/tardis)
[![codecov](https://codecov.io/gh/MatterMiners/tardis/branch/master/graph/badge.svg)](https://codecov.io/gh/MatterMiners/tardis)
[![Documentation Status](https://readthedocs.org/projects/cobald-tardis/badge/?version=latest)](https://cobald-tardis.readthedocs.io/en/latest/?badge=latest)
[![PyPI version](https://badge.fury.io/py/cobald-tardis.svg)](https://badge.fury.io/py/cobald-tardis)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/cobald-tardis.svg?style=flat-square)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/MatterMiners/tardis/blob/master/LICENSE.txt)
[![DOI](https://zenodo.org/badge/132791417.svg)](https://zenodo.org/badge/latestdoi/132791417)

Expand Down
4 changes: 2 additions & 2 deletions docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
#

# You can set these variables from the command line.
SPHINXOPTS =
SPHINXOPTS = -d _build/doctrees
SPHINXBUILD = sphinx-build
SPHINXPROJ = tardis
SOURCEDIR = source
BUILDDIR = build
BUILDDIR = _build

# Put it first so that "make" without argument is like "make help".
help:
Expand Down
7 changes: 6 additions & 1 deletion docs/generate_apidoc.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,7 @@
#!/bin/bash
sphinx-apidoc --force --module-first --separate -o source/ ../tardis
PROJECT=tardis

source ../venv/bin/activate
sphinx-apidoc --force --module-first --separate -o source/api ../${PROJECT} && \
make html && \
open _build/html/index.html
Binary file added docs/pics/TARDIS_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
115 changes: 115 additions & 0 deletions docs/source/adapters/batchsystem.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
.. _ref_batch_system_adapter:

=====================
Batch System Adapters
=====================

Fake Batch System Adapter
-------------------------

.. |FakeBatchSystemAdapter.get_utilization| replace:: :py:meth:`~tardis.adapters.batchsystems.fakebatchsystem.FakeBatchSystemAdapter.get_utilization`

.. |FakeBatchSystemAdapter.get_allocation| replace:: :py:meth:`~tardis.adapters.batchsystems.fakebatchsystem.FakeBatchSystemAdapter.get_allocation`

.. |FakeBatchSystemAdapter.get_machine_status| replace:: :py:meth:`~tardis.adapters.batchsystems.fakebatchsystem.FakeBatchSystemAdapter.get_machine_status`

The :py:class:`~tardis.adapters.batchsystems.fakebatchsystem.FakeBatchSystemAdapter`
implements a batch system adapter that mocks the response of hypothetical batch system.
giffels marked this conversation as resolved.
Show resolved Hide resolved
It can be used for testing purposes and as a demonstrator in workshops and tutorials.

The mocked response to the |FakeBatchSystemAdapter.get_allocation|, |FakeBatchSystemAdapter.get_utilization| and
|FakeBatchSystemAdapter.get_machine_status| API calls is configurable statically in the adapter configuration.

Available configuration options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+----------------+---------------------------------------------------------------------+-----------------+
| Option | Short Description | Requirement |
+================+=====================================================================+=================+
| adapter | Name of the adapter (FakeBatchSystem) | **Required** |
+----------------+---------------------------------------------------------------------+-----------------+
| allocation | Mocked response to |FakeBatchSystemAdapter.get_allocation| call | **Required** |
+----------------+---------------------------------------------------------------------+-----------------+
| utilization | Mocked response to |FakeBatchSystemAdapter.get_utilization| call | **Required** |
+----------------+---------------------------------------------------------------------+-----------------+
| machine_status | Mocked response to |FakeBatchSystemAdapter.get_machine_status| call | **Required** |
+----------------+---------------------------------------------------------------------+-----------------+

Example configuration
~~~~~~~~~~~~~~~~~~~~~

.. code-block:: yaml

BatchSystem:
adapter: FakeBatchSystem
allocation: 1.0
utilization: 1.0
machine_status: Available

HTCondor Batch System Adapter
-----------------------------
.. |HTCondorAdapter.get_utilization| replace:: :py:meth:`~tardis.adapters.batchsystems.htcondor.HTCondorAdapter.get_utilization`

.. |HTCondorAdapter.get_allocation| replace:: :py:meth:`~tardis.adapters.batchsystems.htcondor.HTCondorAdapter.get_allocation`

.. |HTCondorAdapter.get_machine_status| replace:: :py:meth:`~tardis.adapters.batchsystems.htcondor.HTCondorAdapter.get_machine_status`

The :py:class:`~tardis.adapters.batchsystems.htcondor.HTCondorAdapter` implements the TARDIS interface to dynamically
integrate and manage opportunistic resources with the HTCondor batch system.

Information provider for the API calls |HTCondorAdapter.get_utilization|, |HTCondorAdapter.get_allocation| and
|HTCondorAdapter.get_machine_status| is the HTCondor ``condor_status`` command, which is called asynchronously and its
output is cached for a configurable time ``max_age``.

|HTCondorAdapter.get_machine_status| returns the status of the worker node by taking into account the HTCondor
ClassAds ``State`` and ``Activity``. It can take the states ``Available``, ``Draining``, ``Drained`` and
``NotAvailable``.

The allocation and utilization of a worker node is defined as maximum and minimum of the relative ratio of requested
over total resources such as CPU, Memory, Disk, respectively. Which resource ratios to take into account can be
configured via the ``ratios`` option. Any valid HTCondor expression that returns a floating point number is accepted.

Additional options for the condor_status call can be added by using the ``options`` option.

For example

.. code-block:: yaml

options:
pool: htcondor.example

translates into ``condor_status ... -pool htcondor.example``.

Available configuration options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+----------------+-------------------------------------------------------------------------+-----------------+
| Option | Short Description | Requirement |
+================+=========================================================================+=================+
| adapter | Name of the adapter (HTCondor) | **Required** |
+----------------+-------------------------------------------------------------------------+-----------------+
| max_age | Maximum age of the cached ``condor_status`` information in minutes | **Required** |
+----------------+-------------------------------------------------------------------------+-----------------+
| ratios | HTCondor expressions used to determine allocation and utilization | **Required** |
+----------------+-------------------------------------------------------------------------+-----------------+
| options | Additional command line options to add to the ``condor_status`` command | **Optional** |
+----------------+-------------------------------------------------------------------------+-----------------+


Example configuration
~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: yaml

BatchSystem:
adapter: HTCondor
max_age: 1
ratios:
cpu_ratio: Real(TotalSlotCpus-Cpus)/TotalSlotCpus
memory_ratio: Real(TotalSlotMemory-Memory)/TotalSlotMemory
options:
pool: my-htcondor.mysite.mydomain

Your favorite batch system is currently not supported?
Please, have a look at
:ref:`how to contribute.<ref_contribute_batch_system_adapter>`
Loading