Skip to content
This repository has been archived by the owner on Sep 21, 2021. It is now read-only.

Dropwizard082 basiccallbak #1

Open
wants to merge 80 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
640fd1f
Added embeddable build status icon.
sverhoeven May 27, 2013
501113c
Remove build status as jenkins server is not attached to Internet yet.
sverhoeven May 27, 2013
0224793
CopyOption was moved.
sverhoeven May 27, 2013
c347735
Add test for hashCode of config class.
sverhoeven May 29, 2013
3c3aa4c
Add license plugin to generate LICENSE file and license header in files.
sverhoeven May 30, 2013
3eba104
Fix JavaDoc warnings.
sverhoeven May 30, 2013
5ae993d
Octopus has upped jsch version. We do same.
sverhoeven Jun 4, 2013
743c4c5
Job list failed for multiple jobs.
sverhoeven Jun 5, 2013
7130c0c
Download sandbox even when job has an exception.
sverhoeven Jun 5, 2013
8897d03
Make clearer that status callback url is optional.
sverhoeven Jun 5, 2013
10b320c
Octopus forgets about job after it has been cancelled.
sverhoeven Jun 5, 2013
c3225ad
Sandbox has equals so remove ignore tag from test.
sverhoeven Jun 5, 2013
5430287
Remove unused imports.
sverhoeven Jun 10, 2013
300f165
Merge remote-tracking branch 'origin/develop' into develop
sverhoeven Jun 10, 2013
bc82ab4
Make repo in NLeSC organisation the golden standard. Remove repo at h…
sverhoeven Jun 20, 2013
1c90850
Octopus jobs no longer have an UUID.
sverhoeven Jun 26, 2013
5e94a35
Merge remote-tracking branch 'upstream/develop' into develop
sverhoeven Jun 26, 2013
eaf8372
Replaced usage of uuid with simple string, to prevent expectations th…
sverhoeven Jun 27, 2013
9a743ca
Add stop method to poller which will clean up the sandboxes of all p…
sverhoeven Jun 27, 2013
9c7914e
Add test to reproduce the bug that a pending job can not be cancelled.
sverhoeven Jun 28, 2013
05f1455
octopus.end() has been deprecated, replaced with OctopusFactory.endOc…
sverhoeven Jul 16, 2013
23fd2ed
Merge branch 'develop' into issue2
sverhoeven Jul 25, 2013
f775c7b
Octopus has been fixed so pending jobs can be killed.
sverhoeven Jul 30, 2013
adf4c73
Octopus has replaced Properties class usage with Map<String, String>.
sverhoeven Jul 30, 2013
8cc2245
Removed several Sonar issues.
sverhoeven Jul 30, 2013
0f4d3ca
Octopus JobDescription now has no default stdout, stderr so specify i…
sverhoeven Aug 12, 2013
e73f1e3
Added tests.
sverhoeven Aug 12, 2013
0e497e1
Made test for cancelling a pending job a integration test as it uses …
sverhoeven Aug 14, 2013
70acaea
Octopus renamed AbsolutePath to Path and RelativePath to Pathname.
sverhoeven Aug 14, 2013
6d29b0f
Path.resolve() was moved to Pathname.resolve(). New paths must be mad…
sverhoeven Aug 23, 2013
8b8b22d
Creation of schedulers and filesystems don't use URI anymore.
sverhoeven Sep 3, 2013
9e55d1e
Add license header to new files.
sverhoeven Sep 3, 2013
df2134c
Octopus renamed PathName to RelativePath.
sverhoeven Sep 9, 2013
9f54e24
Octopus exceptions where refactored.
sverhoeven Sep 12, 2013
2085b2f
Only bind to localhost
sverhoeven Sep 18, 2013
28e0fcb
Use only one port
sverhoeven Sep 18, 2013
d297e42
Renamed Octopus to Xenon.
sverhoeven Sep 23, 2013
ac01874
Merge remote-tracking branch 'origin/develop' into octopus2xenon
sverhoeven Sep 23, 2013
c245a8f
Add logos.
sverhoeven Sep 23, 2013
aea8825
Replaced last remnants of Octopus.
sverhoeven Sep 24, 2013
e85eae3
Make use of Xenon library from repository.
sverhoeven Sep 25, 2013
11f756b
Git rid of Xenon instructions as Xenon can be downloaded from repo now.
sverhoeven Sep 25, 2013
53ff672
Fix nested list
sverhoeven Sep 25, 2013
e2314b8
Add travis.yml
sverhoeven Sep 25, 2013
2dc9b34
Merge branch 'develop' of github.com:NLeSC/osmium into develop
sverhoeven Sep 25, 2013
9ed580b
Missing location for scheduler in example config.
sverhoeven Sep 25, 2013
332e058
mvn package made jar which is not executable.
sverhoeven Sep 30, 2013
d5ee94a
Got 'java.lang.NoClassDefFoundError: org/sonatype/aether/*'.
sverhoeven Oct 16, 2013
a222a29
Use newer version of powermock
sverhoeven Jan 1, 2014
7d2e141
Get Osmium to compile cleanly
blootsvoets Nov 13, 2014
97fedaf
Removed comments from JSON test files
blootsvoets Nov 13, 2014
280e613
Fix Javadoc 8 errors.
sverhoeven Nov 13, 2014
c5848cc
Merge pull request #7 from blootsvoets/jorisedits
sverhoeven Nov 13, 2014
f0783eb
Removed javadoc 8 workaround, as javadoc is has been made valid in #8.
sverhoeven Nov 13, 2014
ad69803
Added environment variables to Osmium
blootsvoets Feb 9, 2015
b118736
Variable ordering of JobSubmitRequest consistent
blootsvoets Feb 9, 2015
f4c2e44
Test cases take into account environment
blootsvoets Feb 9, 2015
01359c1
Merge pull request #10 from blootsvoets/feature/env_vars
sverhoeven Feb 9, 2015
645d3a0
Style modifications
blootsvoets May 18, 2015
8ec0750
Allow minimal job requests
blootsvoets May 18, 2015
9f82e13
Support for multiple schedulers
blootsvoets May 20, 2015
fe29661
Merge branch 'feature/multiplehosts' into develop
blootsvoets May 20, 2015
d5ee44a
New slimmer banner, fixes #12
sverhoeven May 21, 2015
6b2659a
Multiple launchers: Backwards INCOMPATIBLE (#11)
blootsvoets May 21, 2015
9294c23
Merge branch 'develop' of github.com:NLeSC/osmium into develop
blootsvoets May 21, 2015
e71ad41
Documented launcher for requests in README
blootsvoets May 21, 2015
4a3280d
Fixed reference mistake jobs -> jjobs
blootsvoets May 21, 2015
50ce6e0
Fixed site refs (Magma->Osmium, Xenon->Osmium)
blootsvoets May 21, 2015
96343a1
Findbugs 3.0.1, for recent Java versions
blootsvoets May 21, 2015
1bc5cd5
Test with Java 8
blootsvoets Jul 8, 2015
9ef40aa
openjdk8 not recognized by Travis
blootsvoets Jul 8, 2015
a38b869
Fixed mvn warnings and testing Java 6
blootsvoets Jul 8, 2015
1c45d6b
Consistent spacing pom.xml
blootsvoets Jul 8, 2015
4b9d48a
Explicitly Java 6 is not supported; Java 8 is.
blootsvoets Jul 8, 2015
8a762e1
Enable osmium admin interface by default
blootsvoets Jul 13, 2015
a80e125
Spelling and imports
blootsvoets Jul 13, 2015
20ca7ba
Accept max_time property in JobSubmitRequest
blootsvoets Jul 16, 2015
374c168
Started on updating dropwizard.
sverhoeven Aug 12, 2015
ef80f42
Added tests and performed lived tests. Updated config files for new d…
sverhoeven Aug 18, 2015
b00330e
Merge remote-tracking branch 'origin/develop' into dropwizard082-basi…
sverhoeven Aug 20, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
language: java
jdk:
- oraclejdk7
- openjdk7
- oraclejdk8

6 changes: 4 additions & 2 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ This project includes:
Jetty :: Utilities under Apache Software License - Version 2.0 or Eclipse Public License - Version 1.0
Jetty Orbit :: Servlet API under Apache Software License - Version 2.0 or Eclipse Public License - Version 1.0
Joda time under Apache 2
JSch under Revised BSD
JUL to SLF4J bridge under MIT License
JUnit under Common Public License Version 1.0
Log4j Implemented Over SLF4J under Apache Software Licenses
Expand All @@ -74,8 +75,7 @@ This project includes:
Metrics Servlet under Apache License 2.0
Mockito under The MIT License
Objenesis under Apache 2
octopus under Apache License, Version 2.0
Octopus Job Webservice under Apache License, Version 2.0
Xenon under Apache License, Version 2.0
powermock-api-mockito under The Apache Software License, Version 2.0
powermock-api-support under The Apache Software License, Version 2.0
powermock-core under The Apache Software License, Version 2.0
Expand All @@ -84,3 +84,5 @@ This project includes:
powermock-reflect under The Apache Software License, Version 2.0
SLF4J API Module under MIT License



63 changes: 29 additions & 34 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,48 +1,42 @@
joblauncher
===========
Osmium
======

Web service to submit jobs via a Octopus supported scheduler.
|TravisCILink|_

.. |TravisCILink| image:: https://travis-ci.org/NLeSC/osmium.png
.. _TravisCILink: https://travis-ci.org/NLeSC/osmium

Web service to submit jobs via a Xenon (https://nlesc.github.io/Xenon) supported scheduler.

Submit job to web service using a HTTP POST request.
Reports status of job to submitter using a callback url.

Requirements
------------

- JDK 7 (http://www.java.com)
- JDK 7 or 8 (http://www.java.com)
- Maven 3 (http://maven.apache.org)
- Octopus (https://github.com/NLeSC/octopus)

Install
-------

1. Build Octopus
#. Make copy of 'joblauncher.yml-dist' to 'joblauncher.yml'

.. code-block:: bash
#. Configure Xenon scheduler
#. Configure Xenon sandbox root directory
#. Configure optional callback basic credentials

git clone https://github.com/NLeSC/octopus.git
cd octopus
ant build -Dversion=-1.0
2. Build uber-jar or execute from maven.

2. Add Octopus jar to local maven repository, so it can be used as maven dependencies

.. code-block:: bash

mvn install:install-file -Dfile=dist/octopus-1.0.jar -DartifactId=octopus -Dversion=1.0 -DgroupId=nl.esciencecenter.octopus -Dpackaging=jar -DgeneratePom=true

3. Make copy of 'joblauncher.yml-dist' to 'joblauncher.yml'
3.1 Configure Octopus scheduler
3.2 Configure Octopus sandbox root directory
3.3 Configure optional MAC id/key
4. Build uber-jar or execute from maven.
4.1. Uber-jar, to start on other machine the `octopus-job-ws-2.0.jar` and `joblauncher.yml` files must be copied.
2.1. Uber-jar, to start on other machine the `osmium-3.0.2.jar` and `joblauncher.yml` files must be copied.

.. code-block:: bash

mvn package
java -jar target/octopus-job-ws-2.0.jar server joblauncher.yml
java -jar target/osmium-3.0.2.jar server joblauncher.yml

4.2 Execute from maven

2.2. Execute from maven

.. code-block:: bash

Expand All @@ -67,14 +61,15 @@ Create a json file (query.json)
.. code-block:: json

{
"jobdir": "<absolute path to myjob directory>/",
"launcher": <optional non-default launcher name>,
"jobdir": "<absolute path to myjob directory>",
"executable": "/bin/sh",
"prestaged": ["runme.sh", "input_file"],
"poststaged": ["output_file"],
"stderr": "stderr.txt",
"stdout": "stdout.txt",
"arguments": ["runme.sh"],
"status_callback_url": "http://localhost/job/myjob/status"
"status_callback_url": "<optional url where job status is PUT to>"
}

The `status_callback_url` value is the url where the web service will send the job status with a http PUT request.
Expand All @@ -88,7 +83,7 @@ Then submit it

HTTP/1.1 201 Created
Date: Thu, 23 May 2013 11:50:28 GMT
Location: http://localhost:9998/job/bc66cd43-9d16-4270-813f-734e864e3552
Location: http://localhost:9998/job/local-1234
Content-Type: application/json
Content-Length: 0

Expand All @@ -98,8 +93,8 @@ The `Location` header value is the url where the job can be queried for it's sta
Callback authentication
^^^^^^^^^^^^^^^^^^^^^^^

The status callbacks uses MAC Access Authentication.
The MAC key indentifier and MAC key must be obtained from the provider.
The status callbacks can use basic http authentication.
The credentials must be added to the config file or can be given in the `status_callback_url` of the job submission request.

Status
^^^^^^
Expand All @@ -108,14 +103,15 @@ In the submit response the url is a relative url to the job.

.. code-block:: bash

curl -H "Content-Type: application/json" -H 'Accept: application/json' http://localhost:9998/job/bc66cd43-9d16-4270-813f-734e864e3552
curl -H "Content-Type: application/json" -H 'Accept: application/json' http://localhost:9998/job/local-1234

Example response when job is running:

.. code-block:: json

{
"request": {
"launcher": "local",
"jobdir": "/tmp/jobdir",
"executable": "/bin/sh",
"stderr": "stderr.txt",
Expand Down Expand Up @@ -145,6 +141,7 @@ Example response when job is done:

{
"request": {
"launcher": "local",
"jobdir": "/tmp/jobdir",
"executable": "/bin/sh",
"stderr": "stderr.txt",
Expand Down Expand Up @@ -174,6 +171,7 @@ Example response when job has been canceled (see below for cancel command):

{
"request": {
"launcher": "local",
"jobdir": "/tmp/myjob",
"status_callback_url": null,
"poststaged": [
Expand Down Expand Up @@ -208,7 +206,7 @@ Deletes any generated output in the sandbox where the job was running.

.. code-block:: bash

curl -H "Content-Type: application/json" -H 'Accept: application/json' -X DELETE http://localhost:9998/job/bc66cd43-9d16-4270-813f-734e864e3552
curl -H "Content-Type: application/json" -H 'Accept: application/json' -X DELETE http://localhost:9998/job/local-1234

Documentation
-------------
Expand All @@ -223,9 +221,6 @@ A maven site can be generated with
Integration tests
-----------------

Running the integration tests requires a configuration file called `src/test/resources/integration.props`.
An example configuration is available as `src/test/resources/integration.props-dist`.

Run integration tests with

.. code-block:: bash
Expand Down
44 changes: 33 additions & 11 deletions joblauncher.yml-dist
Original file line number Diff line number Diff line change
@@ -1,17 +1,39 @@
# Example configuration make copy of me
#macs:
#- id: <MAC key identifier>
# key: <MAC key>
# scope: <URL where MAC key should be used>
octopus:
scheduler: local:///
queue: multi
sandboxRoot: file:///tmp/
#callback:
# useInsecureSSL: false
# httpClient:
# keepAlive: "5m"
# basicCredentials:
# -
# username: "someone"
# password: "mypassword"
# hostname: "example.com"
# scheme: "https"
# port: 443
xenon:
launchers:
local:
scheduler:
scheme: local
location: /
queue: multi
sandbox:
scheme: file
location: /
path: /tmp
defaultLauncher: local
preferences:
octopus.adaptors.local.queue.multi.maxConcurrentJobs: 4
xenon.adaptors.local.queue.multi.maxConcurrentJobs: 4
poll:
interval: 30000 # 30 seconds
cancelTimeout: 3600000 # 1 hour
deleteTimeout: 43200000 # 12 hour
http:
port: 9998
server:
applicationConnectors:
- type: http
port: 9998
bindHost: localhost
adminConnectors:
- type: http
port: 9999
bindHost: localhost
Loading