Skip to content
This repository has been archived by the owner on Oct 18, 2021. It is now read-only.

Cluster #42

Closed
wants to merge 4 commits into from
Closed

Cluster #42

wants to merge 4 commits into from

Conversation

bartvm
Copy link
Owner

@bartvm bartvm commented Feb 22, 2016

Includes stuff from both #39 and #38 (sorry to stack PR on PR, I'll merge everything once it's all tested a bit more thoroughly).

There's a submission script (nmt.pbs) to start jobs on the cluster, with the following changes:

  • Make logging ports configurable so that multiple jobs can run on the same device.
  • Random ports for logging, batches and control; this way you can start multiple jobs without having to set different ports for each manually (else they'll get in each other's way on the same node)
  • Use relative file locations in config.json e.g. train/wmt16.de-en.tok.true.clean.shuf.en instead of the absolute path. The submission script then replaces this with the correct path to use on the cluster. A similar script can be used to submit things on Kepler, so that the same configuration files can be used in both cases.
  • environment.yml is a Conda environment that describes the minimum requirements to run the experiment.
  • I added jq to the cluster in $RAP/nmt/bin
  • For some reason the default encoding for jobs is something very weird, so be sure to set the LANG=en_US.UTF-8 in the environment (done in nmt.pbs).

@bartvm
Copy link
Owner Author

bartvm commented Feb 22, 2016

Wrong target branch

@bartvm bartvm closed this Feb 22, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant