GENCODE_DOWNLOAD crashes in stub-run due to double versions.yml overwrite

[Jakob37]

Description of the bug

Running the latest Tomte dev version (commit 95cc7557010bafa54644b) in dev mode, I get crashes like the following:

Command error:
  .command.sh: line 10: versions.yml: cannot overwrite existing file

This seems to be caused by double writes to versions.yml (i.e. both here: cat <<-END_VERSIONS > versions.yml and here wget: $(echo wget -V 2>&1 | grep "GNU Wget" | cut -d" " -f3 > versions.yml)). Together with the recent nf-core template update setting set -C by default, crashing this exact type of double overwrite.

Command used and terminal output

ERROR ~ Error executing process > 'TOMTE:DOWNLOAD_REFERENCES:GTF_DOWNLOAD (gencode)'

Caused by:
  Process `TOMTE:DOWNLOAD_REFERENCES:GTF_DOWNLOAD (gencode)` terminated with an error exit status (1)


Command executed:

  touch GRCh38.primary_assembly.genome.fa
  touch gencode.v46.primary_assembly.annotation.gtf
  cat <<-END_VERSIONS > versions.yml
  "TOMTE:DOWNLOAD_REFERENCES:GTF_DOWNLOAD":
      wget: $(echo wget -V 2>&1 | grep "GNU Wget" | cut -d" " -f3 > versions.yml)
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.sh: line 10: versions.yml: cannot overwrite existing file

Relevant files

No response

System information

No response

[Jakob37]

I also have a different question. We need to be able to run the pipeline offline, i.e. with the NXF_OFFLINE='true' environment variable set.

Is this considered when adding these processes to automatically download reference data?

I have only tried running the stub so far, will see how it behaves for a real run when this is through.

[Lucpen]

Thanks for reporting, I will take a look and try to fix the bug.

Regarding your question, no, it is not taken into account, but as long as you give the pipeline all the input data required, it won't try to download it. You could potentially start the pipeline somewhere with internet, download references, and then upload them to the offline system.

Let me know if you have further questions/info on the bug.

[Jakob37]

I was able to run the pipeline by turning off the set -C option (by removing it from the nextflow.config). Looking in the nf-core Slack though (just searching for "set -C") it is added intentionally to make us developers follow good coding practices - i.e. not writing stdout into files already there within a process.

I think the actual way to solve it would be to in here:

cat <<-END_VERSIONS > versions.yml
  "TOMTE:DOWNLOAD_REFERENCES:GTF_DOWNLOAD":
      wget: $(echo wget -V 2>&1 | grep "GNU Wget" | cut -d" " -f3 > versions.yml)
  END_VERSIONS

remove the second > versions.yml (it is already written to file as declared in the first line).

Haven't tried it out though.

Regarding your question, no, it is not taken into account, but as long as you give the pipeline all the input data required, it won't try to download it. You could potentially start the pipeline somewhere with internet, download references, and then upload them to the offline system.

OK I see! Yes, as long as it does not attempt to download when all is provided that should be fine for us. I'll try it out and get back on that point separately if I don't manage.

[Lucpen]

Thanks for following up on it! If you have the time and realise that it works, feel free to make a PR 😄

[Jakob37]

Thanks for following up on it! If you have the time and realise that it works, feel free to make a PR 😄

Yes, sure! I will focus on other things for a bit now, but if still there when coming back to Tomte I'll make a PR 👌