Deeper, Wider, Faster File Ingestion and Calibration Pipeline

This is legacy file ingestion, data transfer, and calibration pipeline for the Deeper, Wider, Faster Survey. Originally written for the use of DECam data and the Swinburne G2 HPC system by Tyler Pritchard, it now also has limited experimental support for the new ozstar HPC cluster as well as the Subaru HSC camera with contributions from Dany Vohl and Nozomu Tominaga. Use a markdown viewer for best results with this document.

General Schema

• Remote server

  • Monitors for new data
  • Packages raw data from telescope
  • Ships data to local HPC environment (including compression, transfer, parrallelization)

• Local HPC Environment

  • monitors for new recieved data, unpackages
  • initialtes parallelized data reduction
  • monitors for quality,
  • simplistic attempts to automatically improve reduction (WCS mostly)

• Convenience Functions

  • There are some scripts to handle some regular tasks as well, I'll mention what I can remember but include a full dump regardless

Usage

This will focus on CTIO/OzStar as that is the most likely combination going forward. Some differences/caveats at the end. Most python functions have a --help or argparse optional parameters to modify paths and values, please take a look at individual files as nescessary.

Remote Server

1. Log into observer2 @ CTIO.
2. Go to ~/tapritchard and source start_dwf.sh (This initiates the DWF environment)
3. Go to /usr/remote/user/DECamObserver/tapritchard/dwf_prepipe_ozstar
4. Run ./dwf_prepipe_push.py --method='b'
   • I have it set for our observation directory from the DWF 1806 run and a 10:1 compression ratio. for G2, we specify --qs= ...
   • There are different modes, 'b' is batch - send 3-4 at a time, and is the most rebust to all transfer speeds. There is also serial (send the newest file), and parallel (send each file as it comes in, paying no attention to if you start getting behind). Also, end-of-night which will create a list of files on the remote server and transfer anything that was missed during the night.

Local Cluster

1. login to fstars on ozstar end
2. type in lp # (starts the prepipe environment)
3. go to ~/dwf_prepipe
4. 4. run ./dwf_prepipe.py

For G2

• Remote Server:
  • Use /usr/remote/user/DECamObserver/tapritchard/dwf_prepipe instead
• Local
  • When logged in as fstars, use csh to initiate environment

Ozstar Port

The ozstar port was done while on a compressed timescale with limited resources due to the G2 outage. While it is nominally functional, additional work to test and debug is necessary. G2 is fully functional.

HSC

This pipeline was written for HSC and used on g2. It is functional, but will not work on ozstar and could use significant improvements. General thoughs for future usage follow.

• HSC Remote Server

  • Use dwf_pushpollsepNodesVisit.py for transfers
    • This Was written by Nozomu Tominaga from the initial transfer script. Prioritize transfer of all ccd's for a single exposure was a late addition in the last run.

• HSC Local Server

  • Originally written for G2, this will require porting to ozstar
  • I changed the directory structure when archiving, make sure to search for anything with dwf_prepipe/hsc_test/ (the original location) and update accordingly. (e.g. ~/dwf_prepipe/)
  • On the HSC run we were I/O limited. As a quick fix, we implemented a data throttle. There were two modes - ingest a static rate of files or keep a static number of reduction jobs
    • dwf_hsc_prepipe_staticingest.py
    • dwf_hsc_prepipe_staticingest.py
  • To Resolve this in the future, we need to move hscpipe onto the local storage for each node, and modify the pipeline to extract data locally, calibrate locally, and then copy the final file to g2. This is similar to how we allready handle de-compression / j2f jpeg-to-fits conversion
  • A Similar method could be applied to DECam/photpipe as well for a speed benefit

Convenience Functions

• G2/OzStar
  • dwf_prepipe_copycals.py -i <date> -o <date>
    • used to copy calibration files from one night to the next for easy photpipe usage.
  • dwf_prepipe_reproccess.py
    • Given a list of .tar DECam compressed files, un-tar and trigger reprocessing.
• HSC
  • dwf_hsc_getjobs.py - gets the number of fstars hscpipe jobs in queue
  • dwf_hsc_getseeing.py - gets the measured psf value from Igor's Mary
    • this has a hard-coded directory and was done quick on dirty on the night of
  • dwf_hsc_prepipe_convertCorrCorr2.py - Converts WCS system from HSC default to something compatable with Mary/Swarp
  • dwf_hsc_triggerfullred.py - Triggers a full hsc-pipe reduction of a given visit
  • imageinv.py - Gives a list of fields and number of calibrated exposures on them

Configuration files

photpipe Configuration & parameter files are in the photpipe_configs directory

hscpipe minimal config that we used on-sky is hsc_minimal_processCcd.py in the repo