Astropy isssue reading of fits.Z compressed fles

[thespacedoctor]

The ESO archive typically distributes data in fits.Z compressed format. Astropy is currently having problems reading these:

astropy/astropy#10714

astropy/astroquery#1818

I am having to uncompress the data before running soxspipe and would be space/time saving to not have to do this.

Adding this issue to remain myself to check back in the future for a fix.

[almicol]

Hi David, for SOX it would be better to use directly the ESO programmatic interfaces. Using them you will not suffer of such problem. We have available a lot of documentation about the programmatic access, including python scripts and jupyter notebooks. Probably the most relevant for you is the following jupyter notebook:
http://archive.eso.org/programmatic/HOWTO/jupyter/authentication_and_authorisation/
The main site to learn how to use the ESO programmatic interfaces is: http://archive.eso.org/programmatic
where ADQL live queries can be executed, python scripts can be found, etc.
If you need any specific help, please contact me. Thanks and cheers, Alberto

[thespacedoctor]

Hi Alberto, great to hear from you. We are planning to the ESO interfaces for collecting and streaming data to and from Garching, and the resources you link here will be very useful.

My concern here is the direct reading of the FITS .Z compression rather than the use of Astroquery to connect to the ESO archive. Currently, we are mainly using XShooter data downloaded from the SAF to develop the pipeline against. As the Astropy tools can't directly read the .Z compression, we first need to uncompress the data before reading it. Not a huge issue, but would be nice to be able to work directly on the compressed data.

The main issue is here:

astropy/astropy#10714

(I also linked the astroquery issue you are involved with as it too relates to the compression).

[thespacedoctor]

soxspipe will now simply uncompress the .Z files on the fly during reductions.