Choosing gulp exponent

[telegraphic]

Hi Danny,

L. and I have been having some issues with the gulp exponent chosen affecting the Cupy memory. I've found that in some cases, it's best to choose a gulp size where all the channels will fit in a single gulp. In other cases, it's better to have multiple gulps. The exponent ultimately decides if there is a Cupy memory error or not. Is there an optimal way to choose this number, or is trial and error the best way? Also, let me know if posting on #tseti_beta_testers is better to raise these issues -- not sure if Richard or others are still working on the development of this also.

Thanks,
K

[telegraphic]

Hi K,

TLDR:

• I haven’t figured out an optimal way to choose gulp size.
• I would not gulp more than one coarse channel at a time, as the polyfit won’t work as well.
• Gulps that are very small would be prone to SNR calculation quirks.
• You will be able to have overlapping gulp reads with hyperseti 1.1.0

Longer answer with context:

I’m hoping you’re running into an OutOfMemoryError, not a different error? Hyperseti consumes more memory than I think it ought to (and I’d like to resolve this), but it does need more GPU memory than turboseti due to differences in design.

Gulp size will affect the computation of SNR and polynomial fitting. I would not gulp more than one coarse channel at a time (if your data have coarse channels, that is), as the polynomial fit won’t work as well. Gulps that are very small would be prone to SNR calculation quirks.

Here’s a quick overview of why hyperseti uses a fair bit of memory:

A hyperseti ‘gulp’ has the shape:

(N_timesteps, N_beam, N_channels)

Where most of the time N_beam = 1, and N_channels = 2^gulp_exponent

For normal BL data, N_timesteps = 16, N_beam = 1, and if N_channels=2^20, then the data volume is 64 MB per gulp (4 Bytes per float32). That’s very reasonable.

After applying a dedoppler transform, say over -10 to +10 Hz, the shape is:

(N_dedopp, N_beam, N_channels)

For 3 Hz resolution data over 300 seconds, the doppler resolution is 3 Hz / 300 s = 0.01 Hz/s.
So, to fully cover +/- 10 Hz/s drift range optimally, you need 2000 drift trials. In this case, the data array shape is 8 GiB. That is quite large!

Hyperseti makes a copy of this array at various stages. Some of these copies are required, but I think some of them are redundant.

To reduce memory usage, I introduced a ‘stepped’ plan when running dedoppler, where the steps got larger at higher drift rates: https://hyperseti.readthedocs.io/en/latest/pipeline/dedoppler.html#stepped-plan

I haven’t fully tested sensitivity differences between ‘optimal’ and ‘stepped’, but would suggest using ‘stepped’ as the default for high drift rate searches.