e2e tests occasionally flake with out of disk space

[bbrowning]

For whatever reason, we're occasionally running out of disk space in our e2e tests, on a node with 700GB+ free before we start the e2e test. See https://github.com/instructlab/sdg/actions/runs/12261506100/job/34208780366 for an example here.

[nathan-weinberg]

This is something @courtneypacheco has been looking into

[courtneypacheco]

For reference, this is why we run out of space:

instructlab/instructlab#2417 (comment)

tl;dr - it has to do with tmpfs defaults not allowing /tmp to consume more than 50% of the total physical memory we allocate via EBS. The easiest solutions would either be to change the default save data location for the data, or increase the disk space.