A crash during compression will lose all data

[aliddell]

Kyle pointed this out in the chunking RFC document. If we're accumulating to a buffer while we compress and then crash in the middle of accumulating, all the data is lost. It would create slightly more overhead, but maybe if compressing, we should write to disk in a temporary location and allocate a few additional threads for reading data back in and compressing it in the correct location only after a raw chunk has been successfully written.

[nclack]

This is interesting.

There are different kinds of failures. We can't do much about something like a sudden loss of power. We can try to minimize latency to disk (reducing risk) but can't make stronger guarantees.

But, I do want to try to provide guarantees about shutdown behavior when handling e.g. an exception. We could think about what should happen when we have a partially buffered chunk and the runtime goes into shutdown. Can we flush it in a meaningful way?