Error creating block encoder

[satmandu]

We are now using pixz to compress archives for chromebrew, and are noticing this error on i686 and armv7l machines (which may have less memory):

Error creating block encoder

We are using this command line:

https://github.com/skycocker/chromebrew/blob/06c4852f33eb61f88800f26d16e2cebbc8ca131e/bin/crew#L1187
system "tar c#{@verbose} * | pixz -f0.25 -9 > #{pwd}/#{pkg_name}"

(This translates to tar c * | pixz -f0.25 -9 > packagefile.tpxz )

Is there a better way of doing this to avoid these errors?

(I don't get this error when using this command in a docker container of the relevant architecture on machines with plenty of ram.)

[vasi]

Do you know in advance how much RAM and how many cores the system has? You could also try adjusting the compression level, as well as using -q/-f to tune pixz. For a small system, -9 is pretty extra, I typically use -3 even on a beefy box.

(For what it's worth "-f $FRAC" with $FRAC < 1 is usually a bit of a waste. You're generally better off using a lower compression level instead, you'll get similar compression ratios with better speed.)

If you don't know in advance how many CPU/memory exists, I guess we could try to add something to pixz to detect the current amount of RAM and adjust? But this is pretty hard to get right automatically--can't really predict when the user would prefer using all RAM (and swapping out other programs) vs. using only available RAM. Also, the main lever for adjustment is the -LEVEL parameter, and users haven't liked the suggestion of pixz automagically changing that.

Suggestions are welcome!

[satmandu]

The rationale for using the -f $FRAC is to better enable piping a download directly to a tar extract where only some files in the archive would get downloaded and extracted. I hope to use that mechanism to install runtime libraries from large installation archives (like for llvm or gcc). This saves the hassle and technical debt of creating customized runtime library packages for larger packages. If the -f $FRAC isn't useful for making that easier I will remove it.

Re: RAM/cores, I would be shocked if these (i686 & armv7l) systems had more than 2-4 Gb of ram, with a max of 2 cores per system for i686, and possibly more going forward for armv7l.

[uberhacker]

FWIW, a 32-bit system can only physically address 2^32 or 4 gb max. Both i686 and armv7l are 32-bit architectures.