Re: Parallel (de-)compression

[Andreas Enge <andreas@enge.fr>]

On Fri, Dec 04, 2015 at 03:44:38PM +0100, Ludovic Courtès wrote:
> Are you saying that xz-compressing TeX Live to resend it to
> hydra.gnu.org is too CPU-intensive?

That depends on your definition of "too". In any case, on the Novena board
with an SSD attached, CPU is the limiting factor during this phase,
pushing the CPU load on one core to 100% (while the other cores are idle).

> Another option would be to trade compression ratio for reduced CPU usage
> by using, say, ‘xz -2’ or ‘gzip’.
> We did something similar in 5ef9d7d to reduce CPU consumption on the
> front-end.  Usually it’s much less important to reduce CPU consumption
> on the build machines, but your experience seems to suggest otherwise.

If possible, it would be more interesting to leverage the several cores
and not make sacrifices on the compression quality. Note that I also did
not measure the different timings: Compression is, I think, done separately
from sending the compressed file; then it is entirely possible that the
data transfer takes longer than the compression.

Andreas