Re: Parallel (de-)compression

[ludo@gnu.org (Ludovic Courtès)]

Andreas Enge <andreas@enge.fr> skribis:

> on my relatively slow ARM build machine with relatively fast storage (SSD),
> I notice that often there is an xz process taking 100% of CPU, while there
> is never more than 20MB/s written to disk. For instance, texlive-texmf
> takes a very long time to build and install into the store.

Are you saying that xz-compressing TeX Live to resend it to
hydra.gnu.org is too CPU-intensive?

> Would it make sense to switch to a parallel (de-)compression tool to leverage
> higher numbers of cores? We have pbzip2 already in Guix, which is compatible
> with bzip2.

Bzip2 provides a CPU/compression ratio tradeoff that is not as good as
xz, so I would avoid it.

Another option would be to trade compression ratio for reduced CPU usage
by using, say, ‘xz -2’ or ‘gzip’.

We did something similar in 5ef9d7d to reduce CPU consumption on the
front-end.  Usually it’s much less important to reduce CPU consumption
on the build machines, but your experience seems to suggest otherwise.

Thoughts?

Ludo’.