From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Parallel (de-)compression Date: Fri, 04 Dec 2015 15:44:38 +0100 Message-ID: <87twnyi889.fsf@gnu.org> References: <20151202184550.GA5741@debian> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:58602) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a4yX8-0002ME-Ii for guix-devel@gnu.org; Fri, 04 Dec 2015 17:08:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a4yX5-0005c8-UE for guix-devel@gnu.org; Fri, 04 Dec 2015 17:08:54 -0500 In-Reply-To: <20151202184550.GA5741@debian> (Andreas Enge's message of "Wed, 2 Dec 2015 19:45:50 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Andreas Enge Cc: guix-devel@gnu.org Andreas Enge skribis: > on my relatively slow ARM build machine with relatively fast storage (SSD= ), > I notice that often there is an xz process taking 100% of CPU, while there > is never more than 20MB/s written to disk. For instance, texlive-texmf > takes a very long time to build and install into the store. Are you saying that xz-compressing TeX Live to resend it to hydra.gnu.org is too CPU-intensive? > Would it make sense to switch to a parallel (de-)compression tool to leve= rage > higher numbers of cores? We have pbzip2 already in Guix, which is compati= ble > with bzip2. Bzip2 provides a CPU/compression ratio tradeoff that is not as good as xz, so I would avoid it. Another option would be to trade compression ratio for reduced CPU usage by using, say, =E2=80=98xz -2=E2=80=99 or =E2=80=98gzip=E2=80=99. We did something similar in 5ef9d7d to reduce CPU consumption on the front-end. Usually it=E2=80=99s much less important to reduce CPU consumpt= ion on the build machines, but your experience seems to suggest otherwise. Thoughts? Ludo=E2=80=99.