From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id eBm/EwPE3F99TQAA0tVLHw (envelope-from ) for ; Fri, 18 Dec 2020 15:00:19 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id +LOeDwPE3F+QOgAA1q6Kng (envelope-from ) for ; Fri, 18 Dec 2020 15:00:19 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1040794043B for ; Fri, 18 Dec 2020 15:00:19 +0000 (UTC) Received: from localhost ([::1]:46866 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kqHEv-0001Xr-W3 for larch@yhetil.org; Fri, 18 Dec 2020 10:00:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:48702) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kqHEi-0001WP-BZ for guix-devel@gnu.org; Fri, 18 Dec 2020 10:00:04 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33211) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kqHEf-0000TP-Lg; Fri, 18 Dec 2020 10:00:01 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=53268 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kqHEc-0002QW-EN; Fri, 18 Dec 2020 09:59:59 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Pierre Neidhardt Subject: Re: When substitute download + decompression is CPU-bound References: <87im94qbby.fsf@gnu.org> <87sg88ngd1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <871rfrld59.fsf@ambrevar.xyz> <87blevmh2l.fsf@gnu.org> <87sg879rok.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 28 Frimaire an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 18 Dec 2020 15:59:56 +0100 In-Reply-To: <87sg879rok.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Tue, 15 Dec 2020 13:31:23 +0100") Message-ID: <87lfdvb1n7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel , =?utf-8?Q?Nicol=C3=B2?= Balzarotti Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.82 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 1040794043B X-Spam-Score: -2.82 X-Migadu-Scanner: scn0.migadu.com X-TUID: SGw9z0HSO+mP Hi Pierre, Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: > >> Well, =E2=80=98guix publish=E2=80=99 would first need to create multi-me= mber archives, >> right? > > Correct, but it's trivial once the bindings have been implemented. OK. >> Also, lzlib (which is what we use) does not implement parallel >> decompression, AIUI. > > Yes it does, multi-member archives is a non-optional part of the Lzip > specs, and lzlib implemetns all the specs. Nice. >> Even if it did, would we be able to take advantage of it? Currently >> =E2=80=98restore-file=E2=80=99 expects to read an archive stream sequent= ially. > > Yes it works, I just tried this: > > cat big-file.lz | plzip -d -o big-file - > > Decompression happens in parallel. > >> Even if I=E2=80=99m wrong :-), decompression speed would at best be doub= led on >> multi-core machines (wouldn=E2=80=99t help much on low-end ARM devices),= and >> that=E2=80=99s very little compared to the decompression speed achieved = by zstd. > > Why doubled? If the archive has more than CORE-NUMBER segments, then > the decompression duration can be divided by CORE-NUMBER. My laptop has 4 cores, so at best I=E2=80=99d get a 4x speedup, compared to= the 10x speedup with zstd that also comes with much lower resource usage, etc. > All that said, I think we should have both: > > - Parallel lzip support is the easiest to add at this point. > It's the best option for people with low bandwidth. This can benefit > most of the planet I suppose. > > - zstd is best for users with high bandwidth (or with slow hardware). > We need to write the necessary bindings though, so it will take a bit > more time. > > Then the users can choose which compression they prefer, mostly > depending on their hardware and bandwidth. Would you like to give parallel lzip a try? Thanks! Ludo=E2=80=99.