From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id V8fjNHmu2F+tNQAA0tVLHw (envelope-from ) for ; Tue, 15 Dec 2020 12:39:21 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id uHIxMHmu2F94dAAAbx9fmQ (envelope-from ) for ; Tue, 15 Dec 2020 12:39:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7B25C940148 for ; Tue, 15 Dec 2020 12:39:21 +0000 (UTC) Received: from localhost ([::1]:51318 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kp9bs-0008TW-6x for larch@yhetil.org; Tue, 15 Dec 2020 07:39:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:40922) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp9UJ-0000Hq-UH for guix-devel@gnu.org; Tue, 15 Dec 2020 07:31:31 -0500 Received: from relay1-d.mail.gandi.net ([217.70.183.193]:17623) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kp9UH-00078I-ER; Tue, 15 Dec 2020 07:31:31 -0500 X-Originating-IP: 86.247.16.87 Received: from bababa (lfbn-idf2-1-709-87.w86-247.abo.wanadoo.fr [86.247.16.87]) (Authenticated sender: mail@ambrevar.xyz) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 3873F24000F; Tue, 15 Dec 2020 12:31:23 +0000 (UTC) From: Pierre Neidhardt To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: When substitute download + decompression is CPU-bound In-Reply-To: <87blevmh2l.fsf@gnu.org> References: <87im94qbby.fsf@gnu.org> <87sg88ngd1.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <871rfrld59.fsf@ambrevar.xyz> <87blevmh2l.fsf@gnu.org> Date: Tue, 15 Dec 2020 13:31:23 +0100 Message-ID: <87sg879rok.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Received-SPF: pass client-ip=217.70.183.193; envelope-from=mail@ambrevar.xyz; helo=relay1-d.mail.gandi.net X-Spam_score_int: 0 X-Spam_score: -0.1 X-Spam_bar: / X-Spam_report: (-0.1 / 5.0 requ) BAYES_00=-1.9, FROM_SUSPICIOUS_NTLD=0.499, PDS_OTHER_BAD_TLD=1.999, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel , =?utf-8?Q?Nicol=C3=B2?= Balzarotti Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -4.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 7B25C940148 X-Spam-Score: -4.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: vQGhf11Q8+bi --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ludo, Ludovic Court=C3=A8s writes: > Well, =E2=80=98guix publish=E2=80=99 would first need to create multi-mem= ber archives, > right? Correct, but it's trivial once the bindings have been implemented. > Also, lzlib (which is what we use) does not implement parallel > decompression, AIUI. Yes it does, multi-member archives is a non-optional part of the Lzip specs, and lzlib implemetns all the specs. > Even if it did, would we be able to take advantage of it? Currently > =E2=80=98restore-file=E2=80=99 expects to read an archive stream sequenti= ally. Yes it works, I just tried this: =2D-8<---------------cut here---------------start------------->8--- cat big-file.lz | plzip -d -o big-file - =2D-8<---------------cut here---------------end--------------->8--- Decompression happens in parallel. > Even if I=E2=80=99m wrong :-), decompression speed would at best be doubl= ed on > multi-core machines (wouldn=E2=80=99t help much on low-end ARM devices), = and > that=E2=80=99s very little compared to the decompression speed achieved b= y zstd. Why doubled? If the archive has more than CORE-NUMBER segments, then the decompression duration can be divided by CORE-NUMBER. All that said, I think we should have both: =2D Parallel lzip support is the easiest to add at this point. It's the best option for people with low bandwidth. This can benefit most of the planet I suppose. =2D zstd is best for users with high bandwidth (or with slow hardware). We need to write the necessary bindings though, so it will take a bit more time. Then the users can choose which compression they prefer, mostly depending on their hardware and bandwidth. =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl/YrJsSHG1haWxAYW1i cmV2YXIueHl6AAoJEJvc9Jeku8x/hCkH/3rfgE8HJUI4JLXtVhWh9AaNQ6D6zE13 0vKuw5rLLvToPRiGQo8xXQz0VxghZHNoHUL/DoJoyQNDLqE/2RiLGAxqOJAKgGuz M4Jrf6vA7xn2/mdmmyjwJlApa+xAYEkqsmiIe52JxNU4gb/8iWF5Xd1QwNq/R4W5 lsE/syK9EXwr679/VWo/zNksCXCFLNr+tpokp+44ncq9ZjGPc+4KIZTlkO3tFkDk +0h3vQwzQfGmvXq1OPycYX7YEWGYl4sfuSZrM6XgQ14hs0opLR5XTEoXm2hbBLKk tcQnRsy9Fnr/aRMoBomrZYDUWQbIA0J9p7Ymp0W7dyF3swnwq8WUpAk= =kOEn -----END PGP SIGNATURE----- --=-=-=--