From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pierre Neidhardt Subject: Re: Compressing nars with lzip or similar Date: Sat, 15 Dec 2018 13:17:05 +0100 Message-ID: <87k1kbrnlq.fsf@ambrevar.xyz> References: <20181203154335.10366-1-ludo@gnu.org> <87tvju6145.fsf@gnu.org> <87ftv7l6gy.fsf@gmail.com> <87pnua244k.fsf@gnu.org> <65fa6be4-3ddc-703b-708d-e223c2474d0b@crazy-compilers.com> <87tvjgeb12.fsf@ambrevar.xyz> <874lbg2mh3.fsf_-_@gnu.org> <878t0sdthe.fsf@ambrevar.xyz> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42298) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gY8sb-0001hv-Q6 for guix-devel@gnu.org; Sat, 15 Dec 2018 07:17:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gY8sX-0007JP-Ra for guix-devel@gnu.org; Sat, 15 Dec 2018 07:17:13 -0500 In-reply-to: <878t0sdthe.fsf@ambrevar.xyz> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org, 33600@debbugs.gnu.org --=-=-= Content-Type: text/plain I've done some quick research over the various options. - lzip: better than gz and bzip2 for sure, possibly better than xz (at least according to the author). - plzip: for "parallel lzip". With 4 threads I was able to compress icecat 2.5x faster. It used 5x more memory though. The compression ratio is 1-2% worse. - lrzip: it would crash whenever I would change the compression level. Seems less stable. It's as fast as plzip, while being 1-2% less compressed. I don't think it's worth using. All in all, lzip is a definite win over most options. The main question is: lzip or plzip? In my opinion it makes more sense to use plzip, gotta put those cores to some use (on the user side at least)! Not sure what the build farm would think about this though. Finally, plzip can be quite memory-intensive. When passed "--threads=1" however, it gets closer to lzip (bit faster, bit more memory). So maybe the ideal would be to support plzip with a user-settable "threads" option. Thoughts? -- Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlwU8MEACgkQm9z0l6S7 zH+5owgAhdRMia1xxIiCANu8UorM980EiPuGsTAO/VnnLpleGnQsdfQKjU7SWsk+ 3aV9B/zHmyyouA902HTQK0h5HxzXfW2QivrZx7RtkOFXnckbvMjt7epT0UzaaK+v E+zjC5o16DmDiRyYdmqttDnSNL4Kp30zawRDwN+IXYaWm3L1cb6qSU7H4UgmbFng 80MHofqdU/2Uk042pNwZNQUZPjtIWSX611LJGT7YmrfscyDDBhijXTIjiNaIKa6b Tm7Vr7HLN100k3gzPVEBONfrDFMBfOqNHfxy8Y1+79c0eMFqHbvSDKpghxRCrByk lhGLOPPKrvH3LWkJHE2MHh/z8pIb5Q== =gxzl -----END PGP SIGNATURE----- --=-=-=--