From mboxrd@z Thu Jan 1 00:00:00 1970 From: Efraim Flashner Subject: Re: Compress VCS checkouts? Date: Sat, 3 Feb 2018 22:10:39 +0200 Message-ID: <20180203201039.GC1003@macbook41> References: <20180202145011.6961c2c9@centurylink.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="+xNpyl7Qekk2NvDX" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42424) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ei497-0006hx-Ok for guix-devel@gnu.org; Sat, 03 Feb 2018 15:10:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ei492-0003at-OV for guix-devel@gnu.org; Sat, 03 Feb 2018 15:10:45 -0500 Received: from flashner.co.il ([178.62.234.194]:55092) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ei492-0003ab-DG for guix-devel@gnu.org; Sat, 03 Feb 2018 15:10:40 -0500 Content-Disposition: inline In-Reply-To: <20180202145011.6961c2c9@centurylink.net> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Eric Bavier Cc: guix-devel@gnu.org --+xNpyl7Qekk2NvDX Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 02, 2018 at 02:50:11PM -0600, Eric Bavier wrote: > Hello Guix, >=20 > A while back I toyed with the idea of a "Sources DVD" for Guix, much > like Trisquel has. While looking into the feasability of this, I > noticed that the git/svn/cvs/hg checkouts our packages use are not > compressed in the store. So I wrote a simple script to compare the raw > size of all source checkouts with their corresponding xz-compressed > tarballs. =20 >=20 > At the time, I couldn't convince myself the trouble was worth the > effort; we had only 385MiB of uncompressed source checkouts, which > would compress to 135MiB. >=20 > Since that initial investigation, the number of packages using VCS > checkouts has risen from 50 to 389, and the on-disk size of that source > has risen to 3784MiB uncompressed. We could compress that down to > 538MiB. >=20 > Compression in not free, but I think it would be nice for our build > servers who have to store and serve (which, iirc, requires them to > compress it anyhow) this source. >=20 > What do people think? >=20 > `~Eric > name path = size (MiB) tar.xz rel% > footswitch cwrqb...-footswitch-0.1-1.7cb0a93-c= heckout 0.04 0.01 17.5% < snip > > -------------------------------------------------------------------------= ----------------------------------------------------- > TOTALS: = 3783.86 538.07 18.6% (median) In case anyone else had to look at it twice, that's the relative size of the new tarred source compared to the original. The sources don't change very often, so the cost of the initial compression doesn't come up much, and serving compressed data should be much faster. Interestingly, on machines with the store on an external HDD, it might be faster to load and unpack the source to ram than to just copy over USB2. It does make grepping through the source in the store harder, but we already couldn't do that with the tarballs. I think it would be overall a positive change. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --+xNpyl7Qekk2NvDX Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlp2Fz8ACgkQQarn3Mo9 g1HFvQ//Uw9UNHYvaqsmvMnrNY325MSOQeGGkbm29yk0LqhhgU+S4KzYGdkJYhtg H7NGpuFIqKKAWmHPj2RLoLQfxmfyBq3I72E5y5S3riN+A4PW1RGKnsFlc55lg33k zG+UqhwlJKEnydCUJxqthHS7wfqZ7eY19g1p71Vt1nxRnPpEavKEgGhnUMvd7fxN jujLLi89EJFJcEaPmBH10sEfUAHAcfawKLHzoJex7y6F2+kAodEKT6qIL9dp1LDe NL1mi0440walvGcF/HUR5NomobVRLPPfhP5Yt0AnOeqavTQs0OEc/HwopioTBrg+ ckauP4Og17Y5lhMC3qID35nXu3uQUCVG+BKg+WmaGKfgdwv97BnbDwW02on6M9Q/ QQ3vaDbfI43Lcb6+F7GIzTdyfNE+UYudTmGbC57JSonUzFtJHTFFBvXhyRbrgrLn xwR7rZeCCvmMGa7UilcJFHjwYxSQziLS5/7SNe9x/K/zuEZFddz6cqSHfk9cQrSs xram/Ug+G0SXmG+kVNf10LRRQFAjhyaAlwTdm+0zDOfc8L1ZUJmOQvUurXeDRQFF FJFzgUUcTrzVib0OvxqpKQSWG83l9RFSLqY8XSqJPeNb2tKHacoNaZa+Bw6/YKSM sPYvwqwMG+9P5YAxVdMMlrc1o/ZISUidBjC2ehRwpESPEW1qLoQ= =WIhD -----END PGP SIGNATURE----- --+xNpyl7Qekk2NvDX--