From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Compress VCS checkouts? Date: Wed, 07 Feb 2018 22:56:28 +0100 Message-ID: <87y3k4xy43.fsf@gnu.org> References: <20180202145011.6961c2c9@centurylink.net> <87vafbhgu9.fsf@gnu.org> <20180205184922.3ff8e330@centurylink.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53719) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ejXhs-0006b5-KI for guix-devel@gnu.org; Wed, 07 Feb 2018 16:56:45 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ejXho-00068h-5M for guix-devel@gnu.org; Wed, 07 Feb 2018 16:56:44 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:60304) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ejXhn-00068A-V4 for guix-devel@gnu.org; Wed, 07 Feb 2018 16:56:40 -0500 In-Reply-To: <20180205184922.3ff8e330@centurylink.net> (Eric Bavier's message of "Mon, 5 Feb 2018 18:49:22 -0600") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Eric Bavier Cc: guix-devel@gnu.org Hi Eric, Eric Bavier skribis: > On Mon, 05 Feb 2018 11:27:10 +0100 > ludo@gnu.org (Ludovic Court=C3=A8s) wrote: > >> Hello! >>=20 >> Eric Bavier skribis: >>=20 >> > A while back I toyed with the idea of a "Sources DVD" for Guix, much >> > like Trisquel has. While looking into the feasability of this, I >> > noticed that the git/svn/cvs/hg checkouts our packages use are not >> > compressed in the store. So I wrote a simple script to compare the raw >> > size of all source checkouts with their corresponding xz-compressed >> > tarballs.=20=20 >> > >> > At the time, I couldn't convince myself the trouble was worth the >> > effort; we had only 385MiB of uncompressed source checkouts, which >> > would compress to 135MiB. >> > >> > Since that initial investigation, the number of packages using VCS >> > checkouts has risen from 50 to 389, and the on-disk size of that source >> > has risen to 3784MiB uncompressed. We could compress that down to >> > 538MiB. >> > >> > Compression in not free, but I think it would be nice for our build >> > servers who have to store and serve (which, iirc, requires them to >> > compress it anyhow) this source.=20=20 >>=20 >> =E2=80=98guix publish -C=E2=80=99 always compresses store items, unless = they are regular >> files ending in =E2=80=9C.gz=E2=80=9D, =E2=80=9C.xz=E2=80=9D, etc. So t= here=E2=80=99s no problem here. > > I see, ok. > >>=20 >> It seems to me that the only benefit of compressing checkouts would be >> the use case you=E2=80=99re talking about. For everyday use, it=E2=80= =99s actually more >> efficient to leave the uncompressed, so that people don=E2=80=99t end up= doing >> round of compression/decompression when they run =E2=80=9Cguix build=E2= =80=9D. > > Often, people would end up fetching the compressed source from a > substitute server, no? Specifically, they download a compressed nar that contains the uncompressed checkout. > And currently I already need to uncompress what I get from a > substitute server before building. > > But now I'm wondering why we bother with the "repack" portion of > "patch-and-repack"? Good question! It=E2=80=99s mostly because it =E2=80=9Clooked nicer=E2=80= =9D this way (and maybe at the time some of the code did not properly handle non-tarball sources.) I hear we could apply the same =E2=80=9Creasoning=E2=80=9D (if w= e can call it this way) to checkouts. Hmm! >> That said, at worst, this could be addressed by writing a specific tool >> for the job. > > Yes. I was hoping it could be avoided though. The tool could be relatively simple. Ludo=E2=80=99.