Re: Compress VCS checkouts?

[Efraim Flashner <efraim@flashner.co.il>]

[-- Attachment #1: Type: text/plain, Size: 2424 bytes --]

On Fri, Feb 02, 2018 at 02:50:11PM -0600, Eric Bavier wrote:
> Hello Guix,
> 
> A while back I toyed with the idea of a "Sources DVD" for Guix, much
> like Trisquel has. While looking into the feasability of this, I
> noticed that the git/svn/cvs/hg checkouts our packages use are not
> compressed in the store.  So I wrote a simple script to compare the raw
> size of all source checkouts with their corresponding xz-compressed
> tarballs.  
> 
> At the time, I couldn't convince myself the trouble was worth the
> effort; we had only 385MiB of uncompressed source checkouts, which
> would compress to 135MiB.
> 
> Since that initial investigation, the number of packages using VCS
> checkouts has risen from 50 to 389, and the on-disk size of that source
> has risen to 3784MiB uncompressed.  We could compress that down to
> 538MiB.
> 
> Compression in not free, but I think it would be nice for our build
> servers who have to store and serve (which, iirc, requires them to
> compress it anyhow) this source.
> 
> What do people think?
> 
> `~Eric

> name                                  path                                                           size (MiB) tar.xz    rel%
> footswitch                            cwrqb...-footswitch-0.1-1.7cb0a93-checkout                          0.04    0.01   17.5%
< snip >
> ------------------------------------------------------------------------------------------------------------------------------
> TOTALS:                                                                                                3783.86  538.07   18.6% (median)

In case anyone else had to look at it twice, that's the relative size of
the new tarred source compared to the original. The sources don't change
very often, so the cost of the initial compression doesn't come up much,
and serving compressed data should be much faster. Interestingly, on
machines with the store on an external HDD, it might be faster to load
and unpack the source to ram than to just copy over USB2.

It does make grepping through the source in the store harder, but we
already couldn't do that with the tarballs. I think it would be overall
a positive change.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]