* Compress VCS checkouts?
@ 2018-02-02 20:50 Eric Bavier
2018-02-03 20:10 ` Efraim Flashner
2018-02-05 10:27 ` Ludovic Courtès
0 siblings, 2 replies; 5+ messages in thread
From: Eric Bavier @ 2018-02-02 20:50 UTC (permalink / raw)
To: guix-devel
[-- Attachment #1.1: Type: text/plain, Size: 966 bytes --]
Hello Guix,
A while back I toyed with the idea of a "Sources DVD" for Guix, much
like Trisquel has. While looking into the feasability of this, I
noticed that the git/svn/cvs/hg checkouts our packages use are not
compressed in the store. So I wrote a simple script to compare the raw
size of all source checkouts with their corresponding xz-compressed
tarballs.
At the time, I couldn't convince myself the trouble was worth the
effort; we had only 385MiB of uncompressed source checkouts, which
would compress to 135MiB.
Since that initial investigation, the number of packages using VCS
checkouts has risen from 50 to 389, and the on-disk size of that source
has risen to 3784MiB uncompressed. We could compress that down to
538MiB.
Compression in not free, but I think it would be nice for our build
servers who have to store and serve (which, iirc, requires them to
compress it anyhow) this source.
What do people think?
`~Eric
[-- Attachment #1.2: vcs-sizes.txt --]
[-- Type: text/plain, Size: 50208 bytes --]
name path size (MiB) tar.xz rel%
footswitch cwrqb...-footswitch-0.1-1.7cb0a93-checkout 0.04 0.01 17.5%
interrobang lab0p...-interrobang-0.0.0-1.8965437 0.08 0.02 27.2%
mkbootimg imjbx...-android-platform-system-core-7.1.2_r6-checkout 8.68 1.11 12.8%
git-repo rk0vy...-git-repo-1.12.37-checkout 0.47 0.09 19.2%
android-udev-rules i5qbm...-android-udev-rules-20171113-checkout 0.06 0.02 25.0%
fluidsynth ghszj...-fluidsynth-1.1.9-checkout 2.79 0.47 16.8%
faust g8qzg...-faust-0.9.90-checkout 148.59 16.36 11.0%
qjackrcd chq81...-git-checkout 0.25 0.10 38.3%
noise-repellent srfp9...-noise-repellent-0.1.4-checkout 0.15 0.03 18.5%
faust yfjxh...-git-checkout 203.48 22.98 11.3%
filteraudio 80bgn...-filteraudio-0.0.0-1.2fc6695-checkout 1.02 0.15 14.9%
raul 2x87s...-raul-0.8.9-1.4db870b2b-checkout 0.39 0.16 40.6%
bluez-alsa gbva4...-bluez-alsa-1.2.0-checkout 0.31 0.05 16.1%
lv2-mda-epiano 9q4bs...-git-checkout 2.07 1.66 79.9%
lv2-devel 986wf...-git-checkout 1.55 0.30 19.2%
newick-utils qkis8...-newick-utils-1.6-1.da121155-checkout 133.61 24.13 18.1%
libdivsufsort 9ahyi...-git-checkout 0.19 0.03 13.4%
r-chipkernels 9kms2...-r-chipkernels-1.1-1.c9cfcacb6 0.07 0.01 16.2%
bits zadz5...-bits-2.13.0-1.3cc456789-checkout 2.77 0.52 18.9%
piranha bvrk4...-git-checkout 32.86 2.31 7.0%
smithlab-cpp mgvpc...-smithlab-cpp-0.1.728a097-checkout 0.15 0.02 14.2%
bpp-popgen 7ika1...-bpp-popgen-2.2.0-1.e472bac-checkout 0.82 0.09 10.8%
bpp-core njqf5...-bpp-core-2.2.0-1.7d8bced-checkout 1.92 0.22 11.5%
seek gl4rn...-hg-checkout 8.43 0.74 8.8%
bppsuite yq220...-bppsuite-2.2.0-1.c516147-checkout 1.23 0.13 10.3%
mosaik 43sqg...-mosaik-2.2.30 13.57 2.78 20.5%
bpp-phyl kg339...-bpp-phyl-2.2.0-1.0c07167-checkout 3.64 0.37 10.1%
bpp-seq z586w...-bpp-seq-2.2.0-1.6cfa079-checkout 2.32 0.20 8.8%
tremc 0654m...-tremc-0.9.0-2.e06d08d-checkout 0.44 0.24 53.6%
syslinux clcb3...-syslinux-6.04-pre-1.bb41e93-checkout 19.39 9.32 48.1%
vboot-utils 6a6lh...-vboot-utils-R63-10032.B-checkout 63.68 21.70 34.1%
catch 067v7...-catch-1.3.5 2.12 0.18 8.6%
go-gopkg.in-check.v1 6wjyv...-go-gopkg.in-check.v1-0.0.0-0.20d25e2-checkout 0.14 0.03 18.3%
chez-sockets 3il8x...-chez-sockets-0.0-1.bce9688-checkout 0.10 0.02 21.6%
chez-web g2j99...-chez-web-2.0-1.5fd177f-checkout 0.26 0.07 25.2%
cuirass n0ydb...-cuirass-0.0.1-10.9cfea9f 0.27 0.05 18.9%
sfarkxtc v6hfw...-git-checkout 0.05 0.01 27.5%
eschalot yv9sb...-eschalot-1.2.0-1.0bf31d8 0.67 0.19 28.9%
libb2 fn6hl...-libb2-0.0.0-1.60ea749-checkout 1.11 0.20 18.0%
kurly q7p2v...-git-checkout 0.05 0.01 23.0%
mongo-tools l2fv7...-mongo-tools-3.4.0-checkout 17.41 2.96 17.0%
go-gopkg.in-mgo.v2 arawv...-go-gopkg.in-mgo.v2-2016.08.01-checkout 1.99 0.31 15.5%
guile-wiredtiger pjcs0...-guile-wiredtiger-0.6.3-checkout 1.24 1.06 85.6%
stress-make pnwyd...-stress-make-1.0-0.506e6cf-checkout 0.06 0.02 27.6%
emacs-god-mode 4jwih...-emacs-god-mode-20151005.925.1-6cf0807b6-checkout 0.07 0.01 18.3%
emacs-ob-ipython x3pzh...-git-checkout 0.63 0.50 78.1%
emacs-string-inflection 3jay5...-emacs-string-inflection-1.0.6-checkout 0.03 0.00 10.2%
emacs-yasnippet-snippets 88gc0...-emacs-yasnippet-snippets-1-1.885050d3-checkout 0.70 0.06 7.9%
emacs-strace-mode 8dvr6...-emacs-strace-mode-0.0.2-1.6a69b4b-checkout 0.04 0.01 26.8%
emacs-grep-a-lot 560jq...-emacs-grep-a-lot-1.0.7-checkout 0.05 0.01 26.2%
emacs-dream-theme xrcq6...-emacs-dream-theme-0.0.0-1.107a11d-checkout 0.02 0.00 15.0%
emacs-ws-butler n6fzb...-emacs-ws-butler-0.6-checkout 0.03 0.01 18.3%
emacs-julia-mode y8rmq...-emacs-julia-mode-0.3-1.115d4dc8-checkout 0.17 0.02 13.9%
emacs-undo-tree zy83l...-emacs-undo-tree-0.6.6-checkout 0.17 0.03 17.6%
emacs-restclient 6gjw4...-emacs-restclient-1.07a3888-checkout 0.05 0.01 19.1%
emacs-el-search qsr69...-emacs-el-search-0.0-1.f26277b.tar.gz 0.09 0.02 22.1%
emacs-ivy r6j1m...-emacs-ivy-0.10.0-checkout 0.62 0.11 17.6%
emacs-helm-make 1crr4...-emacs-helm-make-0.1.0-1.21c1bfa-checkout 0.02 0.01 20.6%
emacs-helm-c-yasnippet 57mnf...-emacs-helm-c-yasnippet-0.6.7-1.65ca732-checkout 0.06 0.03 48.3%
emacs-writegood-mode xp9b3...-emacs-writegood-mode-checkout 0.02 0.00 20.5%
emacs-butler 45ilp...-emacs-butler-0.2.4 0.03 0.01 18.7%
emacs-slack 26cds...-emacs-slack-0-1.58b1309-checkout 0.40 0.05 12.8%
emacs-danneskjold-theme 90hbr...-emacs-danneskjold-theme-0.0.0-1.8733d2f-checkout 0.54 0.44 81.6%
emacs-circe bsyci...-emacs-circe-2.6-checkout 0.60 0.20 33.9%
emacs-typo 2ixcw...-emacs-typo-1.1 0.03 0.01 20.4%
emacs-memoize 1h1zy...-emacs-memoize-20130421.b55eab0.tar.gz 0.02 0.00 20.9%
emacs-browse-at-remote 7g6c5...-emacs-browse-at-remote-0.9.0-1.b5cff79-checkout 0.04 0.01 16.5%
emacs-request mq5ll...-emacs-request-0.3.0-checkout 0.19 0.04 19.6%
emacs-deferred pzbya...-emacs-deferred-0.3.2 0.20 0.03 16.3%
emacs-dired-hacks qdsww...-emacs-dired-hacks-0.0.1-1.eda6800-checkout 0.22 0.04 18.2%
emacs-graphviz-dot-mode rind2...-emacs-graphviz-dot-mode-0.3.10-1.fdaabbc-checkout 0.57 0.30 51.3%
emacs-lua-mode 07v2y...-emacs-lua-mode-20151025.1-652e299cb.checkout 0.14 0.03 21.6%
emacs-sr-speedbar bljyp...-emacs-sr-speedbar-20161025-0.77a83fb-checkout 0.03 0.01 19.3%
propeller-load 5gwzk...-propeller-load-4c46ecbe79ffbecd2ce918497ace5b956736b5a3-checkout 620.10 67.80 10.9%
proplib p1dkd...-proplib-4c46ecbe79ffbecd2ce918497ace5b956736b5a3-checkout 620.10 67.81 10.9%
binutils-vc4 gfhc6...-binutils-vc4-2.23.51-0.708acc8-checkout 141.97 14.40 10.1%
gcc-vc4 kh6bc...-gcc-vc4-5.4.0-checkout 564.45 71.02 12.6%
spinsim 0dwyg...-spinsim-66915a7ad1a3a2cf990a725bb341fab8d11eb620-checkout 0.75 0.10 13.4%
libjaylink j7zmg...-libjaylink-0.1.0-2.699b700-checkout 0.32 0.05 16.0%
kicad 58h1j...-kicad-4.0-2.5f4599f-checkout 98.86 11.69 11.8%
disorderfs akxf1...-git-checkout 0.08 0.02 21.5%
b43-tools ckiq9...-b43-tools-20140625-1.8dce532-checkout 0.78 0.10 13.0%
ovmf vm3bw...-edk2-20170116-1.13a50a6-checkout 201.26 20.34 10.1%
rkflashtool j8y6m...-rkflashtool-0.0.0-1.094bd64-checkout 0.08 0.01 17.3%
ntk k82sa...-ntk-1.3.0-checkout 8.64 1.93 22.3%
font-go gb8iv...-go-image-20170330-1.f03a046-checkout 16.19 3.07 19.0%
icestorm 59gfn...-icestorm-0.0-1-12b2295c9-checkout 2.82 0.18 6.2%
arachne-pnr nix78...-arachne-pnr-0.0-1-52e69ed20-checkout 0.56 0.06 11.5%
ois kjwgf...-ois-1.3 1.01 0.10 10.2%
open-adventure iiiwy...-open-adventure-2.5-2.d43854f-checkout 1.67 0.12 7.0%
emulation-station h1rjr...-emulation-station-2.0.1-checkout 4.81 0.73 15.1%
badass 3216w...-badass-0.0-0.3c3cd66-checkout 0.02 0.00 20.1%
roguebox-adventures l63ak...-roguebox-adventures-2.1.2-1.19a2c34-checkout 18.18 16.98 93.4%
dosbox w5ik0...-dosbox-0.74.svn3947-checkout 5.05 0.79 15.6%
dia i79dl...-dia-0.97.2-fbc3061-checkout 41.73 7.07 16.9%
byzanz avvhn...-byzanz-0.2-1.f7af3a5-checkout 0.67 0.10 15.4%
gnu-standards kbwrs...-cvs-checkout 0.38 0.10 25.5%
guile-gnunet lrmhb...-guile-gnunet-0.0.383eac2-checkout 0.24 0.04 14.9%
go-github-com-alsm-ioprogress 0jqqf...-git-checkout 0.02 0.00 21.9%
go-github-com-aki237-nscjar 7njw3...-git-checkout 0.02 0.00 14.3%
go-github-com-davidjpeacock-cli qqk5g...-git-checkout 0.30 0.05 15.2%
go-gopkg.in-tomb.v2 0rc3i...-go-gopkg.in-tomb.v2-0.0.0-0.d5d1b58.tar.gz 0.03 0.00 15.5%
go-github.com-jessevdk-go-flags a7cln...-go-github.com-jessevdk-go-flags-1.3.0-checkout 0.24 0.04 18.6%
gpscorrelate r62p1...-git-checkout 0.46 0.13 29.2%
brdf-explorer 34908...-brdf-explorer-1.0.0-1.5b2cd46f3-checkout 18.81 3.68 19.6%
roffit ir6rd...-roffit-e5228388e3faf2b7f1ae5bd048ad46ed565304c6-checkout 0.04 0.01 22.5%
ganv l7rfj...-git-checkout 0.54 0.17 31.9%
guile-wm l0lr7...-guile-wm-1.0-1.f3c7b3b-checkout 0.89 0.18 20.4%
guile-xcb 4idd3...-guile-xcb-1.3-1.db7d5a3-checkout 1.50 0.25 16.4%
guile-dsv 9hgd3...-guile-dsv-0.2.1-checkout 0.30 0.05 17.9%
guile2.0-git plx98...-guile-git-0.0-4.951a32c-checkout 0.36 0.08 21.0%
guile-gdbm-ffi vllal...-guile-gdbm-ffi-20120209.fa1d5b6-checkout 0.06 0.01 23.6%
guile-ics ppb04...-guile-ics-0.1.1-checkout 0.30 0.05 17.2%
guile-miniadapton 5hq5z...-guile-miniadapton-0-1.1b5749422-checkout 0.02 0.00 16.1%
guile-for-guile-emacs afxy2...-guile-for-guile-emacs-20150510.d8d9a8d-checkout 23.97 3.44 14.4%
guile-syntax-highlight b7dsl...-guile-syntax-highlight-0.0.0.a047675-checkout 0.05 0.01 17.7%
guildhall f330d...-guildhall-0-1.2fe2cc539-checkout 1.53 0.20 12.9%
guile-minikanren jcx53...-guile-minikanren-20150424.e844d85-checkout 0.10 0.01 13.0%
mcron2 pyj2k...-mcron2-1.0.8-0.31baff1-checkout 0.27 0.06 22.6%
guile-bash 7p7mk...-guile-bash-0.1.6-0.1eabc56-checkout 0.20 0.03 17.0%
corrode nb842...-corrode-0.0.1-b6699fb-checkout 0.29 0.07 22.7%
cl-yale-haskell vn1sy...-yale-haskell-85f94c72a16c5f70301dd8db04cde9de2d7dd270-checkout 2.64 0.50 18.9%
idris-bifunctors 2b3y9...-idris-bifunctors-0.1-1.53d06a6-checkout 0.14 0.01 5.7%
idris-lens hfs5h...-idris-lens-0.1-1.26f0120-checkout 0.08 0.01 7.9%
idris-lightyear nz2ch...-idris-lightyear-0.1-1.6d65ad1-checkout 0.07 0.01 14.9%
idris-wl-pprint 0422w...-idris-wl-pprint-0.1-1.1d365fc-checkout 0.06 0.01 14.6%
stb-image-for-extempore ca99d...-stb-image-for-extempore-0-1.152a250a7-checkout 3.38 0.69 20.6%
java-aopalliance ybdw8...-java-aopalliance-1.0 0.24 0.02 6.9%
java-cofoja kh0sm...-java-cofoja-1.3-checkout 0.79 0.08 10.2%
java-geronimo-xbean-reflect cyk69...-java-geronimo-xbean-reflect-4.5 4.73 0.22 4.7%
java-jmh iyq7l...-java-jmh-1.17.5-checkout 6.08 0.24 3.9%
classpath 0y51r...-classpath-0.99-1.e7c13ee0c-checkout 65.42 7.16 10.9%
javacc 0x6b3...-javacc-4.1-checkout 2.83 0.57 20.1%
java-jnacl klgqr...-git-checkout 0.15 0.01 5.0%
js-json2 51862...-js-json2-2016-10-28.1-031b1d9-checkout 0.05 0.01 19.5%
mujs x6ss1...-mujs-1.0.2-checkout 0.42 0.09 20.5%
kodi-cli cyi7s...-kodi-cli-1.1-1.104dc23-checkout 0.04 0.01 25.6%
crossguid jwhwq...-crossguid-0.0-1.8f399e8-checkout 0.20 0.04 21.1%
libusb4java 267pc...-git-checkout 0.17 0.02 10.7%
python-lirc hhz9b...-python-lirc-1.2.1-1.4091fe9 0.07 0.02 21.3%
sbcl-cl-unicode npz10...-cl-unicode-0.1.5-1.9fcd06f-checkout 2.65 0.25 9.3%
cl-trivial-gray-streams 2hs2f...-trivial-gray-streams-0.0.0-1.0483ade-checkout 0.05 0.01 15.6%
sbcl-alexandria x91h6...-alexandria-0.0.0-1.926a066-checkout 0.22 0.04 19.3%
gcl bsi2d...-gcl-2.6.12-1.5956140-checkout 28.57 4.44 15.5%
sbcl-cl-uglify-js rwnr0...-git-checkout 0.10 0.02 15.9%
femtolisp iviwg...-femtolisp-0.0.0-1.68c5b12-checkout 2.59 0.25 9.8%
sbcl-parse-js 7hph5...-sbcl-parse-js-fbadc6029bec7039602abfc06c73bb52970998f6-checkout 0.05 0.01 18.6%
selene 8m547...-Selene 0.17 0.02 14.0%
ghmm g8by8...-ghmm-0.9-rc3-0.2341 2.59 0.46 17.7%
esmtp n5lsc...-esmtp-1.2-checkout 0.13 0.04 27.8%
dovecot-libsodium-plugin bn6q7...-dovecot-libsodium-plugin-0.0.0-1.044de73-checkout 0.07 0.01 21.2%
notmuch-addrlookup-c ziksg...-notmuch-addrlookup-c-8-1.88f156d-checkout 0.03 0.01 23.6%
notifymuch 7k1wg...-notifymuch-0.1-1.9d4aaf5-checkout 0.03 0.00 11.0%
blis-haswell 4p35s...-blis-0.2.2-checkout 12.80 0.78 6.1%
maxflow b4s25...-maxflow-3.04-checkout 0.11 0.02 22.0%
reducelcs cg88w...-reducelcs-1.0-1.474f88d-checkout 0.08 0.02 25.6%
psyclpc ln6m2...-psyclpc-20160821-2.61cf9aa-checkout 6.47 1.09 16.8%
dino ay0qv...-dino-0.0-3.f25fadde2-checkout 1.82 0.18 9.8%
libtoxcore q58hf...-libtoxcore-0.0.0-1.755f084-checkout 1.96 0.28 14.2%
microcom rar0g...-microcom-2016.01.09-checkout 0.10 0.02 22.9%
jack-capture n37sl...-jack-capture-0.9.73-checkout 0.23 0.05 21.8%
gx-vintage-fuzz-master-lv2 xfzfs...-gx-vintage-fuzz-master-lv2-0.1-checkout 0.08 0.03 35.0%
non-mixer bg1pp...-non-sequencer-1.9.5-3.10c31e5-checkout 8.04 3.08 38.3%
gx-tone-mender-lv2 rcygs...-gx-tone-mender-lv2-0-1.b6780b4a3-checkout 0.11 0.02 20.3%
gx-overdriver-lv2 r5mf1...-gx-overdriver-lv2-0-1.ed7180198-checkout 0.07 0.02 26.0%
mod-utilities 4m1b4...-mod-utilities-0-2.80ea3ea9f-checkout 6.03 2.27 37.7%
gx-saturator-lv2 4b0nw...-gx-saturator-lv2-0-3.605330f43-checkout 1.84 0.76 41.4%
gx-guvnor-lv2 8fbs3...-gx-guvnor-lv2-0.1-checkout 0.16 0.03 19.2%
gx-switchless-wah-lv2 9ha8q...-gx-switchless-wah-lv2-0-2.7b0869120-checkout 0.06 0.02 25.8%
gx-voodoo-fuzz-lv2 mpyz2...-gx-voodoo-fuzz-lv2-0.1-checkout 0.08 0.02 24.9%
gx-suppa-tone-bender-lv2 dpgpd...-gx-suppa-tone-bender-lv2-0.1-checkout 0.07 0.02 25.7%
libechonest p5rkc...-libechonest-2.3.1-checkout 12.23 11.51 94.1%
gx-push-pull-lv2 0q7as...-gx-push-pull-lv2-0-1.7f76ae206-checkout 0.07 0.02 26.2%
gx-slow-gear-lv2 w7nd3...-gx-slow-gear-lv2-0-3.5d37e775b-checkout 1.66 0.86 51.8%
instantmusic y36yp...-instantmusic-1.0-1.300891d-checkout 0.06 0.01 13.5%
mod-host vbllj...-mod-host-0.10.6-2.299a39774-checkout 0.43 0.06 13.7%
sonivox-eas kw163...-sonivox-eas-1.1.0-checkout 3.17 0.72 22.6%
ingen 8hlqc...-ingen-0.0.0-2.cc4a4db33-checkout 2.22 0.40 17.9%
gx-vbass-preamp-lv2 xgb1m...-gx-vbass-preamp-lv2-0-2.eb999b0ca-checkout 0.73 0.35 48.0%
gx-super-fuzz-lv2 00mjm...-gx-super-fuzz-lv2-0.1-checkout 0.08 0.02 24.6%
gx-hyperion-lv2 s21j4...-gx-hyperion-lv2-0.1-checkout 0.06 0.02 26.2%
muse-sequencer f3lw8...-muse-sequencer-3.0.0-checkout 37.11 6.03 16.3%
tlsdate zq6dn...-tlsdate-0.0.13-checkout 0.84 0.21 25.0%
guix 5krqj...-guix-0.14.0-7.33988f9-checkout 41.56 7.36 17.7%
patches lpvs8...-patches-0.0-1.ef1b8a7-checkout 0.11 0.02 21.8%
pumpa wyps7...-pumpa-0.9.2-checkout 1.76 0.26 14.7%
python-axolotl-curve25519 yla44...-python-axolotl-curve25519-0.1-checkout 0.31 0.07 21.6%
python2-elib.intl dzrar...-git-checkout 0.13 0.03 18.9%
python2-kivy-next skyhx...-python-kivy-next-1.9.1-1.a988c5e-checkout 27.99 17.70 63.2%
freerdp r4xxl...-freerdp-1.1-1.03ab683-checkout 16.32 3.60 22.1%
enki sd00n...-enki-2.0pre-0.afd2d8e-checkout 4.66 3.51 75.2%
ruby-minitar h1wrk...-ruby-minitar-0.5.4-1.e25205ec-checkout 0.15 0.02 15.7%
ruby-nokogumbo nrqzd...-ruby-nokogumbo-1.4.7-1.fb51ff29-checkout 0.07 0.01 15.8%
ruby-minitest-pretty-diff xlbza...-ruby-minitest-pretty-diff-0.1-1.11f32e93-checkout 0.04 0.00 6.3%
ruby-tdiff 7fj7m...-ruby-tdiff-0.3.3-1.b662a604-checkout 0.06 0.01 8.5%
ruby-nokogiri-diff 43pbf...-ruby-nokogiri-diff-0.2.0-1.a38491e4-checkout 0.07 0.00 7.6%
sicp srxd5...-sicp-20170703-1.225c172-checkout 9.61 2.85 29.7%
scheme48-rx pcx0a...-scheme48-rx-0.0.0-2.dd9037f-checkout 0.20 0.04 19.7%
libsemanage 5rjhg...-selinux-20170804-checkout 28.99 3.04 10.5%
python-setools amj5l...-python-setools-4.1.1-checkout 3.44 0.27 7.9%
s-shell rp40m...-s-shell-0.0.0-2.da2e5c2-checkout 0.10 0.02 15.9%
rc 7wvw5...-rc-1.7.4-checkout 0.42 0.11 26.8%
linenoise 5zgrz...-linenoise-1.0-1.2105ce4-checkout 0.06 0.01 23.9%
scsh 1pjgq...-scsh-0.0.0-1.1144324-checkout 0.91 0.18 19.5%
usbredir gsh9j...-git-checkout 0.37 0.06 15.0%
r-catterplots rmccq...-r-catterplots-0-2.40063ec57-checkout 0.81 0.65 80.1%
r-tgstat brklf...-hg-checkout 0.23 0.06 28.1%
r-tgconfig jb4pl...-hg-checkout 0.11 0.02 13.9%
human zr1zd...-human-0.2-1.50c80e6-checkout 0.02 0.00 20.8%
libutf ipc6a...-libutf-0.0.0-1.ff4c606-checkout 1.71 0.16 9.3%
lchat n182g...-lchat-0.0.0-2.25d90f4-checkout 0.05 0.02 39.9%
go-github-com-sasha-s-go-deadlock 42k4i...-go-github-com-sasha-s-go-deadlock-0.1.0-1.03d40e5-checkout 0.04 0.01 24.7%
go-github-com-templexxx-xor 3kv6g...-go-github-com-templexxx-xor-0.1.2-checkout 0.04 0.01 14.5%
go-github-com-golang-snappy lkk0z...-go-github-com-golang-snappy-0.0.0-0.553a641-checkout 0.20 0.05 24.1%
go-golang-org-x-net-internal-iana 5jahp...-go.googlesource.com-net-0.0.0-1.d866cfc-checkout 4.27 0.66 15.5%
go-github-com-klauspost-cpuid 8wnn5...-go-github-com-klauspost-cpuid-1.1-0.eae9b3e-checkout 0.42 0.25 58.5%
go-github-com-bkaradzic-go-lz4 9975v...-go-github-com-bkaradzic-go-lz4-0.0.0-0.7224d8d-checkout 0.62 0.18 29.8%
go-github-com-xtaci-smux vnzhz...-go-github-com-xtaci-smux-1.0.6-0.c3e1824-checkout 0.16 0.12 78.0%
go-golang-org-x-crypto-cast5 bn0nd...-go.googlesource.com-crypto-0.0.0-1.95a4943-checkout 3.46 1.21 34.9%
go-github-com-kardianos-osext rnhvi...-go-github-com-kardianos-osext-0.0.0-1.ae77be6-checkout 0.02 0.00 22.5%
go-github-com-chmduquesne-rollinghash-adler32 9qwvc...-go-github-com-chmduquesne-rollinghash-adler32-0.0.0-1.3dc7875-checkout 0.13 0.02 12.8%
go-github-com-edsrzf-mmap-go qvi0x...-go-github-com-edsrzf-mmap-go-0.0.0-0.0bce6a6-checkout 0.02 0.01 22.9%
go-golang-org-x-net-ipv4 a78xp...-go-golang-org-x-net-ipv4-0.0.0-1.d866cfc-checkout 4.27 0.66 15.5%
go-github-com-pkg-errors d989q...-go-github-com-pkg-errors-0.0.0-1.e881fd5-checkout 0.05 0.01 19.8%
go-github-com-calmh-du vk1xv...-go-github-com-calmh-du-1.0.1-checkout 0.03 0.00 7.9%
go-github-com-minio-sha256-simd 38mjh...-go-github-com-minio-sha256-simd-0.0.0-1.ad98a36-checkout 0.48 0.04 8.1%
go-github-com-oschwald-maxminddb-golang w2qkr...-go-github-com-oschwald-maxminddb-golang-1.2.0-0.26fe5ac-checkout 0.08 0.02 19.2%
go-github-com-lib-pq kl3iv...-go-github-com-lib-pq-0.0.0-1.83612a5-checkout 0.37 0.07 19.8%
go-github-com-tjfoc-gmsm-sm4 j9sdg...-go-github-com-tjfoc-gmsm-sm4-0.0.0-1.98aa888-checkout 0.23 0.05 19.7%
go-github-com-audriusbutkevicius-go-nat-pmp 3sag6...-go-github-com-audriusbutkevicius-go-nat-pmp-0.0.0-0.452c976-checkout 0.01 0.00 18.2%
go-github-com-d4l3k-messagediff ky0nc...-go-github-com-d4l3k-messagediff-1.1.0-0.29f32d8-checkout 0.05 0.01 16.4%
go-golang-org-x-text-unicode-norm ac0n4...-go.googlesource.com-text-0.0.0-1.e19ae14-checkout 25.84 3.23 12.5%
go-github-com-audriusbutkevicius-pfilter nzfpq...-go-github-com-audriusbutkevicius-pfilter-0.0.0-2.9dca34a-checkout 0.01 0.00 20.7%
go-github-com-gogo-protobuf-protoc-gen-gogo 7zacv...-go-github-com-gogo-protobuf-protoc-gen-gogo-0.2-0.efccd33-checkout 23.43 0.80 3.4%
go-github-com-oschwald-geoip2-golang jf28a...-go-github-com-oschwald-geoip2-golang-1.1.0-checkout 0.04 0.01 14.9%
go-github-com-gogo-protobuf if2x5...-go-github-com-gogo-protobuf-0.5-1.35b81a0-checkout 15.17 0.68 4.5%
go-github-com-jackpal-gateway vbzn3...-go-github-com-jackpal-gateway-0.0.0-0.5795ac8-checkout 0.02 0.00 16.5%
go-github-com-petermattis-goid icb8n...-go-github-com-petermattis-goid-0.0.0-0.3db12eb-checkout 0.03 0.01 19.7%
go-github-com-calmh-xdr crqwx...-go-github-com-calmh-xdr-2.0.1-0.08e072f-checkout 0.07 0.01 14.1%
go-github-com-gobwas-glob v3asr...-go-github-com-gobwas-glob-0.0.0-0.51eb1ee-checkout 0.21 0.02 10.1%
go-github-com-stathat-go 3w3df...-go-github-com-stathat-go-0.0.0-0.74669b9-checkout 0.03 0.01 17.4%
go-github-com-syndtr-goleveldb 49g50...-go-github-com-syndtr-goleveldb-0.0.0-2.34011bf-checkout 0.68 0.11 15.8%
go-github-com-kballard-go-shellquote la1q1...-go-github-com-kballard-go-shellquote-0.0.0-0.cd60e84-checkout 0.02 0.00 20.8%
go-github-com-zillode-notify 28mfc...-go-github-com-zillode-notify-0.0.0-1.8fff849-checkout 0.23 0.05 20.0%
go-github-com-audriusbutkevicius-cli c1isd...-go-github-com-audriusbutkevicius-cli-0.0.0-0.7f561c7-checkout 0.08 0.01 17.2%
go-github-com-golang-groupcache-lru hzchs...-go-github-com-golang-groupcache-lru-0.0.0-1.84a468c-checkout 0.13 0.02 17.2%
go-github-com-vitrun-qart-gf256 v7sg9...-go-github-com-vitrun-qart-0.0.0-0.bf64b92-checkout 0.10 0.02 21.0%
go-github-com-rcrowley-go-metrics 3wsdr...-go-github-com-rcrowley-go-metrics-0.0.0-1.e181e09-checkout 0.22 0.03 13.0%
go-golang-org-x-time-rate a3vcg...-go-golang-org-x-time-rate-0.0.0-1.6dc1736-checkout 0.04 0.01 19.9%
go-github-com-thejerf-suture ikp7l...-go-github-com-thejerf-suture-2.0.1-0.87e298c-checkout 0.05 0.01 26.1%
go-golang-org-x-sys-unix 6jzmc...-go-golang-org-x-sys-unix-0.0.0-1.8380141-checkout 4.88 0.23 4.7%
go-github-com-ccding-go-stun 9ikw3...-go-github-com-ccding-go-stun-0.0.0-1.d9bbe8f-checkout 0.07 0.01 18.5%
go-github-com-templexxx-cpufeat igsxi...-go-github-com-templexxx-cpufeat-0.0.0-0.3794dfb-checkout 0.01 0.00 18.4%
go-github-com-klauspost-reedsolomon 0vdn9...-go-github-com-klauspost-reedsolomon-1.6-0.0b30fa7-checkout 0.61 0.10 15.7%
go-github-com-xtaci-kcp-go hqlkv...-go-github-com-xtaci-kcp-go-0.0.0-0.86eebd5-checkout 0.16 0.07 43.6%
go-golang.org-x-crypto-ssh-terminal 287i8...-go-golang.org-x-crypto-ssh-terminal-0.0.0-0.c78caca-checkout 3.14 1.10 35.0%
go-github.com-nsf-termbox-go iqxvl...-go-github.com-nsf-termbox-go-0.0.0-0.4ed959e-checkout 0.13 0.03 19.2%
go-github.com-howeyc-gopass w7047...-go-github.com-howeyc-gopass-0.0.0-0.bf9dde6-checkout 0.04 0.01 26.1%
termite f0zkv...-termite-13-checkout 0.14 0.02 17.4%
texlive-fonts-latex hq540...-svn-checkout 0.09 0.01 12.4%
texlive-metapost 48djy...-svn-checkout 3.91 0.56 14.4%
texlive-fonts-rsfs i5yp4...-svn-checkout 0.05 0.01 15.8%
texlive-latex-trimspaces 4nqlk...-svn-checkout 0.01 0.00 15.7%
texlive-latex-psnfss 2fann...-svn-checkout 0.07 0.02 21.3%
texlive-latex-subfigure n3p72...-svn-checkout 0.16 0.03 21.1%
texlive-latex-tools dd6jq...-svn-checkout 0.94 0.21 22.0%
texlive-latex-colortbl jhvrk...-svn-checkout 0.05 0.01 22.4%
texlive-fonts-amsfonts j8k2p...-svn-checkout 0.69 0.06 9.4%
texlive-latex-float vwvqv...-svn-checkout 0.05 0.01 26.4%
texlive-latex-fncychap xmw59...-texlive-latex-fncychap-44591-checkout 0.03 0.00 17.5%
texlive-latex-oberdiek 3bh3f...-svn-checkout 4.66 0.48 10.3%
texlive-latex-cmap 95ihl...-texlive-latex-cmap-44591-checkout 0.03 0.00 14.7%
texlive-latex-filemod 93qvd...-svn-checkout 0.02 0.00 12.8%
texlive-latex-wasysym nk2rk...-svn-checkout 0.03 0.01 22.2%
texlive-latex-anysize l3hns...-svn-checkout 0.01 0.00 10.5%
texlive-latex-seminar 78cjs...-svn-checkout 0.17 0.03 16.8%
texlive-latex-wrapfig rzfx4...-texlive-latex-wrapfig-44591-checkout 0.03 0.01 26.5%
texlive-tex-texinfo 5dg1w...-svn-checkout 0.40 0.09 23.4%
texlive-latex-dinbrief 6b56j...-texlive-latex-dinbrief-44591-checkout 0.25 0.05 21.5%
texlive-latex-acronym d52jm...-texlive-latex-acronym-44591-checkout 0.07 0.01 21.6%
texlive-latex-changebar zj29x...-svn-checkout 0.12 0.03 21.6%
texlive-fonts-cm 15ld8...-svn-checkout 0.75 0.08 10.2%
texlive-latex-overpic 9j1ln...-svn-checkout 0.01 0.00 12.4%
texlive-fonts-txfonts 6j7ik...-texlive-fonts-txfonts-44591-checkout 0.08 0.01 11.6%
texlive-latex-fontspec bzwda...-svn-checkout 0.34 0.07 21.8%
texlive-latex-amsrefs 7c149...-texlive-latex-amsrefs-44591-checkout 0.47 0.10 20.6%
texlive-latex-footmisc hz7f8...-svn-checkout 0.08 0.02 22.9%
texlive-latex-l3packages qr8j9...-svn-checkout 0.36 0.06 16.3%
texlive-fonts-stmaryrd f7hhc...-svn-checkout 0.03 0.01 18.9%
texlive-latex-jknapltx ysq4b...-svn-checkout 0.04 0.01 21.4%
texlive-latex-bigfoot 3zx72...-texlive-latex-bigfoot-44591-checkout 0.18 0.04 23.5%
texlive-latex-url p7h0j...-svn-checkout 0.02 0.00 24.7%
texlive-latex-amscls vq5l7...-svn-checkout 0.35 0.09 24.3%
texlive-latex-galois 59ksz...-texlive-latex-galois-44591-checkout 0.03 0.01 19.6%
texlive-tex-plain 7zfs9...-svn-checkout 3.79 0.38 10.0%
texlive-latex-ifplatform 5vvm4...-texlive-latex-ifplatform-44591-checkout 0.02 0.01 22.5%
texlive-latex-blindtext ya3xw...-texlive-latex-blindtext-44591-checkout 0.09 0.02 19.2%
texlive-latex-enumitem bzaa3...-texlive-latex-enumitem-44591-checkout 0.05 0.01 21.7%
texlive-latex-etoolbox 1lfbc...-texlive-latex-etoolbox-44591-checkout 0.05 0.01 14.6%
texlive-latex-appendix 115ic...-svn-checkout 0.05 0.01 21.4%
texlive-latex-amsmath 7356r...-svn-checkout 0.28 0.06 21.6%
texlive-latex-preview q8r21...-texlive-latex-preview-44591-checkout 0.09 0.02 24.9%
texlive-latex-base glsk0...-svn-checkout 2.73 0.49 18.1%
texlive-generic-babel-english wg09a...-texlive-generic-babel-english-44591-checkout 0.03 0.01 19.6%
texlive-generic-dehyph-exptl z0cr7...-svn-checkout 0.49 0.08 17.2%
texlive-latex-polyglossia 1dcz8...-texlive-latex-polyglossia-44591-checkout 0.33 0.07 20.8%
texlive-latex-acmart s35zg...-texlive-latex-acmart-1.45-checkout 0.19 0.04 20.8%
texlive-latex-varwidth ad5p7...-texlive-latex-varwidth-44591-checkout 0.02 0.00 20.7%
texlive-latex-mdwtools 5fbjd...-texlive-latex-mdwtools-44591-checkout 0.42 0.10 23.5%
texlive-latex-parskip 3a590...-texlive-latex-parskip-44591-checkout 0.01 0.00 13.7%
texlive-latex-tabulary gmybm...-texlive-latex-tabulary-44591-checkout 0.04 0.01 21.8%
texlive-latex-pstool bv42n...-texlive-latex-pstool-44591-checkout 0.05 0.01 26.6%
texlive-latex-expdlist vj9bl...-texlive-latex-expdlist-44591-checkout 0.06 0.01 16.6%
texlive-latex-multirow w7pdh...-svn-checkout 0.09 0.02 20.0%
texlive-latex-draftwatermark z85s4...-texlive-latex-draftwatermark-44591-checkout 0.03 0.01 21.6%
texlive-latex-framed bhks6...-texlive-latex-framed-44591-checkout 0.03 0.01 24.8%
texlive-generic-epsf nlr8z...-svn-checkout 0.06 0.01 11.5%
texlive-latex-graphics xb9k1...-svn-checkout 0.23 0.05 21.0%
texlive-latex-environ hzi8j...-texlive-latex-environ-44591-checkout 0.03 0.01 22.3%
texlive-latex-babel ap2r5...-svn-checkout 0.38 0.09 23.6%
texlive-latex-filecontents aczcl...-svn-checkout 0.02 0.01 22.6%
texlive-latex-fancybox qz4p4...-svn-checkout 0.03 0.01 21.1%
texlive-latex-titlesec 151gi...-svn-checkout 0.10 0.02 19.2%
texlive-dvips q3g4s...-svn-checkout 6.92 1.10 15.8%
texlive-latex-psfrag z3n4p...-texlive-latex-psfrag-44591-checkout 0.05 0.01 26.2%
texlive-latex-eepic nc1bw...-svn-checkout 0.05 0.01 21.1%
texlive-latex-ucs hcs8p...-texlive-latex-ucs-44591-checkout 2.31 0.26 11.3%
texlive-latex-capt-of 803vw...-texlive-latex-capt-of-44591-checkout 0.01 0.00 9.3%
texlive-fonts-knuth-lib a6d1m...-svn-checkout 0.05 0.01 24.2%
texlive-latex-lh 1r3im...-svn-checkout 0.08 0.01 16.5%
texlive-generic-tex-ini-files v31ah...-svn-checkout 0.02 0.00 18.7%
texlive-latex-xcolor 9j8p8...-svn-checkout 0.34 0.07 21.9%
texlive-latex-eqparbox lcsnm...-texlive-latex-eqparbox-44591-checkout 0.07 0.02 22.7%
texlive-latex-fancyvrb 2kbdl...-svn-checkout 0.18 0.03 17.9%
texlive-metafont-base df56f...-svn-checkout 0.40 0.08 21.1%
texlive-latex-listings kqch1...-svn-checkout 0.94 0.22 24.0%
texlive-latex-g-brief cbypr...-texlive-latex-g-brief-44591-checkout 0.07 0.01 16.5%
texlive-generic-hyph-utf8 rc66j...-svn-checkout 8.22 1.04 12.7%
texlive-generic-ifxetex 77c0h...-svn-checkout 0.01 0.00 16.9%
texlive-latex-upquote h2dw6...-texlive-latex-upquote-44591-checkout 0.01 0.00 19.3%
texlive-latex-fancyhdr jnppk...-svn-checkout 0.02 0.00 16.0%
texlive-latex-amsfonts rgyb8...-svn-checkout 0.19 0.05 24.1%
texlive-latex-cyrillic 9f9mz...-svn-checkout 0.17 0.02 12.2%
texlive-latex-pdfpages gyzn9...-svn-checkout 0.18 0.03 18.5%
texlive-latex-gcite j41dd...-texlive-latex-gcite-44591-checkout 0.03 0.01 26.1%
texlive-latex-supertabular sp25p...-texlive-latex-supertabular-44591-checkout 0.06 0.01 18.2%
texlive-latex-l3kernel apk48...-svn-checkout 2.66 0.44 16.6%
texlive-latex-geometry z88nk...-texlive-latex-geometry-44591-checkout 0.16 0.03 20.0%
texlive-latex-type1cm 8lny2...-texlive-latex-type1cm-44591-checkout 0.02 0.00 12.2%
texlive-latex-eso-pic 414np...-svn-checkout 0.03 0.01 18.8%
texlive-latex-natbib 548pk...-svn-checkout 0.27 0.06 22.3%
texlive-latex-threeparttable 8v3mp...-svn-checkout 0.02 0.00 23.7%
texlive-generic-pdftex 8z4xr...-svn-checkout 0.22 0.03 11.5%
texlive-fontname 6cvf0...-texlive-fontname-44591.tar.gz 1.44 0.11 7.5%
texlive-fonts-ec yfs60...-svn-checkout 1.12 0.10 8.9%
texlive-generic-unicode-data p4d5s...-svn-checkout 2.11 0.21 10.0%
kakoune rj6ks...-kakoune-0.0.0-1.125c8b7-checkout 3.43 0.32 9.2%
go-github.com-mattn-go-runewidth p19nc...-go-github.com-mattn-go-runewidth-0.0.2-checkout 0.08 0.01 17.5%
python-editdistance szas5...-git-checkout 0.11 0.02 14.3%
recode xx34z...-recode-3.7.0.201402-checkout 6.87 0.98 14.2%
cityhash x8c1l...-cityhash-1.1-2.8af9b8c.tar.gz 1.45 0.24 16.5%
tmux-themepack f0590...-tmux-themepack-0.0.0-1.03a3728-checkout 0.10 0.00 4.3%
nyx azabg...-nyx-1.9-1.fea2091-checkout 5.34 4.41 82.7%
git-remote-gcrypt lxnx6...-git-remote-gcrypt-1.0.1-checkout 0.10 0.02 22.8%
git-test-sequence 1hd23...-git-checkout 0.04 0.01 25.6%
git-flow lxd0v...-git-checkout 0.10 0.02 14.9%
myrepos 1rpcr...-myrepos-1.20170129-checkout 0.24 0.05 19.4%
ffmpeg-git ilml0...-ffmpeg-git-3.4-1.3f8874406-checkout 60.70 8.23 13.6%
libsmpeg nl3nn...-libsmpeg-0.4.5-checkout 1.53 0.26 17.0%
vim-scheme 4qzwm...-vim-scheme-0.0.0-1.9382798-checkout 0.06 0.00 6.3%
vim-airline-themes 26h5i...-vim-airline-themes-0.0.0-1.6026eb7-checkout 0.28 0.03 9.0%
vim-neosnippet 8z74v...-vim-neosnippet-4.2-1.1bd7e23-checkout 0.39 0.04 9.5%
vim-luna ih9qf...-vim-luna-0.0.0-1.6336199-checkout 0.04 0.00 11.3%
vim-fugitive kpssr...-vim-fugitive-2.2-1.de6c057-checkout 0.15 0.03 19.1%
vim-context-filetype m8xds...-vim-context-filetype-0.0.0-1.5e85f8c-checkout 0.07 0.01 10.0%
vim-neosnippet-snippets 5b0hf...-vim-neosnippet-snippets-0.0.0-1.8e2b1c0-checkout 0.15 0.03 19.9%
vpnc-scripts zy80j...-git-checkout 0.07 0.02 25.5%
w3m 827af...-w3m-0.5.3+git20170102-checkout 9.35 1.11 11.9%
libwebsockets 1k4c1...-libwebsockets-1.3 3.60 2.60 72.1%
nginx-documentation 8f7pf...-nginx-documentation-2017-04-12-1961-dd4b6c564e10 10.43 1.54 14.8%
wget2 3c93r...-wget2-1.0.0-checkout 42.92 4.87 11.3%
python2-i3-py 2ivq6...-python-i3-py-0.6.5-checkout 0.06 0.01 21.1%
nxbelld h3a3x...-nxbelld-0.1.2-checkout 1.40 0.22 15.4%
dzen jcc21...-dzen-0.9.5-1.488ab66 0.17 0.03 18.6%
xcompmgr h77gz...-xcompmgr-1.1.7 0.08 0.01 19.6%
xf86-video-intel p9fk4...-xf86-video-intel-2.99.917-9-af6d8e9 9.07 0.91 10.1%
xf86-video-freedreno j21yc...-xf86-video-freedreno-1.4.0-1-ccba8f8 0.28 0.05 16.6%
zile-on-guile qcjsz...-zile-on-guile-2.4.14-0.fd09781-checkout 30.74 3.48 11.3%
------------------------------------------------------------------------------------------------------------------------------
TOTALS: 3783.86 538.07 18.6% (median)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Compress VCS checkouts?
2018-02-02 20:50 Compress VCS checkouts? Eric Bavier
@ 2018-02-03 20:10 ` Efraim Flashner
2018-02-05 10:27 ` Ludovic Courtès
1 sibling, 0 replies; 5+ messages in thread
From: Efraim Flashner @ 2018-02-03 20:10 UTC (permalink / raw)
To: Eric Bavier; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 2424 bytes --]
On Fri, Feb 02, 2018 at 02:50:11PM -0600, Eric Bavier wrote:
> Hello Guix,
>
> A while back I toyed with the idea of a "Sources DVD" for Guix, much
> like Trisquel has. While looking into the feasability of this, I
> noticed that the git/svn/cvs/hg checkouts our packages use are not
> compressed in the store. So I wrote a simple script to compare the raw
> size of all source checkouts with their corresponding xz-compressed
> tarballs.
>
> At the time, I couldn't convince myself the trouble was worth the
> effort; we had only 385MiB of uncompressed source checkouts, which
> would compress to 135MiB.
>
> Since that initial investigation, the number of packages using VCS
> checkouts has risen from 50 to 389, and the on-disk size of that source
> has risen to 3784MiB uncompressed. We could compress that down to
> 538MiB.
>
> Compression in not free, but I think it would be nice for our build
> servers who have to store and serve (which, iirc, requires them to
> compress it anyhow) this source.
>
> What do people think?
>
> `~Eric
> name path size (MiB) tar.xz rel%
> footswitch cwrqb...-footswitch-0.1-1.7cb0a93-checkout 0.04 0.01 17.5%
< snip >
> ------------------------------------------------------------------------------------------------------------------------------
> TOTALS: 3783.86 538.07 18.6% (median)
In case anyone else had to look at it twice, that's the relative size of
the new tarred source compared to the original. The sources don't change
very often, so the cost of the initial compression doesn't come up much,
and serving compressed data should be much faster. Interestingly, on
machines with the store on an external HDD, it might be faster to load
and unpack the source to ram than to just copy over USB2.
It does make grepping through the source in the store harder, but we
already couldn't do that with the tarballs. I think it would be overall
a positive change.
--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Compress VCS checkouts?
2018-02-02 20:50 Compress VCS checkouts? Eric Bavier
2018-02-03 20:10 ` Efraim Flashner
@ 2018-02-05 10:27 ` Ludovic Courtès
2018-02-06 0:49 ` Eric Bavier
1 sibling, 1 reply; 5+ messages in thread
From: Ludovic Courtès @ 2018-02-05 10:27 UTC (permalink / raw)
To: Eric Bavier; +Cc: guix-devel
Hello!
Eric Bavier <ericbavier@centurylink.net> skribis:
> A while back I toyed with the idea of a "Sources DVD" for Guix, much
> like Trisquel has. While looking into the feasability of this, I
> noticed that the git/svn/cvs/hg checkouts our packages use are not
> compressed in the store. So I wrote a simple script to compare the raw
> size of all source checkouts with their corresponding xz-compressed
> tarballs.
>
> At the time, I couldn't convince myself the trouble was worth the
> effort; we had only 385MiB of uncompressed source checkouts, which
> would compress to 135MiB.
>
> Since that initial investigation, the number of packages using VCS
> checkouts has risen from 50 to 389, and the on-disk size of that source
> has risen to 3784MiB uncompressed. We could compress that down to
> 538MiB.
>
> Compression in not free, but I think it would be nice for our build
> servers who have to store and serve (which, iirc, requires them to
> compress it anyhow) this source.
‘guix publish -C’ always compresses store items, unless they are regular
files ending in “.gz”, “.xz”, etc. So there’s no problem here.
It seems to me that the only benefit of compressing checkouts would be
the use case you’re talking about. For everyday use, it’s actually more
efficient to leave the uncompressed, so that people don’t end up doing
round of compression/decompression when they run “guix build”.
WDYT?
That said, at worst, this could be addressed by writing a specific tool
for the job.
Ludo’.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Compress VCS checkouts?
2018-02-05 10:27 ` Ludovic Courtès
@ 2018-02-06 0:49 ` Eric Bavier
2018-02-07 21:56 ` Ludovic Courtès
0 siblings, 1 reply; 5+ messages in thread
From: Eric Bavier @ 2018-02-06 0:49 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
On Mon, 05 Feb 2018 11:27:10 +0100
ludo@gnu.org (Ludovic Courtès) wrote:
> Hello!
>
> Eric Bavier <ericbavier@centurylink.net> skribis:
>
> > A while back I toyed with the idea of a "Sources DVD" for Guix, much
> > like Trisquel has. While looking into the feasability of this, I
> > noticed that the git/svn/cvs/hg checkouts our packages use are not
> > compressed in the store. So I wrote a simple script to compare the raw
> > size of all source checkouts with their corresponding xz-compressed
> > tarballs.
> >
> > At the time, I couldn't convince myself the trouble was worth the
> > effort; we had only 385MiB of uncompressed source checkouts, which
> > would compress to 135MiB.
> >
> > Since that initial investigation, the number of packages using VCS
> > checkouts has risen from 50 to 389, and the on-disk size of that source
> > has risen to 3784MiB uncompressed. We could compress that down to
> > 538MiB.
> >
> > Compression in not free, but I think it would be nice for our build
> > servers who have to store and serve (which, iirc, requires them to
> > compress it anyhow) this source.
>
> ‘guix publish -C’ always compresses store items, unless they are regular
> files ending in “.gz”, “.xz”, etc. So there’s no problem here.
I see, ok.
>
> It seems to me that the only benefit of compressing checkouts would be
> the use case you’re talking about. For everyday use, it’s actually more
> efficient to leave the uncompressed, so that people don’t end up doing
> round of compression/decompression when they run “guix build”.
Often, people would end up fetching the compressed source from a
substitute server, no? And currently I already need to uncompress
what I get from a substitute server before building.
But now I'm wondering why we bother with the "repack" portion of
"patch-and-repack"?
> That said, at worst, this could be addressed by writing a specific tool
> for the job.
Yes. I was hoping it could be avoided though.
In the meantime I can continue to monitor the situation.
`~Eric
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Compress VCS checkouts?
2018-02-06 0:49 ` Eric Bavier
@ 2018-02-07 21:56 ` Ludovic Courtès
0 siblings, 0 replies; 5+ messages in thread
From: Ludovic Courtès @ 2018-02-07 21:56 UTC (permalink / raw)
To: Eric Bavier; +Cc: guix-devel
Hi Eric,
Eric Bavier <ericbavier@centurylink.net> skribis:
> On Mon, 05 Feb 2018 11:27:10 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hello!
>>
>> Eric Bavier <ericbavier@centurylink.net> skribis:
>>
>> > A while back I toyed with the idea of a "Sources DVD" for Guix, much
>> > like Trisquel has. While looking into the feasability of this, I
>> > noticed that the git/svn/cvs/hg checkouts our packages use are not
>> > compressed in the store. So I wrote a simple script to compare the raw
>> > size of all source checkouts with their corresponding xz-compressed
>> > tarballs.
>> >
>> > At the time, I couldn't convince myself the trouble was worth the
>> > effort; we had only 385MiB of uncompressed source checkouts, which
>> > would compress to 135MiB.
>> >
>> > Since that initial investigation, the number of packages using VCS
>> > checkouts has risen from 50 to 389, and the on-disk size of that source
>> > has risen to 3784MiB uncompressed. We could compress that down to
>> > 538MiB.
>> >
>> > Compression in not free, but I think it would be nice for our build
>> > servers who have to store and serve (which, iirc, requires them to
>> > compress it anyhow) this source.
>>
>> ‘guix publish -C’ always compresses store items, unless they are regular
>> files ending in “.gz”, “.xz”, etc. So there’s no problem here.
>
> I see, ok.
>
>>
>> It seems to me that the only benefit of compressing checkouts would be
>> the use case you’re talking about. For everyday use, it’s actually more
>> efficient to leave the uncompressed, so that people don’t end up doing
>> round of compression/decompression when they run “guix build”.
>
> Often, people would end up fetching the compressed source from a
> substitute server, no?
Specifically, they download a compressed nar that contains the
uncompressed checkout.
> And currently I already need to uncompress what I get from a
> substitute server before building.
>
> But now I'm wondering why we bother with the "repack" portion of
> "patch-and-repack"?
Good question! It’s mostly because it “looked nicer” this way (and
maybe at the time some of the code did not properly handle non-tarball
sources.) I hear we could apply the same “reasoning” (if we can call it
this way) to checkouts. Hmm!
>> That said, at worst, this could be addressed by writing a specific tool
>> for the job.
>
> Yes. I was hoping it could be avoided though.
The tool could be relatively simple.
Ludo’.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-02-07 21:56 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-02 20:50 Compress VCS checkouts? Eric Bavier
2018-02-03 20:10 ` Efraim Flashner
2018-02-05 10:27 ` Ludovic Courtès
2018-02-06 0:49 ` Eric Bavier
2018-02-07 21:56 ` Ludovic Courtès
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.