Re: 02/02: gnu: next: Compress the executable.

Re: 02/02: gnu: next: Compress the executable.

[Ricardo Wurmus]

Hi Pierre,

> ambrevar pushed a commit to branch master
> in repository guix.
>
> commit 0e8b91dbc45306984d682307d8b40b0e323bb4be
> Author: Pierre Neidhardt <mail@ambrevar.xyz>
> Date:   Thu Sep 5 11:53:52 2019 +0200
>
>     gnu: next: Compress the executable.
>     
>     * gnu/packages/web-browsers.scm (next)[arguments]: Compressing the executable
>       brings it from about 95 MiB to 22 MiB.

I think this commit should be reverted.

Here’s why.  Before your commit:

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix gc --references /gnu/store/bzvg9brphn5dn9q9kw6mirk4bb54wrd2-next-1.3.1
/gnu/store/0ihdh4df3gpczwrpfjxrdcm7lmrr4z8l-sbcl-next-download-manager-1.3.1
/gnu/store/0nxab7i4cmc6zjfmjz58yy0498af74vb-sbcl-log4cl-1.1.2
/gnu/store/199kx1lw836v24n01galpl8bv0km7b6l-sbcl-cl-unicode-0.1.5-1.9fcd06f
/gnu/store/22bb8xb28i6d8b2rb9jdgkxx1hnryz91-sbcl-swap-bytes-1.1
/gnu/store/2ilpaxvwv00j5lmczgc3znmhsqms284h-sbcl-cl-base64-3.3.3
/gnu/store/2xzkgsjynfjvbdj7dq3invk8ic9khad9-sbcl-iterate-20160825
/gnu/store/3695lwpxnc4yj7arvl5gbina5qjvgswj-sbcl-parenscript-2.6-1.061d8e2
/gnu/store/3d8g171h0cmwbgqhd2lpazrc2bffygfg-sbcl-trivia.level0-0.0.0-1.902e0c6
/gnu/store/3h6mz1isvicchrl2bvkzqjwqpjaj4v8c-sbcl-iolib.grovel-0.8.3
/gnu/store/3xsfp5ksd9wf0ib9bygmdcm59cn8nj2l-sbcl-closure-common-20101006-1.e3c5f5f
/gnu/store/3ycw1jw4987bnrsrzqpfyg6wamgr178d-sbcl-closer-mop-1.0.0-1.fac29ce
/gnu/store/42ri80fj80i48rf6xsycn4cmj95ipgls-sbcl-static-vectors-1.8.3-1.0681eac
/gnu/store/4mcnw6b5676z8wmfp1xjs3m0br9gpfnj-sbcl-iolib.common-lisp-0.8.3
/gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib
/gnu/store/4yj8rnffc2basyv6hksyql1f20dr833p-sbcl-trivial-features-0.8
/gnu/store/5afhdhwrp5ydsg53ibr5wfnmxi9avs43-sbcl-cxml-0.0.0-1.00b22bf
/gnu/store/5fma6i5jv4m6fi7m9j4mwz9ls30lx9r7-sbcl-trivial-gray-streams-0.0.0-1.0483ade
/gnu/store/5s2nib1lrd2101bbrivcl17kjx1mspw6-coreutils-8.30
/gnu/store/63vd8vwmf7f9m49npg68lkzbw6y8fykg-sbcl-local-time-1.0.6-1.beac054
/gnu/store/6f1gg0a6228a3514vkz6b560cws4bzx0-sbcl-trivial-garbage-0.21
/gnu/store/6f2x2f4106lyfkbxbn2kb15c6k6qsn9x-sbcl-trivial-cltl2-0.1.1-1.8eec840
/gnu/store/6pxkph74pjv5i0r67j6j2xcq2kab3bl1-sbcl-cxml+dom-0.0.0-1.00b22bf
/gnu/store/6x9rygmjjqk3dxj6p1cks5w81yldm5zk-sbcl-trivial-mimes-1.1.0-1.303f8ac
/gnu/store/74x8v5cj9zf8jlyqwmk5qgixn3sicqw7-sbcl-cl-json-0.5-1.6dfebb9
/gnu/store/7spnpq3b4grnspy7asaidsmsy15yym3v-sbcl-slime-swank-2.22
/gnu/store/8116j9b8vjacrjabr15zpx1097hd4yal-sbcl-smart-buffer-0.0.1-1.09b9a9a
/gnu/store/8jz41f61a67hp3pq1f2kcdhp4s09i4d6-sbcl-ieee-floats-20170924-1.566b51a
/gnu/store/8rqj3fz2nzj25adasdpyqh2svzlxd56a-sbcl-unix-opts-0.1.7
/gnu/store/91kg140kv14hcikmn06r6fs1w90p4l8y-sbcl-named-readtables-0.9-1.4dfb89f
/gnu/store/930c7kmpln4fvb5nfqcskvm0w11cm14w-sbcl-cl-xmlspam-0.0.0-1.ea06abc
/gnu/store/938i4y2br40r3a3ai0klph830a64ak4x-next-1.3.1-lib
/gnu/store/970l1zmfq5c74aknh2bfx34f5fk8m9qp-sbcl-anaphora-0.9.6
/gnu/store/9snfj42ij8kvyhjyh664l09lxdk3yq4f-sbcl-babel-0.5.0
/gnu/store/9yqjy6r4fchryjiy68znj6a9vls5abl4-sbcl-iolib+sockets-0.8.3
/gnu/store/a40sw3w1gmhrzd5qvww62w36rjapczyw-sbcl-iolib+streams-0.8.3
/gnu/store/b12vxsq2ykaij80wp0saqlvdsk5p21bf-sbcl-trivia.balland2006-0.0.0-1.902e0c6
/gnu/store/bjj60qdahw21fdaq8b9bhvj2zi6xih93-sbcl-trivia-0.0.0-1.902e0c6
/gnu/store/bjn6nfhp3j0vjiibs6q0lfxlm8nyd9r6-sbcl-trivia.trivial-0.0.0-1.902e0c6
/gnu/store/bzvg9brphn5dn9q9kw6mirk4bb54wrd2-next-1.3.1
/gnu/store/cjzysi5lh5pkfjwdl6kbxd1iklsv5j4b-sbcl-cl+ssl-0.0.0-1.b81c113
/gnu/store/cxavfbxim8g08438kvqdagck1gr61c9r-sbcl-cl-ppcre-unicode-2.0.11
/gnu/store/dlngg1zc6cn1s0sl65a0yvk9pv6kq35p-sbcl-bordeaux-threads-0.8.6-1.5dce49f
/gnu/store/dnjyxpjf0ypd0w7sj9g83zk7yh2ba1x6-sbcl-cl-css-0.1-1.8fe654c
/gnu/store/dxbgbh9yfnf56mycvqnizwdly719lany-sbcl-iolib.base-0.8.3
/gnu/store/fz65d87rx6k1qh17cdbki7sk0jmqpxvx-sbcl-trivia.level1-0.0.0-1.902e0c6
/gnu/store/g44dw0w7873rjw0xa63yl5g8cbb0drwq-sbcl-quri-0.1.0-1.76b7510
/gnu/store/g6jv6jhq0fsga1wa17psx0xzr5yhji3z-sbcl-flexi-streams-1.0.16
/gnu/store/gaz7r55v39ngm00s0x641lb3y8i4ni4g-sbcl-iolib-0.8.3
/gnu/store/gsb7wswwk6qj8f17wq4mj47r3r9hq7dr-sbcl-puri-20180228
/gnu/store/gv9mgqvvaj3xrybnaqnz92k4pwxd5swn-sbcl-cl-cookie-0.9.10-1.cea55ae
/gnu/store/h2j8gy8d2yrmrklavrik1y864zg7qwqd-sbcl-1.5.6
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28
/gnu/store/i3nbdq5mzf9w75pcwj4vmmk9ig477y57-sbcl-cxml+test-0.0.0-1.00b22bf
/gnu/store/iqrad7j28rs3ybjbxqiy9ng8n7568cn3-sbcl-dexador-0.9.10-1.a2714d1
/gnu/store/jvvyq487ainy1y2y1slf3lnizjhwb9s4-sbcl-cl-utilities-0.0.0-1.dce2d2f
/gnu/store/kaxbi425n6389mk6wdvylf1npfgayv1h-sbcl-ironclad-0.46
/gnu/store/kg104rhqzvmnpqxk91ayvwxzsrncr0f4-sqlite-3.26.0
/gnu/store/kivhppqdnc05j5h55843m9iq4m5sfqlw-sbcl-cffi-0.19.0
/gnu/store/lgd8f6lg8cc0wq1p88ypq7cxl68yz84a-sbcl-alexandria-0.0.0-1.926a066
/gnu/store/lp0zkbgzj8w9ciyzad3xz37746qr323c-sbcl-introspect-environment-0.1-1.fff42f8
/gnu/store/lx1bmdxxkq6ydhmbmfb14aw2gwfss3b2-sbcl-cl-reexport-0.1-1.312f366
/gnu/store/m1w2cxc4h7f36pgdwpmaqp8iynbngvzs-sbcl-iolib+multiplex-0.8.3
/gnu/store/m9ngbfyi8wrwiwyr7dkfnhn5fyfalq3r-openssl-1.0.2p
/gnu/store/maw58fp2xfhgijjl7ddb53hia3gldi65-sbcl-cl-markup-0.1-1.e0eb7de
/gnu/store/mkxpi2n0xhy3hdjbj89f5zpndyyl87fb-sbcl-type-i-0.1-1.dea233f
/gnu/store/n5yy521im972r4cky8ipgpgg3l5l4xcx-sbcl-iolib.asdf-0.8.3
/gnu/store/n7wcp2npwf4b5x4cywwi48mah1g35jwz-libfixposix-0.4.3
/gnu/store/nq4lsyipmfb0q7g26ra45rwwqrh3x8zw-zlib-1.2.11
/gnu/store/pn86r0syfrwaq2ny7czr8cjdids9y8dx-sbcl-cl-fad-0.7.5
/gnu/store/q19l04vd2za80mk1845pz7r8cz29qk43-bash-minimal-4.4.23
/gnu/store/qc0wlrr3q5z3sg42447njb7kd41xhfnl-sbcl-idna-0.2.2
/gnu/store/qfy9imvn3x3ycnmzl3kpj69550rm0qj7-sbcl-cl-str-0.8-1.3d5ec86
/gnu/store/r2qwa2qd5hrq6kk9fscljlsnn747mnrd-sbcl-iolib.conf-0.8.3
/gnu/store/rbz9hscv4wg71b4ymgzycicnqsi6fm4n-sbcl-trivia.level2-0.0.0-1.902e0c6
/gnu/store/rdw3fax1cdnr8q35s88773z2sx2dbkg1-sbcl-fast-http-0.2.0-1.f9e7597
/gnu/store/rnl0z3qkwx303lpz6dy68r78ls63kmww-sbcl-iolib+syscalls-0.8.3
/gnu/store/rw2y9fy9mmgcd7gil3vyn0k0h5lqs5xp-sbcl-fast-io-1.0.0-1.dc3a71d
/gnu/store/s9f4nxk7s3djgnxymg7vqy16b7vslbw9-sbcl-mk-string-metrics-0.1.2
/gnu/store/vcrnpjv9qzswvjlgazvw98vm8v909xav-sbcl-split-sequence-1.4.1
/gnu/store/vmkwl8546jm943qda0r4pvnl64qldwyi-sbcl-cl-sqlite-0.2-1.c738e66
/gnu/store/vn6lfph8x0i7cm2spyi205rp3sybprfq-sbcl-lparallel-2.8.4
/gnu/store/w3gmvxvr37kmdlyvcdxrvzhh424w0qw4-sbcl-usocket-0.7.1-1.86e7efb
/gnu/store/wn3crbgl4idv9lb1wvzz2x0v2mjv4h4l-sbcl-cl-unicode-base-0.1.5-1.9fcd06f
/gnu/store/wq56c5pinx0ccaavr5ralglam30sfrcs-sbcl-nibbles-0.14
/gnu/store/wsyla0p9cpg05ar3rpby7x3pwfd4zlw3-sbcl-chipz-0.8-1.75dfbc6
/gnu/store/wzqv2sq4lm6bwkk7i6ls85czxprmjvv0-sbcl-chunga-1.1.7
/gnu/store/xn1p8pw6snzn2pnbij66sdsa1h07hr5h-sbcl-lisp-namespace-0.1-1.28107ca
/gnu/store/xqalg7y5yh83846c2kwip6kfgwz6wxdi-cl-dbus-20190408-1.24b452d
/gnu/store/xv72rxxmi5b5pp8kphnbmg0zv3whl8y9-sbcl-trivial-clipboard-0.0.0.0-2.5af3415
/gnu/store/yr0pv0sq4fr50iavynnv6pwwxxa6dg0q-sbcl-proc-parse-0.0.0-1.ac36368
/gnu/store/yyryqxy2sf9kwbxq5kfcfh3wm0vg0wpa-sbcl-cl-ppcre-2.0.11
/gnu/store/z94yn3iarygf8a5nh6wjfwdr5a15fx7j-sbcl-cxml+klacks-0.0.0-1.00b22bf
/gnu/store/zaj2wi6rs214rah4c8an62hn95lfx4mp-sbcl-cxml+xml-0.0.0-1.00b22bf
/gnu/store/zraa220x950amls86xflrk5jpzn5s938-sbcl-xsubseq-0.0.1-1.5ce430b
--8<---------------cut here---------------end--------------->8---

After your commit:

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix gc --references /gnu/store/sypf1iq80b2i192jp0mxm43bk6dj6fcc-next-1.3.1
/gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib
/gnu/store/a0rffysgyr66q5fjgm1iibjm4ma1jbk3-next-1.3.1-lib
/gnu/store/h2j8gy8d2yrmrklavrik1y864zg7qwqd-sbcl-1.5.6
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28
/gnu/store/nq4lsyipmfb0q7g26ra45rwwqrh3x8zw-zlib-1.2.11
/gnu/store/sypf1iq80b2i192jp0mxm43bk6dj6fcc-next-1.3.1
--8<---------------cut here---------------end--------------->8---

This means that “guix gc” will collect more than it should, leaving
users with a broken Next browser.

-- 
Ricardo

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]

Oh, my bad, I did the test with "guix size" instead of "guix gc
--references".
I'm actually confused why guix size report dependencies that are not in
"guix gc --references"?

I'll revert the commit just now.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Oh, my bad, I did the test with "guix size" instead of "guix gc
> --references".
> I'm actually confused why guix size report dependencies that are not in
> "guix gc --references"?

“guix size” lists dependencies recursively, like “guix gc -R”, whereas
“guix gc --references” lists only direct dependencies.

Ludo’.

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

Hmmm, OK, but why would indirect dependencies be garbage-collected?

Also in the case of Next there is something fishy: direct dependencies
include none of the Common Lisp libraries, and none of them _depends
indirectly_ on the those libraries, so how come they show up in the
listing of indirect dependencies?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Also in the case of Next there is something fishy: direct dependencies
> include none of the Common Lisp libraries, and none of them _depends
> indirectly_ on the those libraries, so how come they show up in the
> listing of indirect dependencies?

Hmm, reading myself again I realize this was poorly phrased.  Allow me
to explain that again.  In

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix gc --references /gnu/store/sypf1iq80b2i192jp0mxm43bk6dj6fcc-next-1.3.1
/gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib
/gnu/store/a0rffysgyr66q5fjgm1iibjm4ma1jbk3-next-1.3.1-lib
/gnu/store/h2j8gy8d2yrmrklavrik1y864zg7qwqd-sbcl-1.5.6
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28
/gnu/store/nq4lsyipmfb0q7g26ra45rwwqrh3x8zw-zlib-1.2.11
/gnu/store/sypf1iq80b2i192jp0mxm43bk6dj6fcc-next-1.3.1
--8<---------------cut here---------------end--------------->8---

we see that next-1.3.1-lib is a dependency, which holds references to
all the other libraries.

So my question is, why would `guix gc` collect sqlite, libfixposix,
etc. if they are indirect dependencies?

In other words?  Wasn't I right to check for dependencies with `guix
size` (or `guix gc -R`) instead of `guix gc --references`?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Hmm, reading myself again I realize this was poorly phrased.  Allow me
> to explain that again.  In
>
> ./pre-inst-env guix gc --references /gnu/store/sypf1iq80b2i192jp0mxm43bk6dj6fcc-next-1.3.1
> /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib
> /gnu/store/a0rffysgyr66q5fjgm1iibjm4ma1jbk3-next-1.3.1-lib
> /gnu/store/h2j8gy8d2yrmrklavrik1y864zg7qwqd-sbcl-1.5.6
> /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28
> /gnu/store/nq4lsyipmfb0q7g26ra45rwwqrh3x8zw-zlib-1.2.11
> /gnu/store/sypf1iq80b2i192jp0mxm43bk6dj6fcc-next-1.3.1
>
> we see that next-1.3.1-lib is a dependency, which holds references to
> all the other libraries.
>
> So my question is, why would `guix gc` collect sqlite, libfixposix,
> etc. if they are indirect dependencies?
>
> In other words?  Wasn't I right to check for dependencies with `guix
> size` (or `guix gc -R`) instead of `guix gc --references`?

‘guix size’ and ‘guix gc -R’ show you the whole closure of the store
item, so you might not realize that some of the things that ought to be
direct dependencies are now in fact indirect dependencies.

If sqlite ought to be a direct dependency and is now, in fact, an
indirect dependency, things won’t break right away: sqlite won’t be
deleted as long as next is live.

But you’ll already run into problems: grafting will yield a broken next,
as in <https://issues.guix.gnu.org/issue/33848>.

Furthermore, sqlite might eventually vanish entirely from the closure of
next, as a consequence of changes in a dependency, and at that point
running the GC may remove sqlite and thus break next.  ‘guix pack’ would
also produce an incomplete pack.

To draw a parallel, it’s as if SBCL’s GC were unable to see some of the
pointers to the heap, unpredictably: your code might keep running for a
while, but sooner or later, it’ll break badly.  :-)

HTH,
Ludo’.

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

And... I have so many more questions!

> ‘guix size’ and ‘guix gc -R’ show you the whole closure of the store
> item, so you might not realize that some of the things that ought to be
> direct dependencies are now in fact indirect dependencies.
>
> If sqlite ought to be a direct dependency and is now, in fact, an
> indirect dependency, things won’t break right away: sqlite won’t be
> deleted as long as next is live.
>
> But you’ll already run into problems: grafting will yield a broken next,
> as in <https://issues.guix.gnu.org/issue/33848>.

I think the aforementioned issue is different: it's about store paths
that are written within Common Lisp code, which only happens here for
the next-gtk-webkit executable.  This is not related to SQLite or
others, which are visible to the reference scanner.

I don't understand how grafting could cause a problem: next-1.3.1-lib
would still be present, right?

> Furthermore, sqlite might eventually vanish entirely from the closure of
> next, as a consequence of changes in a dependency, and at that point
> running the GC may remove sqlite and thus break next.  ‘guix pack’ would
> also produce an incomplete pack.

If sqlite vanishes from the closure, it means the Next does not need
it.  Why would it be a problem then?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> And... I have so many more questions!
>
>> ‘guix size’ and ‘guix gc -R’ show you the whole closure of the store
>> item, so you might not realize that some of the things that ought to be
>> direct dependencies are now in fact indirect dependencies.
>>
>> If sqlite ought to be a direct dependency and is now, in fact, an
>> indirect dependency, things won’t break right away: sqlite won’t be
>> deleted as long as next is live.
>>
>> But you’ll already run into problems: grafting will yield a broken next,
>> as in <https://issues.guix.gnu.org/issue/33848>.
>
> I think the aforementioned issue is different: it's about store paths
> that are written within Common Lisp code, which only happens here for
> the next-gtk-webkit executable.  This is not related to SQLite or
> others, which are visible to the reference scanner.
>
> I don't understand how grafting could cause a problem: next-1.3.1-lib
> would still be present, right?

Same story: if the reference to ‘next-1.3.1-lib’ is invisible to the
scanner or grafting code (due to compression, or due use of a non-ASCII
compatible string encoding), the ‘next-1.3.1-lib’ might vanish.

Ludo’.

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Same story: if the reference to ‘next-1.3.1-lib’ is invisible to the
> scanner or grafting code (due to compression, or due use of a non-ASCII
> compatible string encoding), the ‘next-1.3.1-lib’ might vanish.

But it's not, as shown in the first post by Ricardo.
next-1.3.1-lib is directly referenced by the asdf build system, there is
no risk for it to disappear.

So I'm still wondering how "guix size" could show dependencies that
could potentially be garbage-collected.  Sorry if I'm missing the
obvious :p

Can we re-enable core compression for Next then?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

Duh, got it: if next-1.3.1-lib is grafted, then the path to SQLite,
libfixposix, etc. _inside the compress binary_ will not be updated.
So if we garbage-collect after that, the binary will try to FFI-load
non-existing libraries.

Note that this is only a problem because Next depends on FFI libraries.
Compressing pure-SBCL binaries should not be a problem.

Thanks for your help!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Ludovic Courtès]

Howdy!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Duh, got it: if next-1.3.1-lib is grafted, then the path to SQLite,
> libfixposix, etc. _inside the compress binary_ will not be updated.
> So if we garbage-collect after that, the binary will try to FFI-load
> non-existing libraries.

Exactly!

> Note that this is only a problem because Next depends on FFI libraries.
> Compressing pure-SBCL binaries should not be a problem.

It’s a problem in general: any store reference in a compressed file is
invisible to the GC and to the grafting code.

Ludo’.

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

Howdy!

> It’s a problem in general: any store reference in a compressed file is
> invisible to the GC and to the grafting code.

I meant that pure Common Lisp binaries are self-contained, so they can be
compressed because they don't have any useful reference.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hi Pierre,

Ludovic Courtès <ludo@gnu.org> writes:

> Howdy!
>
> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> Duh, got it: if next-1.3.1-lib is grafted, then the path to SQLite,
>> libfixposix, etc. _inside the compress binary_ will not be updated.
>> So if we garbage-collect after that, the binary will try to FFI-load
>> non-existing libraries.
>
> Exactly!
>
>> Note that this is only a problem because Next depends on FFI libraries.
>> Compressing pure-SBCL binaries should not be a problem.
>
> It’s a problem in general: any store reference in a compressed file is
> invisible to the GC and to the grafting code.
>
> Ludo’.

In case you were not thinking about it, Btrfs compression gives you the
best of both worlds -- Guix operates on seemingly uncompressed data,
while in reality everything (not just SBCL binaries) is compressed to
reduce disk space usage and speed up read/writes.

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

True.

I've been using Btrfs for my data for a little while and I'm very happy
with it.

I wonder how Btrfs fares for a Guix system.  In many ways, Guix
supersedes many of the features of Btrfs (snapshots and deduplication in
particular).  So I wonder if it's not redundant and possibly incurs a
waste of energy.

What's your experience, Maxim?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

On Sun, Sep 29, 2019 at 03:43:45PM +0200, Pierre Neidhardt wrote:
> True.
> 
> I've been using Btrfs for my data for a little while and I'm very happy
> with it.
> 
> I wonder how Btrfs fares for a Guix system.  In many ways, Guix
> supersedes many of the features of Btrfs (snapshots and deduplication in
> particular).  So I wonder if it's not redundant and possibly incurs a
> waste of energy.
> 
> What's your experience, Maxim?
> 

Not Maxim, but I use btrfs also on my Guix Systems. I have the
compression set to lz4, which doesn't compress a lot but does
compress/decompress with minimal CPU overhead so I feel like I'm getting
some benefit from it. In general compression can "speed up" disk
operations since you can write easily compressible data faster than
regular data.

Other than that after boot-up I run 'sudo chattr +C /tmp' to make my
/tmp dir not copy-on-write, which helps with building packages. I also
have a daily cron job to balance my disk and to scrub it. Overall I
don't really use very many features that aren't available on ext4.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]

Thanks for your feedback, Efraim.

I'd be interested in knowing the difference between a compressed
/gnu/store and an uncompressed one.

I believe the `compsize` utility can give us an idea:

  sudo compsize /gnu/store

(It's a Guix package.)
Would you like to try it, Efraim?  Or Maxim?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hello Pierre!

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> True.
>
> I've been using Btrfs for my data for a little while and I'm very happy
> with it.
>
> I wonder how Btrfs fares for a Guix system.  In many ways, Guix
> supersedes many of the features of Btrfs (snapshots and deduplication in
> particular).  So I wonder if it's not redundant and possibly incurs a
> waste of energy.
>
> What's your experience, Maxim?

I like that Btrfs allows to set different namespaces (think of LVM
logical volumes) on the fly as subvolumes.  I use snapshots as a mean of
backups, (using the btrfs send/receive mechanism to backup the snapshots
(differentially) to external storage).

I haven't noticed much slow down (if at all?), and the 'lzo' compression
keeps the /gnu/store size 30% smaller or so.

That sums it for now, I think.

HTH!

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> I like that Btrfs allows to set different namespaces (think of LVM
> logical volumes) on the fly as subvolumes.  I use snapshots as a mean of
> backups, (using the btrfs send/receive mechanism to backup the snapshots
> (differentially) to external storage).

Aren't btrfs snapshots overlapping with Guix generations?

>
> I haven't noticed much slow down (if at all?), and the 'lzo' compression
> keeps the /gnu/store size 30% smaller or so.

This is pretty good!  Alright, you sold it to me! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

Pierre,

Pierre Neidhardt 写道：
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> I like that Btrfs allows to set different namespaces (think of 
>> LVM
>> logical volumes) on the fly as subvolumes.  I use snapshots as 
>> a mean of
>> backups, (using the btrfs send/receive mechanism to backup the 
>> snapshots
>> (differentially) to external storage).
>
> Aren't btrfs snapshots overlapping with Guix generations?

A few ‘traditional’ distributions use btrfs snapshots to do what 
Guix does natively and much better: system roll-backs.  Reversing 
that to call btrfs's features ‘overlapping’ with Guix seems very 
forced to me :-)

Guix generations don't even protect the one valuable part: the 
human-written configuration that created it.  You need to do that 
yourself (probably with git).  Everything else is just insanely 
convenient caching.

Generations also don't allow you to btrfs send/receive, which I 
think was Maxim's main point.  If there's another KISSy way to 
back up whole Guix Systems over the Internet, I don't know of it 
(rsync can't, nor can borg or restic or…, and everything else is 
too much work :o).

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Generations also don't allow you to btrfs send/receive, which I 
> think was Maxim's main point.  If there's another KISSy way to 
> back up whole Guix Systems over the Internet, I don't know of it 
> (rsync can't, nor can borg or restic or…, and everything else is 
> too much work :o).

Question: What is the point of backing up a Guix system if the
config.scm suffices?  Is there something that is not contained in config.scm?

Beside Internet connection cost and time, of course! ;)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

Pierre,

Pierre Neidhardt 写道：
> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>
>> Generations also don't allow you to btrfs send/receive, which I 
>> think was Maxim's main point.  If there's another KISSy way to 
>> back up whole Guix Systems over the Internet, I don't know of 
>> it 
>> (rsync can't, nor can borg or restic or…, and everything else 
>> is 
>> too much work :o).
>
> Question: What is the point of backing up a Guix system if the
> config.scm suffices?

I did blatantly invite that question, didn't I :-)

> Is there something that is not contained in config.scm?

So. much. state.

‘Oh, right, /etc/networkmanager's gone now.  Let me get my 
notebook with passwords.  Oh, that reminds me: all user passwords 
are empty, have to change that ASAP.  Phew.  Oh, I have to 
reinstall my printer, good thing I have time for that while people 
are angrily calling me about deadlines.  What's this about changed 
SSH host ke—’

That's trivial stuff I thought of while writing, and it already 
adds up to minutes of human work (=downtime).  All at the most 
inconvenient and stressful time.  Now multiply that by the number 
of things I forgot.

I don't consider a back-up that can't be restored while drunk a 
back-up.

> […] time, of course! ;)

I've learnt the hard way over the years that, for me, nothing else 
matters.

♪,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 65 bytes --]

Fair enough! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Maxim Cournoyer]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>
>> Generations also don't allow you to btrfs send/receive, which I 
>> think was Maxim's main point.  If there's another KISSy way to 
>> back up whole Guix Systems over the Internet, I don't know of it 
>> (rsync can't, nor can borg or restic or…, and everything else is 
>> too much work :o).
>
> Question: What is the point of backing up a Guix system if the
> config.scm suffices?  Is there something that is not contained in config.scm?
>
> Beside Internet connection cost and time, of course! ;)

Hi Pierre!

I've used a full backup to migrate from hard drives in the past; it's
actually faster to sync drives than fetch packages off the internet, and
my system is exactly in the same *state* as it was left (including those
bookmarks in Icecat, those cached emails, etc.).

Keep in mind that multiple subvolumes can be used to hold your 'home' or
other data partitons, making these individually easy to snapshot and
backup.

Maxim

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

Hi Maxim!

> I've used a full backup to migrate from hard drives in the past; it's
> actually faster to sync drives than fetch packages off the internet, and
> my system is exactly in the same *state* as it was left (including those
> bookmarks in Icecat, those cached emails, etc.).
>
> Keep in mind that multiple subvolumes can be used to hold your 'home' or
> other data partitons, making these individually easy to snapshot and
> backup.

I personally prefer to restore the home data (bookmarks, emails...) in a
functional, declarative manner (for now via scripts, later maybe via
Shepherd user services if I find time / money to work on this).  It's
more work, but if we get user services we could factor this work so that
everyone benefits from it.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: btrfs and Guix features [was: gnu: next: Compress the executable.]

[Maxim Cournoyer]

Hello again,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Hi Maxim!
>
>> I've used a full backup to migrate from hard drives in the past; it's
>> actually faster to sync drives than fetch packages off the internet, and
>> my system is exactly in the same *state* as it was left (including those
>> bookmarks in Icecat, those cached emails, etc.).
>>
>> Keep in mind that multiple subvolumes can be used to hold your 'home' or
>> other data partitons, making these individually easy to snapshot and
>> backup.
>
> I personally prefer to restore the home data (bookmarks, emails...) in a
> functional, declarative manner (for now via scripts, later maybe via
> Shepherd user services if I find time / money to work on this).  It's
> more work, but if we get user services we could factor this work so that
> everyone benefits from it.

I agree with you, that declarative is nice; but it's still no substitute
for a backup IMO.  I'm already using a private git + GNU Stow to deploy
my dot files, and have been meaning to try the Guix Home Manager [0]
Guix extension (via a channel) by Julien Lepiller.

[0]  https://framagit.org/tyreunom/guix-home-manager

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

On Thu, Oct 03, 2019 at 12:01:43AM +0900, Maxim Cournoyer wrote:
> Hello Pierre!
> 
> Pierre Neidhardt <mail@ambrevar.xyz> writes:
> 
> > True.
> >
> > I've been using Btrfs for my data for a little while and I'm very happy
> > with it.
> >
> > I wonder how Btrfs fares for a Guix system.  In many ways, Guix
> > supersedes many of the features of Btrfs (snapshots and deduplication in
> > particular).  So I wonder if it's not redundant and possibly incurs a
> > waste of energy.
> >
> > What's your experience, Maxim?
<snip>
> 
> I haven't noticed much slow down (if at all?), and the 'lzo' compression
> keeps the /gnu/store size 30% smaller or so.
> 
> That sums it for now, I think.
> 

What mount options do you have? I realized i have compression enabled
but 'sudo compsize /gnu/store' doesn't show any compression happening.

(file-system
 (device (file-system-label "root"))
 (mount-point "/")
 (type "btrfs")
 (options "autodefrag,compress=lzo,discard,ssd_spread"))

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Bengt Richter]

On +2019-10-03 10:09:30 +0300, Efraim Flashner wrote:
> On Thu, Oct 03, 2019 at 12:01:43AM +0900, Maxim Cournoyer wrote:
> > Hello Pierre!
> > 
> > Pierre Neidhardt <mail@ambrevar.xyz> writes:
> > 
> > > True.
> > >
> > > I've been using Btrfs for my data for a little while and I'm very happy
> > > with it.
> > >
> > > I wonder how Btrfs fares for a Guix system.  In many ways, Guix
> > > supersedes many of the features of Btrfs (snapshots and deduplication in
> > > particular).  So I wonder if it's not redundant and possibly incurs a
> > > waste of energy.
> > >
> > > What's your experience, Maxim?
> <snip>
> > 
> > I haven't noticed much slow down (if at all?), and the 'lzo' compression
> > keeps the /gnu/store size 30% smaller or so.
> > 
> > That sums it for now, I think.
> > 
> 
> What mount options do you have? I realized i have compression enabled
> but 'sudo compsize /gnu/store' doesn't show any compression happening.
> 
> (file-system
>  (device (file-system-label "root"))
>  (mount-point "/")
>  (type "btrfs")
>  (options "autodefrag,compress=lzo,discard,ssd_spread"))
>

I am wondering if there is some low level losetup size ambivalence involved.

I.e., isn't e.g. ext4 smart enough to defer actual allocation of physical page blocks
when blocks are pure zero, and just store the fact of their existence in inode
metadata?

If that's  right, then it would seem that it would be possible to access both
the full physical size and alternatively just the sum on non-zero blocks' sizes.

One would think that someone has taken advantage of this to let losetup
over-book physical backing blocks in a mounted loopback device.

Is there a version of df that has an option to output both sizes of
a sparsely non-zero file?

HTH trigger some useful thought sussing out that 30% :-)
--
Regards,
Bengt Richter

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

Bengt Richter <bokr@bokr.com> writes:

> I.e., isn't e.g. ext4 smart enough to defer actual allocation of physical page blocks
> when blocks are pure zero, and just store the fact of their existence in inode
> metadata?

Yes, ext4 supports that.

> Is there a version of df that has an option to output both sizes of
> a sparsely non-zero file?

I don't think df can do that, bu GNU `du' can, see the --apparent-size /
--bytes options.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hello again,

Efraim Flashner <efraim@flashner.co.il> writes:

> On Thu, Oct 03, 2019 at 12:01:43AM +0900, Maxim Cournoyer wrote:
>> Hello Pierre!
>> 
>> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>> 
>> > True.
>> >
>> > I've been using Btrfs for my data for a little while and I'm very happy
>> > with it.
>> >
>> > I wonder how Btrfs fares for a Guix system.  In many ways, Guix
>> > supersedes many of the features of Btrfs (snapshots and deduplication in
>> > particular).  So I wonder if it's not redundant and possibly incurs a
>> > waste of energy.
>> >
>> > What's your experience, Maxim?
> <snip>
>> 
>> I haven't noticed much slow down (if at all?), and the 'lzo' compression
>> keeps the /gnu/store size 30% smaller or so.
>> 
>> That sums it for now, I think.
>> 
>
> What mount options do you have? I realized i have compression enabled
> but 'sudo compsize /gnu/store' doesn't show any compression happening.
>
> (file-system
>  (device (file-system-label "root"))
>  (mount-point "/")
>  (type "btrfs")
>  (options "autodefrag,compress=lzo,discard,ssd_spread"))

I'm just using the 'compress=lzo' option.

Here's the output of 'compsize' on my system:

--8<---------------cut here---------------start------------->8---
time sudo compsize /gnu/store
Processed 2039520 files, 446555 regular extents (1329746 refs), 925862 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       90%       49G          54G         128G       
none       100%       44G          44G          96G       
lzo         49%      5.2G          10G          31G       

real    1m18.454s
user    0m6.375s
sys     0m52.852sn
--8<---------------cut here---------------end--------------->8---

So my total compression rate would be 10%, which is not as good as the
30% I had on mind.  I think that 30% I had on mind was based on sending
an uncompressed root file system to a compressed backup drive (using
Btrfs' lzo compression).

But it appears most of my /gnu/store content is not using compression at
all, perhaps because Btrfs will not compress files it knows wouldn't
compress well (e.g. already compressed archives).

HTH,

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> But it appears most of my /gnu/store content is not using compression at
> all, perhaps because Btrfs will not compress files it knows wouldn't
> compress well (e.g. already compressed archives).

Hmmm... If you've got a lot of source archives in your store, this is
would be true, but libraries and binaries compress very well in general,
so I would also expect more than 10%.  Have you tried cleaning up some
of the compressed archives?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> But it appears most of my /gnu/store content is not using compression at
>> all, perhaps because Btrfs will not compress files it knows wouldn't
>> compress well (e.g. already compressed archives).
>
> Hmmm... If you've got a lot of source archives in your store, this is
> would be true, but libraries and binaries compress very well in general,
> so I would also expect more than 10%.  Have you tried cleaning up some
> of the compressed archives?

I haven't, but running compsize on my /home gives a similar
result... Weird. I don't keep much already compressed (videos, photos,
compressed archives, etc.) files in my home, AFAICT.

It'd be useful if 'compsize' could tell us the nature of the files that
we not compressed.

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

Maybe you can chain a shell `find' + compsize on specific folders / file types.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hi Pierre,

The problem where using Btrfs compression wasn't effective for
/gnu/store (or anything else) was caused by the init RAM disk not
honoring options on the root file system.

I could notice about this when running just 'mount', and the options I
gave in my config.scm were missing.

Since this was fixed with commit
900ef20b1da66ad71145082c883dc12f31fafa54, retesting compsize gives
better results :-).

Here are the results when using the zstd compression (default level, 3)
on the /gnu/store of this relatively new machine:

--8<---------------cut here---------------start------------->8---
sudo compsize /gnu/store
Password: 
Processed 1166359 files, 379224 regular extents (1025522 refs), 543008 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       65%       30G          46G         107G       
none       100%       23G          23G          52G       
zstd        29%      6.8G          22G          55G
--8<---------------cut here---------------end--------------->8---

We see that there's 23 GiB that aren't compressed.  I attribute those to
already compressed files in the store, that Btrfs doesn't consider worth
compressing.

For the compressible files, Zstd achieves an impressive 29% compression
ratio, while overall the compression ration is 65%.

The last column shows how much space a traditional file system would
need to store the content referenced.

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 206 bytes --]

This is fantastic, thanks a lot for the feedback!

Speaking of which, should we update the manual and/or templates to
encourage users to switch to Btrfs?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> This is fantastic, thanks a lot for the feedback!
>
> Speaking of which, should we update the manual and/or templates to
> encourage users to switch to Btrfs?

Perhaps!  I don't really have an opinion on this :-)

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 98 bytes --]

Can you share your operating system declaration?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: 02/02: gnu: next: Compress the executable.

[Maxim Cournoyer]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Can you share your operating system declaration?

Sorry for the delay; here it is.  I've anonymized some of the
information such as SSH public keys and usernames.

--8<---------------cut here---------------start------------->8---
;; This is an operating system configuration template
;; for a "desktop" setup with GNOME and Xfce where the
;; root partition is encrypted with LUKS.

(use-modules (guix store)
             (gnu)
             (gnu packages bash)
             (gnu packages version-control)
             (gnu system nss)
             (srfi srfi-1))

(use-service-modules admin desktop docker linux ssh xorg)
(use-package-modules android certs docker java linux nfs ratpoison)

(define %my-desktop-services
  (remove (lambda (service)
            (eq? (service-kind service) gdm-service-type))
          %desktop-services))

(operating-system
  (host-name "myhost")
  (timezone "America/Montreal")
  (locale "en_US.utf8")

  ;; Choose US English keyboard layout.  The "altgr-intl"
  ;; variant provides dead keys for accented characters.
  (keyboard-layout (keyboard-layout "dvorak"))

  ;; Use the UEFI variant of GRUB with the EFI System
  ;; Partition mounted on /boot/efi.
  (bootloader (bootloader-configuration
               (bootloader grub-efi-bootloader)
               (target "/boot/efi")
               (keyboard-layout keyboard-layout)))

  ;; Specify a mapped device for the encrypted root partition.
  ;; The UUID is that returned by 'cryptsetup luksUUID'.
  (mapped-devices
   (list (mapped-device
          (source (uuid "f85c0627-1f6f-48b9-a2c2-6c12594a7bd1"))
          (target "btrfs-pool-1")
          (type luks-device-mapping))
         (mapped-device
          (source (uuid "73b08e1a-ca2f-4d46-845a-44443fe14cd7"))
          (target "btrfs-pool-4")
          (type luks-device-mapping))))

  (file-systems (cons*
                 ;; For EFI firmware.
                 (file-system
                   (device (uuid "209E-67AD" 'fat))
                   (mount-point "/boot/efi")
                   (type "vfat"))

                 ;; Main system, on a 500 GB SSD (dev/sda).
                 (file-system
                   (device (file-system-label "btrfs-pool-1"))
                   (mount-point "/")
                   (type "btrfs")
                   (options "subvol=rootfs,compress=zstd")
                   (dependencies mapped-devices))
                 (file-system
                   (device (file-system-label "btrfs-pool-1"))
                   (mount-point "/home")
                   (type "btrfs")
                   (options "subvol=homefs,compress=zstd")
                   (dependencies mapped-devices))

                 ;; 1000 GB drive for builds (/dev/nvme0n1).  Shared
                 ;; between jenkins-home, jenkins-build and
                 ;; docker-cache subvolumes.
                 (file-system
                   (device (file-system-label "btrfs-pool-4"))
                   (mount-point "/home/jenkins-user")
                   (create-mount-point? #t)
                   (type "btrfs")
                   (options "subvol=jenkins-home,compress=zstd")
                   (dependencies mapped-devices))
                 (file-system
                   (device (file-system-label "btrfs-pool-4"))
                   (mount-point "/home/jenkins-user/workspace")
                   (create-mount-point? #t)
                   (type "btrfs")
                   (options "subvol=jenkins-build,compress=zstd")
                   (dependencies mapped-devices))
                 (file-system
                   (device (file-system-label "btrfs-pool-4"))
                   (mount-point "/var/lib/docker")
                   (create-mount-point? #t)
                   (type "btrfs")
                   (options "subvol=docker-cache,compress=zstd")
                   (dependencies mapped-devices))

                 ;; NFS mounts for caching the state and downloads of
                 ;; Yocto.
                 ;; FIXME: Must be manually mounted.
                 (file-system
                   (device "server:/mnt/scratch/yocto-sstate")
                   (mount-point "/mnt/scratch/yocto-sstate")
                   (create-mount-point? #t)
                   (type "nfs")
                   (mount? #f)
                   (options "soft")
                   (flags '(no-exec)))
                 (file-system
                   (device "server:/mnt/scratch/yocto-dldir")
                   (mount-point "/mnt/scratch/yocto-dldir")
                   (create-mount-point? #t)
                   (type "nfs")
                   (mount? #f)
                   (options "soft")
                   (flags '(no-exec)))

                 %base-file-systems))

  (swap-devices '("/swap/swapfile"))

  (users (cons* (user-account
                 (name "myuser")
                 (group "users")
                 (supplementary-groups '("dialout" "wheel" "netdev"
                                         "audio" "video"
                                         "kvm" "docker"
                                         "adbusers")))
                (user-account
                 (name "jenkins-user")
                 (comment "User for a Jenkins build slave")
                 (home-directory "/home/jenkins-user")
                 (group "users")
                 (supplementary-groups '("netdev" "kvm" "docker")))
                %base-user-accounts))

  (groups (cons* (user-group (system? #t)
                             (name "adbusers"))
                 %base-groups))

  ;; This is where we specify system-wide packages.
  (packages (cons* ratpoison
                   nss-certs            ;for HTTPS access
                   btrfs-progs
                   nfs-utils
                   cqfd
                   docker-cli
                   git
                   git-repo
                   openjdk12
                   %base-packages))

  ;; SSH, Docker
  (services
   (cons*
    (extra-special-file "/bin/bash"
                        (file-append bash "/bin/bash"))
    (service rottlog-service-type)
    (service earlyoom-service-type)
    (service openssh-service-type
             (openssh-configuration
              (port-number 22)
              (permit-root-login #t)
              (authorized-keys
               `(("myuser" ,(local-file "some-key.pub"))
                 ;; Give access to the Jenkins master.
                 ("jenkins-user" ,(plain-file "jenkins.pub"
                                               "ssh-rsa AAAAB3NzaC1yc2EAAAADAQA\
[...]
YK+l20fjZSu198/keqjnlTIWryC479GI3 jenkins@jenkins-user.mtl.sfl"))))))
    (service docker-service-type)
    ;; (set-xorg-configuration (xorg-configuration
    ;;                       (keyboard-layout keyboard-layout)))

    ;; TODO: mcron jobs for cleaning up old docker containers, stale
    ;; /tmp files
    (service slim-service-type
             (slim-configuration
              (auto-login? #f)
              (default-user "mcournoyer")
              (xorg-configuration
               (xorg-configuration
                (keyboard-layout keyboard-layout)))))
    (service guix-publish-service-type
             (guix-publish-configuration
              (host "0.0.0.0")))        ;listen on all interfaces
    (modify-services %my-desktop-services
      (guix-service-type config =>
                         (guix-configuration
                          (inherit config)
                          (authorized-keys
                           (cons (local-file "some-key.pub")
                                 %default-authorized-guix-keys))
                          (extra-options '("--max-jobs=8"))))
      ;; Enable using adb as a simple user with a multitude of devices.
      (udev-service-type config =>
                         (udev-configuration
                          (inherit config)
                          (rules (cons* android-udev-rules
                                        (udev-configuration-rules config))))))))

  ;; Allow resolution of '.local' host names with mDNS.
  (name-service-switch %mdns-host-lookup-nss))
--8<---------------cut here---------------end--------------->8---

Note that to have my root partition mounted on a subvolume, you'll need
my (yet to be merged) patches available at:
https://issues.guix.info/issue/37305.  I'll post a fresh, rebased v3
(hopefully the last!) series shortly.

Maxim

Re: 02/02: gnu: next: Compress the executable.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

Thanks Maxim, this is a great config.scm! :)

> Note that to have my root partition mounted on a subvolume, you'll need
> my (yet to be merged) patches available at:
> https://issues.guix.info/issue/37305.  I'll post a fresh, rebased v3
> (hopefully the last!) series shortly.

I see that you've posted a new series.
I've just sent patch 40236@debbugs.gnu.org to edit the template and use
Btrfs + Zstd.

Thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]