Several questions on Guix(SD)

Several questions on Guix(SD)

[Björn Höfling]

Dear list,

since I watched that LibrePlanet video on GuixSD, I installed it in a
QEMU environment and I'm very fascinated about it. It's a nice concept,
having package management the functional and FSDG way. And that concept
is reality. It works!

I'm trying to learn to use and understand the system, and found out
several things for myself or through documentation or other people's
question on the list. But there is a lot left which I would like to ask
here. I hope it is not too much:

 * Do you have the concept of package maintainers, like in Debian?
   People who are responsible for each package, apply patches, etc?

 * Are there "recommmended"/"sugggested" dependencies? For example,
   diffoscope complained that "'readelf' is not on the path. Falling
   back to binary comparison." That could be a candidate for a package
   not needed at compile/runtime but that would enhance the usage if
   present.

 * I have for example in my store:
   * /gnu/store/abc...-tar-1.29/
   * /gnu/store/xyz...-tar-1.29/

   Can I somehow figure out what's the differnce between them, how
   they came into the system, which one is "newer"? I tried diffoscope
   and found some marginal, crypted hex-differences.

 * Can I somehow get a store history? I.e. a (ordered) list of
   installation progress. "ls -ltr /gnu/store" was my first try
   but it obviously fails.

 * Is there a log for the deamon? I only found the
   "/var/log/guix/drvs", which seam to be only logs of each compilation.
   BTW, what do the two letters of the subfolders mean? 

 * Is there a list of builds available that are not reproducible? Or is
   that (eq? _ '()) ?

 * Are any other Hydra instances running other than the GNU ones that
   could be challenged?

 * If I would like to set up my own Hydra server, how much storage space
   takes the whole hydra-archive of substitutes (for each platform)? Or
   only the %base-packages?

 * How much time takes it to recompile all packages in Hydra?


Thank you in advance,

Björn

Re: Several questions on Guix(SD)

[Leo Famulari]

Welcome, Björn!

On Mon, Aug 22, 2016 at 10:53:50PM +0200, Björn Höfling wrote:
>  * Do you have the concept of package maintainers, like in Debian?
>    People who are responsible for each package, apply patches, etc?

No, we don't have a concept of this. In practice, people pay attention
to the packages they care about. This information can be obtained with
`git blame` and `git log`.

Some of us subscribe to security-related mailing lists, and apply
security updates across the package tree. More people should do this,
please :)

>  * Are there "recommmended"/"sugggested" dependencies? For example,
>    diffoscope complained that "'readelf' is not on the path. Falling
>    back to binary comparison." That could be a candidate for a package
>    not needed at compile/runtime but that would enhance the usage if
>    present.

All dependencies are specified explicitly in the package definitions and
the build systems.

We do have a concept of run-time and build-time dependencies. See the
text about "inputs":
<https://www.gnu.org/software/guix/manual/html_node/package-Reference.html#package-Reference>

Speaking of caring about packages, I care about diffoscope. I will look
into adding readelf to the diffoscope package definition.

>  * I have for example in my store:
>    * /gnu/store/abc...-tar-1.29/
>    * /gnu/store/xyz...-tar-1.29/
> 
>    Can I somehow figure out what's the differnce between them, how
>    they came into the system, which one is "newer"? I tried diffoscope
>    and found some marginal, crypted hex-differences.

You can inspect the relationships a given store item has to other store
items with the '--referrers', '--requisites', '--references' options to
`guix gc`:
<https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-gc.html#Invoking-guix-gc>

I'm not sure if it's possible to go backwards from a store item to the
Guix Git commit it was built from. Advice wanted!

>  * Can I somehow get a store history? I.e. a (ordered) list of
>    installation progress. "ls -ltr /gnu/store" was my first try
>    but it obviously fails.

Yes, there is `guix package --list-generations`. For more, see
<https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-package.html#Invoking-guix-package>
<https://www.gnu.org/software/guix/manual/html_node/Features.html#Features>

>  * Is there a log for the deamon? I only found the
>    "/var/log/guix/drvs", which seam to be only logs of each compilation.
>    BTW, what do the two letters of the subfolders mean? 

The log output of the daemon goes wherever your daemons normally log to.
On most recent versions of the major distros, this is handled by
systemd / journald.

Package build logs can be obtained like so:

`guix build foo --log-file --no-grafts`

... but you still have the problem of knowing which version of Guix
produced the store item you are interested in. I'm interested to hear
better answers to this question.

>  * Is there a list of builds available that are not reproducible? Or is
>    that (eq? _ '()) ?

We don't have much infrastructure for tracking reproducible builds yet.
Help wanted!

We do have `guix challenge`, for challenging a binary substitute server.
Also, packages can be built with '--rounds=x' and '--check', which
repeat the build and compare the outputs to see if they are bit-for-bit
identical.

<https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-build.html#Invoking-guix-build>
<https://www.gnu.org/software/guix/manual/html_node/Common-Build-Options.html#Common-Build-Options>

>  * Are any other Hydra instances running other than the GNU ones that
>    could be challenged?

I don't know of any public instances. There are some private instances
running in institutional settings.

There is `guix publish`, which is not a full continuous integration
system like Hydra, but rather a simple HTTP server that you can use to
serve your binary substitutes:

<https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-publish.html#Invoking-guix-publish>

>  * If I would like to set up my own Hydra server, how much storage space
>    takes the whole hydra-archive of substitutes (for each platform)? Or
>    only the %base-packages?
> 
>  * How much time takes it to recompile all packages in Hydra?

I don't know. Others can answer this.

And, others can add to my answers as well.

Re: Several questions on Guix(SD)

[Björn Höfling]

Leo, thank you for your answers. I would like some more clarifications:

On Mon, 22 Aug 2016 17:49:21 -0400
Leo Famulari <leo@famulari.name> wrote:

> 
> >  * Is there a log for the deamon? I only found the
> >    "/var/log/guix/drvs", which seam to be only logs of each
> > compilation. BTW, what do the two letters of the subfolders mean? 
> 
> The log output of the daemon goes wherever your daemons normally log
> to. On most recent versions of the major distros, this is handled by
> systemd / journald.

My distro in that case is GuixSD, using shepherd. So I found information
about guix-daemon start and stop events in /var/log/shepherd.log. But I
want some more information about the daemon. Is guix-daemon so silent?

I figured out that I can (modify-services %base-services ...) for
adding the "--debug" extra-options to guix-daemon. Still I see only
start/stop log entries in shepherd.log. What am I missing?


> >  * Is there a list of builds available that are not reproducible?
> > Or is that (eq? _ '()) ?
> 
> We don't have much infrastructure for tracking reproducible builds
> yet. Help wanted!
> 
> We do have `guix challenge`, for challenging a binary substitute
> server. Also, packages can be built with '--rounds=x' and '--check',
> which repeat the build and compare the outputs to see if they are
> bit-for-bit identical.

So you mean you can't tell which builds are not reproducible? Couldn't
Hydra just build each package two times? Or does that cost too much?

Björn

Re: Several questions on Guix(SD)

[Leo Famulari]

On Thu, Aug 25, 2016 at 11:03:32PM +0200, Björn Höfling wrote:
> Leo, thank you for your answers. I would like some more clarifications:
> 
> On Mon, 22 Aug 2016 17:49:21 -0400
> Leo Famulari <leo@famulari.name> wrote:
> > We don't have much infrastructure for tracking reproducible builds
> > yet. Help wanted!
> > 
> > We do have `guix challenge`, for challenging a binary substitute
> > server. Also, packages can be built with '--rounds=x' and '--check',
> > which repeat the build and compare the outputs to see if they are
> > bit-for-bit identical.
> 
> So you mean you can't tell which builds are not reproducible? Couldn't
> Hydra just build each package two times? Or does that cost too much?

I'm not familiar with the configuration of the Hydra CI system, so I
don't know if it would be easy or hard to make it build everything twice
and check the results.

But, we would still need some infrastructure to remember the results of
the comparison, and to distinguish it from other types of failures, and
I don't _think_ we have that yet.

There is a Google Summer of Code project that aims to replace Hydra with
a Guile Scheme program called Cuirass:
https://notabug.org/mthl/cuirass

We want to support development of Cuirass until we can "dog food" it,
and then we may start building features on top of it such as a
reproducibility checker.

I've CC-ed the Cuirass author in my reply in case he has some comments.
He has also sent some reports the guix-devel mailing list that you may
find interesting.

Re: Several questions on Guix(SD)

[Leo Famulari]

On Thu, Aug 25, 2016 at 11:03:32PM +0200, Björn Höfling wrote:
> Leo, thank you for your answers. I would like some more clarifications:
> 
> On Mon, 22 Aug 2016 17:49:21 -0400
> Leo Famulari <leo@famulari.name> wrote:
> > The log output of the daemon goes wherever your daemons normally log
> > to. On most recent versions of the major distros, this is handled by
> > systemd / journald.
> 
> My distro in that case is GuixSD, using shepherd. So I found information
> about guix-daemon start and stop events in /var/log/shepherd.log. But I
> want some more information about the daemon. Is guix-daemon so silent?
> 
> I figured out that I can (modify-services %base-services ...) for
> adding the "--debug" extra-options to guix-daemon. Still I see only
> start/stop log entries in shepherd.log. What am I missing?

Somebody else could answer this better than me, but I'll give it a shot.
The guix-daemon is indeed laconic, even with --debug. Here, is what it
prints when I start it manually and run `guix gc -d /gnu/store/...-hello`:

---
# /root/.guix-profile/bin/guix-daemon --build-users-group=guixbuild --debug
extra chroot directories: ''
automatic deduplication set to 1
listening on `/var/guix/daemon-socket/socket'
accepted connection from pid 9343, user leo
1 operations
---

On GuixSD, it seems that the daemon just prints to the console. I have a
non-graphical system and the daemon's output is rather annoying in the
rare case that I am sitting at the machine.

If you are looking for logs of package building, you can do `guix build
foo --log-file`. You will often need to add `--no-grafts` to see the
"real" build log instead of the log of the grafting operation.

Does that help answer your question at all?

Re: Several questions on Guix(SD)

[Björn Höfling]

Hi Leo,

On Fri, 26 Aug 2016 19:23:11 -0400
Leo Famulari <leo@famulari.name> wrote:

> On Thu, Aug 25, 2016 at 11:03:32PM +0200, Björn Höfling wrote:
> > Leo, thank you for your answers. I would like some more
> > clarifications:
> > 
> > On Mon, 22 Aug 2016 17:49:21 -0400
> > Leo Famulari <leo@famulari.name> wrote:
> > > The log output of the daemon goes wherever your daemons normally
> > > log to. On most recent versions of the major distros, this is
> > > handled by systemd / journald.
> > 
> > My distro in that case is GuixSD, using shepherd. So I found
> > information about guix-daemon start and stop events
> > in /var/log/shepherd.log. But I want some more information about
> > the daemon. Is guix-daemon so silent?
> > 
> > I figured out that I can (modify-services %base-services ...) for
> > adding the "--debug" extra-options to guix-daemon. Still I see only
> > start/stop log entries in shepherd.log. What am I missing?
> 
> Somebody else could answer this better than me, but I'll give it a
> shot. The guix-daemon is indeed laconic, even with --debug. Here, is
> what it prints when I start it manually and run `guix gc
> -d /gnu/store/...-hello`:
> 
> ---
> # /root/.guix-profile/bin/guix-daemon --build-users-group=guixbuild
> --debug extra chroot directories: ''
> automatic deduplication set to 1
> listening on `/var/guix/daemon-socket/socket'
> accepted connection from pid 9343, user leo
> 1 operations
> ---
> 
> On GuixSD, it seems that the daemon just prints to the console. I
> have a non-graphical system and the daemon's output is rather
> annoying in the rare case that I am sitting at the machine.
> 
> If you are looking for logs of package building, you can do `guix
> build foo --log-file`. You will often need to add `--no-grafts` to
> see the "real" build log instead of the log of the grafting operation.
> 
> Does that help answer your question at all?

I also noticed that it prints to the console. I hoped that it would
print a bit more output to some logfile. But that's fine for now, that
answers my question. Thank you.

Björn

Re: Several questions on Guix(SD)

[Ludovic Courtès]

Hello!

Leo Famulari <leo@famulari.name> skribis:

> On Mon, Aug 22, 2016 at 10:53:50PM +0200, Björn Höfling wrote:

[...]

>>  * Are any other Hydra instances running other than the GNU ones that
>>    could be challenged?
>
> I don't know of any public instances. There are some private instances
> running in institutional settings.

Pjotr Prins runs a semi-public instance of ‘guix publish’ at
<http://guix.genenetwork.org/>.

Note that ‘guix challenge’ can also compare with locally-built binaries
available on your computer (‘guix build --check’ does that too), and you
could run ‘guix publish’ as well.  :-)

Thanks for your feedback!

Ludo’.

Re: Several questions on Guix(SD)

[Florian Paul Schmidt]

[-- Attachment #1.1: Type: text/plain, Size: 831 bytes --]

On 08/22/2016 10:53 PM, Björn Höfling wrote:
> Dear list,
> 
>  * How much time takes it to recompile all packages in Hydra?

I only have a rough estimate. I didn't compile all packages in hydra,
but I attempted on several occasions to compile all packages of a
certain git revision of guix for my own platform. Machine: VM with 8
cores of

model name      : Intel Xeon E312xx (Sandy Bridge)
cpu MHz         : 3411.482
cache size      : 4096 KB

fps@guix ~$ free -m
              total        used        free      shared  buff/cache
available
Mem:          13699         295       11665           0        1738
 13144
Swap:             0           0           0

The build took on the order of a couple of days, somewhere between 2 and
5 if i remember correctly.

Regards,
Flo

-- 
https://fps.io


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]