Copying whole /gnu/store from USB does not work

Copying whole /gnu/store from USB does not work

[Pjotr Prins]

Hi all,

A binary install should be easy because all properly installed Guix
packages are independent from the underlying distribution - a package
can be created which contains a package with all its dependencies. The
only real dependency is the (Linux) kernel - which should have a
compatible API. Fortunately the fundamental API does not change that
often, so a binary distribution can be copied from machine to
machine. Of course, also the build target archicture has to match.
With a binary distribution it is possible to simply unpack the
package, create the build group and permissions.

I thought I would be able to bootstrap using a USB image and copy
/gnu/store to a local hard disk.

To bootstrap I have successfully downloaded a GNU Guix tarball 

    wget http://alpha.gnu.org/gnu/guix/gnu-usb-install-0.8.x86_64-linux.xz
    xz -d gnu-usb-install-0.8.x86_64-linux.xz 

mount it with kpartx

    kpartx -v -a gnu-usb-install-0.8.x86_64-linux
    mkdir mnt 
    mkdir tmp
    mount /dev/mapper/loop0p1 mnt
    cp -vau mnt/gnu tmp
    mv tmp/gnu/ /

and should be able to run Guix with

    /gnu/store/(...)-guix-0.8(...)/bin/guix --help

Actually this now works the first time.

There is a problem currently: right after copying the files to /gnu
they work find, e.g. 

    /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash --version
    GNU bash, version 4.3.33(1)-release (x86_64-unknown-linux-gnu)

Nice! 

when I run, however,

    /gnu/store/wnsbzddp3izv3xr3gqf2rar4fyd4shl7-guix-0.8.1.f1082ec/bin/guix package -i bash

it runs and I get

    The following package will be installed:     
       bash 4.3.33  /gnu/store/7cc229218glj3kzp6zafzs5mc4241wqk-bash-4.3.33
    substitute-binary: updating list of substitutes from 'http://hydra.gnu.org'...

    The following derivations will be built:
    /gnu/store/gkfzdr01ga0m1c2vl8dh6p3cspjik7pp-profile.drv
    /gnu/store/kw632ijb16nracg8kbd2z3qqarv72b9c-info-dir.drv
    The following files will be downloaded:
    /gnu/store/7cc229218glj3kzp6zafzs5mc4241wqk-bash-4.3.33
    /gnu/store/1j6njwwhcqr04c2m0868400p74pv36k2-module-import-compiled
    /gnu/store/z2mbq8915vvh0fj6sz3iqiifrdm9k7np-guile-2.0.11
    (...)
    substitute error: executing `/gnu/store/wnsbzddp3izv3xr3gqf2rar4fyd4shl7-guix-0.8.1.f1082ec/libexec/guix/substitute-binary': No such file or directory
    guix package: error: build failed: unexpected EOF reading a line

Looking at

    head -1 /gnu/store/wnsbzddp3izv3xr3gqf2rar4fyd4shl7-guix-0.8.1.f1082ec/libexec/guix/substitute-binary
#!/gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash

it calls bash and now check

    /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash
    bash: /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash: No such file or directory

while

    ldd /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash
        linux-vdso.so.1 =>  (0x00007fff739ec000)
        libreadline.so.6 => /gnu/store/fznw2a1ig05jkc2yrw8knzcb99akcjx7-readline-6.3/lib/libreadline.so.6 (0x00007f0084bd5000)
        libhistory.so.6 => /gnu/store/fznw2a1ig05jkc2yrw8knzcb99akcjx7-readline-6.3/lib/libhistory.so.6 (0x00007f00849cc000)
        libncursesw.so.5 => /gnu/store/ksnhghfy6hkk8mmxd4nacyd8c1smdp6p-ncurses-5.9/lib/libncursesw.so.5 (0x00007f008476a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f008455b000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f008419b000)
        /gnu/store/q2mm1wv4a3g0b29yv0rjybfjh8kr07qi-glibc-2.20/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0084e1d000)

Oooops. Where does libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 come from? 

Also the other bash stopped working too

    /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash --version

and ldd shows links outside /gnu (when it worked they are all linked inside /gnu).

Also the guix binary stopped working. Same issue.

Can anyone explain this? I tried multiple USB images and it is all the same.

Note I am not building, only installing binaries.

Pj.

Re: Copying whole /gnu/store from USB does not work

[Ludovic Courtès]

Hi!

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> I thought I would be able to bootstrap using a USB image and copy
> /gnu/store to a local hard disk.

To “bootstrap,” it’s best to install Guix on top of your distro, and
build/download packages from there.  (Actually bootstrapping doesn’t
require any special action with Guix.)

> mount it with kpartx
>
>     kpartx -v -a gnu-usb-install-0.8.x86_64-linux
>     mkdir mnt 
>     mkdir tmp
>     mount /dev/mapper/loop0p1 mnt
>     cp -vau mnt/gnu tmp
>     mv tmp/gnu/ /

That cannot work.

The reason is that the store is really two things: /gnu/store, and
/var/guix/db/db.sqlite.  The latter contains meta-data about the former,
notably the list of valid store items.

The valid way to do that is with ‘guix archive’ (info "(guix) Invoking
guix archive"), or using the ‘populate-store’ and ‘register-closure’ as
is done in (gnu build vm).

HTH,
Ludo’.

PS: note that the latest version is 0.8.1, not 0.8.

Re: Copying whole /gnu/store from USB does not work

[Pjotr Prins]

Hi Ludo,

On Fri, Apr 10, 2015 at 01:48:34PM +0200, Ludovic Courtès wrote:
> Hi!
> 
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > I thought I would be able to bootstrap using a USB image and copy
> > /gnu/store to a local hard disk.
> 
> To ???bootstrap,??? it???s best to install Guix on top of your distro, and
> build/download packages from there.  (Actually bootstrapping doesn???t
> require any special action with Guix.)

Well yes, you need to install all prerequisites and that is not always
trivial. Try bootstrapping on an old Centos install, for example.
Can't even get Docker going there.

It would be nice to have an installable binary tar ball for Guix. Just
unpack in root and go...

> > mount it with kpartx
> >
> >     kpartx -v -a gnu-usb-install-0.8.x86_64-linux
> >     mkdir mnt 
> >     mkdir tmp
> >     mount /dev/mapper/loop0p1 mnt
> >     cp -vau mnt/gnu tmp
> >     mv tmp/gnu/ /
> 
> That cannot work.

I am merely trying to be clever ;)

> The reason is that the store is really two things: /gnu/store, and
> /var/guix/db/db.sqlite.  The latter contains meta-data about the former,
> notably the list of valid store items.

Aha. That makes sense. If I also copy the db.sqlite it might just
work then.

> The valid way to do that is with ???guix archive??? (info "(guix) Invoking
> guix archive"), or using the ???populate-store??? and ???register-closure??? as
> is done in (gnu build vm).

OK, I'll take a look. Thanks. 

Pj.

PS I am rather excited that you have LLVM going and Python packages
and that Ricardo is adding bioinformatics packages. Way to go!! Now we
need to add Ruby gems and GNU Guix will be the default deployment
system for many.

Re: Copying whole /gnu/store from USB does not work

[David Thompson]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Now we need to add Ruby gems and GNU Guix will be the default
> deployment system for many.

Help wanted. ;)

Two major problems to deal with:

1) Propagated inputs: Dynamic languages typically don't have an
equivalent to RUNPATH for ELF binaries.  Thus, for Python (and others),
we have to propagate all of the python libraries along with each package
so that they can all be found in a particular load path.  This in
unideal.  I haven't found a way to hardcode the location of libraries as
part of the build process.  If it can't be reasonably solved, we'll just
propagate the inputs like every other dynamic language.

Do you know a way that we could avoid this for Ruby gems?

2) Circular dependencies: Rspec is a particularly important Ruby gem to
package.  Pretty much every Ruby project uses it for their test suite.
However, a lot of Rspec's dependencies use Rspec!  The only thing I've
come up with is to produce a set of private packages for all of Rspec's
dependencies whose build processes skip the test suite.  Then, with
Rspec packaged, create public versions of the packages with Rspec as a
native input and the test suite enabled.  The dependency tree isn't
exactly shallow, though.

The Rspec maintainer failed to understand this issue, because the system
in place at rubygems.org doesn't run test suites or build the gems at
all.  People just upload the final product and no one notices that
there's nasty circular dependencies.

Thoughts?

Nice to see you poking around with Guix again, Pjotr.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Copying whole /gnu/store from USB does not work (SOLVED)

[Pjotr Prins]

> > The reason is that the store is really two things: /gnu/store, and
> > /var/guix/db/db.sqlite.  The latter contains meta-data about the former,
> > notably the list of valid store items.
> 
> Aha. That makes sense. If I also copy the db.sqlite it might just
> work then.

It works rather well and is easy. This means we can just bootstrap
from an existing /gnu tree in thu USB distribution provided the
kernel/arch is compatible.

At build time 'guix package -i guix' complained it did not have enough
builders. The guix-daemon did not honour the -M 8 switch, so I had to
add 30 build users on a 32 core machine. Now it is happy.

Pj.

Re: Copying whole /gnu/store from USB does not work (SOLVED)

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> At build time 'guix package -i guix' complained it did not have enough
> builders. The guix-daemon did not honour the -M 8 switch, so I had to
> add 30 build users on a 32 core machine. Now it is happy.

Could it be that you’re using 0.8 instead of 0.8.1?  The latter fixed a
bug in that area, whereby -M would default to the number of cores.

Ludo’.

Self-contained Guix tarball

[Ludovic Courtès]

[-- Attachment #1.1: Type: text/plain, Size: 1325 bytes --]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> Well yes, you need to install all prerequisites and that is not always
> trivial. Try bootstrapping on an old Centos install, for example.
> Can't even get Docker going there.
>
> It would be nice to have an installable binary tar ball for Guix. Just
> unpack in root and go...

I’ve given that a try since it’s actually quite simple to do.  The
procedure in the attached file produces a self-contained tarball.  It
contains the closure of Guix, as well as /var/guix and an initial
/root/.guix-profile from which one can use the ‘guix’ commands.

I’ve uploaded the result (for x86_64) at:

  http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz
  http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz.sig
  SHA1: afde62b3e9f97f9ae0c57eaccfaca842b157cd2f

To install Guix on a foreign distro, follow these steps:

  1. Download the tarball (38 MiB.)

  2. As root, run:
       cd /
       tar xf /path/to/guix-tarball-0.8.1.1140.tar.lz
     You need lzip and GNU tar.

  3. Setup the daemon as explained in the manual, and run it from
     /root/.guix-profile/bin/guix-daemon.

And hopefully, that’s it.

I haven’t actually tried it, so feedback is welcome!  :-)

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: tarball creation --]
[-- Type: text/x-scheme, Size: 3477 bytes --]

(use-modules (guix)
             (guix profiles)
             (gnu packages base)
             (gnu packages compression)
             (gnu packages package-management))

(define* (self-contained-tarball #:key (guix guix))
  (mlet %store-monad ((profile (profile-derivation
                                (manifest
                                 (list (package->manifest-entry guix))))))
    (gexp->derivation "guix-tarball.tar.lz"
                      #~(begin
                          (use-modules (guix build store-copy)
                                       (guix build utils)
                                       (gnu build install)
                                       (srfi srfi-26))

                          (define %root
                            (string-append (getcwd) "/root"))
                          (define %root-profile
                            "/var/guix/profiles/per-user/root")

                          (define (mkdir-p* dir)
                            (mkdir-p (string-append %root "/" dir)))

                          (define (symlink* old new)
                            (symlink old (string-append %root "/" new)))

                          (setenv "PATH"
                                  (string-append #$guix "/sbin:"
                                                 #$tar "/bin:" #$lzip "/bin"))

                          ;; Populate the store.
                          (populate-store '("profile") %root)
                          (for-each (cut register-closure %root <>)
                                    '("profile"))

                          ;; 'guix-register' registers profiles as GC roots
                          ;; but the symlink target uses $TMPDIR.  Fix that.
                          (delete-file
                           (string-append %root
                                          "/var/guix/gcroots/profiles"))
                          (symlink* "/var/guix/profiles"
                                    "/var/guix/gcroots/profiles")

                          ;; Make root's profile, which makes it a GC root.
                          (mkdir-p* %root-profile)
                          (symlink* #$profile
                                    (string-append %root-profile
                                                   "/guix-profile-1-link"))
                          (symlink* (string-append %root-profile
                                                   "/guix-profile-1-link")
                                    (string-append %root-profile
                                                   "/guix-profile"))

                          (mkdir-p* "/root")
                          (symlink* (string-append %root-profile
                                                   "/guix-profile")
                                    "/root/.guix-profile")

                          ;; Create the tarball.  Use GNU format so there's no
                          ;; file name length limitation.
                          (chdir %root)
                          (zero? (system* "tar" "--lzip" "--format=gnu"
                                          "-cvf" #$output ".")))
                      #:references-graphs `(("profile" ,profile))
                      #:modules '((guix build utils)
                                  (guix build store-copy)
                                  (gnu build install)))))

;; (pk (with-store store
;;       (run-with-store store (self-contained-tarball))))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Self-contained Guix tarball

[Pjotr Prins]

The tar-ball works a treat!! Simply download and

   tar -C / --lzip -xvf guix-tarball-0.8.1.1140.tar.lz 
   export ~/.guix-profile/bin:$PATH

Add the build users, update the /gnu/store permissions and start the
daemon and that is it (which could be covered by a simple shell
script).

This should work on ANY Linux, provided kernel and architecture are
compatible.

Very nice. GNU Guix out of a box. No more installation pain. 

The tar-ball should go in the standard download directory. I would
name the tar ball with something 'binary', maybe add the supported
Linux versions (3.x?) and the architecture in the name. Just to be
complete. There may be other editions later.

Pj.

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> The tar-ball works a treat!! Simply download and
>
>    tar -C / --lzip -xvf guix-tarball-0.8.1.1140.tar.lz 
>    export ~/.guix-profile/bin:$PATH
>
> Add the build users, update the /gnu/store permissions

Normally guix-daemon adjusts the permissions/ownership of /gnu/store
automatically nowadays, so that part shouldn’t even be needed.

> Very nice. GNU Guix out of a box. No more installation pain. 
>
> The tar-ball should go in the standard download directory. I would
> name the tar ball with something 'binary', maybe add the supported
> Linux versions (3.x?) and the architecture in the name. Just to be
> complete. There may be other editions later.

Yes, something like guix-binary-0.8.2.x86_64.tar.gz?

I guess we’ll publish it for the next release.

Thanks for your feedback!

Ludo’.

Re: Self-contained Guix tarball

[Pjotr Prins]

On Mon, Apr 13, 2015 at 11:57:20AM +0200, Ludovic Courtès wrote:
> Yes, something like guix-binary-0.8.2.x86_64.tar.gz?
> 
> I guess we???ll publish it for the next release.
> 
> Thanks for your feedback!

So far, I have installed the tarball on 5 machines. And counting...

It is beautiful.

Pj.

Re: Self-contained Guix tarball

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> Very nice. GNU Guix out of a box. No more installation pain. 
>>
>> The tar-ball should go in the standard download directory. I would
>> name the tar ball with something 'binary', maybe add the supported
>> Linux versions (3.x?) and the architecture in the name. Just to be
>> complete. There may be other editions later.
>
> Yes, something like guix-binary-0.8.2.x86_64.tar.gz?
>
> I guess we’ll publish it for the next release.

Thank you very much for this!

Could we add a package to Guix to build this?  If that is undesirable
for some reason, could we add a command to build it, and jobs on Hydra
to build it automatically on all platforms supported by the build farm,
analogous to the 'usb-image' jobs?

      Mark

Re: Self-contained Guix tarball

[Thompson, David]

On Sun, Apr 12, 2015 at 4:07 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>
> I’ve given that a try since it’s actually quite simple to do.  The
> procedure in the attached file produces a self-contained tarball.  It
> contains the closure of Guix, as well as /var/guix and an initial
> /root/.guix-profile from which one can use the ‘guix’ commands.
>
> I’ve uploaded the result (for x86_64) at:
>
>   http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz
>   http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz.sig
>   SHA1: afde62b3e9f97f9ae0c57eaccfaca842b157cd2f
>
> To install Guix on a foreign distro, follow these steps:
>
>   1. Download the tarball (38 MiB.)
>
>   2. As root, run:
>        cd /
>        tar xf /path/to/guix-tarball-0.8.1.1140.tar.lz
>      You need lzip and GNU tar.
>
>   3. Setup the daemon as explained in the manual, and run it from
>      /root/.guix-profile/bin/guix-daemon.
>
> And hopefully, that’s it.
>
> I haven’t actually tried it, so feedback is welcome!  :-)

It works perfectly!

- Dave

Re: Self-contained Guix tarball

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Could we add a package to Guix to build this?  If that is undesirable
> for some reason, could we add a command to build it, and jobs on Hydra
> to build it automatically on all platforms supported by the build farm,
> analogous to the 'usb-image' jobs?

Commit b607593 adds Hydra jobs, and commit 9d3fb6c adds the code and a
Makefile target:

  make guix-binary.x86_64-linux.tar.xz

or similar builds the binary tarball.

We’ll have to figure out how to handle that for all the systems, for the
release.

Thanks,
Ludo’.

Re: Self-contained Guix tarball

[Ludovic Courtès]

I’ve documented installation from the binary tarball (which I plan to
distribute for the next release) in commit 09722b1.  Please let me know
what you think.

Ludo’.


2.1 Binary Installation
=======================

This section describes how to install Guix on an arbitrary system from a
self-contained tarball providing binaries for Guix and for all its
dependencies.  This is often quicker than installing from source, which
is described in the next sections.  The only requirement is to have
GNU tar and Xz.

   Installing goes along these lines:

  1. Download the binary tarball from
     ‘ftp://alpha.gnu.org/gnu/guix/guix-binary-0.8.2.SYSTEM.tar.xz’(1),
     where SYSTEM is ‘x86_64-linux’ for an ‘x86_64’ machine already
     running the kernel Linux, and so on.

  2. As ‘root’, run:

          # cd /
          # tar xf guix-binary-0.8.2.SYSTEM.tar.xz

     This creates ‘/gnu/store’ (*note The Store::), ‘/var/guix’, and
     ‘/root/.guix-profile’.  ‘/root/.guix-profile’ is a ready-to-use
     profile for ‘root’ where Guix is installed.

  3. Setup the daemon as explained below (*note Setting Up the
     Daemon::), and run it:

          # /root/.guix-profile/bin/guix-daemon --build-users-group=guix-builder

  4. Make the ‘guix’ command available to other users on the machine,
     for instance with:

          # mkdir -p /usr/local/bin
          # cd /usr/local/bin
          # ln -s /root/.guix-profile/bin/guix

   And that’s it!

   There are two things to note.  First, don’t unpack the tarball on a
working Guix system since that would overwrite its own essential files.
Second, the ‘guix’ package must remain available in ‘root’’s profile, or
it would become subject to garbage collection—in which case you would
find yourself badly handicapped by the lack of the ‘guix’ command.

   The tarball in question can be (re)produced simply by running the
following command in the Guix source tree:

     make guix-binary.SYSTEM.tar.xz

   ---------- Footnotes ----------

   (1) As usual, make sure to download the associated ‘.sig’ file and to
verify the authenticity of the tarball against it!

Re: Self-contained Guix tarball

[Pjotr Prins]

Great :). I would make it a little clearer that this is
'bootstrapping' and hype it a little more that now there is no reason
NOT to install Guix. Only 100Mb on your HDD.

The warning about overwriting a Guix installation should be earlier.
Maybe point out that if you want to move a package with dependencies
see another part of the documentation.

BTW when Nix decided to go for a meta-database they lost something. I
know it has good reasons (performance mostly) but it took away the
self-containedness of packages. It would be nice just to be able to
copy/del packages and rebuild the meta information. Do we have
something like that? 

Pj.

On Wed, Apr 15, 2015 at 11:31:09PM +0200, Ludovic Courtès wrote:
> I???ve documented installation from the binary tarball (which I plan to
> distribute for the next release) in commit 09722b1.  Please let me know
> what you think.
> 
> Ludo???.
> 
> 
> 2.1 Binary Installation
> =======================
> 
> This section describes how to install Guix on an arbitrary system from a
> self-contained tarball providing binaries for Guix and for all its
> dependencies.  This is often quicker than installing from source, which
> is described in the next sections.  The only requirement is to have
> GNU tar and Xz.
> 
>    Installing goes along these lines:
> 
>   1. Download the binary tarball from
>      ???ftp://alpha.gnu.org/gnu/guix/guix-binary-0.8.2.SYSTEM.tar.xz???(1),
>      where SYSTEM is ???x86_64-linux??? for an ???x86_64??? machine already
>      running the kernel Linux, and so on.
> 
>   2. As ???root???, run:
> 
>           # cd /
>           # tar xf guix-binary-0.8.2.SYSTEM.tar.xz
> 
>      This creates ???/gnu/store??? (*note The Store::), ???/var/guix???, and
>      ???/root/.guix-profile???.  ???/root/.guix-profile??? is a ready-to-use
>      profile for ???root??? where Guix is installed.
> 
>   3. Setup the daemon as explained below (*note Setting Up the
>      Daemon::), and run it:
> 
>           # /root/.guix-profile/bin/guix-daemon --build-users-group=guix-builder
> 
>   4. Make the ???guix??? command available to other users on the machine,
>      for instance with:
> 
>           # mkdir -p /usr/local/bin
>           # cd /usr/local/bin
>           # ln -s /root/.guix-profile/bin/guix
> 
>    And that???s it!
> 
>    There are two things to note.  First, don???t unpack the tarball on a
> working Guix system since that would overwrite its own essential files.
> Second, the ???guix??? package must remain available in ???root??????s profile, or
> it would become subject to garbage collection???in which case you would
> find yourself badly handicapped by the lack of the ???guix??? command.
> 
>    The tarball in question can be (re)produced simply by running the
> following command in the Guix source tree:
> 
>      make guix-binary.SYSTEM.tar.xz
> 
>    ---------- Footnotes ----------
> 
>    (1) As usual, make sure to download the associated ???.sig??? file and to
> verify the authenticity of the tarball against it!
> 

-- 

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> Great :). I would make it a little clearer that this is
> 'bootstrapping' and hype it a little more that now there is no reason
> NOT to install Guix. Only 100Mb on your HDD.

Not sure how to do that, would you like to propose actual text?
The thing is, I want it to remain accurate and factual.

> The warning about overwriting a Guix installation should be earlier.

Done in 5dc3ce5.

> Maybe point out that if you want to move a package with dependencies
> see another part of the documentation.

What do you meaning by moving a package with dependencies?

> BTW when Nix decided to go for a meta-database they lost something. I
> know it has good reasons (performance mostly) but it took away the
> self-containedness of packages. It would be nice just to be able to
> copy/del packages and rebuild the meta information. Do we have
> something like that? 

This part is the same as Nix.  The database is here to store meta-data
about store items, notably the list of references found in a store item.
Determining this list requires scanning all of the store item’s
contents, which takes time proportional to the number/size of files it
contains, so the database can hardly be avoided.

Thanks,
Ludo’.

Re: Self-contained Guix tarball

[Pjotr Prins]

On Sat, Apr 18, 2015 at 11:23:50PM +0200, Ludovic Courtès wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > Great :). I would make it a little clearer that this is
> > 'bootstrapping' and hype it a little more that now there is no reason
> > NOT to install Guix. Only 100Mb on your HDD.
> 
> Not sure how to do that, would you like to propose actual text?
> The thing is, I want it to remain accurate and factual.

The current text is fine. 

> What do you meaning by moving a package with dependencies?

I am thinking about Nix-style closures. But it may only confuse
things. I don't think the Guix manual covers closures.

> > BTW when Nix decided to go for a meta-database they lost something. I
> > know it has good reasons (performance mostly) but it took away the
> > self-containedness of packages. It would be nice just to be able to
> > copy/del packages and rebuild the meta information. Do we have
> > something like that? 
> 
> This part is the same as Nix.  The database is here to store meta-data
> about store items, notably the list of references found in a store item.
> Determining this list requires scanning all of the store item???s
> contents, which takes time proportional to the number/size of files it
> contains, so the database can hardly be avoided.

Yes, I understand. But would it be possible to regenerate the database
from an existing /gnu/store? You can see I like to mess around with
files ;). With closures a rebuild should not be necessary, but as a
wary system administrator I know I will need it at some point.

Pj.

Re: Self-contained Guix tarball

[Taylan Ulrich Bayırlı/Kammer]

ludo@gnu.org (Ludovic Courtès) writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> BTW when Nix decided to go for a meta-database they lost something. I
>> know it has good reasons (performance mostly) but it took away the
>> self-containedness of packages. It would be nice just to be able to
>> copy/del packages and rebuild the meta information. Do we have
>> something like that? 
>
> This part is the same as Nix.  The database is here to store meta-data
> about store items, notably the list of references found in a store item.
> Determining this list requires scanning all of the store item’s
> contents, which takes time proportional to the number/size of files it
> contains, so the database can hardly be avoided.

Just throwing this out there:

Would it be possible to bundle meta-data files together with store
items?

I guess one could in principle make each store item a directory with any
number of meta-data files plus the actual content of the store item (a
file or directory), but I suppose it would be too big an overhaul of the
design to make this worthwhile.

Taylan

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Sat, Apr 18, 2015 at 11:23:50PM +0200, Ludovic Courtès wrote:

[...]

>> What do you meaning by moving a package with dependencies?
>
> I am thinking about Nix-style closures. But it may only confuse
> things. I don't think the Guix manual covers closures.

I think “Invoking guix archive” covers what you have in mind.  Maybe it
should be made more visible somehow?

https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-archive.html

>> This part is the same as Nix.  The database is here to store meta-data
>> about store items, notably the list of references found in a store item.
>> Determining this list requires scanning all of the store item???s
>> contents, which takes time proportional to the number/size of files it
>> contains, so the database can hardly be avoided.
>
> Yes, I understand. But would it be possible to regenerate the database
> from an existing /gnu/store?

In theory, yes.

> You can see I like to mess around with files ;). With closures a
> rebuild should not be necessary, but as a wary system administrator I
> know I will need it at some point.

What do you mean by “rebuild should not be necessary”?  If you remove
files from the store manually?

Thanks,
Ludo’.

Re: Self-contained Guix tarball

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Would it be possible to bundle meta-data files together with store
> items?
>
> I guess one could in principle make each store item a directory with any
> number of meta-data files plus the actual content of the store item (a
> file or directory), but I suppose it would be too big an overhaul of the
> design to make this worthwhile.

I suspect this would make GC inefficient (lots of disk seeks to
determine references/referrers compared to queries of the SQLite
database.)

Another (opposite :-)) option is to make /gnu/store a read-only bind
mount on GuixSD.  Commit 3392ce5 does that.  This will prevent
accidental modifications of the store.

Thanks,
Ludo’.

Re: Self-contained Guix tarball

[Pjotr Prins]

> What do you mean by ???rebuild should not be necessary????  If you
> remove files from the store manually?

When you administrate a large amount of servers things can go wrong
due to system failures, failed backup recoveries, hacking attempts and
adminstrators trying to be clever ;). Murphy's law dictates that the
store and the sqlitedb meta information will go out of sync. For
production setups it is necessary to be able to recover from backups,
but as an intermediate recovery step it would be really nice if Guix
could recover its meta information from an existing store - assuming
only the DB is corrupt (google for sqlite corrupted databases). It is
especially nice when your backups are out of sync too.

guix archive looks good, but for speedy deployment it can happen an 
adminstrator would have a simple use case of:

- Copying 2 stores to 1 machine
- Rebuild database

to do it quick and dirty. Since all software packages are isolated
this would make a really good use case even if (with some trouble) you
could use guix archive for that. I prefer quick and dirty.

I am not pushing for this functionality directly, but I would
certainly like to have it when I need it :)

Maybe I am being pathetic (a lot of people won't appreciate the
problem), but under time pressure which huge deployments - this is
what is going to be wanted.

On Mon, Apr 20, 2015 at 10:49:05PM +0200, Ludovic Courtès wrote:
> taylanbayirli@gmail.com (Taylan Ulrich "Bay??rl??/Kammer") skribis:
> 
> > Would it be possible to bundle meta-data files together with store
> > items?
> >
> > I guess one could in principle make each store item a directory with any
> > number of meta-data files plus the actual content of the store item (a
> > file or directory), but I suppose it would be too big an overhaul of the
> > design to make this worthwhile.
> 
> I suspect this would make GC inefficient (lots of disk seeks to
> determine references/referrers compared to queries of the SQLite
> database.)

Yes, Nix switched to using sqlitedb because of the GC. It is also
useful to search current versions of installed packages quickly. Even
so, I think it should be viewed as an index. The state of the machine
is what is *sitting* in the store. That would be the correct design.

Meta information can go out of sync. Therefore we should not assume
they are in sync.

And it is simple to achieve. Just allow for database rebuilds.

> Another (opposite :-)) option is to make /gnu/store a read-only bind
> mount on GuixSD.  Commit 3392ce5 does that.  This will prevent
> accidental modifications of the store.

That is a good solution for end-users. Not for administrators. So
adminstrators will circumvent it.

Remember Guix is a hackable system. That is true for the store too.

Pj.

Re: Self-contained Guix tarball

[Pjotr Prins]

guix archive copies via ssh it appears. Would it be an idea to use
rsync instead with the --delete switch. That way a copy is efficient
and faithful. You really want the same package quaranteed. I trust
guix, but I don't trust systems.

You give an example of copying profiles. How about copying a store
with and without profile linked packages? That is use case:

- copy all current packages in use on the server (in profiles)

and use case:

- copy all files, even those not linked to profiles and up for GC

When that works that would be the way forward, but it does not drop my
wish for being able to regenerate the database.

Pj.

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> When you administrate a large amount of servers things can go wrong
> due to system failures, failed backup recoveries, hacking attempts and
> adminstrators trying to be clever ;). Murphy's law dictates that the
> store and the sqlitedb meta information will go out of sync. For
> production setups it is necessary to be able to recover from backups,
> but as an intermediate recovery step it would be really nice if Guix
> could recover its meta information from an existing store - assuming
> only the DB is corrupt (google for sqlite corrupted databases). It is
> especially nice when your backups are out of sync too.

The important thing is that currently, the DB is authoritative.  So it
cannot be corrupt (that would be equivalent to having lost /gnu/store
altogether), and thus it cannot be repaired.

What *can* be repaired is the store: for instance, if a store item is
tampered with.  The daemon has code to do it, but the Guix client tools
do not expose it yet.

> guix archive looks good, but for speedy deployment it can happen an 
> adminstrator would have a simple use case of:
>
> - Copying 2 stores to 1 machine
> - Rebuild database
>
> to do it quick and dirty. Since all software packages are isolated
> this would make a really good use case even if (with some trouble) you
> could use guix archive for that. I prefer quick and dirty.
>
> I am not pushing for this functionality directly, but I would
> certainly like to have it when I need it :)

I don’t think it could work the way you envision it.  What kind of
deployment do you have in mind?  For whole system deployment, one can
obviously use ‘guix system’.

>> I suspect this would make GC inefficient (lots of disk seeks to
>> determine references/referrers compared to queries of the SQLite
>> database.)
>
> Yes, Nix switched to using sqlitedb because of the GC.

I think it’s been there “forever” (at least since I started contributing
in 2008.)

> It is also useful to search current versions of installed packages
> quickly. Even so, I think it should be viewed as an index. The state
> of the machine is what is *sitting* in the store. That would be the
> correct design.
>
> Meta information can go out of sync. Therefore we should not assume
> they are in sync.

Again, the store can go “out of sync,” but the DB itself is
authoritative currently.

And it’s important that it be this way.  One example is that build
processes can write their outputs to the store; if the build process
fails, there are still those files in the store, but the DB won’t have
recorded them as valid, so they can be swept on the next GC.

>> Another (opposite :-)) option is to make /gnu/store a read-only bind
>> mount on GuixSD.  Commit 3392ce5 does that.  This will prevent
>> accidental modifications of the store.
>
> That is a good solution for end-users. Not for administrators. So
> adminstrators will circumvent it.

Well, administrators won’t be able to circumvent it accidentally, at
least.

I understand this framework really constrains what sysadmins can do, and
in particular prevents them from doing “quick-and-dirty hacks.”  I think
we should strive to find the UIs that allow for quick hacks while not
compromising the store’s integrity.

WDYT?

Thanks,
Ludo’.

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> guix archive copies via ssh it appears.

No, it does nothing remotely.  The manual has an example showing how to
use it over SSH, but it’s just an example...

> Would it be an idea to use rsync instead with the --delete
> switch.

... and it has a discussion of bandwidth efficiency and --missing.
:-)

> You give an example of copying profiles. How about copying a store
> with and without profile linked packages? That is use case:
>
> - copy all current packages in use on the server (in profiles)

That’s possible, by passing ‘guix archive’ the list of profiles.

> and use case:
>
> - copy all files, even those not linked to profiles and up for GC

Even those not linked?  What for?  :-)

> When that works that would be the way forward, but it does not drop my
> wish for being able to regenerate the database.

My reading of this is different: we need more/better tools to transfer
whole subsets of a store among different machines (remember we already
have ‘guix publish’ and ‘guix offload’ in that area.)

Once we have those, I think you may stop worrying about the database.  :-)

WDYT?

Ludo’.

Re: Self-contained Guix tarball

[Pjotr Prins]

On Tue, Apr 21, 2015 at 10:11:29AM +0200, Ludovic Courtès wrote:
> The important thing is that currently, the DB is authoritative.  So it
> cannot be corrupt (that would be equivalent to having lost /gnu/store
> altogether), and thus it cannot be repaired.

The point really is that it is not so hard to achieve rebuilding the
database. Anyone on this list thinks that should be possible? I am
asking the big systems guys :)

> What *can* be repaired is the store: for instance, if a store item is
> tampered with.  The daemon has code to do it, but the Guix client tools
> do not expose it yet.

There are other tools that do that. It would be a nice feature, even
so.

> I don???t think it could work the way you envision it.  What kind of
> deployment do you have in mind?  For whole system deployment, one can
> obviously use ???guix system???.

It can work if we rebuild the database from a store.

> >> I suspect this would make GC inefficient (lots of disk seeks to
> >> determine references/referrers compared to queries of the SQLite
> >> database.)
> >
> > Yes, Nix switched to using sqlitedb because of the GC.
> 
> I think it???s been there ???forever??? (at least since I started contributing
> in 2008.)

I remember the switch to sqlite and questioned it in my mind at the
time.

> > It is also useful to search current versions of installed packages
> > quickly. Even so, I think it should be viewed as an index. The state
> > of the machine is what is *sitting* in the store. That would be the
> > correct design.
> >
> > Meta information can go out of sync. Therefore we should not assume
> > they are in sync.
> 
> Again, the store can go ???out of sync,??? but the DB itself is
> authoritative currently.

Yes. It should be. But it would be good to rebuild from an existing
store consisting of packages. All I am saying :)

> >> Another (opposite :-)) option is to make /gnu/store a read-only bind
> >> mount on GuixSD.  Commit 3392ce5 does that.  This will prevent
> >> accidental modifications of the store.
> >
> > That is a good solution for end-users. Not for administrators. So
> > adminstrators will circumvent it.
> 
> Well, administrators won???t be able to circumvent it accidentally, at
> least.

True.

> I understand this framework really constrains what sysadmins can do, and
> in particular prevents them from doing ???quick-and-dirty hacks.???  I think
> we should strive to find the UIs that allow for quick hacks while not
> compromising the store???s integrity.
> 
> WDYT?

The Unix way is transparency and dirty hacks. That is why systems guys
and gurus generally don't like systemd, for example, even if it has
great features. I think that is a nice comparison. Another one would
be the Microsoft settings management infrastructure. 

To make guix loved by system deployments you have to cater for quick
and dirty. If everything has to be handled by an opaque daemon - even
if it is FOSS - they are not going to like it enough. I am speaking
for myself here (of course), but I know enough systems guys (and
girls) and how they think. One of the great features of Guix is its
store and that it is easy to grasp how it works. Handling everything
through a daemon makes sense from a user perspective - but you should
also allow for short-cuts. As unintuitive as it may sound, it is the
Unix way ;)

It may also allow for people creating other store-based tools in time.

Anyway, just raising the point now. We should not be against dirty
hacks out of principle or purism. That will hurt acceptance in my
opinion.

Pj.

Re: Self-contained Guix tarball

[Pjotr Prins]

On Tue, Apr 21, 2015 at 10:18:51AM +0200, Ludovic Courtès wrote:
> No, it does nothing remotely.  The manual has an example showing how to
> use it over SSH, but it???s just an example...
> 
> > Would it be an idea to use rsync instead with the --delete
> > switch.
> 
> ... and it has a discussion of bandwidth efficiency and --missing.
> :-)

Ah, OK.

> > You give an example of copying profiles. How about copying a store
> > with and without profile linked packages? That is use case:
> >
> > - copy all current packages in use on the server (in profiles)
> 
> That???s possible, by passing ???guix archive??? the list of profiles.

Maybe an example for all profiles on the system? But yes, this is
doable.

> > and use case:
> >
> > - copy all files, even those not linked to profiles and up for GC
> 
> Even those not linked?  What for?  :-)

Caching. I know guix does not reload files already in the store.

> > When that works that would be the way forward, but it does not drop my
> > wish for being able to regenerate the database.
> 
> My reading of this is different: we need more/better tools to transfer
> whole subsets of a store among different machines (remember we already
> have ???guix publish??? and ???guix offload??? in that area.)
> 
> Once we have those, I think you may stop worrying about the database.  :-)

True if you ignore my message about the state being in two places and
the database can go out of sync/be corrupt.

Rebuilding the database from a store layout would take many of my
concerns away. What do we have against it?

Pj.

Re: Self-contained Guix tarball

[Ricardo Wurmus]

Pjotr Prins writes:

> On Tue, Apr 21, 2015 at 10:11:29AM +0200, Ludovic Courtès wrote:
>> The important thing is that currently, the DB is authoritative.  So it
>> cannot be corrupt (that would be equivalent to having lost /gnu/store
>> altogether), and thus it cannot be repaired.
>
> The point really is that it is not so hard to achieve rebuilding the
> database. Anyone on this list thinks that should be possible? I am
> asking the big systems guys :)

You could rebuild the database only if the contents of the /gnu/store
directory were authoritative (which they are not).  The database only
contains four tables and none of the entries are magical:

  sqlite> .fullschema
  CREATE TABLE ValidPaths (
      id               integer primary key autoincrement not null,
      path             text unique not null,
      hash             text not null,
      registrationTime integer not null,
      deriver          text,
      narSize          integer
  );
  CREATE TABLE Refs (
      referrer  integer not null,
      reference integer not null,
      primary key (referrer, reference),
      foreign key (referrer) references ValidPaths(id) on delete cascade,
      foreign key (reference) references ValidPaths(id) on delete restrict
  );
  CREATE TABLE DerivationOutputs (
      drv  integer not null,
      id   text not null, -- symbolic output id, usually "out"
      path text not null,
      primary key (drv, id),
      foreign key (drv) references ValidPaths(id) on delete cascade
  );
  CREATE TABLE FailedPaths (
      path text primary key not null,
      time integer not null
  );
  CREATE INDEX IndexReferrer on Refs(referrer);
  CREATE INDEX IndexReference on Refs(reference);
  CREATE INDEX IndexDerivationOutputs on DerivationOutputs(path);
  CREATE TRIGGER DeleteSelfRefs before delete on ValidPaths
    begin
      delete from Refs where referrer = old.id and reference = old.id;
    end;
  /* No STAT tables available */

To create records in Refs you would obviously need to know the
relationship between packages, but you can get that by using Guix
and the package modules as a library.

Someone daring enough to mess with the store contents (by manually
adding, altering or removing stuff) without going through the daemon
would also have to take care of manipulating database records in the
appropriate manner.  I fail to see, however, in what situation this
would be desirable to a system administrator.

Sysadmins don't expect, for example, to be able to delete files from a
system where software is managed with RPM *without* also having to
modify the RPM database to avoid problems.

If I were to sync /gnu/store/ across different machines with rsync, I'd
make sure to keep any additional state by also copying the
localstatedir: /var/guix/.  But as I said before, I fail to see a good
reason to do this.

> It may also allow for people creating other store-based tools in time.

Guix can be used as a library and with the store monad it is possible to
programmatically manipulate the store.  I expect this to become even
easier once the daemon is rewritten in Scheme (which is bound to result
in more reusable parts that could be glued together by adventurous
sysadmins).

~~ Ricardo

Re: Self-contained Guix tarball

[Pjotr Prins]

On Tue, Apr 21, 2015 at 12:08:39PM +0200, Ricardo Wurmus wrote:
> You could rebuild the database only if the contents of the /gnu/store
> directory were authoritative (which they are not).  The database only
> contains four tables and none of the entries are magical:

That is good.

> Someone daring enough to mess with the store contents (by manually
> adding, altering or removing stuff) without going through the daemon
> would also have to take care of manipulating database records in the
> appropriate manner.  I fail to see, however, in what situation this
> would be desirable to a system administrator.
> 
> Sysadmins don't expect, for example, to be able to delete files from a
> system where software is managed with RPM *without* also having to
> modify the RPM database to avoid problems.

You are wrong here. Not wanting to start disputing minor points, but
people do clean out directories and install new software in existing
distributions. Guix gets away from the latter, but not from the first.

> If I were to sync /gnu/store/ across different machines with rsync, I'd
> make sure to keep any additional state by also copying the
> localstatedir: /var/guix/.  But as I said before, I fail to see a good
> reason to do this.

That is a problem. Anyone else see what I mean, at all?

> > It may also allow for people creating other store-based tools in time.
> 
> Guix can be used as a library and with the store monad it is possible to
> programmatically manipulate the store.  I expect this to become even
> easier once the daemon is rewritten in Scheme (which is bound to result
> in more reusable parts that could be glued together by adventurous
> sysadmins).

Sounds good :)

Pj.

Re: Self-contained Guix tarball

[Andreas Enge]

On Tue, Apr 21, 2015 at 12:17:54PM +0200, Pjotr Prins wrote:
> On Tue, Apr 21, 2015 at 12:08:39PM +0200, Ricardo Wurmus wrote:
> > If I were to sync /gnu/store/ across different machines with rsync, I'd
> > make sure to keep any additional state by also copying the
> > localstatedir: /var/guix/.  But as I said before, I fail to see a good
> > reason to do this.
> That is a problem. Anyone else see what I mean, at all?

I wonder whether we should not (at least by default) keep all the additional
state somewhere under /gnu/store instead of /var/guix. Or maybe just the
database part.

Andreas

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> To make guix loved by system deployments you have to cater for quick
> and dirty.

Not sure.  GuixSD is clearly a departure from “traditional Unix.”  I
agree that we have to work to make this acceptable and intelligible to
those used to traditional Unix.

That said, what it brings, notably reproducible system deployments with
‘guix system’, would hopefully be considered a selling point to system
administrators used to the imperative way.

Thanks,
Ludo’.

Re: Self-contained Guix tarball

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> True if you ignore my message about the state being in two places and
> the database can go out of sync/be corrupt.
>
> Rebuilding the database from a store layout would take many of my
> concerns away. What do we have against it?

It’s not about being for or against.  :-)

The important state is in the database.  So if something is to be
rebuilt, it’s the store, not the database.

The database cannot be “guessed” from the set of files found under
/gnu/store because not all of them are valid.

This is the current design, and it’s consistent.  One could come up with
a different design, like Taylan suggested, but we have yet to see what
could be better than this.

Ludo’.