* Copying whole /gnu/store from USB does not work @ 2015-04-10 8:46 Pjotr Prins 2015-04-10 11:48 ` Ludovic Courtès 0 siblings, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-10 8:46 UTC (permalink / raw) To: guix-devel Hi all, A binary install should be easy because all properly installed Guix packages are independent from the underlying distribution - a package can be created which contains a package with all its dependencies. The only real dependency is the (Linux) kernel - which should have a compatible API. Fortunately the fundamental API does not change that often, so a binary distribution can be copied from machine to machine. Of course, also the build target archicture has to match. With a binary distribution it is possible to simply unpack the package, create the build group and permissions. I thought I would be able to bootstrap using a USB image and copy /gnu/store to a local hard disk. To bootstrap I have successfully downloaded a GNU Guix tarball wget http://alpha.gnu.org/gnu/guix/gnu-usb-install-0.8.x86_64-linux.xz xz -d gnu-usb-install-0.8.x86_64-linux.xz mount it with kpartx kpartx -v -a gnu-usb-install-0.8.x86_64-linux mkdir mnt mkdir tmp mount /dev/mapper/loop0p1 mnt cp -vau mnt/gnu tmp mv tmp/gnu/ / and should be able to run Guix with /gnu/store/(...)-guix-0.8(...)/bin/guix --help Actually this now works the first time. There is a problem currently: right after copying the files to /gnu they work find, e.g. /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash --version GNU bash, version 4.3.33(1)-release (x86_64-unknown-linux-gnu) Nice! when I run, however, /gnu/store/wnsbzddp3izv3xr3gqf2rar4fyd4shl7-guix-0.8.1.f1082ec/bin/guix package -i bash it runs and I get The following package will be installed: bash 4.3.33 /gnu/store/7cc229218glj3kzp6zafzs5mc4241wqk-bash-4.3.33 substitute-binary: updating list of substitutes from 'http://hydra.gnu.org'... The following derivations will be built: /gnu/store/gkfzdr01ga0m1c2vl8dh6p3cspjik7pp-profile.drv /gnu/store/kw632ijb16nracg8kbd2z3qqarv72b9c-info-dir.drv The following files will be downloaded: /gnu/store/7cc229218glj3kzp6zafzs5mc4241wqk-bash-4.3.33 /gnu/store/1j6njwwhcqr04c2m0868400p74pv36k2-module-import-compiled /gnu/store/z2mbq8915vvh0fj6sz3iqiifrdm9k7np-guile-2.0.11 (...) substitute error: executing `/gnu/store/wnsbzddp3izv3xr3gqf2rar4fyd4shl7-guix-0.8.1.f1082ec/libexec/guix/substitute-binary': No such file or directory guix package: error: build failed: unexpected EOF reading a line Looking at head -1 /gnu/store/wnsbzddp3izv3xr3gqf2rar4fyd4shl7-guix-0.8.1.f1082ec/libexec/guix/substitute-binary #!/gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash it calls bash and now check /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash bash: /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash: No such file or directory while ldd /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash linux-vdso.so.1 => (0x00007fff739ec000) libreadline.so.6 => /gnu/store/fznw2a1ig05jkc2yrw8knzcb99akcjx7-readline-6.3/lib/libreadline.so.6 (0x00007f0084bd5000) libhistory.so.6 => /gnu/store/fznw2a1ig05jkc2yrw8knzcb99akcjx7-readline-6.3/lib/libhistory.so.6 (0x00007f00849cc000) libncursesw.so.5 => /gnu/store/ksnhghfy6hkk8mmxd4nacyd8c1smdp6p-ncurses-5.9/lib/libncursesw.so.5 (0x00007f008476a000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f008455b000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f008419b000) /gnu/store/q2mm1wv4a3g0b29yv0rjybfjh8kr07qi-glibc-2.20/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0084e1d000) Oooops. Where does libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 come from? Also the other bash stopped working too /gnu/store/nx4zd42igyb7ghmv4mxv6ncg8wr7ypa1-bash-4.3.33/bin/bash --version and ldd shows links outside /gnu (when it worked they are all linked inside /gnu). Also the guix binary stopped working. Same issue. Can anyone explain this? I tried multiple USB images and it is all the same. Note I am not building, only installing binaries. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Copying whole /gnu/store from USB does not work 2015-04-10 8:46 Copying whole /gnu/store from USB does not work Pjotr Prins @ 2015-04-10 11:48 ` Ludovic Courtès 2015-04-10 13:14 ` Pjotr Prins 0 siblings, 1 reply; 31+ messages in thread From: Ludovic Courtès @ 2015-04-10 11:48 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Hi! Pjotr Prins <pjotr.public12@thebird.nl> skribis: > I thought I would be able to bootstrap using a USB image and copy > /gnu/store to a local hard disk. To “bootstrap,” it’s best to install Guix on top of your distro, and build/download packages from there. (Actually bootstrapping doesn’t require any special action with Guix.) > mount it with kpartx > > kpartx -v -a gnu-usb-install-0.8.x86_64-linux > mkdir mnt > mkdir tmp > mount /dev/mapper/loop0p1 mnt > cp -vau mnt/gnu tmp > mv tmp/gnu/ / That cannot work. The reason is that the store is really two things: /gnu/store, and /var/guix/db/db.sqlite. The latter contains meta-data about the former, notably the list of valid store items. The valid way to do that is with ‘guix archive’ (info "(guix) Invoking guix archive"), or using the ‘populate-store’ and ‘register-closure’ as is done in (gnu build vm). HTH, Ludo’. PS: note that the latest version is 0.8.1, not 0.8. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Copying whole /gnu/store from USB does not work 2015-04-10 11:48 ` Ludovic Courtès @ 2015-04-10 13:14 ` Pjotr Prins 2015-04-10 13:41 ` David Thompson ` (2 more replies) 0 siblings, 3 replies; 31+ messages in thread From: Pjotr Prins @ 2015-04-10 13:14 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Hi Ludo, On Fri, Apr 10, 2015 at 01:48:34PM +0200, Ludovic Courtès wrote: > Hi! > > Pjotr Prins <pjotr.public12@thebird.nl> skribis: > > > I thought I would be able to bootstrap using a USB image and copy > > /gnu/store to a local hard disk. > > To ???bootstrap,??? it???s best to install Guix on top of your distro, and > build/download packages from there. (Actually bootstrapping doesn???t > require any special action with Guix.) Well yes, you need to install all prerequisites and that is not always trivial. Try bootstrapping on an old Centos install, for example. Can't even get Docker going there. It would be nice to have an installable binary tar ball for Guix. Just unpack in root and go... > > mount it with kpartx > > > > kpartx -v -a gnu-usb-install-0.8.x86_64-linux > > mkdir mnt > > mkdir tmp > > mount /dev/mapper/loop0p1 mnt > > cp -vau mnt/gnu tmp > > mv tmp/gnu/ / > > That cannot work. I am merely trying to be clever ;) > The reason is that the store is really two things: /gnu/store, and > /var/guix/db/db.sqlite. The latter contains meta-data about the former, > notably the list of valid store items. Aha. That makes sense. If I also copy the db.sqlite it might just work then. > The valid way to do that is with ???guix archive??? (info "(guix) Invoking > guix archive"), or using the ???populate-store??? and ???register-closure??? as > is done in (gnu build vm). OK, I'll take a look. Thanks. Pj. PS I am rather excited that you have LLVM going and Python packages and that Ricardo is adding bioinformatics packages. Way to go!! Now we need to add Ruby gems and GNU Guix will be the default deployment system for many. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Copying whole /gnu/store from USB does not work 2015-04-10 13:14 ` Pjotr Prins @ 2015-04-10 13:41 ` David Thompson 2015-04-12 10:28 ` Copying whole /gnu/store from USB does not work (SOLVED) Pjotr Prins 2015-04-12 20:07 ` Self-contained Guix tarball Ludovic Courtès 2 siblings, 0 replies; 31+ messages in thread From: David Thompson @ 2015-04-10 13:41 UTC (permalink / raw) To: Pjotr Prins, Ludovic Courtès; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> writes: > Now we need to add Ruby gems and GNU Guix will be the default > deployment system for many. Help wanted. ;) Two major problems to deal with: 1) Propagated inputs: Dynamic languages typically don't have an equivalent to RUNPATH for ELF binaries. Thus, for Python (and others), we have to propagate all of the python libraries along with each package so that they can all be found in a particular load path. This in unideal. I haven't found a way to hardcode the location of libraries as part of the build process. If it can't be reasonably solved, we'll just propagate the inputs like every other dynamic language. Do you know a way that we could avoid this for Ruby gems? 2) Circular dependencies: Rspec is a particularly important Ruby gem to package. Pretty much every Ruby project uses it for their test suite. However, a lot of Rspec's dependencies use Rspec! The only thing I've come up with is to produce a set of private packages for all of Rspec's dependencies whose build processes skip the test suite. Then, with Rspec packaged, create public versions of the packages with Rspec as a native input and the test suite enabled. The dependency tree isn't exactly shallow, though. The Rspec maintainer failed to understand this issue, because the system in place at rubygems.org doesn't run test suites or build the gems at all. People just upload the final product and no one notices that there's nasty circular dependencies. Thoughts? Nice to see you poking around with Guix again, Pjotr. -- David Thompson Web Developer - Free Software Foundation - http://fsf.org GPG Key: 0FF1D807 Support the FSF: https://fsf.org/donate ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Copying whole /gnu/store from USB does not work (SOLVED) 2015-04-10 13:14 ` Pjotr Prins 2015-04-10 13:41 ` David Thompson @ 2015-04-12 10:28 ` Pjotr Prins 2015-04-12 12:29 ` Ludovic Courtès 2015-04-12 20:07 ` Self-contained Guix tarball Ludovic Courtès 2 siblings, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-12 10:28 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel > > The reason is that the store is really two things: /gnu/store, and > > /var/guix/db/db.sqlite. The latter contains meta-data about the former, > > notably the list of valid store items. > > Aha. That makes sense. If I also copy the db.sqlite it might just > work then. It works rather well and is easy. This means we can just bootstrap from an existing /gnu tree in thu USB distribution provided the kernel/arch is compatible. At build time 'guix package -i guix' complained it did not have enough builders. The guix-daemon did not honour the -M 8 switch, so I had to add 30 build users on a 32 core machine. Now it is happy. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Copying whole /gnu/store from USB does not work (SOLVED) 2015-04-12 10:28 ` Copying whole /gnu/store from USB does not work (SOLVED) Pjotr Prins @ 2015-04-12 12:29 ` Ludovic Courtès 0 siblings, 0 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-12 12:29 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > At build time 'guix package -i guix' complained it did not have enough > builders. The guix-daemon did not honour the -M 8 switch, so I had to > add 30 build users on a 32 core machine. Now it is happy. Could it be that you’re using 0.8 instead of 0.8.1? The latter fixed a bug in that area, whereby -M would default to the number of cores. Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Self-contained Guix tarball 2015-04-10 13:14 ` Pjotr Prins 2015-04-10 13:41 ` David Thompson 2015-04-12 10:28 ` Copying whole /gnu/store from USB does not work (SOLVED) Pjotr Prins @ 2015-04-12 20:07 ` Ludovic Courtès 2015-04-13 6:54 ` Pjotr Prins ` (2 more replies) 2 siblings, 3 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-12 20:07 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel [-- Attachment #1.1: Type: text/plain, Size: 1325 bytes --] Pjotr Prins <pjotr.public12@thebird.nl> skribis: > Well yes, you need to install all prerequisites and that is not always > trivial. Try bootstrapping on an old Centos install, for example. > Can't even get Docker going there. > > It would be nice to have an installable binary tar ball for Guix. Just > unpack in root and go... I’ve given that a try since it’s actually quite simple to do. The procedure in the attached file produces a self-contained tarball. It contains the closure of Guix, as well as /var/guix and an initial /root/.guix-profile from which one can use the ‘guix’ commands. I’ve uploaded the result (for x86_64) at: http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz.sig SHA1: afde62b3e9f97f9ae0c57eaccfaca842b157cd2f To install Guix on a foreign distro, follow these steps: 1. Download the tarball (38 MiB.) 2. As root, run: cd / tar xf /path/to/guix-tarball-0.8.1.1140.tar.lz You need lzip and GNU tar. 3. Setup the daemon as explained in the manual, and run it from /root/.guix-profile/bin/guix-daemon. And hopefully, that’s it. I haven’t actually tried it, so feedback is welcome! :-) Thanks, Ludo’. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1.2: tarball creation --] [-- Type: text/x-scheme, Size: 3477 bytes --] (use-modules (guix) (guix profiles) (gnu packages base) (gnu packages compression) (gnu packages package-management)) (define* (self-contained-tarball #:key (guix guix)) (mlet %store-monad ((profile (profile-derivation (manifest (list (package->manifest-entry guix)))))) (gexp->derivation "guix-tarball.tar.lz" #~(begin (use-modules (guix build store-copy) (guix build utils) (gnu build install) (srfi srfi-26)) (define %root (string-append (getcwd) "/root")) (define %root-profile "/var/guix/profiles/per-user/root") (define (mkdir-p* dir) (mkdir-p (string-append %root "/" dir))) (define (symlink* old new) (symlink old (string-append %root "/" new))) (setenv "PATH" (string-append #$guix "/sbin:" #$tar "/bin:" #$lzip "/bin")) ;; Populate the store. (populate-store '("profile") %root) (for-each (cut register-closure %root <>) '("profile")) ;; 'guix-register' registers profiles as GC roots ;; but the symlink target uses $TMPDIR. Fix that. (delete-file (string-append %root "/var/guix/gcroots/profiles")) (symlink* "/var/guix/profiles" "/var/guix/gcroots/profiles") ;; Make root's profile, which makes it a GC root. (mkdir-p* %root-profile) (symlink* #$profile (string-append %root-profile "/guix-profile-1-link")) (symlink* (string-append %root-profile "/guix-profile-1-link") (string-append %root-profile "/guix-profile")) (mkdir-p* "/root") (symlink* (string-append %root-profile "/guix-profile") "/root/.guix-profile") ;; Create the tarball. Use GNU format so there's no ;; file name length limitation. (chdir %root) (zero? (system* "tar" "--lzip" "--format=gnu" "-cvf" #$output "."))) #:references-graphs `(("profile" ,profile)) #:modules '((guix build utils) (guix build store-copy) (gnu build install))))) ;; (pk (with-store store ;; (run-with-store store (self-contained-tarball)))) [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 818 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-12 20:07 ` Self-contained Guix tarball Ludovic Courtès @ 2015-04-13 6:54 ` Pjotr Prins 2015-04-13 9:57 ` Ludovic Courtès 2015-04-13 15:47 ` Thompson, David 2015-04-15 21:31 ` Ludovic Courtès 2 siblings, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-13 6:54 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel The tar-ball works a treat!! Simply download and tar -C / --lzip -xvf guix-tarball-0.8.1.1140.tar.lz export ~/.guix-profile/bin:$PATH Add the build users, update the /gnu/store permissions and start the daemon and that is it (which could be covered by a simple shell script). This should work on ANY Linux, provided kernel and architecture are compatible. Very nice. GNU Guix out of a box. No more installation pain. The tar-ball should go in the standard download directory. I would name the tar ball with something 'binary', maybe add the supported Linux versions (3.x?) and the architecture in the name. Just to be complete. There may be other editions later. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-13 6:54 ` Pjotr Prins @ 2015-04-13 9:57 ` Ludovic Courtès 2015-04-13 10:05 ` Pjotr Prins 2015-04-13 13:29 ` Mark H Weaver 0 siblings, 2 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-13 9:57 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > The tar-ball works a treat!! Simply download and > > tar -C / --lzip -xvf guix-tarball-0.8.1.1140.tar.lz > export ~/.guix-profile/bin:$PATH > > Add the build users, update the /gnu/store permissions Normally guix-daemon adjusts the permissions/ownership of /gnu/store automatically nowadays, so that part shouldn’t even be needed. > Very nice. GNU Guix out of a box. No more installation pain. > > The tar-ball should go in the standard download directory. I would > name the tar ball with something 'binary', maybe add the supported > Linux versions (3.x?) and the architecture in the name. Just to be > complete. There may be other editions later. Yes, something like guix-binary-0.8.2.x86_64.tar.gz? I guess we’ll publish it for the next release. Thanks for your feedback! Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-13 9:57 ` Ludovic Courtès @ 2015-04-13 10:05 ` Pjotr Prins 2015-04-13 13:29 ` Mark H Weaver 1 sibling, 0 replies; 31+ messages in thread From: Pjotr Prins @ 2015-04-13 10:05 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel On Mon, Apr 13, 2015 at 11:57:20AM +0200, Ludovic Courtès wrote: > Yes, something like guix-binary-0.8.2.x86_64.tar.gz? > > I guess we???ll publish it for the next release. > > Thanks for your feedback! So far, I have installed the tarball on 5 machines. And counting... It is beautiful. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-13 9:57 ` Ludovic Courtès 2015-04-13 10:05 ` Pjotr Prins @ 2015-04-13 13:29 ` Mark H Weaver 2015-04-14 21:33 ` Ludovic Courtès 1 sibling, 1 reply; 31+ messages in thread From: Mark H Weaver @ 2015-04-13 13:29 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel ludo@gnu.org (Ludovic Courtès) writes: > Pjotr Prins <pjotr.public12@thebird.nl> skribis: > >> Very nice. GNU Guix out of a box. No more installation pain. >> >> The tar-ball should go in the standard download directory. I would >> name the tar ball with something 'binary', maybe add the supported >> Linux versions (3.x?) and the architecture in the name. Just to be >> complete. There may be other editions later. > > Yes, something like guix-binary-0.8.2.x86_64.tar.gz? > > I guess we’ll publish it for the next release. Thank you very much for this! Could we add a package to Guix to build this? If that is undesirable for some reason, could we add a command to build it, and jobs on Hydra to build it automatically on all platforms supported by the build farm, analogous to the 'usb-image' jobs? Mark ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-13 13:29 ` Mark H Weaver @ 2015-04-14 21:33 ` Ludovic Courtès 0 siblings, 0 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-14 21:33 UTC (permalink / raw) To: Mark H Weaver; +Cc: guix-devel Mark H Weaver <mhw@netris.org> skribis: > Could we add a package to Guix to build this? If that is undesirable > for some reason, could we add a command to build it, and jobs on Hydra > to build it automatically on all platforms supported by the build farm, > analogous to the 'usb-image' jobs? Commit b607593 adds Hydra jobs, and commit 9d3fb6c adds the code and a Makefile target: make guix-binary.x86_64-linux.tar.xz or similar builds the binary tarball. We’ll have to figure out how to handle that for all the systems, for the release. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-12 20:07 ` Self-contained Guix tarball Ludovic Courtès 2015-04-13 6:54 ` Pjotr Prins @ 2015-04-13 15:47 ` Thompson, David 2015-04-15 21:31 ` Ludovic Courtès 2 siblings, 0 replies; 31+ messages in thread From: Thompson, David @ 2015-04-13 15:47 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel On Sun, Apr 12, 2015 at 4:07 PM, Ludovic Courtès <ludo@gnu.org> wrote: > > I’ve given that a try since it’s actually quite simple to do. The > procedure in the attached file produces a self-contained tarball. It > contains the closure of Guix, as well as /var/guix and an initial > /root/.guix-profile from which one can use the ‘guix’ commands. > > I’ve uploaded the result (for x86_64) at: > > http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz > http://www.fdn.fr/~lcourtes/software/guix/guix-tarball-0.8.1.1140.tar.lz.sig > SHA1: afde62b3e9f97f9ae0c57eaccfaca842b157cd2f > > To install Guix on a foreign distro, follow these steps: > > 1. Download the tarball (38 MiB.) > > 2. As root, run: > cd / > tar xf /path/to/guix-tarball-0.8.1.1140.tar.lz > You need lzip and GNU tar. > > 3. Setup the daemon as explained in the manual, and run it from > /root/.guix-profile/bin/guix-daemon. > > And hopefully, that’s it. > > I haven’t actually tried it, so feedback is welcome! :-) It works perfectly! - Dave ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-12 20:07 ` Self-contained Guix tarball Ludovic Courtès 2015-04-13 6:54 ` Pjotr Prins 2015-04-13 15:47 ` Thompson, David @ 2015-04-15 21:31 ` Ludovic Courtès 2015-04-16 5:33 ` Pjotr Prins 2 siblings, 1 reply; 31+ messages in thread From: Ludovic Courtès @ 2015-04-15 21:31 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel I’ve documented installation from the binary tarball (which I plan to distribute for the next release) in commit 09722b1. Please let me know what you think. Ludo’. 2.1 Binary Installation ======================= This section describes how to install Guix on an arbitrary system from a self-contained tarball providing binaries for Guix and for all its dependencies. This is often quicker than installing from source, which is described in the next sections. The only requirement is to have GNU tar and Xz. Installing goes along these lines: 1. Download the binary tarball from ‘ftp://alpha.gnu.org/gnu/guix/guix-binary-0.8.2.SYSTEM.tar.xz’(1), where SYSTEM is ‘x86_64-linux’ for an ‘x86_64’ machine already running the kernel Linux, and so on. 2. As ‘root’, run: # cd / # tar xf guix-binary-0.8.2.SYSTEM.tar.xz This creates ‘/gnu/store’ (*note The Store::), ‘/var/guix’, and ‘/root/.guix-profile’. ‘/root/.guix-profile’ is a ready-to-use profile for ‘root’ where Guix is installed. 3. Setup the daemon as explained below (*note Setting Up the Daemon::), and run it: # /root/.guix-profile/bin/guix-daemon --build-users-group=guix-builder 4. Make the ‘guix’ command available to other users on the machine, for instance with: # mkdir -p /usr/local/bin # cd /usr/local/bin # ln -s /root/.guix-profile/bin/guix And that’s it! There are two things to note. First, don’t unpack the tarball on a working Guix system since that would overwrite its own essential files. Second, the ‘guix’ package must remain available in ‘root’’s profile, or it would become subject to garbage collection—in which case you would find yourself badly handicapped by the lack of the ‘guix’ command. The tarball in question can be (re)produced simply by running the following command in the Guix source tree: make guix-binary.SYSTEM.tar.xz ---------- Footnotes ---------- (1) As usual, make sure to download the associated ‘.sig’ file and to verify the authenticity of the tarball against it! ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-15 21:31 ` Ludovic Courtès @ 2015-04-16 5:33 ` Pjotr Prins 2015-04-18 21:23 ` Ludovic Courtès 0 siblings, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-16 5:33 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Great :). I would make it a little clearer that this is 'bootstrapping' and hype it a little more that now there is no reason NOT to install Guix. Only 100Mb on your HDD. The warning about overwriting a Guix installation should be earlier. Maybe point out that if you want to move a package with dependencies see another part of the documentation. BTW when Nix decided to go for a meta-database they lost something. I know it has good reasons (performance mostly) but it took away the self-containedness of packages. It would be nice just to be able to copy/del packages and rebuild the meta information. Do we have something like that? Pj. On Wed, Apr 15, 2015 at 11:31:09PM +0200, Ludovic Courtès wrote: > I???ve documented installation from the binary tarball (which I plan to > distribute for the next release) in commit 09722b1. Please let me know > what you think. > > Ludo???. > > > 2.1 Binary Installation > ======================= > > This section describes how to install Guix on an arbitrary system from a > self-contained tarball providing binaries for Guix and for all its > dependencies. This is often quicker than installing from source, which > is described in the next sections. The only requirement is to have > GNU tar and Xz. > > Installing goes along these lines: > > 1. Download the binary tarball from > ???ftp://alpha.gnu.org/gnu/guix/guix-binary-0.8.2.SYSTEM.tar.xz???(1), > where SYSTEM is ???x86_64-linux??? for an ???x86_64??? machine already > running the kernel Linux, and so on. > > 2. As ???root???, run: > > # cd / > # tar xf guix-binary-0.8.2.SYSTEM.tar.xz > > This creates ???/gnu/store??? (*note The Store::), ???/var/guix???, and > ???/root/.guix-profile???. ???/root/.guix-profile??? is a ready-to-use > profile for ???root??? where Guix is installed. > > 3. Setup the daemon as explained below (*note Setting Up the > Daemon::), and run it: > > # /root/.guix-profile/bin/guix-daemon --build-users-group=guix-builder > > 4. Make the ???guix??? command available to other users on the machine, > for instance with: > > # mkdir -p /usr/local/bin > # cd /usr/local/bin > # ln -s /root/.guix-profile/bin/guix > > And that???s it! > > There are two things to note. First, don???t unpack the tarball on a > working Guix system since that would overwrite its own essential files. > Second, the ???guix??? package must remain available in ???root??????s profile, or > it would become subject to garbage collection???in which case you would > find yourself badly handicapped by the lack of the ???guix??? command. > > The tarball in question can be (re)produced simply by running the > following command in the Guix source tree: > > make guix-binary.SYSTEM.tar.xz > > ---------- Footnotes ---------- > > (1) As usual, make sure to download the associated ???.sig??? file and to > verify the authenticity of the tarball against it! > -- ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-16 5:33 ` Pjotr Prins @ 2015-04-18 21:23 ` Ludovic Courtès 2015-04-19 8:18 ` Pjotr Prins 2015-04-19 13:34 ` Taylan Ulrich Bayırlı/Kammer 0 siblings, 2 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-18 21:23 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > Great :). I would make it a little clearer that this is > 'bootstrapping' and hype it a little more that now there is no reason > NOT to install Guix. Only 100Mb on your HDD. Not sure how to do that, would you like to propose actual text? The thing is, I want it to remain accurate and factual. > The warning about overwriting a Guix installation should be earlier. Done in 5dc3ce5. > Maybe point out that if you want to move a package with dependencies > see another part of the documentation. What do you meaning by moving a package with dependencies? > BTW when Nix decided to go for a meta-database they lost something. I > know it has good reasons (performance mostly) but it took away the > self-containedness of packages. It would be nice just to be able to > copy/del packages and rebuild the meta information. Do we have > something like that? This part is the same as Nix. The database is here to store meta-data about store items, notably the list of references found in a store item. Determining this list requires scanning all of the store item’s contents, which takes time proportional to the number/size of files it contains, so the database can hardly be avoided. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-18 21:23 ` Ludovic Courtès @ 2015-04-19 8:18 ` Pjotr Prins 2015-04-19 20:09 ` Ludovic Courtès 2015-04-19 13:34 ` Taylan Ulrich Bayırlı/Kammer 1 sibling, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-19 8:18 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel On Sat, Apr 18, 2015 at 11:23:50PM +0200, Ludovic Courtès wrote: > Pjotr Prins <pjotr.public12@thebird.nl> skribis: > > > Great :). I would make it a little clearer that this is > > 'bootstrapping' and hype it a little more that now there is no reason > > NOT to install Guix. Only 100Mb on your HDD. > > Not sure how to do that, would you like to propose actual text? > The thing is, I want it to remain accurate and factual. The current text is fine. > What do you meaning by moving a package with dependencies? I am thinking about Nix-style closures. But it may only confuse things. I don't think the Guix manual covers closures. > > BTW when Nix decided to go for a meta-database they lost something. I > > know it has good reasons (performance mostly) but it took away the > > self-containedness of packages. It would be nice just to be able to > > copy/del packages and rebuild the meta information. Do we have > > something like that? > > This part is the same as Nix. The database is here to store meta-data > about store items, notably the list of references found in a store item. > Determining this list requires scanning all of the store item???s > contents, which takes time proportional to the number/size of files it > contains, so the database can hardly be avoided. Yes, I understand. But would it be possible to regenerate the database from an existing /gnu/store? You can see I like to mess around with files ;). With closures a rebuild should not be necessary, but as a wary system administrator I know I will need it at some point. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-19 8:18 ` Pjotr Prins @ 2015-04-19 20:09 ` Ludovic Courtès 0 siblings, 0 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-19 20:09 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > On Sat, Apr 18, 2015 at 11:23:50PM +0200, Ludovic Courtès wrote: [...] >> What do you meaning by moving a package with dependencies? > > I am thinking about Nix-style closures. But it may only confuse > things. I don't think the Guix manual covers closures. I think “Invoking guix archive” covers what you have in mind. Maybe it should be made more visible somehow? https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-archive.html >> This part is the same as Nix. The database is here to store meta-data >> about store items, notably the list of references found in a store item. >> Determining this list requires scanning all of the store item???s >> contents, which takes time proportional to the number/size of files it >> contains, so the database can hardly be avoided. > > Yes, I understand. But would it be possible to regenerate the database > from an existing /gnu/store? In theory, yes. > You can see I like to mess around with files ;). With closures a > rebuild should not be necessary, but as a wary system administrator I > know I will need it at some point. What do you mean by “rebuild should not be necessary”? If you remove files from the store manually? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-18 21:23 ` Ludovic Courtès 2015-04-19 8:18 ` Pjotr Prins @ 2015-04-19 13:34 ` Taylan Ulrich Bayırlı/Kammer 2015-04-20 20:49 ` Ludovic Courtès 1 sibling, 1 reply; 31+ messages in thread From: Taylan Ulrich Bayırlı/Kammer @ 2015-04-19 13:34 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel ludo@gnu.org (Ludovic Courtès) writes: > Pjotr Prins <pjotr.public12@thebird.nl> skribis: > >> BTW when Nix decided to go for a meta-database they lost something. I >> know it has good reasons (performance mostly) but it took away the >> self-containedness of packages. It would be nice just to be able to >> copy/del packages and rebuild the meta information. Do we have >> something like that? > > This part is the same as Nix. The database is here to store meta-data > about store items, notably the list of references found in a store item. > Determining this list requires scanning all of the store item’s > contents, which takes time proportional to the number/size of files it > contains, so the database can hardly be avoided. Just throwing this out there: Would it be possible to bundle meta-data files together with store items? I guess one could in principle make each store item a directory with any number of meta-data files plus the actual content of the store item (a file or directory), but I suppose it would be too big an overhaul of the design to make this worthwhile. Taylan ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-19 13:34 ` Taylan Ulrich Bayırlı/Kammer @ 2015-04-20 20:49 ` Ludovic Courtès 2015-04-21 7:03 ` Pjotr Prins 0 siblings, 1 reply; 31+ messages in thread From: Ludovic Courtès @ 2015-04-20 20:49 UTC (permalink / raw) To: Taylan Ulrich "Bayırlı/Kammer"; +Cc: guix-devel taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis: > Would it be possible to bundle meta-data files together with store > items? > > I guess one could in principle make each store item a directory with any > number of meta-data files plus the actual content of the store item (a > file or directory), but I suppose it would be too big an overhaul of the > design to make this worthwhile. I suspect this would make GC inefficient (lots of disk seeks to determine references/referrers compared to queries of the SQLite database.) Another (opposite :-)) option is to make /gnu/store a read-only bind mount on GuixSD. Commit 3392ce5 does that. This will prevent accidental modifications of the store. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-20 20:49 ` Ludovic Courtès @ 2015-04-21 7:03 ` Pjotr Prins 2015-04-21 7:19 ` Pjotr Prins 2015-04-21 8:11 ` Ludovic Courtès 0 siblings, 2 replies; 31+ messages in thread From: Pjotr Prins @ 2015-04-21 7:03 UTC (permalink / raw) To: guix-devel > What do you mean by ???rebuild should not be necessary???? If you > remove files from the store manually? When you administrate a large amount of servers things can go wrong due to system failures, failed backup recoveries, hacking attempts and adminstrators trying to be clever ;). Murphy's law dictates that the store and the sqlitedb meta information will go out of sync. For production setups it is necessary to be able to recover from backups, but as an intermediate recovery step it would be really nice if Guix could recover its meta information from an existing store - assuming only the DB is corrupt (google for sqlite corrupted databases). It is especially nice when your backups are out of sync too. guix archive looks good, but for speedy deployment it can happen an adminstrator would have a simple use case of: - Copying 2 stores to 1 machine - Rebuild database to do it quick and dirty. Since all software packages are isolated this would make a really good use case even if (with some trouble) you could use guix archive for that. I prefer quick and dirty. I am not pushing for this functionality directly, but I would certainly like to have it when I need it :) Maybe I am being pathetic (a lot of people won't appreciate the problem), but under time pressure which huge deployments - this is what is going to be wanted. On Mon, Apr 20, 2015 at 10:49:05PM +0200, Ludovic Courtès wrote: > taylanbayirli@gmail.com (Taylan Ulrich "Bay??rl??/Kammer") skribis: > > > Would it be possible to bundle meta-data files together with store > > items? > > > > I guess one could in principle make each store item a directory with any > > number of meta-data files plus the actual content of the store item (a > > file or directory), but I suppose it would be too big an overhaul of the > > design to make this worthwhile. > > I suspect this would make GC inefficient (lots of disk seeks to > determine references/referrers compared to queries of the SQLite > database.) Yes, Nix switched to using sqlitedb because of the GC. It is also useful to search current versions of installed packages quickly. Even so, I think it should be viewed as an index. The state of the machine is what is *sitting* in the store. That would be the correct design. Meta information can go out of sync. Therefore we should not assume they are in sync. And it is simple to achieve. Just allow for database rebuilds. > Another (opposite :-)) option is to make /gnu/store a read-only bind > mount on GuixSD. Commit 3392ce5 does that. This will prevent > accidental modifications of the store. That is a good solution for end-users. Not for administrators. So adminstrators will circumvent it. Remember Guix is a hackable system. That is true for the store too. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 7:03 ` Pjotr Prins @ 2015-04-21 7:19 ` Pjotr Prins 2015-04-21 8:18 ` Ludovic Courtès 2015-04-21 8:11 ` Ludovic Courtès 1 sibling, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-21 7:19 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel guix archive copies via ssh it appears. Would it be an idea to use rsync instead with the --delete switch. That way a copy is efficient and faithful. You really want the same package quaranteed. I trust guix, but I don't trust systems. You give an example of copying profiles. How about copying a store with and without profile linked packages? That is use case: - copy all current packages in use on the server (in profiles) and use case: - copy all files, even those not linked to profiles and up for GC When that works that would be the way forward, but it does not drop my wish for being able to regenerate the database. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 7:19 ` Pjotr Prins @ 2015-04-21 8:18 ` Ludovic Courtès 2015-04-21 8:44 ` Pjotr Prins 0 siblings, 1 reply; 31+ messages in thread From: Ludovic Courtès @ 2015-04-21 8:18 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > guix archive copies via ssh it appears. No, it does nothing remotely. The manual has an example showing how to use it over SSH, but it’s just an example... > Would it be an idea to use rsync instead with the --delete > switch. ... and it has a discussion of bandwidth efficiency and --missing. :-) > You give an example of copying profiles. How about copying a store > with and without profile linked packages? That is use case: > > - copy all current packages in use on the server (in profiles) That’s possible, by passing ‘guix archive’ the list of profiles. > and use case: > > - copy all files, even those not linked to profiles and up for GC Even those not linked? What for? :-) > When that works that would be the way forward, but it does not drop my > wish for being able to regenerate the database. My reading of this is different: we need more/better tools to transfer whole subsets of a store among different machines (remember we already have ‘guix publish’ and ‘guix offload’ in that area.) Once we have those, I think you may stop worrying about the database. :-) WDYT? Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 8:18 ` Ludovic Courtès @ 2015-04-21 8:44 ` Pjotr Prins 2015-04-21 12:16 ` Ludovic Courtès 0 siblings, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-21 8:44 UTC (permalink / raw) To: guix-devel On Tue, Apr 21, 2015 at 10:18:51AM +0200, Ludovic Courtès wrote: > No, it does nothing remotely. The manual has an example showing how to > use it over SSH, but it???s just an example... > > > Would it be an idea to use rsync instead with the --delete > > switch. > > ... and it has a discussion of bandwidth efficiency and --missing. > :-) Ah, OK. > > You give an example of copying profiles. How about copying a store > > with and without profile linked packages? That is use case: > > > > - copy all current packages in use on the server (in profiles) > > That???s possible, by passing ???guix archive??? the list of profiles. Maybe an example for all profiles on the system? But yes, this is doable. > > and use case: > > > > - copy all files, even those not linked to profiles and up for GC > > Even those not linked? What for? :-) Caching. I know guix does not reload files already in the store. > > When that works that would be the way forward, but it does not drop my > > wish for being able to regenerate the database. > > My reading of this is different: we need more/better tools to transfer > whole subsets of a store among different machines (remember we already > have ???guix publish??? and ???guix offload??? in that area.) > > Once we have those, I think you may stop worrying about the database. :-) True if you ignore my message about the state being in two places and the database can go out of sync/be corrupt. Rebuilding the database from a store layout would take many of my concerns away. What do we have against it? Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 8:44 ` Pjotr Prins @ 2015-04-21 12:16 ` Ludovic Courtès 0 siblings, 0 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-21 12:16 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > True if you ignore my message about the state being in two places and > the database can go out of sync/be corrupt. > > Rebuilding the database from a store layout would take many of my > concerns away. What do we have against it? It’s not about being for or against. :-) The important state is in the database. So if something is to be rebuilt, it’s the store, not the database. The database cannot be “guessed” from the set of files found under /gnu/store because not all of them are valid. This is the current design, and it’s consistent. One could come up with a different design, like Taylan suggested, but we have yet to see what could be better than this. Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 7:03 ` Pjotr Prins 2015-04-21 7:19 ` Pjotr Prins @ 2015-04-21 8:11 ` Ludovic Courtès 2015-04-21 8:37 ` Pjotr Prins 1 sibling, 1 reply; 31+ messages in thread From: Ludovic Courtès @ 2015-04-21 8:11 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > When you administrate a large amount of servers things can go wrong > due to system failures, failed backup recoveries, hacking attempts and > adminstrators trying to be clever ;). Murphy's law dictates that the > store and the sqlitedb meta information will go out of sync. For > production setups it is necessary to be able to recover from backups, > but as an intermediate recovery step it would be really nice if Guix > could recover its meta information from an existing store - assuming > only the DB is corrupt (google for sqlite corrupted databases). It is > especially nice when your backups are out of sync too. The important thing is that currently, the DB is authoritative. So it cannot be corrupt (that would be equivalent to having lost /gnu/store altogether), and thus it cannot be repaired. What *can* be repaired is the store: for instance, if a store item is tampered with. The daemon has code to do it, but the Guix client tools do not expose it yet. > guix archive looks good, but for speedy deployment it can happen an > adminstrator would have a simple use case of: > > - Copying 2 stores to 1 machine > - Rebuild database > > to do it quick and dirty. Since all software packages are isolated > this would make a really good use case even if (with some trouble) you > could use guix archive for that. I prefer quick and dirty. > > I am not pushing for this functionality directly, but I would > certainly like to have it when I need it :) I don’t think it could work the way you envision it. What kind of deployment do you have in mind? For whole system deployment, one can obviously use ‘guix system’. >> I suspect this would make GC inefficient (lots of disk seeks to >> determine references/referrers compared to queries of the SQLite >> database.) > > Yes, Nix switched to using sqlitedb because of the GC. I think it’s been there “forever” (at least since I started contributing in 2008.) > It is also useful to search current versions of installed packages > quickly. Even so, I think it should be viewed as an index. The state > of the machine is what is *sitting* in the store. That would be the > correct design. > > Meta information can go out of sync. Therefore we should not assume > they are in sync. Again, the store can go “out of sync,” but the DB itself is authoritative currently. And it’s important that it be this way. One example is that build processes can write their outputs to the store; if the build process fails, there are still those files in the store, but the DB won’t have recorded them as valid, so they can be swept on the next GC. >> Another (opposite :-)) option is to make /gnu/store a read-only bind >> mount on GuixSD. Commit 3392ce5 does that. This will prevent >> accidental modifications of the store. > > That is a good solution for end-users. Not for administrators. So > adminstrators will circumvent it. Well, administrators won’t be able to circumvent it accidentally, at least. I understand this framework really constrains what sysadmins can do, and in particular prevents them from doing “quick-and-dirty hacks.” I think we should strive to find the UIs that allow for quick hacks while not compromising the store’s integrity. WDYT? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 8:11 ` Ludovic Courtès @ 2015-04-21 8:37 ` Pjotr Prins 2015-04-21 10:08 ` Ricardo Wurmus 2015-04-21 12:12 ` Ludovic Courtès 0 siblings, 2 replies; 31+ messages in thread From: Pjotr Prins @ 2015-04-21 8:37 UTC (permalink / raw) To: guix-devel On Tue, Apr 21, 2015 at 10:11:29AM +0200, Ludovic Courtès wrote: > The important thing is that currently, the DB is authoritative. So it > cannot be corrupt (that would be equivalent to having lost /gnu/store > altogether), and thus it cannot be repaired. The point really is that it is not so hard to achieve rebuilding the database. Anyone on this list thinks that should be possible? I am asking the big systems guys :) > What *can* be repaired is the store: for instance, if a store item is > tampered with. The daemon has code to do it, but the Guix client tools > do not expose it yet. There are other tools that do that. It would be a nice feature, even so. > I don???t think it could work the way you envision it. What kind of > deployment do you have in mind? For whole system deployment, one can > obviously use ???guix system???. It can work if we rebuild the database from a store. > >> I suspect this would make GC inefficient (lots of disk seeks to > >> determine references/referrers compared to queries of the SQLite > >> database.) > > > > Yes, Nix switched to using sqlitedb because of the GC. > > I think it???s been there ???forever??? (at least since I started contributing > in 2008.) I remember the switch to sqlite and questioned it in my mind at the time. > > It is also useful to search current versions of installed packages > > quickly. Even so, I think it should be viewed as an index. The state > > of the machine is what is *sitting* in the store. That would be the > > correct design. > > > > Meta information can go out of sync. Therefore we should not assume > > they are in sync. > > Again, the store can go ???out of sync,??? but the DB itself is > authoritative currently. Yes. It should be. But it would be good to rebuild from an existing store consisting of packages. All I am saying :) > >> Another (opposite :-)) option is to make /gnu/store a read-only bind > >> mount on GuixSD. Commit 3392ce5 does that. This will prevent > >> accidental modifications of the store. > > > > That is a good solution for end-users. Not for administrators. So > > adminstrators will circumvent it. > > Well, administrators won???t be able to circumvent it accidentally, at > least. True. > I understand this framework really constrains what sysadmins can do, and > in particular prevents them from doing ???quick-and-dirty hacks.??? I think > we should strive to find the UIs that allow for quick hacks while not > compromising the store???s integrity. > > WDYT? The Unix way is transparency and dirty hacks. That is why systems guys and gurus generally don't like systemd, for example, even if it has great features. I think that is a nice comparison. Another one would be the Microsoft settings management infrastructure. To make guix loved by system deployments you have to cater for quick and dirty. If everything has to be handled by an opaque daemon - even if it is FOSS - they are not going to like it enough. I am speaking for myself here (of course), but I know enough systems guys (and girls) and how they think. One of the great features of Guix is its store and that it is easy to grasp how it works. Handling everything through a daemon makes sense from a user perspective - but you should also allow for short-cuts. As unintuitive as it may sound, it is the Unix way ;) It may also allow for people creating other store-based tools in time. Anyway, just raising the point now. We should not be against dirty hacks out of principle or purism. That will hurt acceptance in my opinion. Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 8:37 ` Pjotr Prins @ 2015-04-21 10:08 ` Ricardo Wurmus 2015-04-21 10:17 ` Pjotr Prins 2015-04-21 12:12 ` Ludovic Courtès 1 sibling, 1 reply; 31+ messages in thread From: Ricardo Wurmus @ 2015-04-21 10:08 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins writes: > On Tue, Apr 21, 2015 at 10:11:29AM +0200, Ludovic Courtès wrote: >> The important thing is that currently, the DB is authoritative. So it >> cannot be corrupt (that would be equivalent to having lost /gnu/store >> altogether), and thus it cannot be repaired. > > The point really is that it is not so hard to achieve rebuilding the > database. Anyone on this list thinks that should be possible? I am > asking the big systems guys :) You could rebuild the database only if the contents of the /gnu/store directory were authoritative (which they are not). The database only contains four tables and none of the entries are magical: sqlite> .fullschema CREATE TABLE ValidPaths ( id integer primary key autoincrement not null, path text unique not null, hash text not null, registrationTime integer not null, deriver text, narSize integer ); CREATE TABLE Refs ( referrer integer not null, reference integer not null, primary key (referrer, reference), foreign key (referrer) references ValidPaths(id) on delete cascade, foreign key (reference) references ValidPaths(id) on delete restrict ); CREATE TABLE DerivationOutputs ( drv integer not null, id text not null, -- symbolic output id, usually "out" path text not null, primary key (drv, id), foreign key (drv) references ValidPaths(id) on delete cascade ); CREATE TABLE FailedPaths ( path text primary key not null, time integer not null ); CREATE INDEX IndexReferrer on Refs(referrer); CREATE INDEX IndexReference on Refs(reference); CREATE INDEX IndexDerivationOutputs on DerivationOutputs(path); CREATE TRIGGER DeleteSelfRefs before delete on ValidPaths begin delete from Refs where referrer = old.id and reference = old.id; end; /* No STAT tables available */ To create records in Refs you would obviously need to know the relationship between packages, but you can get that by using Guix and the package modules as a library. Someone daring enough to mess with the store contents (by manually adding, altering or removing stuff) without going through the daemon would also have to take care of manipulating database records in the appropriate manner. I fail to see, however, in what situation this would be desirable to a system administrator. Sysadmins don't expect, for example, to be able to delete files from a system where software is managed with RPM *without* also having to modify the RPM database to avoid problems. If I were to sync /gnu/store/ across different machines with rsync, I'd make sure to keep any additional state by also copying the localstatedir: /var/guix/. But as I said before, I fail to see a good reason to do this. > It may also allow for people creating other store-based tools in time. Guix can be used as a library and with the store monad it is possible to programmatically manipulate the store. I expect this to become even easier once the daemon is rewritten in Scheme (which is bound to result in more reusable parts that could be glued together by adventurous sysadmins). ~~ Ricardo ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 10:08 ` Ricardo Wurmus @ 2015-04-21 10:17 ` Pjotr Prins 2015-04-21 12:08 ` Andreas Enge 0 siblings, 1 reply; 31+ messages in thread From: Pjotr Prins @ 2015-04-21 10:17 UTC (permalink / raw) To: guix-devel On Tue, Apr 21, 2015 at 12:08:39PM +0200, Ricardo Wurmus wrote: > You could rebuild the database only if the contents of the /gnu/store > directory were authoritative (which they are not). The database only > contains four tables and none of the entries are magical: That is good. > Someone daring enough to mess with the store contents (by manually > adding, altering or removing stuff) without going through the daemon > would also have to take care of manipulating database records in the > appropriate manner. I fail to see, however, in what situation this > would be desirable to a system administrator. > > Sysadmins don't expect, for example, to be able to delete files from a > system where software is managed with RPM *without* also having to > modify the RPM database to avoid problems. You are wrong here. Not wanting to start disputing minor points, but people do clean out directories and install new software in existing distributions. Guix gets away from the latter, but not from the first. > If I were to sync /gnu/store/ across different machines with rsync, I'd > make sure to keep any additional state by also copying the > localstatedir: /var/guix/. But as I said before, I fail to see a good > reason to do this. That is a problem. Anyone else see what I mean, at all? > > It may also allow for people creating other store-based tools in time. > > Guix can be used as a library and with the store monad it is possible to > programmatically manipulate the store. I expect this to become even > easier once the daemon is rewritten in Scheme (which is bound to result > in more reusable parts that could be glued together by adventurous > sysadmins). Sounds good :) Pj. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 10:17 ` Pjotr Prins @ 2015-04-21 12:08 ` Andreas Enge 0 siblings, 0 replies; 31+ messages in thread From: Andreas Enge @ 2015-04-21 12:08 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel On Tue, Apr 21, 2015 at 12:17:54PM +0200, Pjotr Prins wrote: > On Tue, Apr 21, 2015 at 12:08:39PM +0200, Ricardo Wurmus wrote: > > If I were to sync /gnu/store/ across different machines with rsync, I'd > > make sure to keep any additional state by also copying the > > localstatedir: /var/guix/. But as I said before, I fail to see a good > > reason to do this. > That is a problem. Anyone else see what I mean, at all? I wonder whether we should not (at least by default) keep all the additional state somewhere under /gnu/store instead of /var/guix. Or maybe just the database part. Andreas ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Self-contained Guix tarball 2015-04-21 8:37 ` Pjotr Prins 2015-04-21 10:08 ` Ricardo Wurmus @ 2015-04-21 12:12 ` Ludovic Courtès 1 sibling, 0 replies; 31+ messages in thread From: Ludovic Courtès @ 2015-04-21 12:12 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> skribis: > To make guix loved by system deployments you have to cater for quick > and dirty. Not sure. GuixSD is clearly a departure from “traditional Unix.” I agree that we have to work to make this acceptable and intelligible to those used to traditional Unix. That said, what it brings, notably reproducible system deployments with ‘guix system’, would hopefully be considered a selling point to system administrators used to the imperative way. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2015-04-21 12:17 UTC | newest] Thread overview: 31+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-04-10 8:46 Copying whole /gnu/store from USB does not work Pjotr Prins 2015-04-10 11:48 ` Ludovic Courtès 2015-04-10 13:14 ` Pjotr Prins 2015-04-10 13:41 ` David Thompson 2015-04-12 10:28 ` Copying whole /gnu/store from USB does not work (SOLVED) Pjotr Prins 2015-04-12 12:29 ` Ludovic Courtès 2015-04-12 20:07 ` Self-contained Guix tarball Ludovic Courtès 2015-04-13 6:54 ` Pjotr Prins 2015-04-13 9:57 ` Ludovic Courtès 2015-04-13 10:05 ` Pjotr Prins 2015-04-13 13:29 ` Mark H Weaver 2015-04-14 21:33 ` Ludovic Courtès 2015-04-13 15:47 ` Thompson, David 2015-04-15 21:31 ` Ludovic Courtès 2015-04-16 5:33 ` Pjotr Prins 2015-04-18 21:23 ` Ludovic Courtès 2015-04-19 8:18 ` Pjotr Prins 2015-04-19 20:09 ` Ludovic Courtès 2015-04-19 13:34 ` Taylan Ulrich Bayırlı/Kammer 2015-04-20 20:49 ` Ludovic Courtès 2015-04-21 7:03 ` Pjotr Prins 2015-04-21 7:19 ` Pjotr Prins 2015-04-21 8:18 ` Ludovic Courtès 2015-04-21 8:44 ` Pjotr Prins 2015-04-21 12:16 ` Ludovic Courtès 2015-04-21 8:11 ` Ludovic Courtès 2015-04-21 8:37 ` Pjotr Prins 2015-04-21 10:08 ` Ricardo Wurmus 2015-04-21 10:17 ` Pjotr Prins 2015-04-21 12:08 ` Andreas Enge 2015-04-21 12:12 ` Ludovic Courtès
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.