From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: Workflow management with GNU Guix Date: Fri, 28 Oct 2016 15:15:15 +0200 Message-ID: <87zilobonw.fsf@gnu.org> References: <87wpmzhdk2.fsf@gnu.org> <87wpmvyocy.fsf@gnu.org> <87wpgwimn3.fsf@gnu.org> <87wpgvs2nz.fsf@gnu.org> <87twbz9qi3.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:48221) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c070E-0007Mo-5j for guix-devel@gnu.org; Fri, 28 Oct 2016 09:15:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c070A-0004mp-1I for guix-devel@gnu.org; Fri, 28 Oct 2016 09:15:22 -0400 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:33290) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c0709-0004mc-Uj for guix-devel@gnu.org; Fri, 28 Oct 2016 09:15:17 -0400 In-Reply-To: <87twbz9qi3.fsf@gnu.org> (Roel Janssen's message of "Wed, 26 Oct 2016 15:41:40 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Roel Janssen Cc: guix-devel@gnu.org Hi! Roel Janssen skribis: > Ludovic Court=C3=A8s writes: [...] >> IIUC, (guix workflows) from the tarball you sent executes workflows in >> the current environment, as opposed to creating a derivation that would >> actually perform the workflow. What motivated this approach? > > The short answer: > Lack of time to implement it properly ;). > > The slightly longer answer: > I want to avoid storing results in the store, because we could be > analyzing files of 100GB or more that we do not want to copy into the > store, neither do we want to store the results of the run in the store. Good point! > I now realize we could only put the derivation in the store, and not the > build output itself.. A derivation has to get its inputs from the store, and to write its output to the store. There=E2=80=99s no other option. So I guess that=E2=80=99s an argument in favor of the approach you chose. >> Workflows could compiled to derivations, which in turn could be =E2=80= =9Cbuilt=E2=80=9D, >> and their build result would be the workflow=E2=80=99s output file. >> >> I guess in practice it only works if users of the cluster can build >> derivations on the cluster and have them scheduled on compute nodes. >> >> Thoughts? > > For building derivations, I think we need super user privileges, right? Well guix-daemon needs to run as root, unless --disable-chroot is used. > Why can't the scripts "just" output the environment variables required as > @code{guix package --search-paths} provides, and then run the commands > with the newly set environment? Fundamentally, a derivation just describes a command, its arguments, its dependencies, its outputs, and its environment variables. So you=E2=80=99re right: you can very much run a derivation =E2=80=9Cby han= d=E2=80=9D instead of letting the daemon do it on your behalf. The only difference is that you won=E2=80=99t have write access to the store. Here=E2=80=99s an example: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(guix) scheme@(guile-user)> ,use(gnu packages base) scheme@(guile-user)> (define s (open-connection)) scheme@(guile-user)> (package-derivation s coreutils) $4 =3D # /gnu/store/81pkzgzjwbnxfd5izgmgam8hfmjn20v8-coreutils-8.25-deb= ug /gnu/store/apx87qb8g3f6x0gbx555qpnfm1wkdv4v-coreutils-8.25 5baea00> scheme@(guile-user)> (derivation-builder $4) $5 =3D "/gnu/store/ik15p8lrbk6jfa3fs3x34m78lj2c0ix1-guile-2.0.11/bin/guile" scheme@(guile-user)> (derivation-builder-arguments $4) $6 =3D ("--no-auto-compile" "-L" "/gnu/store/mn706n39l8z37w8wdqcm9v8pg6zcn3= 3v-module-import" "/gnu/store/1a559p1yki9x1g676r8z0p3cf1f3pq7l-coreutils-8.= 25-guile-builder") scheme@(guile-user)> (apply system* $5 $6) [ Well, chdir to /tmp or something before trying it at home=E2=80=A6 ] --8<---------------cut here---------------end--------------->8--- Ludo=E2=80=99.