From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roel Janssen Subject: Re: Workflow management with GNU Guix Date: Fri, 28 Oct 2016 16:40:57 +0200 Message-ID: <874m3wh6yu.fsf@gnu.org> References: <87wpmzhdk2.fsf@gnu.org> <87wpmvyocy.fsf@gnu.org> <87wpgwimn3.fsf@gnu.org> <87wpgvs2nz.fsf@gnu.org> <87twbz9qi3.fsf@gnu.org> <87zilobonw.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:45240) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c08Jx-0000Yn-32 for guix-devel@gnu.org; Fri, 28 Oct 2016 10:39:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c08Jw-0005AV-79 for guix-devel@gnu.org; Fri, 28 Oct 2016 10:39:49 -0400 In-reply-to: <87zilobonw.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Ludovic Courtès writes: > Hi! > > Roel Janssen skribis: > >> Ludovic Courtès writes: > > [...] > >>> IIUC, (guix workflows) from the tarball you sent executes workflows in >>> the current environment, as opposed to creating a derivation that would >>> actually perform the workflow. What motivated this approach? >> >> The short answer: >> Lack of time to implement it properly ;). >> >> The slightly longer answer: >> I want to avoid storing results in the store, because we could be >> analyzing files of 100GB or more that we do not want to copy into the >> store, neither do we want to store the results of the run in the store. > > Good point! > >> I now realize we could only put the derivation in the store, and not the >> build output itself.. > > A derivation has to get its inputs from the store, and to write its > output to the store. There’s no other option. > > So I guess that’s an argument in favor of the approach you chose. Can't a derivation write its output to some other place than the store? Maybe by running it "by hand"? >>> Workflows could compiled to derivations, which in turn could be “built”, >>> and their build result would be the workflow’s output file. >>> >>> I guess in practice it only works if users of the cluster can build >>> derivations on the cluster and have them scheduled on compute nodes. >>> >>> Thoughts? >> >> For building derivations, I think we need super user privileges, right? > > Well guix-daemon needs to run as root, unless --disable-chroot is used. Yeah ok.. But as long as the guix-daemon doesn't build any derivation it doesn't need super user privileges ;). >> Why can't the scripts "just" output the environment variables required as >> @code{guix package --search-paths} provides, and then run the commands >> with the newly set environment? > > Fundamentally, a derivation just describes a command, its arguments, its > dependencies, its outputs, and its environment variables. > > So you’re right: you can very much run a derivation “by hand” instead of > letting the daemon do it on your behalf. The only difference is that > you won’t have write access to the store. And that's fine, because we don't want to write the output to the store :). So, the workflow language should create a derivation, but then guix-daemon should not execute the derivation, but instead, the workflow execution engine can do it so it can avoid writing the output to the store.. right? Kind regards, Roel Janssen