From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roel Janssen Subject: Re: Workflow management with GNU Guix Date: Wed, 26 Oct 2016 15:41:40 +0200 Message-ID: <87twbz9qi3.fsf@gnu.org> References: <87wpmzhdk2.fsf@gnu.org> <87wpmvyocy.fsf@gnu.org> <87wpgwimn3.fsf@gnu.org> <87wpgvs2nz.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:56168) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzORQ-00079M-MI for guix-devel@gnu.org; Wed, 26 Oct 2016 09:40:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bzORP-0002U5-Lf for guix-devel@gnu.org; Wed, 26 Oct 2016 09:40:28 -0400 In-reply-to: <87wpgvs2nz.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Ludovic Courtès writes: > Roel Janssen skribis: > >> I realize I never shared my proof-of-concept implementation. I attached >> my motivations for having a workflow language in Guix, and my code. > > Nice work, thanks for sharing! > >> The subcommand "guix workflow" does not work (yet) here. I currently >> execute a workflow directly from the REPL. >> >> A final point to note is that I would like to do a second attempt at >> designing the workflow language, changing the way we can execute >> programs. > > IIUC, (guix workflows) from the tarball you sent executes workflows in > the current environment, as opposed to creating a derivation that would > actually perform the workflow. What motivated this approach? The short answer: Lack of time to implement it properly ;). The slightly longer answer: I want to avoid storing results in the store, because we could be analyzing files of 100GB or more that we do not want to copy into the store, neither do we want to store the results of the run in the store. I now realize we could only put the derivation in the store, and not the build output itself.. > Workflows could compiled to derivations, which in turn could be “built”, > and their build result would be the workflow’s output file. > > I guess in practice it only works if users of the cluster can build > derivations on the cluster and have them scheduled on compute nodes. > > Thoughts? For building derivations, I think we need super user privileges, right? Why can't the scripts "just" output the environment variables required as @code{guix package --search-paths} provides, and then run the commands with the newly set environment? Kind regards, Roel Janssen