From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:44340) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ibs7T-0002B6-CT for gwl-devel@gnu.org; Mon, 02 Dec 2019 15:16:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ibs7R-00080G-QV for gwl-devel@gnu.org; Mon, 02 Dec 2019 15:16:31 -0500 Received: from mail-qk1-x729.google.com ([2607:f8b0:4864:20::729]:37138) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ibs7R-00080C-JO for gwl-devel@gnu.org; Mon, 02 Dec 2019 15:16:29 -0500 Received: by mail-qk1-x729.google.com with SMTP id m188so949778qkc.4 for ; Mon, 02 Dec 2019 12:16:29 -0800 (PST) MIME-Version: 1.0 References: <20191202200236.ghnn3cbirecgjfnj@thebird.nl> In-Reply-To: <20191202200236.ghnn3cbirecgjfnj@thebird.nl> From: Josh Marshall Date: Mon, 2 Dec 2019 15:16:17 -0500 Message-ID: Subject: Re: How do I support building a guix package over multiple machines in a cloud environment? Content-Type: text/plain; charset="UTF-8" List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gwl-devel-bounces+kyle=kyleam.com@gnu.org Sender: "gwl-devel" To: pjotr.public12@thebird.nl Cc: gwl-devel@gnu.org I'll start with packaging nextflow. That seems like the easy starting point, will be needed if I want to switch my personal work systems to guix, and I still shouldn't get deep into guix development challenges since it is all still new. On Mon, Dec 2, 2019 at 3:02 PM wrote: > > Yeah, if you read my BLOG below you can also simply split packaging > (read deployment) from the pipeline (runner). It would already be > worthwile to show how Guix can be mixed with Nextflow and how you > could use the same method to create reproducible containers. > > That is a real win! > > In the next step try to package Nextflow itself. > > Or, alternatively, port pipelines to GWL. Mind, GWL needs development. > Nextflow is an amazing (Cloud) tool that can benefit from > reproducible deployment. > > Pj. > > > On Mon, Dec 02, 2019 at 08:00:07PM +0100, zimoun wrote: > > Hi (again) Josh :-) > > > > On Mon, 2 Dec 2019 at 19:38, Josh Marshall > > wrote: > > > > > He uses it as a bioinformatics workflow to generate some analysis. It > > > > GWL should work for this use case. o/ > > > > > Is this kind of use case supported? If so, how so? Is nextflow not > > > practical to keep? Please, someone catch me up here so I can start to > > > write code to help him out. If this goes well, my company could > > > integrate for gwl/guix in our work, which would be amazing. > > > > Netxflow [1] is a Domain Specific Language (DSL): you write "rules" > > and how these rules are combined together. In the bioinformatics > > field, Snakemake [2] seems more popular. Other alternatives are CWL > > [3], WDL [4], etc. > > > > Basically, you describe: > > - what is the inputs > > - what is the outputs > > - how to process the inputs to produce the outputs > > > > You can find examples there [*]. It uses the WISP syntax [5] but it > > perfectly works with a Scheme-syntax if you prefer parenthesis. ;-) > > > > [*] https://guixwl.org/ > > > > > > However, you should be interested by this blog post [#] by Pjotr using > > Guix and CWL and other niceties! > > > > [#] https://hpc.guix.info/blog/2019/01/creating-a-reproducible-workflow-with-cwl/ > > > > AFAIK, Nextflow is not yet packaged in Guix. One direction is to > > package it and then use the workflow described in Nextflow DSL in the > > spirit of [#]. One other direction is to rewrite the workflow using > > the GWL DSL. It depends a bit on what is your final aim. > > > > > > Hope that helps. > > simon > > > > [1] https://www.nextflow.io/ > > [2] https://snakemake.readthedocs.io/en/stable/ > > [3] https://www.commonwl.org/ > > [4] http://www.openwdl.org/ > > [5] https://srfi.schemers.org/srfi-119/srfi-119.html > >