Re: How do I support building a guix package over multiple machines in a cloud environment?

[Josh Marshall <joshua.r.marshall.1991@gmail.com>]

Looking at https://lists.gnu.org/archive/html/gwl-devel/2019-01/msg00034.html
the use case I'm looking at explicitly requires the input files to be
hashed and tracked manually, as if a package.  The actual pipeline
doesn't change much if at all, but those large data files must be
tracked.  Nextflow is the current fad pipeline, but it would be nice
to have some fully magical reproducible way to just re-use any DSL, as
there are a ton used and it would be nice to not replicate
xkcd.com/927 .  Still reading over everything.  I'm going to get a
direct plan for supporting this use case today.

I wish I had work time for this rather than vacation time.  The
technology is fascinating.

On Mon, Dec 2, 2019 at 2:00 PM zimoun <zimon.toutoune@gmail.com> wrote:
>
> Hi (again) Josh :-)
>
> On Mon, 2 Dec 2019 at 19:38, Josh Marshall
> <joshua.r.marshall.1991@gmail.com> wrote:
>
> > He uses it as a bioinformatics workflow to generate some analysis.  It
>
> GWL should work for this use case. o/
>
> > Is this kind of use case supported?  If so, how so?  Is nextflow not
> > practical to keep?  Please, someone catch me up here so I can start to
> > write code to help him out.  If this goes well, my company could
> > integrate for gwl/guix in our work, which would be amazing.
>
> Netxflow [1] is a Domain Specific Language (DSL): you write "rules"
> and how these rules are combined together. In the bioinformatics
> field, Snakemake [2] seems more popular. Other alternatives are CWL
> [3], WDL [4], etc.
>
> Basically, you describe:
>  - what is the inputs
>  - what is the outputs
>  - how to process the inputs to produce the outputs
>
> You can find examples there [*]. It uses the WISP syntax [5] but it
> perfectly works with a Scheme-syntax if you prefer parenthesis. ;-)
>
> [*] https://guixwl.org/
>
>
> However, you should be interested by this blog post [#] by Pjotr using
> Guix and CWL and other niceties!
>
> [#] https://hpc.guix.info/blog/2019/01/creating-a-reproducible-workflow-with-cwl/
>
> AFAIK, Nextflow is not yet packaged in Guix. One direction is to
> package it and then use the workflow described in Nextflow DSL in the
> spirit of [#]. One other direction is to rewrite the workflow using
> the GWL DSL. It depends a bit on what is your final aim.
>
>
> Hope that helps.
> simon
>
> [1] https://www.nextflow.io/
> [2] https://snakemake.readthedocs.io/en/stable/
> [3] https://www.commonwl.org/
> [4] http://www.openwdl.org/
> [5] https://srfi.schemers.org/srfi-119/srfi-119.html