Guix Workflow Language ?

Guix Workflow Language ?

[zimoun]

Dear,

Thank you to provide a workflow manager!
It is fun. :-)

I am currently investigating to describe basic and simple workflows.
And there is a couple of solutions, from python-oriented Snakemake
with Conda capabilities to Common Workflow Language or Workflow
Description Language.

https://snakemake.readthedocs.io/en/latest/
http://www.commonwl.org
https://software.broadinstitute.org/wdl/

And there is already some pipelines elsewhere, e.g.,
https://view.commonwl.org/workflows?search=rnaseq
https://github.com/UMCUGenetics/GGR-cwl

Moreover, some hyper-specialised and classical genomics pipelines are
also available elsewhere, e.g.,
http://bioinformatics.mdc-berlin.de/pigx/


Well, my question is: does it appear to you reasonable to write a
front-end for CWL ?


Because it seems hard to have some room in this landscape, even if I
think that the Guix-way is the most scientific (reproducible, open,
etc.).
I mean, there is room for a guix-backend engine (another CWL
implementation), I guess.

So, during a raining week-end, I gave a look to the specs of CWL and
why not start by define a subset, but then I have failed to write a
parser of it in Guile.


Hum? another question: does it exist an easy-to-use Guile library for
parsing YAML-like files ?


I have found Racket ones, but it was not clear to me how to do with
Guile (without reinventing the wheel).


What do you think about feeding GWL engine by CWL pipeline ?


Thank you for any advice.

All the best,
simon

Re: Guix Workflow Language ?

[Roel Janssen]

Dear Zimoun,

zimoun writes:

> Dear,
>
> Thank you to provide a workflow manager!
> It is fun. :-)

Thanks for looking at the GWL.

>
> I am currently investigating to describe basic and simple workflows.
> And there is a couple of solutions, from python-oriented Snakemake
> with Conda capabilities to Common Workflow Language or Workflow
> Description Language.
>
> https://snakemake.readthedocs.io/en/latest/
> http://www.commonwl.org
> https://software.broadinstitute.org/wdl/
>
> And there is already some pipelines elsewhere, e.g.,
> https://view.commonwl.org/workflows?search=rnaseq
> https://github.com/UMCUGenetics/GGR-cwl

Here's an example of a GWL workflow:
https://github.com/UMCUGenetics/gwl-atacseq

> Moreover, some hyper-specialised and classical genomics pipelines are
> also available elsewhere, e.g.,
> http://bioinformatics.mdc-berlin.de/pigx/
>
>
> Well, my question is: does it appear to you reasonable to write a
> front-end for CWL ?
>
>
> Because it seems hard to have some room in this landscape, even if I
> think that the Guix-way is the most scientific (reproducible, open,
> etc.).
> I mean, there is room for a guix-backend engine (another CWL
> implementation), I guess.

The way I see it, there are two ways to go about this:

1. Make workflows written in GWL (in Scheme) run on a CWL execution
engine.  So basically, produce a YAML file adhering to the CWL spec.

This way we can, at the very least, convert GWL workflows to CWL
workflows -- producing wide compatibility for workflow execution.

This is doable, as long as we put some limitations on the GWL workflow.
It has to have clearly defined 'inputs' and 'outputs', and we probably
have to generate a Docker container with the software dependencies.

2. Let CWL workflows run on top of GWL.  I think this is a lot harder,
because the software deployment is unclear.

>
> So, during a raining week-end, I gave a look to the specs of CWL and
> why not start by define a subset, but then I have failed to write a
> parser of it in Guile.
>
>
> Hum? another question: does it exist an easy-to-use Guile library for
> parsing YAML-like files ?
>
>
> I have found Racket ones, but it was not clear to me how to do with
> Guile (without reinventing the wheel).
>
>
> What do you think about feeding GWL engine by CWL pipeline ?

I am not sure what you mean by this.  Do you mean, run a CWL workflow
using the GWL?  Then my question would be:  Where does the software come
from?  Guix, or some other package manager?  What would the added
benefit of GWL be?

> Thank you for any advice.
>
> All the best,
> simon

Thanks for your interest!

Kind regards,
Roel Janssen

Re: Guix Workflow Language ?

[zimoun]

Dear Roel,

Thank you for your comments.

I was imaging your point 2. And the softwares come from Guix.
The added benefit was: a controlled and reproducible environment.
In other words, the added benefit came from the GuixWorkflow (the
engine of workflow), and not from the Language (lisp EDSL).
But maybe it is a wrong way.

From my experience, the classical strategy of writing pipelines is to
adapt an already existing workflow for one another particular
question. We fetch bits here and there, do some ugly and dirty hacks
to have some results; then depending on them, a cleaner pipeline is
written (or not! :-) or other pieces are tested.
Again from my experience, there is (at least) 3 issues: the number of
tools to learn and know enough to be able to adapt; the bits/pieces
already available; the environment/dependencies and how they are
managed.

In this context, since 'lispy' syntax is not mainstream (and will
never be), it appears to me as a hard position. That's why I asked if
a Guix-backend workflow engine for CWL specs is doable. Run CWL specs
workflow on the top of the GWL engine.

However, I got your point, I guess.
You mean: it is a lot of work with unclear benefits over existing engines.


Therefore, your point 1. reverses "my issue".
Once the pipeline is well-established, write it with GWL! :-)
Next, if it is possible to convert this GWL specs pipeline to CWL one
[+ Docker] (with softwares coming from Guix), then we can enjoy the
CWL-world engine capabilities.
The benefit of that is from two sides: run the pipeline with different
engines; and produce a clean docker image.

So , instead of working on improving the GWL engine (adding features
about efficiency, Grid,  Amazon, etc.) which is a very tough task, the
doable plan would be to add an "exporter".
Right ?


Another question, do you think it is doable to write "importers" ?

I am not sure that the metaphor is good enough, but do you think it is
a feasible goal from the existing GWL to go towards a kind of `Pandoc
of workflows` ? also packing the softwares.

And a start should be:
 - write a parser for (subset of) CWL yaml file and obtain the GWL
representation of the workflow
 - write a exporter to CWL + Docker image

What do you think ?


About the parser, I haven't found yet an easy-to-use Guile lib for
parsing YAML-like files. Any pointer ? Adapt some Racket ones ?



Thank you for your insights.

All the best,
simon

Re: Guix Workflow Language ?

[Ricardo Wurmus]

Hi,

zimoun <zimon.toutoune@gmail.com> writes:

> In this context, since 'lispy' syntax is not mainstream (and will
> never be), it appears to me as a hard position.

We’ve got you covered here: the GWL has built-in support for Wisp, a
pretty language extension for Guile.  It also comes with a bunch of
extra syntax support to make the definition of workflows easier.

Here’s a convoluted artificial example:

--8<---------------cut here---------------start------------->8---
define-module
    test

use-modules
    guix workflows
    guix processes
    gnu packages bioinformatics
    gnu packages python

process: simple-test
    package-inputs
        list python samtools
    data-inputs
        list "sample.bam" "hg38.fa" "abc"
    procedure #---{python}
import os

def hello():
  print "hello from python 3"
  print GWL['data-inputs']
  print GWL['name']

hello()
---

workflow: example-workflow
    processes
        list simple-test
--8<---------------cut here---------------end--------------->8---

Put this in a file called “test.wisp” and add the directory to the
GUIX_WORKFLOW_PATH and you’re good to go.

Note that the “simple-test” process definition embeds Python code.  A
number of other languages can be supported easily.

I don’t think syntax should hold you back.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Guix Workflow Language ?

[Roel Janssen]

zimoun writes:

> Dear Roel,
>
> Thank you for your comments.
>
> I was imaging your point 2. And the softwares come from Guix.
> The added benefit was: a controlled and reproducible environment.
> In other words, the added benefit came from the GuixWorkflow (the
> engine of workflow), and not from the Language (lisp EDSL).
> But maybe it is a wrong way.

I get that point.  Maybe it's then a better idea to write the workflow
in CWL (like you would do), and use Guix to generate Docker containers.

Then you do get the benefit of Guix's strong reproducibility and
composability forscientific software, plus you get to keep writing the
workflow in CWL. :-)

>
>From my experience, the classical strategy of writing pipelines is to
> adapt an already existing workflow for one another particular
> question. We fetch bits here and there, do some ugly and dirty hacks
> to have some results; then depending on them, a cleaner pipeline is
> written (or not! :-) or other pieces are tested.
> Again from my experience, there is (at least) 3 issues: the number of
> tools to learn and know enough to be able to adapt; the bits/pieces
> already available; the environment/dependencies and how they are
> managed.
>
> In this context, since 'lispy' syntax is not mainstream (and will
> never be), it appears to me as a hard position. That's why I asked if
> a Guix-backend workflow engine for CWL specs is doable. Run CWL specs
> workflow on the top of the GWL engine.

This is a good question, but how can you describe the origin of a
software package in CWL?  In the GWL, we use the Scheme symbols, and the
Guix programming interface directly, but that is unavailable in CWL.

This is a real problem that I don't see we can easily solve.


>
> However, I got your point, I guess.
> You mean: it is a lot of work with unclear benefits over existing engines.

So, I think it's impossible to express the deployment of a software
program in CWL.  It is not as expressive as GWL in this regard.
Translating to a precise Guix package recipe and its dependencies is
very hard from what we can write in CWL.

If I am mistaken here, please let me know.  Maybe we can figure
something out.

>
>
> Therefore, your point 1. reverses "my issue".
> Once the pipeline is well-established, write it with GWL! :-)
> Next, if it is possible to convert this GWL specs pipeline to CWL one
> [+ Docker] (with softwares coming from Guix), then we can enjoy the
> CWL-world engine capabilities.
> The benefit of that is from two sides: run the pipeline with different
> engines; and produce a clean docker image.
>
> So , instead of working on improving the GWL engine (adding features
> about efficiency, Grid,  Amazon, etc.) which is a very tough task, the
> doable plan would be to add an "exporter".
> Right ?

The plan is to implement back-ends, or 'process-engines' for GWL to work
with AWS, Kubernetes, Grid (this one is already supported).

These back-ends are surprisingly easy to write, because the Guix
programming interface allows us to generate virtual machines,
containers, or simply store items if Guix is available locally.

We also implemented a Bash-engine that can generate Bash scripts for
every step of the workflow.  That in combination with the variety of
deployment options solves most of the challenges.

>
>
> Another question, do you think it is doable to write "importers" ?
>
> I am not sure that the metaphor is good enough, but do you think it is
> a feasible goal from the existing GWL to go towards a kind of `Pandoc
> of workflows` ? also packing the softwares.
>
> And a start should be:
>  - write a parser for (subset of) CWL yaml file and obtain the GWL
> representation of the workflow
>  - write a exporter to CWL + Docker image
>
> What do you think ?

Maybe.  But in CWL we cannot describe precise software packages.  So
translating these things to Guix is hard.

>
>
> About the parser, I haven't found yet an easy-to-use Guile lib for
> parsing YAML-like files. Any pointer ? Adapt some Racket ones ?

I don't know of one, sorry.


> Thank you for your insights.
>
> All the best,
> simon

Thanks!

Kind regards,
Roel Janssen

RE: Guix Workflow Language ?

[Cook, Malcolm]

Hi,

Watching this thread and trying to take the pulse of GWL.

Where should I look?

https://git.roelj.com/guix/gwl has little documentation - it does say " GWL has a built-in getting-started guide. To use it, run: guix workflow --web-interface" - but supposing we just want to read some documentation

https://www.guixwl.org/ is 503

Workflow management with GNU Guix https://archive.fosdem.org/2017/schedule/event/guixworkflowmanagement/ is interesting but not documentation

Can someone please catch me up?

Thx,

~malcolm_cook@stowers.org

 > -----Original Message-----
 > From: Guix-devel [mailto:guix-devel-bounces+mec=stowers.org@gnu.org]
 > On Behalf Of Roel Janssen
 > Sent: Thursday, January 25, 2018 4:05 PM
 > To: zimoun <zimon.toutoune@gmail.com>
 > Cc: guix-devel@gnu.org
 > Subject: Re: Guix Workflow Language ?
 > 
 > 
 > zimoun writes:
 > 
 > > Dear Roel,
 > >
 > > Thank you for your comments.
 > >
 > > I was imaging your point 2. And the softwares come from Guix.
 > > The added benefit was: a controlled and reproducible environment.
 > > In other words, the added benefit came from the GuixWorkflow (the
 > > engine of workflow), and not from the Language (lisp EDSL).
 > > But maybe it is a wrong way.
 > 
 > I get that point.  Maybe it's then a better idea to write the workflow
 > in CWL (like you would do), and use Guix to generate Docker containers.
 > 
 > Then you do get the benefit of Guix's strong reproducibility and
 > composability forscientific software, plus you get to keep writing the
 > workflow in CWL. :-)
 > 
 > >
 > >>From my experience, the classical strategy of writing pipelines is to
 > > adapt an already existing workflow for one another particular
 > > question. We fetch bits here and there, do some ugly and dirty hacks
 > > to have some results; then depending on them, a cleaner pipeline is
 > > written (or not! :-) or other pieces are tested.
 > > Again from my experience, there is (at least) 3 issues: the number of
 > > tools to learn and know enough to be able to adapt; the bits/pieces
 > > already available; the environment/dependencies and how they are
 > > managed.
 > >
 > > In this context, since 'lispy' syntax is not mainstream (and will
 > > never be), it appears to me as a hard position. That's why I asked if
 > > a Guix-backend workflow engine for CWL specs is doable. Run CWL specs
 > > workflow on the top of the GWL engine.
 > 
 > This is a good question, but how can you describe the origin of a
 > software package in CWL?  In the GWL, we use the Scheme symbols, and
 > the
 > Guix programming interface directly, but that is unavailable in CWL.
 > 
 > This is a real problem that I don't see we can easily solve.
 > 
 > 
 > >
 > > However, I got your point, I guess.
 > > You mean: it is a lot of work with unclear benefits over existing engines.
 > 
 > So, I think it's impossible to express the deployment of a software
 > program in CWL.  It is not as expressive as GWL in this regard.
 > Translating to a precise Guix package recipe and its dependencies is
 > very hard from what we can write in CWL.
 > 
 > If I am mistaken here, please let me know.  Maybe we can figure
 > something out.
 > 
 > >
 > >
 > > Therefore, your point 1. reverses "my issue".
 > > Once the pipeline is well-established, write it with GWL! :-)
 > > Next, if it is possible to convert this GWL specs pipeline to CWL one
 > > [+ Docker] (with softwares coming from Guix), then we can enjoy the
 > > CWL-world engine capabilities.
 > > The benefit of that is from two sides: run the pipeline with different
 > > engines; and produce a clean docker image.
 > >
 > > So , instead of working on improving the GWL engine (adding features
 > > about efficiency, Grid,  Amazon, etc.) which is a very tough task, the
 > > doable plan would be to add an "exporter".
 > > Right ?
 > 
 > The plan is to implement back-ends, or 'process-engines' for GWL to work
 > with AWS, Kubernetes, Grid (this one is already supported).
 > 
 > These back-ends are surprisingly easy to write, because the Guix
 > programming interface allows us to generate virtual machines,
 > containers, or simply store items if Guix is available locally.
 > 
 > We also implemented a Bash-engine that can generate Bash scripts for
 > every step of the workflow.  That in combination with the variety of
 > deployment options solves most of the challenges.
 > 
 > >
 > >
 > > Another question, do you think it is doable to write "importers" ?
 > >
 > > I am not sure that the metaphor is good enough, but do you think it is
 > > a feasible goal from the existing GWL to go towards a kind of `Pandoc
 > > of workflows` ? also packing the softwares.
 > >
 > > And a start should be:
 > >  - write a parser for (subset of) CWL yaml file and obtain the GWL
 > > representation of the workflow
 > >  - write a exporter to CWL + Docker image
 > >
 > > What do you think ?
 > 
 > Maybe.  But in CWL we cannot describe precise software packages.  So
 > translating these things to Guix is hard.
 > 
 > >
 > >
 > > About the parser, I haven't found yet an easy-to-use Guile lib for
 > > parsing YAML-like files. Any pointer ? Adapt some Racket ones ?
 > 
 > I don't know of one, sorry.
 > 
 > 
 > > Thank you for your insights.
 > >
 > > All the best,
 > > simon
 > 
 > Thanks!
 > 
 > Kind regards,
 > Roel Janssen

Re: Guix Workflow Language ?

[Pjotr Prins]

On Thu, Jan 25, 2018 at 11:04:36PM +0100, Roel Janssen wrote:
> I get that point.  Maybe it's then a better idea to write the workflow
> in CWL (like you would do), and use Guix to generate Docker containers.
> 
> Then you do get the benefit of Guix's strong reproducibility and
> composability forscientific software, plus you get to keep writing the
> workflow in CWL. :-)

I think the CWL is not particularly great, mostly because its design
is too open ended (they include a Javascript interpreter - so it is
YAML + JS) and 'complex'. Nor does it help/enforce analysis to be
deterministic.

CWL has community backing. But it could well implode on itself. So
many people are working on it (nominally) for years, and so little has
it got to show for. The promise is truly shared pipelines - and, so
far, it has not happened.  But maybe by sheer grit they will get
there. I know people who are working hard to make it happen.

I think that if you are not drinking the CWL cool-aid, GWL is a great
alternative. But it needs LISP and it may need a bit more development
to make it a smooth experience. If more people help out I am sure we
can get there.  What would be a great pipeline that has general
interest? How about using GWL on the build farm, for example?

I don't think the way forward is adding CWL-style YAML support to GWL.
I think the way forward is to be able to run CWL workflows inside GWL,
i.e., a CWL runner is treated as a single step inside GWL. That way we
get the best of both worlds. Treat CWL as a piece of software on its
own and provision the Guix container to match. Interestingly, this is
pretty much possible today. (Note that there will be some complexity
in error handling because both GWL and CWL will submit jobs to a
cluster environment).

Pj.

Re: Guix Workflow Language ?

[zimoun]

Dear,

Thank you for all your explanations.


Concerning the point about the 'lisppy' syntax.

Thank you Ricardo to point out the WISP initiative. Even if I watched
all the previous guile/guix FOSDEM videos, I have not realized that it
should be an elegant path to reduce the gap.

As Christopher Lemmer Webber said during the presentation, WISP-style
seems less scary for non-lisper people. :-)

If someone needs pointers:
 https://srfi.schemers.org/srfi-119/srfi-119.html
 http://www.draketo.de/english/wisp



Concerning the point about the origin of a software package in CWL.

I am not sure to understand well all the mechanisms.

If I understand well, GWL tracks the packages with `package-inputs'
(Guix API etc.). And it seems possible somehow to fix a specific version
of a software (e.g., commit hash of Guix tree).

About CWL, I have never tried myself and it seems harder. Right ?
However, the specs mentions the `SoftwarePackage' field:
http://www.commonwl.org/v1.0/Workflow.html#SoftwareRequirement
Is it not enough ?



@Cook, Malcolm

Hum? https://www.guixwl.org/getting-started is not currently 503.
If it is and you want to give a quick look, try:
https://git.roelj.com/guix/gwl/src/master/gwl/www/pages/getting-started.md



Concerning to be or not to be CWL.

Thank you for the comments. I am giving a look at CWL because I have
read the Piotr's paper mentioned here. :-)

Quote:
     GWL is a great alternative. But it needs LISP and it may need a bit
     more development to make it a smooth experience. If more people
     help out I am sure we can get there.

I totally agree ! That's why I am asking :-)

As Pjotr also said: "The promise is truly shared pipelines - and, so
far, it has not happened." and from my opinion, there is 3 issues: "the
number of tools to learn and know enough to be able to adapt; the
bits/pieces already available; the environment/dependencies and how they
are managed.". If one point is not strong enough, then all fails.

Currently, GWL is the strongest available about env/deps management.
However, Lisp is not mainstream, especially with Bio* and few
pieces/workflow are already available.

That's why I asked if "does it appear to you reasonable to write a
front-end for CWL ?". Because:
 - CWL appears to me enough simple;
 - the CWL community seems large (at least larger than GWL ;-)

Well, the "lisp scaryness" should be fixed by WISP.
The community is work in progress :-)

From my point of view, GWL is two sides:
 - the Guix Workflow, the engine of worklows which is already awesome !!
 - the Workflow Language, the lisp EDSL which is hard to buy for the
 non-lispers.

If I understand well, the future which is described is: improve the
engine with the current description language. Since the manpower is not
extensible and Guix allows to decrease the pain when extending the
engine, I think that the inputs (language) deserve more love. Well,
people who are doing is the future :-)


Last, I do not understand how 2 workflow engines can co-exist. It is
error-prone and a spaghetti plate that I will not eat. :-)



Happy FOSDEM !! (for people who are going)
Hope that nice ideas will be discussed during the GWL session. :-)


All the best,
simon

Re: Guix Workflow Language ?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Currently, GWL is the strongest available about env/deps management.
> However, Lisp is not mainstream, especially with Bio* and few
> pieces/workflow are already available.

Well, things can change.  Did you know that Ross Ihaka of GNU R fame has
been thinking out loud about a Lisp system as a successor to R?

See https://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-2008.pdf
and https://www.stat.auckland.ac.nz/%7Eihaka/downloads/JSM-2010.pdf

A lisp dialect is used in one of the most impressive editors, and lispy
languages have seen something of a renaissance with Clojure (also check
out Kawa), the rise of Racket, and possibly the appearance of Guix ;)

It’s not mainstream *yet*, but what becomes mainstream is really
determined by users who demonstrate that the tool is up to the task.

> From my point of view, GWL is two sides:
>  - the Guix Workflow, the engine of worklows which is already awesome !!
>  - the Workflow Language, the lisp EDSL which is hard to buy for the
>  non-lispers.

The two are closely linked.  Workflows in the GWL are described as
Scheme values, exactly as packages in Guix are Scheme values.  What
emerges from these linked values is a graph.  In the case of Guix that’s
a huge network of software, whereas in the GWL the graph is an
executable workflow.

> Last, I do not understand how 2 workflow engines can co-exist. It is
> error-prone and a spaghetti plate that I will not eat. :-)

The GWL describes the relations between processes as workflows.  An
execution engine then makes sense of workflows, e.g. by running its
processes on a cluster, spawning individual processes in isolated
environments (“containers!”), pre-processing the data inputs
(e.g. fetching files from a data repository or checking for staleness),
… these things are really independent of the workflows themselves.

But you really do need to have that network of processes first, and
that’s what the GWL builds up as a live datastructure from Scheme
values.  And that’s why it’s much less useful to try to compile a CWL
definition to the GWL — you’d lose most of the features the GWL can
provide.

> Happy FOSDEM !! (for people who are going)
> Hope that nice ideas will be discussed during the GWL session. :-)

I’ll be there and ready for a chat :)

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Guix Workflow Language ?

[zimoun]

Hi,

Thank you Ricardo for the pointers and the explanations.
Some I already knew. Other not.

About Lisp, my heart follows you. My reason is still hesitating. :-)

Well, that said :-)


I am asking if it should be affordable to organize all the available workflows.

For example, two are there
https://github.com/UMCUGenetics/guix-workflows/tree/master/umcu/workflows
Another here https://github.com/UMCUGenetics/gwl-atacseq
etc. etc. (quick search does not find more but I am sure there are)

This means a way to structure with a central repo: category (or not), etc.
And the second step should be a Web search as
https://view.commonwl.org/workflows

Is it affordable ?


All the best,
simon