Re: Guix Workflow Language ?

[Roel Janssen <roel@gnu.org>]

zimoun writes:

> Dear Roel,
>
> Thank you for your comments.
>
> I was imaging your point 2. And the softwares come from Guix.
> The added benefit was: a controlled and reproducible environment.
> In other words, the added benefit came from the GuixWorkflow (the
> engine of workflow), and not from the Language (lisp EDSL).
> But maybe it is a wrong way.

I get that point.  Maybe it's then a better idea to write the workflow
in CWL (like you would do), and use Guix to generate Docker containers.

Then you do get the benefit of Guix's strong reproducibility and
composability forscientific software, plus you get to keep writing the
workflow in CWL. :-)

>
>From my experience, the classical strategy of writing pipelines is to
> adapt an already existing workflow for one another particular
> question. We fetch bits here and there, do some ugly and dirty hacks
> to have some results; then depending on them, a cleaner pipeline is
> written (or not! :-) or other pieces are tested.
> Again from my experience, there is (at least) 3 issues: the number of
> tools to learn and know enough to be able to adapt; the bits/pieces
> already available; the environment/dependencies and how they are
> managed.
>
> In this context, since 'lispy' syntax is not mainstream (and will
> never be), it appears to me as a hard position. That's why I asked if
> a Guix-backend workflow engine for CWL specs is doable. Run CWL specs
> workflow on the top of the GWL engine.

This is a good question, but how can you describe the origin of a
software package in CWL?  In the GWL, we use the Scheme symbols, and the
Guix programming interface directly, but that is unavailable in CWL.

This is a real problem that I don't see we can easily solve.


>
> However, I got your point, I guess.
> You mean: it is a lot of work with unclear benefits over existing engines.

So, I think it's impossible to express the deployment of a software
program in CWL.  It is not as expressive as GWL in this regard.
Translating to a precise Guix package recipe and its dependencies is
very hard from what we can write in CWL.

If I am mistaken here, please let me know.  Maybe we can figure
something out.

>
>
> Therefore, your point 1. reverses "my issue".
> Once the pipeline is well-established, write it with GWL! :-)
> Next, if it is possible to convert this GWL specs pipeline to CWL one
> [+ Docker] (with softwares coming from Guix), then we can enjoy the
> CWL-world engine capabilities.
> The benefit of that is from two sides: run the pipeline with different
> engines; and produce a clean docker image.
>
> So , instead of working on improving the GWL engine (adding features
> about efficiency, Grid,  Amazon, etc.) which is a very tough task, the
> doable plan would be to add an "exporter".
> Right ?

The plan is to implement back-ends, or 'process-engines' for GWL to work
with AWS, Kubernetes, Grid (this one is already supported).

These back-ends are surprisingly easy to write, because the Guix
programming interface allows us to generate virtual machines,
containers, or simply store items if Guix is available locally.

We also implemented a Bash-engine that can generate Bash scripts for
every step of the workflow.  That in combination with the variety of
deployment options solves most of the challenges.

>
>
> Another question, do you think it is doable to write "importers" ?
>
> I am not sure that the metaphor is good enough, but do you think it is
> a feasible goal from the existing GWL to go towards a kind of `Pandoc
> of workflows` ? also packing the softwares.
>
> And a start should be:
>  - write a parser for (subset of) CWL yaml file and obtain the GWL
> representation of the workflow
>  - write a exporter to CWL + Docker image
>
> What do you think ?

Maybe.  But in CWL we cannot describe precise software packages.  So
translating these things to Guix is hard.

>
>
> About the parser, I haven't found yet an easy-to-use Guile lib for
> parsing YAML-like files. Any pointer ? Adapt some Racket ones ?

I don't know of one, sorry.


> Thank you for your insights.
>
> All the best,
> simon

Thanks!

Kind regards,
Roel Janssen