RE: Guix Workflow Language ?

["Cook, Malcolm" <MEC@stowers.org>]

Hi,

Watching this thread and trying to take the pulse of GWL.

Where should I look?

https://git.roelj.com/guix/gwl has little documentation - it does say " GWL has a built-in getting-started guide. To use it, run: guix workflow --web-interface" - but supposing we just want to read some documentation

https://www.guixwl.org/ is 503

Workflow management with GNU Guix https://archive.fosdem.org/2017/schedule/event/guixworkflowmanagement/ is interesting but not documentation

Can someone please catch me up?

Thx,

~malcolm_cook@stowers.org

 > -----Original Message-----
 > From: Guix-devel [mailto:guix-devel-bounces+mec=stowers.org@gnu.org]
 > On Behalf Of Roel Janssen
 > Sent: Thursday, January 25, 2018 4:05 PM
 > To: zimoun <zimon.toutoune@gmail.com>
 > Cc: guix-devel@gnu.org
 > Subject: Re: Guix Workflow Language ?
 > 
 > 
 > zimoun writes:
 > 
 > > Dear Roel,
 > >
 > > Thank you for your comments.
 > >
 > > I was imaging your point 2. And the softwares come from Guix.
 > > The added benefit was: a controlled and reproducible environment.
 > > In other words, the added benefit came from the GuixWorkflow (the
 > > engine of workflow), and not from the Language (lisp EDSL).
 > > But maybe it is a wrong way.
 > 
 > I get that point.  Maybe it's then a better idea to write the workflow
 > in CWL (like you would do), and use Guix to generate Docker containers.
 > 
 > Then you do get the benefit of Guix's strong reproducibility and
 > composability forscientific software, plus you get to keep writing the
 > workflow in CWL. :-)
 > 
 > >
 > >>From my experience, the classical strategy of writing pipelines is to
 > > adapt an already existing workflow for one another particular
 > > question. We fetch bits here and there, do some ugly and dirty hacks
 > > to have some results; then depending on them, a cleaner pipeline is
 > > written (or not! :-) or other pieces are tested.
 > > Again from my experience, there is (at least) 3 issues: the number of
 > > tools to learn and know enough to be able to adapt; the bits/pieces
 > > already available; the environment/dependencies and how they are
 > > managed.
 > >
 > > In this context, since 'lispy' syntax is not mainstream (and will
 > > never be), it appears to me as a hard position. That's why I asked if
 > > a Guix-backend workflow engine for CWL specs is doable. Run CWL specs
 > > workflow on the top of the GWL engine.
 > 
 > This is a good question, but how can you describe the origin of a
 > software package in CWL?  In the GWL, we use the Scheme symbols, and
 > the
 > Guix programming interface directly, but that is unavailable in CWL.
 > 
 > This is a real problem that I don't see we can easily solve.
 > 
 > 
 > >
 > > However, I got your point, I guess.
 > > You mean: it is a lot of work with unclear benefits over existing engines.
 > 
 > So, I think it's impossible to express the deployment of a software
 > program in CWL.  It is not as expressive as GWL in this regard.
 > Translating to a precise Guix package recipe and its dependencies is
 > very hard from what we can write in CWL.
 > 
 > If I am mistaken here, please let me know.  Maybe we can figure
 > something out.
 > 
 > >
 > >
 > > Therefore, your point 1. reverses "my issue".
 > > Once the pipeline is well-established, write it with GWL! :-)
 > > Next, if it is possible to convert this GWL specs pipeline to CWL one
 > > [+ Docker] (with softwares coming from Guix), then we can enjoy the
 > > CWL-world engine capabilities.
 > > The benefit of that is from two sides: run the pipeline with different
 > > engines; and produce a clean docker image.
 > >
 > > So , instead of working on improving the GWL engine (adding features
 > > about efficiency, Grid,  Amazon, etc.) which is a very tough task, the
 > > doable plan would be to add an "exporter".
 > > Right ?
 > 
 > The plan is to implement back-ends, or 'process-engines' for GWL to work
 > with AWS, Kubernetes, Grid (this one is already supported).
 > 
 > These back-ends are surprisingly easy to write, because the Guix
 > programming interface allows us to generate virtual machines,
 > containers, or simply store items if Guix is available locally.
 > 
 > We also implemented a Bash-engine that can generate Bash scripts for
 > every step of the workflow.  That in combination with the variety of
 > deployment options solves most of the challenges.
 > 
 > >
 > >
 > > Another question, do you think it is doable to write "importers" ?
 > >
 > > I am not sure that the metaphor is good enough, but do you think it is
 > > a feasible goal from the existing GWL to go towards a kind of `Pandoc
 > > of workflows` ? also packing the softwares.
 > >
 > > And a start should be:
 > >  - write a parser for (subset of) CWL yaml file and obtain the GWL
 > > representation of the workflow
 > >  - write a exporter to CWL + Docker image
 > >
 > > What do you think ?
 > 
 > Maybe.  But in CWL we cannot describe precise software packages.  So
 > translating these things to Guix is hard.
 > 
 > >
 > >
 > > About the parser, I haven't found yet an easy-to-use Guile lib for
 > > parsing YAML-like files. Any pointer ? Adapt some Racket ones ?
 > 
 > I don't know of one, sorry.
 > 
 > 
 > > Thank you for your insights.
 > >
 > > All the best,
 > > simon
 > 
 > Thanks!
 > 
 > Kind regards,
 > Roel Janssen