From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roel Janssen Subject: Re: Guix Workflow Language ? Date: Thu, 25 Jan 2018 23:04:36 +0100 Message-ID: <87tvv9lhmz.fsf@gnu.org> References: <874lnbqauw.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49371) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eepdY-0007xW-7i for guix-devel@gnu.org; Thu, 25 Jan 2018 17:04:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eepdT-0006h5-9a for guix-devel@gnu.org; Thu, 25 Jan 2018 17:04:48 -0500 In-reply-to: List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: zimoun Cc: guix-devel@gnu.org zimoun writes: > Dear Roel, > > Thank you for your comments. > > I was imaging your point 2. And the softwares come from Guix. > The added benefit was: a controlled and reproducible environment. > In other words, the added benefit came from the GuixWorkflow (the > engine of workflow), and not from the Language (lisp EDSL). > But maybe it is a wrong way. I get that point. Maybe it's then a better idea to write the workflow in CWL (like you would do), and use Guix to generate Docker containers. Then you do get the benefit of Guix's strong reproducibility and composability forscientific software, plus you get to keep writing the workflow in CWL. :-) > >>From my experience, the classical strategy of writing pipelines is to > adapt an already existing workflow for one another particular > question. We fetch bits here and there, do some ugly and dirty hacks > to have some results; then depending on them, a cleaner pipeline is > written (or not! :-) or other pieces are tested. > Again from my experience, there is (at least) 3 issues: the number of > tools to learn and know enough to be able to adapt; the bits/pieces > already available; the environment/dependencies and how they are > managed. > > In this context, since 'lispy' syntax is not mainstream (and will > never be), it appears to me as a hard position. That's why I asked if > a Guix-backend workflow engine for CWL specs is doable. Run CWL specs > workflow on the top of the GWL engine. This is a good question, but how can you describe the origin of a software package in CWL? In the GWL, we use the Scheme symbols, and the Guix programming interface directly, but that is unavailable in CWL. This is a real problem that I don't see we can easily solve. > > However, I got your point, I guess. > You mean: it is a lot of work with unclear benefits over existing engines. So, I think it's impossible to express the deployment of a software program in CWL. It is not as expressive as GWL in this regard. Translating to a precise Guix package recipe and its dependencies is very hard from what we can write in CWL. If I am mistaken here, please let me know. Maybe we can figure something out. > > > Therefore, your point 1. reverses "my issue". > Once the pipeline is well-established, write it with GWL! :-) > Next, if it is possible to convert this GWL specs pipeline to CWL one > [+ Docker] (with softwares coming from Guix), then we can enjoy the > CWL-world engine capabilities. > The benefit of that is from two sides: run the pipeline with different > engines; and produce a clean docker image. > > So , instead of working on improving the GWL engine (adding features > about efficiency, Grid, Amazon, etc.) which is a very tough task, the > doable plan would be to add an "exporter". > Right ? The plan is to implement back-ends, or 'process-engines' for GWL to work with AWS, Kubernetes, Grid (this one is already supported). These back-ends are surprisingly easy to write, because the Guix programming interface allows us to generate virtual machines, containers, or simply store items if Guix is available locally. We also implemented a Bash-engine that can generate Bash scripts for every step of the workflow. That in combination with the variety of deployment options solves most of the challenges. > > > Another question, do you think it is doable to write "importers" ? > > I am not sure that the metaphor is good enough, but do you think it is > a feasible goal from the existing GWL to go towards a kind of `Pandoc > of workflows` ? also packing the softwares. > > And a start should be: > - write a parser for (subset of) CWL yaml file and obtain the GWL > representation of the workflow > - write a exporter to CWL + Docker image > > What do you think ? Maybe. But in CWL we cannot describe precise software packages. So translating these things to Guix is hard. > > > About the parser, I haven't found yet an easy-to-use Guile lib for > parsing YAML-like files. Any pointer ? Adapt some Racket ones ? I don't know of one, sorry. > Thank you for your insights. > > All the best, > simon Thanks! Kind regards, Roel Janssen