From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Cook, Malcolm" Subject: RE: Guix Workflow Language ? Date: Thu, 25 Jan 2018 22:23:34 +0000 Message-ID: References: <874lnbqauw.fsf@gnu.org> <87tvv9lhmz.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53582) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eepw0-0003C6-Uw for guix-devel@gnu.org; Thu, 25 Jan 2018 17:23:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eepvz-0005ek-DU for guix-devel@gnu.org; Thu, 25 Jan 2018 17:23:52 -0500 In-Reply-To: <87tvv9lhmz.fsf@gnu.org> Content-Language: en-US List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Roel Janssen , zimoun Cc: "guix-devel@gnu.org" Hi, Watching this thread and trying to take the pulse of GWL. Where should I look? https://git.roelj.com/guix/gwl has little documentation - it does say " GWL= has a built-in getting-started guide. To use it, run: guix workflow --web-= interface" - but supposing we just want to read some documentation https://www.guixwl.org/ is 503 Workflow management with GNU Guix https://archive.fosdem.org/2017/schedule/= event/guixworkflowmanagement/ is interesting but not documentation Can someone please catch me up? Thx, ~malcolm_cook@stowers.org > -----Original Message----- > From: Guix-devel [mailto:guix-devel-bounces+mec=3Dstowers.org@gnu.org] > On Behalf Of Roel Janssen > Sent: Thursday, January 25, 2018 4:05 PM > To: zimoun > Cc: guix-devel@gnu.org > Subject: Re: Guix Workflow Language ? >=20 >=20 > zimoun writes: >=20 > > Dear Roel, > > > > Thank you for your comments. > > > > I was imaging your point 2. And the softwares come from Guix. > > The added benefit was: a controlled and reproducible environment. > > In other words, the added benefit came from the GuixWorkflow (the > > engine of workflow), and not from the Language (lisp EDSL). > > But maybe it is a wrong way. >=20 > I get that point. Maybe it's then a better idea to write the workflow > in CWL (like you would do), and use Guix to generate Docker containers. >=20 > Then you do get the benefit of Guix's strong reproducibility and > composability forscientific software, plus you get to keep writing the > workflow in CWL. :-) >=20 > > > >>From my experience, the classical strategy of writing pipelines is to > > adapt an already existing workflow for one another particular > > question. We fetch bits here and there, do some ugly and dirty hacks > > to have some results; then depending on them, a cleaner pipeline is > > written (or not! :-) or other pieces are tested. > > Again from my experience, there is (at least) 3 issues: the number of > > tools to learn and know enough to be able to adapt; the bits/pieces > > already available; the environment/dependencies and how they are > > managed. > > > > In this context, since 'lispy' syntax is not mainstream (and will > > never be), it appears to me as a hard position. That's why I asked if > > a Guix-backend workflow engine for CWL specs is doable. Run CWL specs > > workflow on the top of the GWL engine. >=20 > This is a good question, but how can you describe the origin of a > software package in CWL? In the GWL, we use the Scheme symbols, and > the > Guix programming interface directly, but that is unavailable in CWL. >=20 > This is a real problem that I don't see we can easily solve. >=20 >=20 > > > > However, I got your point, I guess. > > You mean: it is a lot of work with unclear benefits over existing engi= nes. >=20 > So, I think it's impossible to express the deployment of a software > program in CWL. It is not as expressive as GWL in this regard. > Translating to a precise Guix package recipe and its dependencies is > very hard from what we can write in CWL. >=20 > If I am mistaken here, please let me know. Maybe we can figure > something out. >=20 > > > > > > Therefore, your point 1. reverses "my issue". > > Once the pipeline is well-established, write it with GWL! :-) > > Next, if it is possible to convert this GWL specs pipeline to CWL one > > [+ Docker] (with softwares coming from Guix), then we can enjoy the > > CWL-world engine capabilities. > > The benefit of that is from two sides: run the pipeline with different > > engines; and produce a clean docker image. > > > > So , instead of working on improving the GWL engine (adding features > > about efficiency, Grid, Amazon, etc.) which is a very tough task, the > > doable plan would be to add an "exporter". > > Right ? >=20 > The plan is to implement back-ends, or 'process-engines' for GWL to work > with AWS, Kubernetes, Grid (this one is already supported). >=20 > These back-ends are surprisingly easy to write, because the Guix > programming interface allows us to generate virtual machines, > containers, or simply store items if Guix is available locally. >=20 > We also implemented a Bash-engine that can generate Bash scripts for > every step of the workflow. That in combination with the variety of > deployment options solves most of the challenges. >=20 > > > > > > Another question, do you think it is doable to write "importers" ? > > > > I am not sure that the metaphor is good enough, but do you think it is > > a feasible goal from the existing GWL to go towards a kind of `Pandoc > > of workflows` ? also packing the softwares. > > > > And a start should be: > > - write a parser for (subset of) CWL yaml file and obtain the GWL > > representation of the workflow > > - write a exporter to CWL + Docker image > > > > What do you think ? >=20 > Maybe. But in CWL we cannot describe precise software packages. So > translating these things to Guix is hard. >=20 > > > > > > About the parser, I haven't found yet an easy-to-use Guile lib for > > parsing YAML-like files. Any pointer ? Adapt some Racket ones ? >=20 > I don't know of one, sorry. >=20 >=20 > > Thank you for your insights. > > > > All the best, > > simon >=20 > Thanks! >=20 > Kind regards, > Roel Janssen