From mboxrd@z Thu Jan 1 00:00:00 1970 From: zimoun Subject: [GWL] (random) next steps? Date: Fri, 14 Dec 2018 20:16:14 +0100 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:39087) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gXswq-0006MB-Lc for guix-devel@gnu.org; Fri, 14 Dec 2018 14:16:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gXswn-0003po-I8 for guix-devel@gnu.org; Fri, 14 Dec 2018 14:16:32 -0500 Received: from mail-wm1-x344.google.com ([2a00:1450:4864:20::344]:55366) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gXswm-0003pQ-SM for guix-devel@gnu.org; Fri, 14 Dec 2018 14:16:29 -0500 Received: by mail-wm1-x344.google.com with SMTP id y139so6646158wmc.5 for ; Fri, 14 Dec 2018 11:16:28 -0800 (PST) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Guix Devel Dear Guixers, ... or at least some of them :-) Here, I would like to collect some discussions or ideas about the Guix Workflow Language (GWL) and the next steps of this awesome tool! For those who do not know. About workflow language, Wikipedia says: https://en.wikipedia.org/wiki/Scientific_workflow_system About GWL, basically the idea is to apply Guix functional principles to data processing. More details here: https://www.guixwl.org/ Roel Janssen is the original author of the GWL and now the project is part of GNU. Well, I narrow the Ludo's notes from the Paris' meeting and add commentaries as it was suggested. :-) ** HPC, =E2=80=9Cworkflows=E2=80=9D, and all that - overview & status - supporting =E2=80=9Cthe cloud=E2=80=9D + service that produces Docker/Singularity images + todo: produce layered Docker images like Nix folks - workflows + snakemake doesn=E2=80=99t handle software deployment + or does so through Docker + GWL =3D workflow + deployment + add support for Docker + add =E2=80=9Crecency=E2=80=9D checks + data storage: IRODS? - Docker arguments + security: handling patient data with untrusted =E2=80=9C:latest=E2=80= =9D images + Guix allows for =E2=80=9Cbisect=E2=80=9D 1. Even if I am not a big fan of WISP because I remember difficulties to catch "parenthesis closing" issue last time I tried, now I am fan of what Ricardo showed! Let push out the wisp-way of GWL... or not. :-) What are the opinions ? (pa (ren (the (sis)))) vs pa: ren: the: sis With wisp, the workflow description seems close to CWL, which is an argumen= t ;-) Last time, I check, CWL files seems flat yaml-like files and they lack programmable extension; as Snakemake provides with Python for example. And GWL-wisp will have both: easy syntax and programmable stuff, because it is still lisp under the hood. https://www.draketo.de/proj/wisp/ https://www.commonwl.org/ https://snakemake.readthedocs.io/en/stable/getting_started/examples.html 2. One of the lacking feature of GWL is kind-of content addressable store (CAS) for data. Another workflow language named FunFlow (baked as Haskell-DSL) implements such kind of ideas. To quote explanations by Ricardo: "we could copy for the GWL (thus avoiding the need for recency checks). The GWL lacks a data provenance story and a CAS could fit the bill." https://github.com/tweag/funflow 3. The project OpenMole about parametric explorations seems implementing an IPFS way to deal with the data. Not sure what does it mean. :-) https://openmole.org/ Talking about data, Zenodo is always around. ;-) https://zenodo.org/ 4. Some GWL scripts are already there. Could we centralize them to one repo? Even if they are not clean. I mean something in this flavor: https://github.com/common-workflow-language/workflows 5. I recently have discovered the ELisp package `s.el` via the blog post: http://kitchingroup.cheme.cmu.edu/blog/2018/05/14/f-strings-in-emacs-lisp/ or other said: https://github.com/alphapapa/elexandria/blob/master/elexandria.el#L224 Does it appear to you a right path to write a "formater" in this flavour instead of the `string-append` ? I mean, e.g., `(system ,(string-command "gzip ${data-inputs} -c > ${outputs}")) instead of e.g., `(system ,(string-append "gzip " data-inputs " -c > " outputs)) It seems more on the flavour of Snakemake. 6. The graph of dependencies between the processes/units/rules is written by hand. What should be the best strategy to capture it ? By files "=C3=A0 la" Snakemake ? Other ? 7. Does it appear to you a good idea to provide a command `guix workflow pack`= ? to produce an archive with the binaries or the commit hashes of the channels, etc. Last, the webpage [1] points to gwl-devel mailing list which seems broken. Does the gwl-devel need to be activated and the GWL discussion will append there or everything stay here to not scattered too much. [1] https://www.guixwl.org/community What do you think? What is do-able? Science-fiction dream? Thank you. Have a nice week-end, simon -- Then, just pointers/threads (that I am aware of) to remind what the list already discussed. https://lists.gnu.org/archive/html/help-guix/2016-02/msg00019.html https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00380.html https://lists.gnu.org/archive/html/guix-devel/2016-10/msg00947.html https://lists.gnu.org/archive/html/guix-devel/2016-10/msg01248.html https://lists.gnu.org/archive/html/guix-devel/2018-01/msg00371.html https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00177.html https://lists.gnu.org/archive/html/help-guix/2018-05/msg00241.html