From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:48214) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVyvO-0002nq-6G for gwl-devel@gnu.org; Wed, 29 May 2019 09:47:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVyvN-000245-52 for gwl-devel@gnu.org; Wed, 29 May 2019 09:47:26 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:52733) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVyvM-00022v-W7 for gwl-devel@gnu.org; Wed, 29 May 2019 09:47:25 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id 532EE160031 for ; Wed, 29 May 2019 15:47:20 +0200 (CEST) Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id nRf_-58O-lEa for ; Wed, 29 May 2019 15:47:19 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS for ; Wed, 29 May 2019 15:47:19 +0200 (CEST) From: Ricardo Wurmus Date: Wed, 29 May 2019 15:47:18 +0200 Message-ID: <87a7f5l6e1.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Subject: Next steps for the GWL List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gwl-devel-bounces+kyle=kyleam.com@gnu.org Sender: "gwl-devel" To: gwl-devel@gnu.org Hi, I=E2=80=99m going to use the GWL in the next few days to rewrite the PiGx R= NAseq pipeline from Snakemake. This will likely show me what features are still missing from the GWL and what implemented features are awkward to use. My goals for the future (in no particular order) are as follows: * add support for running workflows from a file (without GUIX_WORKFLOW_PATH) * replace the =E2=80=9Cqsub=E2=80=9D-based Grid Engine execution engine wit= h proper DRMAA bindings * add support for running remote jobs via Guile SSH * add support for spawning jobs on AWS or Azure or via Kubernetes * tighter integration with Guix features, e.g. to export a container image per process via =E2=80=9Cguix pack=E2=80=9D or to pack up the whole= workflow as a relocatable executable. * inversion of control: enable workflow designers to use the GWL as a library, so that the =E2=80=9Cguix workflow=E2=80=9D user interface does = not need to be used at all (see PiGx for an example). * explore the use of inferiors =E2=80=94 the GWL should be usable with any version of Guix that may be installed, not just the version that was used at compilation time. Can we use =E2=80=9Cguix repl=E2=80=9D and inf= eriors, perhaps? * add support for executing processes in isolated environments (containers) =E2=80=94 this requires a better understanding of process in= puts. What do you think about this? Who would like to help flesh out these ideas? -- Ricardo