From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:49282) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hYrMQ-0005hD-BF for gwl-devel@gnu.org; Thu, 06 Jun 2019 08:19:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hYrMP-0005LS-2b for gwl-devel@gnu.org; Thu, 06 Jun 2019 08:19:14 -0400 Received: from c2062.mx.srv.dfn.de ([194.95.238.172]:59619) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hYrMO-0005JS-Oc for gwl-devel@gnu.org; Thu, 06 Jun 2019 08:19:13 -0400 References: <87a7f5l6e1.fsf@mdc-berlin.de> <87o93eiqvz.fsf@mdc-berlin.de> From: Ricardo Wurmus In-Reply-To: Date: Thu, 6 Jun 2019 14:19:04 +0200 Message-ID: <87muiukitj.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Subject: Re: Next steps for the GWL List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gwl-devel-bounces+kyle=kyleam.com@gnu.org Sender: "gwl-devel" To: zimoun Cc: Pjotr Prins , gwl-devel@gnu.org Hi simon, > (+ Pjotr because I am sure he has an interesting opinion but not sure > he closely reads this list ;-) > > On Mon, 3 Jun 2019 at 18:18, Ricardo Wurmus > wrote: > >> > - what about a bridge with CWL? >> >> I=E2=80=99m open to this idea, but it would need to be well-defined. Wh= at does >> it really mean? Generating CWL files from GWL workflows? That really >> shouldn=E2=80=99t be too hard. Anything else, however, is hard for me to >> imagine. > > Well, I point out previous threads about this topic: > > https://lists.gnu.org/archive/html/guix-devel/2018-01/msg00428.html > https://lists.gnu.org/archive/html/gwl-devel/2019-02/msg00019.html > > 1- > Generating CWL from GWL should be nice. It should ease the use of > already in-place platform and tools (AWS, etc.) Generating CWL from GWL should be easy, but it=E2=80=99s also not all that useful. The GWL takes care of software deployment, so not only should we generate CWL files but also generate (and upload?) Docker images and make the CWL file reference them. The tooling for CWL=E2=80=A6 seems a little less substantial and focused th= an it first appears. The cwltool can only run CWL workflows locally =E2=80=94 no DRMAA, no AWS. All the other runners that are listed on the CWL website are either very limited or very large environments where CWL execution is not necessarily the primary purpose (cf Galaxy or Arvados). Still, I think it=E2=80=99s the most meanigful connection the GWL can have = with the CWL: using the GWL as a high-level representation which =E2=80=9Ccompil= es=E2=80=9D down to a lower-level representation of CWL + Docker images when needed. > 2- > Use CWL as a process. A lot of work have been done by Pjotr and > reported here [1] > > > [1] https://guix-hpc.bordeaux.inria.fr/blog/2019/01/creating-a-reproducib= le-workflow-with-cwl/ Yes, this works, of course, but that=E2=80=99s a level of integration that= =E2=80=99s extremely limited, in my opinion. Using Guix with the CWL is fine as the blog post demonstrates, but there is very little to be gained and much to be lost when embedding CWL in a GWL workflow. The only thing this enables is reusing existing CWL workflows as a GWL =E2=80=9Cprocess=E2= =80=9D. There is no meaningful integration =E2=80=93 the embedded CWL workflow is a second-class citizen that cannot benefit from any of the GWL features. If the CWL workflow is connected to the GWL via cwltool then the only way to run the workflow on a DRMAA-supported cluster or a bunch of SSH-connected servers, or AWS EC2 instances is to wrap it up in a GWL context. The GWL treats the process as its smallest unit of organisation, so a CWL workflow that=E2=80=99s run as a GWL process cannot really be scaled. If the user has a different CWL execution environment (such as an Arvados installation), the CWL workflow embedded in the GWL will not be able to make use of it. It would forever be tied to the particular version of cwltool in Guix. I=E2=80=99d rather not advocate this use of the CWL in the GWL. It might s= ound good (=E2=80=9CThe GWL is compatible with the CWL!=E2=80=9D), but ultimatel= y it=E2=80=99s a really awkward connection that is bound to lead to a great deal of frustration. Does this make sense? I don=E2=80=99t want to be dismissive. It would be great if we could come = up with something that=E2=80=99s mutually beneficial for CWL users and GWL use= rs alike, but I feel that our options are very limited. I=E2=80=99m still ope= n to ideas and use-case scenarios. -- Ricardo