From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id lB+mFE1MWmCEIwAA0tVLHw (envelope-from ) for ; Tue, 23 Mar 2021 20:15:09 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 8Cf7D01MWmBtEwAA1q6Kng (envelope-from ) for ; Tue, 23 Mar 2021 20:15:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 29B30C812 for ; Tue, 23 Mar 2021 21:15:08 +0100 (CET) Received: from localhost ([::1]:53544 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOnQf-00053W-Se for larch@yhetil.org; Tue, 23 Mar 2021 16:15:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48332) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOnQE-00050t-9R for gwl-devel@gnu.org; Tue, 23 Mar 2021 16:14:38 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21117) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOnQ9-0007VW-G2; Tue, 23 Mar 2021 16:14:38 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1616530465; cv=none; d=zohomail.com; s=zohoarc; b=S9fjpdPecEI+9xf6Uoh1keny5O4A6rdwHVuL11vXbFVLDpeb6AstovK/9/w6zaYSl8ls1Zo/+5RbZ30V5X26YJv3dwnk/5L6Xi/ttrn52mOh2OLHFZ/CY0yF+RcXd6fhGGSqAuqMW6PwgPCpDONXiHNsRzaLa23dZ13NYHqu/mY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1616530465; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=lthhgX1sDO+NFC5rBLkXn637RGww7ASfE+yuQFQ31+s=; b=SnEv6c2210VnmgcDQBNFwulKPJAFDvMqNn2g8xKMQaOMkwiAiubVi4oU1WPCHuH6xhIJSyWSstWOlrV8K8WWn61inNGVUMBopS6tRfgghrGgpItf9HB8jPnTTmbPtms3UBvVXtzGZX6ScTmQI91KbPB1TP2G4nVtRZTATIuw7mg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1616530465; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=lthhgX1sDO+NFC5rBLkXn637RGww7ASfE+yuQFQ31+s=; b=aLE/Q8mj+zUP/A/4WpUx8uu0mpzo+GxPDnrPMk4Eqx+gN0aGRIvKhtlqmLNetqfv aNcqf9TZRSUsz6IZGfUtngU8JK9lC06vMcnhlJjgA+j+ioDhOzlbpx1uXWaWY2if0lr mQA5GKFX1a/R1GIdNLNfZiZHB7cDDPYXwV259KZ0= Received: from localhost (p54ad4f2a.dip0.t-ipconnect.de [84.173.79.42]) by mx.zohomail.com with SMTPS id 161653046281281.7255518089587; Tue, 23 Mar 2021 13:14:22 -0700 (PDT) References: <87pmzpvknf.fsf@elephly.net> <17010cba54fe3607be33eecceeb23dd8fffb1ab5.camel@gnu.org> User-agent: mu4e 1.4.14; emacs 27.1 From: Ricardo Wurmus To: Roel Janssen Subject: Re: Getting started with GWL 0.3.0 In-reply-to: <17010cba54fe3607be33eecceeb23dd8fffb1ab5.camel@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 23 Mar 2021 21:14:20 +0100 Message-ID: <87k0pxvd9f.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616530509; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=lthhgX1sDO+NFC5rBLkXn637RGww7ASfE+yuQFQ31+s=; b=CrlIKD2lL/WQmIXdpxbJjgMpF6kMoO7fUWmQlznfoVkmr3lQ/OPI5nYUK2OnTsMBe144ei 5et+he4cG+DCAR8s7JKn38265KdrGdIqLuZWOeui39SGXN0kND4g/jqU7Pw9ty49s85mXD QnkuSgf3+kQ4Iwbt7MJ14pBY5BlnCWQ9fYe4GUv6eDxYTlPQxd/kXez6yZEIGthKPYK6BN oXi27RbKkcBcebAdDD5ieHYkbq4R7tBWYywjj1hEJQ/6f/f17UAk50W+mE4fO1cMsGmQVs TjBqZRMS//Us9wCgzwUfP8AkMHOJfvuTRRJc5tqmFwb3v3gbEUyiYrdGGbXzoA== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1616530509; a=rsa-sha256; cv=pass; b=EsDi5XZkK7lfxD/c6HFdGCzvNFhui4j6jIq1c74XpryPgarupwdnf58CBPW8ZQqKjI6GYQ bukG0JtBa0NZYqAEzta5IJAznu71Htq5v7b9GtTP2OwY50OJ/VSKkwhluD2DkKHlNsGZol YIdAqkjB0wOxXn3IsjopN/XqV5pn009iZhmOwoF2YB9T1iflYAHS05xwYR0FdGADc91MuA WWVJlYpJs092euqaZ7yMzsqOchGxUZSOa1KwNi3g1c8nsRKA81EpycBeyaZGH+ShELCHyC 2mYr4Vw1JSVAGdTHrva/Q/cUw/6W4zNAz+3qY3nAW9DVjfN9G8FxdnwwcDzQmw== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b="aLE/Q8mj"; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.62 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b="aLE/Q8mj"; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: 29B30C812 X-Spam-Score: -3.62 X-Migadu-Scanner: scn0.migadu.com X-TUID: 65tferUVrK8g Roel Janssen writes: > On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote: >>=20 >> Before you get too enthusiastic about the GWL, though, I=E2=80=99d like = to >> note >> that 0.3.0 has a few known bugs that are already fixed in the >> repository. I=E2=80=99ve been putting off making a new release until ei= ther >> Guile-AWS or Guile-DRMAA are ready and usable with the GWL. > > Is there a feature-branch to try out GWL with Guile-DRMAA? :) Unfortunately not yet. I haven=E2=80=99t been 100% successful with the only DRMAA-enabled cluster = that I have access to, and it turns out that it=E2=80=99s not as simple as SGE= =E2=80=99s =E2=80=9Chold_jid=E2=80=9D. It=E2=80=99s no longer =E2=80=9Cfire and forget=E2=80=9D, which is a bit sa= d, but that=E2=80=99s how DRMAA works. We need a run-time component that keeps track of submitted jobs and their status and actively starts held jobs when the prerequisites have finished. It=E2=80=99s not clear to me if and how we should persist workflow state. = The GWL will submit all jobs to the scheduler in a held state and then change their status when its their turn. I wonder if and how we should handle the case where the GWL runtime monitor dies and is restarted. The easiest way is to simply kill all queued up jobs, but I don=E2=80=99t k= now if there=E2=80=99s a better approach. Ideas? --=20 Ricardo