From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id iDhbFjLCj2DZLAEAgWs5BA (envelope-from ) for ; Mon, 03 May 2021 11:28:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id Gc0EEjLCj2CgAwAA1q6Kng (envelope-from ) for ; Mon, 03 May 2021 09:28:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C4D101FE76 for ; Mon, 3 May 2021 11:28:17 +0200 (CEST) Received: from localhost ([::1]:44926 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ldUsC-0002uS-3w for larch@yhetil.org; Mon, 03 May 2021 05:28:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35154) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ldUos-00013O-Bz for gwl-devel@gnu.org; Mon, 03 May 2021 05:24:50 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21163) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ldUop-0003ce-2g for gwl-devel@gnu.org; Mon, 03 May 2021 05:24:49 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1620033880; cv=none; d=zohomail.com; s=zohoarc; b=B3u1Sx7/BL4RgSqGvEeXDNRvEaK01xv/DLqnBj06C6JQKjliY2v63BV70UQu8QAqzdnibuFjLdYQzefg6tCs/+F7Ondf+FWAXVgx70Dd+8GFCozL562TuP6exLj+QKI0N1DW6PRnTSGNBU/T4I0+Yh3OlWfuLZGUrZY89znHujc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1620033880; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=w649mELyCNj3kheI/AW1rx+CgwlGFXgrPZSBOiUmIYo=; b=PXfG0tjw93n7vnELY/QfYlBbfJqKuvCwjIbcKjimamIN6X5p7vEOG/JNRMNM2SEW4MkZT+nhfoChQjjfP9QZQvbyahQQBPMcawg9b/FGcdxj9Wyz+AJpKh25jBuB/NWG0t+MQ+nygISJy10iSrYkGH2kS254JSxlTVTAZlrFTUc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1620033880; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=w649mELyCNj3kheI/AW1rx+CgwlGFXgrPZSBOiUmIYo=; b=JN2yjbX4uqAy1nSf94UASwptIHvb5ksZktYxOJ77s0bJE8Bhk8EcO9ku29Xy1LSd nnAw8eSRA/s7/LegxAdN4kzgH8Uuy+bCRFoBb+IkH6AWi9u58fTMFEpEaXjreoC3oQK r5a317zwnKp/eSK5KzTXh93O82GSgZUhDdNGn2LY= Received: from localhost (p4fd5aec5.dip0.t-ipconnect.de [79.213.174.197]) by mx.zohomail.com with SMTPS id 1620033877821507.17976360704256; Mon, 3 May 2021 02:24:37 -0700 (PDT) User-agent: mu4e 1.4.15; emacs 27.2 From: Ricardo Wurmus To: gwl-devel@gnu.org Subject: RFC: Support for DRMAA backend X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 03 May 2021 11:24:35 +0200 Message-ID: <877dkgkw4s.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1620034097; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=w649mELyCNj3kheI/AW1rx+CgwlGFXgrPZSBOiUmIYo=; b=MrZKmMNx23KsmXqPCAtK0cFWi7NVjD3CHuQkeqWNht7arysvdjxVtf7vjL2mVkjq0GaXVa kEgLbGhnVHX5Yxt9prQpjrElEfO2eZwgbr2+a35u/JNSMmMHTG+/Fp+N2UKp+UqmD4I5sR fx7z3RyiulwPTgAxkIQPwWYrkgjf+tWv51+PDM6JeLjmB621uavPkyHuRcEFi4QBKIKTgC 1gApHCvzTq4oBYCiVux/7WADnbqLa30wIYXNBhK86BQsTmKtC+WQ6V8Y6FexQrGGMIy5Fj tk4XKCUIgeGtYfqKky48QOso7Dv8hBhbd/vFuUGJrxUODXGVG3moFic3q2fESg== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1620034097; a=rsa-sha256; cv=pass; b=lRDfG7osOpRJ7xByNM7Y8+mj45z9ZNOEsXpojPYda8SG4BaYx2znwYXbisi/hGo7Kh84Nm pyR9AE45NTynOBub67221+vpOqWKm+e9TVAgxzJ+iYJafaqrWgVsm+N3jvjZhmAJfcMeWX Zlzinme+STKAyJ2gOlYBAuQNsg+9T9gnylIU0OrlO1Ri5J4msfG2PgTjaCkN/2qj8vTKgz zRUMT4lI/i4tNKUya1yqgHWjgdvCIBPWVd/4bbJTqvmN9mDcBIz/LCGDVbbTj9qgMt7D/x X3qppxyqgvul5bAzWbIJb2+no+ptYG6E0/gv+9fWDj6R6dMgug31az/5t4eNTQ== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=JN2yjbX4; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.66 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=JN2yjbX4; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: C4D101FE76 X-Spam-Score: -3.66 X-Migadu-Scanner: scn0.migadu.com X-TUID: Xi6bChffPRSP Hello there, I pushed a new branch =E2=80=9Cwip-drmaa=E2=80=9D, which lets GWL use an HP= C=20 scheduler via DRMAA. Set the GWL_USE_DRMAA variable and use the=20 default execution engine. (In the future this variable will=20 disappear once we have agreed on a nicer way to select DRMAA=20 execution.) I only tested this with a local Slurm cluster on my laptop, but it=20 seems to work fine. I=E2=80=99d be happy to hear test reports! Before this can make its way into the main branch we have to=20 figure out if the concept of an execution engine still makes sense=20 the way it is implemented. Execution engines currently are only used to build custom scripts=20 (or script wrappers) =E2=80=94 but using DRMAA does not require any new=20 kind of script. The old Grid Engine backend builds the usual=20 workflow scripts and a set of job scripts that call those scripts=20 after passing Grid Engine settings. For DRMAA we only need=20 run-time support. I wonder if we should scrap execution engines as they currently=20 exist; we would only ever generate the same kind of workflow=20 scripts and move all execution-related work (like submitting jobs=20 to a scheduler) to, well, execution time. This also means that=20 the workflow scripts would not need to be modified when picking a=20 different execution backend, which I think is good. Comments? --=20 Ricardo