unofficial mirror of gwl-devel@gnu.org
 help / color / mirror / Atom feed
From: Ricardo Wurmus <rekado@elephly.net>
To: gwl-devel@gnu.org
Subject: RFC: Support for DRMAA backend
Date: Mon, 03 May 2021 11:24:35 +0200	[thread overview]
Message-ID: <877dkgkw4s.fsf@elephly.net> (raw)

Hello there,

I pushed a new branch “wip-drmaa”, which lets GWL use an HPC 
scheduler via DRMAA.  Set the GWL_USE_DRMAA variable and use the 
default execution engine.  (In the future this variable will 
disappear once we have agreed on a nicer way to select DRMAA 
execution.)

I only tested this with a local Slurm cluster on my laptop, but it 
seems to work fine.  I’d be happy to hear test reports!

Before this can make its way into the main branch we have to 
figure out if the concept of an execution engine still makes sense 
the way it is implemented.

Execution engines currently are only used to build custom scripts 
(or script wrappers) — but using DRMAA does not require any new 
kind of script.  The old Grid Engine backend builds the usual 
workflow scripts and a set of job scripts that call those scripts 
after passing Grid Engine settings.  For DRMAA we only need 
run-time support.

I wonder if we should scrap execution engines as they currently 
exist; we would only ever generate the same kind of workflow 
scripts and move all execution-related work (like submitting jobs 
to a scheduler) to, well, execution time.  This also means that 
the workflow scripts would not need to be modified when picking a 
different execution backend, which I think is good.

Comments?

-- 
Ricardo


                 reply	other threads:[~2021-05-03  9:28 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.guixwl.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=877dkgkw4s.fsf@elephly.net \
    --to=rekado@elephly.net \
    --cc=gwl-devel@gnu.org \
    --subject='Re: RFC: Support for DRMAA backend' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

unofficial mirror of gwl-devel@gnu.org

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/gwl/0 gwl/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 gwl gwl/ https://yhetil.org/gwl \
		gwl-devel@gnu.org
	public-inbox-index gwl

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.gwl


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git