From: Phil <phil@beadling.co.uk>
To: Yasuaki Kudo <yasu@yasuaki.com>
Cc: help-guix@gnu.org
Subject: Re: Guix for Corporate "Batch Jobs"?
Date: Tue, 08 Mar 2022 23:18:50 +0000 [thread overview]
Message-ID: <87sfrs2ftx.fsf@beadling.co.uk> (raw)
In-Reply-To: <506EEA75-0A98-4F04-9BC2-C7588FF9A65C@yasuaki.com>
Hi Yasu,
Yasuaki Kudo writes:
> Hi,
>
> In many so-called Application Support jobs in the enterprises, one of the core responsibilities is to see through the daily completion of "batch jobs" - those I/O heavy processes that take a long time to run, even with parallel processing.
>
> And at the core of it is to "re-run" the jobs, after due troubleshooting.
>
> In many workplaces I have seen, teams ended up writing their own job schedulers based on cron or used proprietary software such as Autosys (and in Japan, there are local brews such as A-Auto, if I remember the name correctly).
Not sure if this is exactly what you're looking for - but Guix in my
experience can sit at the centre of a tech-stack for providing software
on machines, and then batch-running that software in a very predictable way.
However Guix is currenty first and foremost a command-line tool, so I
find myself augmenting it with other standard offerings to produce
familiar front-ends for triggers, job processing, management, etc.
A few examples below.
I oversee the use of Guix in an enterprise environment. Initially it
was used to build/test our software and also provide deployments with
dependencies etc. We wrapped Guix builds in Jenkins, which in-turn
integrates with our source control to trigger Guix using a standard
branch workflow developers are used to. Guix fetches and caches any
build dependencies making subsequent builds faster, and making artifacts
available via a Guix substitute server to servers across the enterprise.
More recently and probably more useful to you - I've been looking at
taking the build outputs and making them available as batch jobs using
Guix Workflow Language (https://guixwl.org) - which is a good fit if
your batches are compute jobs with well defined inputs, numerous
dependent stages, and the requirement to reproduce identical numerical
output. GWL provides lots of cool features - it's somewhat like Autosys
in that it is declarative - defining dependencies (and thus an order)
between different workflow processes etc. I don't think GWL can memoize
different processes in a workflow tho - so running a workflow several
times results in all workflow processes being run, as far as I know.
The point is you should be guaranteed the same result with the same
inputs, every time.
I tend to wrap the GWL scripts in Rundeck (job scheduler) to allow
less-technical staff to re-run batches through a web app or to construct
a daily schedule for overnight/regression tests etc, rather than use the
guix command line.
Note GWL isn't designed to be used if the aim of your batch jobs is to
have a side-effect on the server you're running on. We only use it to
produce results from calculations. This is different to Autosys where
each job could be entirely made-up of side-effects which change the
state of the server itself.
HTH,
Phil.
next prev parent reply other threads:[~2022-03-08 23:19 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-08 21:16 Guix for Corporate "Batch Jobs"? Yasuaki Kudo
2022-03-08 23:18 ` Phil [this message]
2022-03-09 8:20 ` Ricardo Wurmus
2022-03-09 8:49 ` Yasuaki Kudo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sfrs2ftx.fsf@beadling.co.uk \
--to=phil@beadling.co.uk \
--cc=help-guix@gnu.org \
--cc=yasu@yasuaki.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).