From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id QGcEJ27kJ2LHcwAAgWs5BA (envelope-from ) for ; Wed, 09 Mar 2022 00:19:10 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id uBr5H27kJ2KuUAEAG6o9tA (envelope-from ) for ; Wed, 09 Mar 2022 00:19:10 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DEB5114FE3 for ; Wed, 9 Mar 2022 00:19:09 +0100 (CET) Received: from localhost ([::1]:54878 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nRj6i-0001pY-De for larch@yhetil.org; Tue, 08 Mar 2022 18:19:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:48516) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nRj6W-0001pO-S3 for help-guix@gnu.org; Tue, 08 Mar 2022 18:18:56 -0500 Received: from [2a00:1450:4864:20::431] (port=33651 helo=mail-wr1-x431.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nRj6U-0002hJ-GD for help-guix@gnu.org; Tue, 08 Mar 2022 18:18:56 -0500 Received: by mail-wr1-x431.google.com with SMTP id j17so497722wrc.0 for ; Tue, 08 Mar 2022 15:18:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=references:user-agent:from:to:cc:subject:in-reply-to:message-id :date:mime-version; bh=bd0BFjY4p8ogCvbxDCPDcaQmtUQ769cBV51idNQq5jo=; b=XK0HKcK0zHVr4xZUCZNnfE0d3eBP3zHXOP9pgS8jeLQcsxjtW8CBPh+p763GO2Lrxr VManvfu6cpuzSw2nUC3Y5mHAK5PkOE0Q5q/fXpelQ7AP9c4jw9T7/qavEYqT44fhwdFR 7x23qu5GiwzF4GDOFeZPnFS79wm6qjK7vBsC7yVvQi/Cbky3ZlgobveyRBsq2fD6qL+e aMlhTj+FJt3G0bIU9s8oj/OUOkOUP+Hi7GEdjZtea/IE8Gm5nwKN8a8KpTSolmWyCtcU wJaKmZY9CD1ZVWvJcmtV0hZGj+PgOI70yS9k07xNGeqaVm3G8ZixrQ45/9nDjBFQ9A+6 HPEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:message-id:date:mime-version; bh=bd0BFjY4p8ogCvbxDCPDcaQmtUQ769cBV51idNQq5jo=; b=fU+WbYRpfVX01l5eR1aeTZ3Q2bv2Lj8uBWfWX4IPHf4Q5lEq0T5dnNLh85+qIfsKXv mrEJy2dUgwMEm+Ub82wvPfr+NVPMfXxlLjg+gwiTKgiV0sz54bYCrTnht3eFO675c5Ny kdo81K9fEWzTEeQ/XVtDFh6VIFUSC8fAhN/Afq26DaXkA937EjprK4AAXKHXFyYdJne2 mE9pKl+0I2jROAo1yhLRp3MrjvBkSGVIpjjouCcHRRpcjP5ZamKACu99+69BoRsIEtZv nwF2OWfY5FRE/hp732e6IKCSXwZVRYTx4raWY+GLnEEuIEIx3ZmV6Of9w74JntoykEpo AJBg== X-Gm-Message-State: AOAM531A/D7qobpIiVPTkjKteuVc627nhtLvOIcLi9h5MaXIerCc/Qsa EYnR2bRz83m18tN5kdhAz94q+ikZR7Pc2t4r X-Google-Smtp-Source: ABdhPJwv4yyRmkeuzEBZl5OdfGGNppw6PfP8hSXdIw8D1tq3d21Bu4aDloejl8B49p17YJxnJItocQ== X-Received: by 2002:a05:6000:156e:b0:1f0:5e29:6e28 with SMTP id 14-20020a056000156e00b001f05e296e28mr14258731wrz.706.1646781532428; Tue, 08 Mar 2022 15:18:52 -0800 (PST) Received: from xps13 (host-2-103-29-35.as13285.net. [2.103.29.35]) by smtp.gmail.com with ESMTPSA id f8-20020adfb608000000b0020229d72a4esm177428wre.38.2022.03.08.15.18.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 15:18:51 -0800 (PST) References: <506EEA75-0A98-4F04-9BC2-C7588FF9A65C@yasuaki.com> User-agent: mu4e 1.4.15; emacs 27.2 From: Phil To: Yasuaki Kudo Subject: Re: Guix for Corporate "Batch Jobs"? In-reply-to: <506EEA75-0A98-4F04-9BC2-C7588FF9A65C@yasuaki.com> Message-ID: <87sfrs2ftx.fsf@beadling.co.uk> Date: Tue, 08 Mar 2022 23:18:50 +0000 MIME-Version: 1.0 Content-Type: text/plain X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::431 (failed) Received-SPF: none client-ip=2a00:1450:4864:20::431; envelope-from=phil@beadling.co.uk; helo=mail-wr1-x431.google.com X-Spam_score_int: -4 X-Spam_score: -0.5 X-Spam_bar: / X-Spam_report: (-0.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, PDS_HP_HELO_NORDNS=0.659, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix@gnu.org Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1646781550; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=bd0BFjY4p8ogCvbxDCPDcaQmtUQ769cBV51idNQq5jo=; b=ZEGTdDe7UhszkaKiGvzMzhrOh23OUiiOY/qZ8sCZ0sZ8+iSWefGVZh1xKMvGbB/PPy/lzV 9kwOvnpI4uaFLc7wKoCK7GOkzdv9ZejBCf6li48DUNL9/tCUB4H9aesXgOCRyds1LD941t KAiyfCCu0AImxHKGdgZwOOlmRHi2scmSKcGA3+EpVidka1HTTq9B2jBkulokc88AccRV4W P5VUEZlL+DIkVnAcd4HC4mli2yD4MZ++53sBLq6u+IifEFcJ1whVpzrnk3sdH0+JRxjjRd GRTCIewIT/DQ87pkiX/HUl0HBtOEMoFJZ5ZnFBIsi9ke1K0nnt43sCDLPB3Qug== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1646781550; a=rsa-sha256; cv=none; b=NwL/HqlhEXdULBZe70elupJHKgu9wFnjta+3U2KK/q7XJqzVX7EKgN+xyJIyUBmXjzMGZp CItZ6GJSUDh+CoUNIqT34FRL9RK6di8AXCLThLtqNLnoX27sVWPAdh20Sqr1lNyivpEreT R0z2eUrgLF6cZCP5VuiLKK3/kHacGU5e5BIh6ZjF0+TZmAikuhqLDLn+tlIJnV1CNslX6t dqYEODSzcN2GWph05ial1TajbZtQPe++G19L482+L0rGYvf4Sd10iTWSXTWfw8L0zyE7Q5 uhZSn4M0VAMDrU1QqY7erMzySjXQpU1qQDQ1yMO4Fl+q1BBoqJP0KxcOhZaViA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=XK0HKcK0; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.71 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=XK0HKcK0; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: DEB5114FE3 X-Spam-Score: -2.71 X-Migadu-Scanner: scn1.migadu.com X-TUID: g0rcQIcQLdYY Hi Yasu, Yasuaki Kudo writes: > Hi, > > In many so-called Application Support jobs in the enterprises, one of the core responsibilities is to see through the daily completion of "batch jobs" - those I/O heavy processes that take a long time to run, even with parallel processing. > > And at the core of it is to "re-run" the jobs, after due troubleshooting. > > In many workplaces I have seen, teams ended up writing their own job schedulers based on cron or used proprietary software such as Autosys (and in Japan, there are local brews such as A-Auto, if I remember the name correctly). Not sure if this is exactly what you're looking for - but Guix in my experience can sit at the centre of a tech-stack for providing software on machines, and then batch-running that software in a very predictable way. However Guix is currenty first and foremost a command-line tool, so I find myself augmenting it with other standard offerings to produce familiar front-ends for triggers, job processing, management, etc. A few examples below. I oversee the use of Guix in an enterprise environment. Initially it was used to build/test our software and also provide deployments with dependencies etc. We wrapped Guix builds in Jenkins, which in-turn integrates with our source control to trigger Guix using a standard branch workflow developers are used to. Guix fetches and caches any build dependencies making subsequent builds faster, and making artifacts available via a Guix substitute server to servers across the enterprise. More recently and probably more useful to you - I've been looking at taking the build outputs and making them available as batch jobs using Guix Workflow Language (https://guixwl.org) - which is a good fit if your batches are compute jobs with well defined inputs, numerous dependent stages, and the requirement to reproduce identical numerical output. GWL provides lots of cool features - it's somewhat like Autosys in that it is declarative - defining dependencies (and thus an order) between different workflow processes etc. I don't think GWL can memoize different processes in a workflow tho - so running a workflow several times results in all workflow processes being run, as far as I know. The point is you should be guaranteed the same result with the same inputs, every time. I tend to wrap the GWL scripts in Rundeck (job scheduler) to allow less-technical staff to re-run batches through a web app or to construct a daily schedule for overnight/regression tests etc, rather than use the guix command line. Note GWL isn't designed to be used if the aim of your batch jobs is to have a side-effect on the server you're running on. We only use it to produce results from calculations. This is different to Autosys where each job could be entirely made-up of side-effects which change the state of the server itself. HTH, Phil.