From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kHAjJ8VqKGILlwAAgWs5BA (envelope-from ) for ; Wed, 09 Mar 2022 09:52:21 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id 4IXtH8VqKGLSJwAAG6o9tA (envelope-from ) for ; Wed, 09 Mar 2022 09:52:21 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EE488223B9 for ; Wed, 9 Mar 2022 09:52:20 +0100 (CET) Received: from localhost ([::1]:37942 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nRs3P-0002Te-5s for larch@yhetil.org; Wed, 09 Mar 2022 03:52:19 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40370) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nRs0f-0000Hw-8d for help-guix@gnu.org; Wed, 09 Mar 2022 03:49:29 -0500 Received: from mail1.g12.pair.com ([66.39.4.99]:12890) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nRs0d-0000Ez-Ei for help-guix@gnu.org; Wed, 09 Mar 2022 03:49:28 -0500 Received: from mail1.g12.pair.com (localhost [127.0.0.1]) by mail1.g12.pair.com (Postfix) with ESMTP id 9435171E5D0; Wed, 9 Mar 2022 03:49:26 -0500 (EST) Received: from smtpclient.apple (pw126233219027.20.panda-world.ne.jp [126.233.219.27]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail1.g12.pair.com (Postfix) with ESMTPSA id 531E57457DE; Wed, 9 Mar 2022 03:49:26 -0500 (EST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Yasuaki Kudo Mime-Version: 1.0 (1.0) Subject: Re: Guix for Corporate "Batch Jobs"? Date: Wed, 9 Mar 2022 17:49:23 +0900 Message-Id: References: <87sfrs2ftx.fsf@beadling.co.uk> In-Reply-To: <87sfrs2ftx.fsf@beadling.co.uk> To: Phil X-Mailer: iPhone Mail (19D52) Received-SPF: none client-ip=66.39.4.99; envelope-from=yasu@yasuaki.com; helo=mail1.g12.pair.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix@gnu.org Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1646815941; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=knWtAjJfQZ5EPerDVuMCrruIafaGIqvQQBC2moNgUC4=; b=QvUMyjbUErL7cS2susjRLJvxosox7Qo9267Pm63uq7N4lEA1Q7aSB3pfbQMKlsXe0I9LEQ BY9U1Q2Mm5IVBoljG1dAB64kIojWSPyz6/lcsC8rblAREAU4E+Sl/mm9VvT1DrJWL39Pvx 51dCH7OhpnDsffjEbxtLz78g2TMAS1GQWJL7YmvyGlWCquLOgtFwE+rOrlFpiWSNt0PqjW +2IQ2t7LmOX3F0mcPidz56si0IJ1OC8xFnxXPinjS3TvggqzLlx3h6U8zgRY5JXr+tRC1u dTxsQYfjOfTNFkYFLvwxHTldRsMABbw2Al77KaLzbVadA39pLc4gwVzy3ox52w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1646815941; a=rsa-sha256; cv=none; b=fNWLkcUY9E2FWqW9QjsmPWN1HDuI4Uocr9eU7JL8xGC2euegmENQeSDhMBd8I16mW/bK8x xexLgW0885df8+f6NYUEgxMJ0u8uN/ENH1GcWMTAfFG2zhhpVxDpifAg2RND1m7XztTtmB nG1x/9crAz3JlmuXlhz2BUdIJmjavgA2pARetvzExCtLuKSIR3TutIie6hju14Aq3AdNIp S26n/RC4W1P9zvN3tsP4tFHjr+j55BKGkz+H/556a7gd41TRjiHliKXYQFXbcdH6VLrHXx RlE2xzt/3VK45NwZxhBlgZ68DGGf5gUVhrJuL3MACih1ZZqqUPWHzBE7gJyOvQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.91 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: EE488223B9 X-Spam-Score: -1.91 X-Migadu-Scanner: scn0.migadu.com X-TUID: VSK3yqHeunGq Hi Phil, Thank you so much, yes, this does help! I was thinking of profit/loss simulations for millions of transactions at la= rge financial companies. They typically have purpose-built libraries writte= n in C and rely on server farms and beefy databases. The acceptable range o= f input for such systems are quite limited and they do fail due to bad data,= wrong assumptions of dates, business events, and so forth. And I am always looking for a good place to start for international worker c= ooperatives spread around the globe. Providing 24 hour "dev/op" network wit= h Guix as one of the core competencies might do=F0=9F=98=84 -Yasu > On Mar 9, 2022, at 08:18, Phil wrote: >=20 > =EF=BB=BFHi Yasu, >=20 > Yasuaki Kudo writes: >=20 >> Hi, >>=20 >> In many so-called Application Support jobs in the enterprises, one of the= core responsibilities is to see through the daily completion of "batch jobs= " - those I/O heavy processes that take a long time to run, even with parall= el processing. >>=20 >> And at the core of it is to "re-run" the jobs, after due troubleshooting.= >>=20 >> In many workplaces I have seen, teams ended up writing their own job sche= dulers based on cron or used proprietary software such as Autosys (and in Ja= pan, there are local brews such as A-Auto, if I remember the name correctly)= . >=20 > Not sure if this is exactly what you're looking for - but Guix in my > experience can sit at the centre of a tech-stack for providing software > on machines, and then batch-running that software in a very predictable wa= y. >=20 > However Guix is currenty first and foremost a command-line tool, so I > find myself augmenting it with other standard offerings to produce > familiar front-ends for triggers, job processing, management, etc. >=20 > A few examples below. >=20 > I oversee the use of Guix in an enterprise environment. Initially it > was used to build/test our software and also provide deployments with > dependencies etc. We wrapped Guix builds in Jenkins, which in-turn > integrates with our source control to trigger Guix using a standard > branch workflow developers are used to. Guix fetches and caches any > build dependencies making subsequent builds faster, and making artifacts > available via a Guix substitute server to servers across the enterprise. >=20 > More recently and probably more useful to you - I've been looking at > taking the build outputs and making them available as batch jobs using > Guix Workflow Language (https://guixwl.org) - which is a good fit if > your batches are compute jobs with well defined inputs, numerous > dependent stages, and the requirement to reproduce identical numerical > output. GWL provides lots of cool features - it's somewhat like Autosys > in that it is declarative - defining dependencies (and thus an order) > between different workflow processes etc. I don't think GWL can memoize > different processes in a workflow tho - so running a workflow several > times results in all workflow processes being run, as far as I know. > The point is you should be guaranteed the same result with the same > inputs, every time. >=20 > I tend to wrap the GWL scripts in Rundeck (job scheduler) to allow > less-technical staff to re-run batches through a web app or to construct > a daily schedule for overnight/regression tests etc, rather than use the > guix command line. >=20 > Note GWL isn't designed to be used if the aim of your batch jobs is to > have a side-effect on the server you're running on. We only use it to > produce results from calculations. This is different to Autosys where > each job could be entirely made-up of side-effects which change the > state of the server itself. >=20 > HTH, > Phil.