From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id SKb/Fj1j+2N0cwAAbAwnHQ (envelope-from ) for ; Sun, 26 Feb 2023 14:48:45 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id +DQhFj1j+2OBOwAAG6o9tA (envelope-from ) for ; Sun, 26 Feb 2023 14:48:45 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E2BF0D921 for ; Sun, 26 Feb 2023 14:48:44 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=tilde.club header.s=mail header.b=PwHEWIBu; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=tilde.club ARC-Seal: i=1; s=key1; d=yhetil.org; t=1677419325; a=rsa-sha256; cv=none; b=N7T8tqUKN7tbS+Cqkzhi8AdLpgOR1bfDlOYERZmh0JeOv8okAw6RYNJpFDpdb4oJ0J4zvl b9lJXCyaOiUy8UVLM7PBSTleL04YR7PbQQh/uMmwU6jRfmiNxlJLLo4uq1OGIkYfphE9Vn DoN5phYC49VyzJd88IUyI4+Pob3XT1OTcHiFSzSoEnlg/bU/wmMS1JdVFrYeKYLkeQkqvW rFvr0HKt5vvESGosQpxws1hsM4vh4E1lDcfkzzfCuS0rT4CHspo1DV6Fw5gVKAkuymBQnL 22zIbgjoNAJ8BLt8hixNuQYE3wqUyLuGN6wM2jqmZGDnHB7OnXLJz8yzD0z4BA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=tilde.club header.s=mail header.b=PwHEWIBu; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=tilde.club ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1677419325; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=0ZsLXPI5gWB2RuTYuAXgQUHJs6lbBiad3SiZaeFdnCM=; b=e2MNB1u05Bl/Wmo8fp25EP7YFloM7X0eA2EydXO0/xISYfcLr9FLjDTKponnuz8cw2l6XK V4jOuzITqAgJKscS4uhYX11C+yS1YWEe581F3+wHGUFY4sDO3MYx9eQ2TmOuPo3rjnzHq/ UqQbQSOF265Mr+vUXt46WyS+ChfX9nkKV6LBLau4ulaILDRek+onBTYI0wrRIICXNOreZG 6kbXWkRukedNHyaJtUDGFcAYfDU7OCiA4Usvhyv/eW+mY9QLD/7iiuaw1PkX1q2Gjaavac nZBU5ZKj3dDPYFdSAxfED4SpMdHFwza+JA72jXyvCOX6Sypzq+eV9Sq9nRGtQw== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pWHO1-00071A-GL; Sun, 26 Feb 2023 08:48:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pVu4T-0004U1-Ru for guix-devel@gnu.org; Sat, 25 Feb 2023 07:54:37 -0500 Received: from tilde.club ([2607:5300:204:4340::114]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pVu4O-0004nQ-27 for guix-devel@gnu.org; Sat, 25 Feb 2023 07:54:37 -0500 Received: by tilde.club (Postfix, from userid 5378) id 6243A22312293; Sat, 25 Feb 2023 12:54:31 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 tilde.club 6243A22312293 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tilde.club; s=mail; t=1677329671; bh=qWsQZHLybalAV+uzxd1ACNAQALAfpecF3NZ4g6KqIqE=; h=From:To:Subject:Date:From; b=PwHEWIBumAJpjsXaZ6mZxs+Zb82wOZ/DGTnmu9i/67h/8vTPz/iLhH5685WKSdjCo blY8pt3G+SIaAxJ/1LqYGMKeZrJDW0B/G1v7tgVRtfg7Eia3EREV3BsfODFHHS1iJ/ pPvp9xxEnOan6Z0g0ymSPNvg+dD5LMMlyG5snz10= From: Ulf Herrman To: guix-devel@gnu.org Subject: [PATCH 0/3] [shepherd] improve race-free spawn+wait User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) Date: Sat, 25 Feb 2023 06:54:25 -0600 Message-ID: <87pm9x29dq.fsf@tilde.club> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Received-SPF: pass client-ip=2607:5300:204:4340::114; envelope-from=striness@tilde.club; helo=tilde.club X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sun, 26 Feb 2023 08:48:17 -0500 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: X-Migadu-Queue-Id: E2BF0D921 X-Spam-Score: -12.19 X-Migadu-Spam-Score: -12.19 X-Migadu-Scanner: scn0.migadu.com List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-TUID: cjmIkEDpG2qm --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain These patches fill out shepherd's procedures for running processes to completion. They add a replacement for 'system' to complement the existing replacement for 'system*', and add a 'fork+exec+wait-process' procedure so that the flexibility of that family of procedures is available for this use case as well. It also improves error handling in the event that an exception occurs while spawning a process in the process monitor, which would normally kill that essential fiber. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-service-Propagate-exceptions-while-spawning-in-proce.patch Content-Transfer-Encoding: quoted-printable From=2064370a98dfc17f0531de7397a38362c03a1d89bc Mon Sep 17 00:00:00 2001 From: ulfvonbelow Date: Sat, 25 Feb 2023 00:42:41 -0600 Subject: [PATCH 1/3] service: Propagate exceptions while spawning in process monitor. * modules/shepherd/service.scm (unboxed-errors): new procedure. (boxed-errors): new syntax. (process-monitor): use it to propagate exceptions from fork+exec-command = via reply channel. (spawn-via-monitor): new procedure. (spawn-command): use it. =2D-- modules/shepherd/service.scm | 47 ++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 10 deletions(-) diff --git a/modules/shepherd/service.scm b/modules/shepherd/service.scm index fd2ef1b..196ee44 100644 =2D-- a/modules/shepherd/service.scm +++ b/modules/shepherd/service.scm @@ -1825,6 +1825,24 @@ otherwise by updating its state." ;; loop so we don't miss any terminated child process. (loop))))) =20 +(define-syntax-rule (boxed-errors exps ...) + (catch #t + (lambda () + (call-with-values + (lambda () + exps ...) + (lambda results + (list 'success results)))) + (lambda args + (list 'exception args)))) + +(define unboxed-errors + (match-lambda + (('success vals) + (apply values vals)) + (('exception args) + (apply throw args)))) + (define (process-monitor channel) "Run a process monitor that handles requests received over @var{channel}= ." (let loop ((waiters vlist-null)) @@ -1860,11 +1878,17 @@ otherwise by updating its state." waiters))) =20 (('spawn command reply) =2D ;; Spawn COMMAND; send its exit status to REPLY when it terminate= s. =2D ;; This operation is atomic: the WAITERS table is updated before =2D ;; termination of PID can possibly be handled. =2D (let ((pid (fork+exec-command command))) =2D (loop (vhash-consv pid reply waiters)))) + ;; Spawn COMMAND; send the spawn result (pid or exception) to REPLY; + ;; send its exit status to REPLY when it terminates. This operatio= n is + ;; atomic: the WAITERS table is updated before termination of PID c= an + ;; possibly be handled. + (let ((result (boxed-errors (fork+exec-command command)))) + (put-message reply result) + (match result + (('exception . _) + (loop waiters)) + (('success (pid)) + (loop (vhash-consv pid reply waiters)))))) =20 (('await pid reply) ;; Await the termination of PID and send its status on REPLY. @@ -1900,14 +1924,17 @@ context. The process monitoring fiber is responsib= le for handling @code{SIGCHLD} and generally dealing with process creation and termination= ." (call-with-process-monitor (lambda () exp ...))) =20 +(define (spawn-via-monitor command) + (let ((reply (make-channel))) + (put-message (current-process-monitor) + `(spawn ,command ,reply)) + (unboxed-errors (get-message reply)) + (get-message reply))) + (define (spawn-command program . arguments) "Like 'system*' but do not block while waiting for PROGRAM to terminate." (if (current-process-monitor) =2D (let ((reply (make-channel))) =2D (put-message (current-process-monitor) =2D `(spawn ,(cons program arguments) =2D ,reply)) =2D (get-message reply)) + (spawn-via-monitor (cons program arguments)) (apply system* program arguments))) =20 (define default-process-termination-grace-period =2D-=20 2.38.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-service-accept-fork-exec-command-argument-list-in-mo.patch Content-Transfer-Encoding: quoted-printable From=2051ee63ace6f3f52eb196c990664cc6b9af3d3683 Mon Sep 17 00:00:00 2001 From: ulfvonbelow Date: Sat, 25 Feb 2023 00:46:27 -0600 Subject: [PATCH 2/3] service: accept fork+exec-command argument list in monitor. Sometimes it's necessary to run startup / shutdown programs as a certain us= er, in a certain directory, with certain environment variables, etc. Shepherd currently provides a replacement for system* that won't race against the child process auto-reaper, but this lacks the flexibility Shepherd users are used to. * modules/shepherd/service.scm (process-monitor): treat command instead as argument list to fork+exec-command. (spawn-via-monitor): update to new convention. (fork+exec+wait-command): new procedure. =2D-- modules/shepherd/service.scm | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/modules/shepherd/service.scm b/modules/shepherd/service.scm index 196ee44..a36e486 100644 =2D-- a/modules/shepherd/service.scm +++ b/modules/shepherd/service.scm @@ -94,6 +94,7 @@ default-process-termination-grace-period exec-command fork+exec-command + fork+exec+wait-command default-pid-file-timeout read-pid-file make-system-constructor @@ -1877,12 +1878,12 @@ otherwise by updating its state." vlist-null waiters))) =20 =2D (('spawn command reply) + (('spawn args reply) ;; Spawn COMMAND; send the spawn result (pid or exception) to REPLY; ;; send its exit status to REPLY when it terminates. This operatio= n is ;; atomic: the WAITERS table is updated before termination of PID c= an ;; possibly be handled. =2D (let ((result (boxed-errors (fork+exec-command command)))) + (let ((result (boxed-errors (apply fork+exec-command args)))) (put-message reply result) (match result (('exception . _) @@ -1924,19 +1925,26 @@ context. The process monitoring fiber is responsib= le for handling @code{SIGCHLD} and generally dealing with process creation and termination= ." (call-with-process-monitor (lambda () exp ...))) =20 =2D(define (spawn-via-monitor command) +(define (spawn-via-monitor arguments) (let ((reply (make-channel))) (put-message (current-process-monitor) =2D `(spawn ,command ,reply)) + `(spawn ,arguments ,reply)) (unboxed-errors (get-message reply)) (get-message reply))) =20 (define (spawn-command program . arguments) "Like 'system*' but do not block while waiting for PROGRAM to terminate." (if (current-process-monitor) =2D (spawn-via-monitor (cons program arguments)) + (spawn-via-monitor (list (cons program arguments))) (apply system* program arguments))) =20 +(define (fork+exec+wait-command command . arguments) + "Like 'fork+exec' but also wait for PROGRAM to terminate, giving its exit +status." + (if (current-process-monitor) + (spawn-via-monitor (cons command arguments)) + (waitpid (apply fork+exec-command command arguments)))) + (define default-process-termination-grace-period ;; Default process termination "grace period" before we send SIGKILL. (make-parameter 5)) =2D-=20 2.38.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0003-service-add-spawn-shell-command-replacement-for-syst.patch Content-Transfer-Encoding: quoted-printable From=20177592ee9d4b7fc6dcc80e545e8ad615a1d6786c Mon Sep 17 00:00:00 2001 From: ulfvonbelow Date: Sat, 25 Feb 2023 00:56:57 -0600 Subject: [PATCH 3/3] service: add spawn-shell-command replacement for `system'. We already have a replacement for `system*' that avoids racing, but not for `system'. * configure.ac (SHELL): new substitution variable. * modules/shepherd/system.scm.in (%shell-filename): new variable. * modules/shepherd/service.scm (spawn-shell-command, real-system): new procedures. * modules/shepherd.scm (main): replace `system' with `spawn-shell-command'. =2D-- configure.ac | 1 + modules/shepherd.scm | 7 +++++-- modules/shepherd/service.scm | 13 +++++++++++++ modules/shepherd/system.scm.in | 5 ++++- 4 files changed, 23 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 6f681dc..19c177a 100644 =2D-- a/configure.ac +++ b/configure.ac @@ -32,6 +32,7 @@ guilemoduledir=3D"${datarootdir}/guile/site/$GUILE_EFFECT= IVE_VERSION" guileobjectdir=3D"${libdir}/guile/$GUILE_EFFECTIVE_VERSION/site-ccache" AC_SUBST([guilemoduledir]) AC_SUBST([guileobjectdir]) +AC_SUBST([SHELL]) =20 dnl Check for extra dependencies. GUILE_MODULE_AVAILABLE([have_fibers], [(fibers)]) diff --git a/modules/shepherd.scm b/modules/shepherd.scm index cce0507..1f6342e 100644 =2D-- a/modules/shepherd.scm +++ b/modules/shepherd.scm @@ -420,8 +420,10 @@ already ~a threads running, disabling 'signalfd' suppo= rt") =20 ;; Replace the default 'system*' binding with one that ;; cooperates instead of blocking on 'waitpid'. =2D (let ((real-system* system*)) + (let ((real-system* system*) + (real-system system)) (set! system* spawn-command) + (set! system spawn-shell-command) =20 ;; Restore 'system*' after fork. (set! primitive-fork @@ -430,7 +432,8 @@ already ~a threads running, disabling 'signalfd' suppor= t") (let ((result (real-fork))) (when (zero? result) (set! primitive-fork real-fork) =2D (set! system* real-system*)) + (set! system* real-system*) + (set! system real-system)) result))))) =20 (run-daemon #:socket-file socket-file diff --git a/modules/shepherd/service.scm b/modules/shepherd/service.scm index a36e486..f8df3a9 100644 =2D-- a/modules/shepherd/service.scm +++ b/modules/shepherd/service.scm @@ -81,6 +81,7 @@ handle-SIGCHLD with-process-monitor spawn-command + spawn-shell-command %precious-signals register-services provided-by @@ -1938,6 +1939,18 @@ context. The process monitoring fiber is responsibl= e for handling (spawn-via-monitor (list (cons program arguments))) (apply system* program arguments))) =20 +(define real-system system) + +(define* (spawn-shell-command #:optional command) + "Like 'system' but do not block while waiting for COMMAND to terminate." + (if (current-process-monitor) + (if command + (spawn-command %shell-filename "-c" command) + #t) + (if command + (real-system command) + (real-system)))) + (define (fork+exec+wait-command command . arguments) "Like 'fork+exec' but also wait for PROGRAM to terminate, giving its exit status." diff --git a/modules/shepherd/system.scm.in b/modules/shepherd/system.scm.in index 29357aa..4646e81 100644 =2D-- a/modules/shepherd/system.scm.in +++ b/modules/shepherd/system.scm.in @@ -41,7 +41,8 @@ unblock-signals set-blocked-signals with-blocked-signals =2D without-automatic-finalization)) + without-automatic-finalization + %shell-filename)) =20 ;; The constants. (define RB_AUTOBOOT @RB_AUTOBOOT@) @@ -328,3 +329,5 @@ Turning finalization off shuts down the finalization th= read as a side effect." exp ...) (lambda () (%set-automatic-finalization-enabled?! enabled?))))) + +(define %shell-filename "@SHELL@") =2D-=20 2.38.1 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQHIBAEBCAAyFiEEn6BUn0yca1D9JsMa1lV76sJM9mgFAmP6BQEUHHN0cmluZXNz QHRpbGRlLmNsdWIACgkQ1lV76sJM9miehgv9FWFCpVMJsxBMBZ0th1Rm7DAma/yc q1bevjBLmzP5q68WYcJ+vEfXhblrdr7w7usQ6uYlBSJNAgY87qi3dqSbrd3nxu6U qOiY369/YNvuRuPg16DGYpbk+r8ieKcgYi/WgxtzERJ6OazSy+ICJFfg1QBP5BNR zfcUaJTdcCG5osFQt+o8uREvLtz2oe7ZwagbEPA6y4oXoZr7ZcSmnCkhkmZvk+ew TEGYGnyq5cIZddBR1GAKEgDn5WK6wcKHPMBuZR3/5xCD9765N2r7LJt6hC80ArX7 NCMLA/8Ej+ka4c61UZDJgWL8etzF6ebfzeOrlUEaIcjhUjEiT244MVJED7Dhd5Zk GsNSC2vmlllfUpTHUOHPM/hVlCpd2qWngDv7iZU5IY1Fo0CT5zZWrpgWAo+16t9A S5oxi9K3O4dPffYSK6KlSA65lDxRsenoD/FmIXGts0TPhiZWTR4nZi/lenp++6bB F3S/wYkzF+0Jlc0Xt1XL8Slp++u6f8QEQXVr =cuVC -----END PGP SIGNATURE----- --==-=-=--