From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id iKmwHRHEPWKT6wAAgWs5BA (envelope-from ) for ; Fri, 25 Mar 2022 14:30:57 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id SR3/GhHEPWI2FwAA9RJhRA (envelope-from ) for ; Fri, 25 Mar 2022 14:30:57 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E3A65160FF for ; Fri, 25 Mar 2022 14:30:52 +0100 (CET) Received: from localhost ([::1]:42888 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nXk1j-0001oi-Fn for larch@yhetil.org; Fri, 25 Mar 2022 09:30:51 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43736) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nXk0B-0001lq-EA for guix-devel@gnu.org; Fri, 25 Mar 2022 09:29:15 -0400 Received: from [2607:f8b0:4864:20::735] (port=35390 helo=mail-qk1-x735.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nXk09-0004eh-8p; Fri, 25 Mar 2022 09:29:14 -0400 Received: by mail-qk1-x735.google.com with SMTP id g8so5952848qke.2; Fri, 25 Mar 2022 06:29:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=AJE0m2324jvIgBTJ8BGVAWY2JEGZ3oOlmNOuNeWrXGo=; b=ACoWrn0QLrRJ8rFW20AXub9opBqfCrRqV0p/cRwO6W930gg3pt2IQPfTMCz0SKOUHw OlQKbbpj7aUy+i7LqQ7trW67m0JFm0uNQN0XCGo13ewdf4g/tklEw2w6D7C3VZ4+laz8 XM1oNHiHtPMUuGyKQG7cl54oNgE/cPvw1InK6+ClvLonSxqUVW+dqalvk64yiUNPoDT6 fkIXdxB/WNhTLb0aWOKkaKkvZXT4W+euSH6KisRZlgUcIbrikGYo1GgQB+su2AwBVfob h0bsi/3c+kUHnUXVNMmAYEdMS2QRfJeN4/0/gUEBtTurIlzsrJ5yas0nRiXTr6pYrrHB 2HHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=AJE0m2324jvIgBTJ8BGVAWY2JEGZ3oOlmNOuNeWrXGo=; b=D3+dgTGg/FVRpT31fX1kxVe0bJ1BVKSkjb8m90+YUnFCDVKAvOMbhBLvd5zerjr2rL yEKkA9w1X5fh+iur5Sn09d4DZP0F9BTCJptMQwcyI0e6zwY0KduElGPK/tIvnIllMh9k tLOIAM1uAwojasken22WOeNZB8iWh2mdx6W8aXJzIxfmljRjLkKHPd83jWcCrfJuBf5t LiaQEHH6Z+XvyXqx/m2RXOFudt91/U1C5I6G0fCa0uGFBWLaVdZnLehZjg+ZoAS+tFga 39dLYuzTNZcaS+kOX+1GUcMHkN0en0JlMM/YPQanPtqozRrRuxvbLLrKKOzPW2zww11z v43g== X-Gm-Message-State: AOAM530Diym4P5DJuSzY34Sb67G6wZcZZWFmGhetqa/4xv/rY+EDzgmw g0w0KJT+XcRsuYGX5esrcxr72XZqQBo= X-Google-Smtp-Source: ABdhPJwlbMInavE0aX4OEIWUoUeDDusVkSE/IV71TJR3UJE7Gm37mQ5krpL+gINwzivXK1sLxe6D4w== X-Received: by 2002:a05:620a:2487:b0:67b:3110:b5ca with SMTP id i7-20020a05620a248700b0067b3110b5camr6884178qkn.279.1648214950464; Fri, 25 Mar 2022 06:29:10 -0700 (PDT) Received: from hurd (dsl-157-48.b2b2c.ca. [66.158.157.48]) by smtp.gmail.com with ESMTPSA id m14-20020a05622a054e00b002e2072cffe6sm4675829qtx.5.2022.03.25.06.29.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 06:29:10 -0700 (PDT) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: The Shepherd on Fibers References: <87ee2sfg9d.fsf@inria.fr> Date: Fri, 25 Mar 2022 09:29:09 -0400 In-Reply-To: <87ee2sfg9d.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Wed, 23 Mar 2022 23:36:30 +0100") Message-ID: <87cziai2je.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Host-Lookup-Failed: Reverse DNS lookup failed for 2607:f8b0:4864:20::735 (failed) Received-SPF: pass client-ip=2607:f8b0:4864:20::735; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qk1-x735.google.com X-Spam_score_int: -6 X-Spam_score: -0.7 X-Spam_bar: / X-Spam_report: (-0.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_HP_HELO_NORDNS=0.659, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1648215053; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=AJE0m2324jvIgBTJ8BGVAWY2JEGZ3oOlmNOuNeWrXGo=; b=hfSUh02uwU0qKz7XX2OHyikzB8WfS27nCb4BHkUsm4IfHvxTJo54/kGu4lJxOPtZ1DO8jS LFf2tn3gT4p6u6pS2BkwbeT72U24hF+CpeHj7EMoJ2E45DPOcEU1vmaKS/O6SVr0usiB0Z 6wP/HP5xbG17Yr5JCkaph10yqfIm98Ro1LgMFyuM20Le0Gj64fwHfbdZ97FRpNFH0et1JZ h4dDIjmxaWe2+kVJbRRSt3iC01/dZbc0wGvr/68e/LuaIOO3+ajvY8IihwuTO5YqgmDa2b v4QTW/IQTfCJKDUlIsz/zbmh+NrWjQUxm0ivUkuFY+YWGi2YyLmKSu4o6Y+AZQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1648215053; a=rsa-sha256; cv=none; b=PeQ3BvEYntnx5arszrJg5vZVLc8BJhOuvGxxfo1Tws+hubPuGeEZgg7taSiKtFy1ZpJ5rC ssUH8QMc2+WSxNUa7Z6cFmzxFWzPjKOdjhukgw3uqRHojZP6nDzHktDxUqXCeKTVWOIAcL yBN/tqc/0NbFdA5BO1VZnve+IcQNL3Cy6MH2HrTqdK5sTJVMyT+JZjGFfK5N0uNreBsVL0 D5qUm9QsYkNmvVrUYykbvDa8jWv74MjFxoCJIUMZpYOn9x70yvvlOCB822C+1ABUIMc40W R+d7x2djPOba3Rtk3nCn+bBhAuILIEQsalqeRA3/6x8eEVVFAhholVegTqd32Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ACoWrn0Q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.88 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ACoWrn0Q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: E3A65160FF X-Spam-Score: -5.88 X-Migadu-Scanner: scn0.migadu.com X-TUID: hD3ijP9WZcvW Hi Ludo! Ludovic Court=C3=A8s writes: > Hello Guix! > > I have pushed a =E2=80=98wip-fibers=E2=80=99 branch of the Shepherd: > > https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=3Dwip-fibers > > The goal is to make shepherd (the daemon) use Fibers=C2=B9 for concurrenc= y. Very exciting work! I can't wait to have my 5 min boot reduced (mostly busy-waiting on the child-hurd VM to start) ;-). > To illustrate that, it introduces a new, incredible feature: logging! > As you know, shepherd had (euphemism ahead) limited support for logging, > essentially in the form of #:log-file, which would redirect a daemon=E2= =80=99s > stdout/stderr to a file, and also in the form of =E2=80=9Clet=E2=80=99s h= ope the daemon > talks to syslogd=E2=80=9D. Here I was able to trivially add logging, suc= h that > anything spawned by =E2=80=98fork+exec-command=E2=80=99 is logged, with t= imestamps and > all (the logger is a fiber that calls =E2=80=98read-line=E2=80=99 in a lo= op, which is > automagically non-blocking=E2=80=94delightful!). The next fun step (besi= des > logging) will be adding inetd-style and/or systemd-style =E2=80=9Csocket > activation=E2=80=9D. Interesting; in a (unmerged) patch I had made to mcron, I had extended the select loop in a way that it'd detect available output and use a suspendable port to accomplish this; perhaps Fibers would be a cleaner/more straightforward way to accomplish this. Also, I was particularly happy about how flexible the timestamp/metadata prefixed to each line could be configured by a user; you may want to steal ideas from it! [0] [0] https://lists.gnu.org/archive/html/bug-mcron/2021-08/msg00008.html > Fibers is used in a single-threaded fashion, which is the main > constraint for shepherd since it forks. That also means that fibers > cannot be preempted, so it=E2=80=99s fully cooperative scheduling. I see. I was puzzle by this configure.ac check: --8<---------------cut here---------------start------------->8--- +dnl Check for extra dependencies. +GUILE_MODULE_AVAILABLE([have_fibers], [(fibers)]) +if test "x$have_fibers" !=3D "xyes"; then + AC_MSG_ERROR([Fibers is missing; please install it.]) +fi + +dnl Make sure Fibers does not create POSIX threads: since shepherd +dnl forks, it must be single-threaded. +AC_CACHE_CHECK([whether Fibers might create POSIX threads], + [ac_cv_fibers_creates_pthreads], + [GUILE_CHECK([retval], + [(use-modules (fibers)) + (set! (@ (ice-9 threads) call-with-new-thread) + (lambda _ (throw 'new-thread!))) + (run-fibers (lambda () (spawn-fiber (lambda () 1))) + #:parallelism 1 #:hz 0)]) + if test "$retval" =3D 0; then + ac_cv_fibers_creates_pthreads=3D"no" + else + ac_cv_fibers_creates_pthreads=3D"yes" + fi]) +if test "x$ac_cv_fibers_creates_pthreads" =3D "xyes"; then + AC_MSG_ERROR([Fibers creates POSIX threads behind our back; aborting.]) +fi --8<---------------cut here---------------end--------------->8--- In which scenario would Fibers create unwanted POSIX threads? Is it that older versions of Fibers had that problem? If so, the message could be more explicit (Fibers too old, you need Fibers >=3D N). > There=E2=80=99s one catch: Fibers is currently Linux-only. The good news= is > that work has been done to port it to other kernels via libevent=C2=B2. > Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd. Sounds promising. > I=E2=80=99ve done some Guix System testing in VMs and didn=E2=80=99t noti= ce any major > issues. I=E2=80=99d like to merge that branch in =E2=80=98master=E2=80= =99 and to eventually > release it as 0.9.0 (with or without socket activation, we=E2=80=99ll see= .) > Hopefully, we could be running it within a couple of weeks. > > > Thoughts? Nice! Here's a very naive review (I'm not yet comfortable with the continuation barriers and others Fibers specifics): 1. In fae59fb * build: Capture the source and object directories of Fibers. Why must we even capture the Fibers source/compiled objects directories? Can't we let Guile and GUILE_LOAD_PATH (or the FHS) handle this for us? Messing with the load path system dynamically directly is usually a code smell, in my book. 2. nitpick; in "service: 'read-pid-file' no longer blocks": --8<---------------cut here---------------start------------->8--- -;; Copyright (C) 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludo= vic Court=C3=A8s +;; Copyright (C) 2013-2022 Ludovic Court=C3=A8s --8<---------------cut here---------------end--------------->8--- While I agree this reads better, I've recently stumbled in info '(maintain) Copyright Notices' on: You can use a range (=E2=80=982008-2010=E2=80=99) instead of listing = individual years (=E2=80=982008, 2009, 2010=E2=80=99) if and only if: 1) every year in th= e range, inclusive, really is a =E2=80=9Ccopyrightable=E2=80=9D year that would b= e listed individually; _and_ 2) you make an explicit statement in a =E2=80=98READ= ME=E2=80=99 file about this usage. So you'd want to add an explicit statement in the README about it. 3. I was a bit skeptical about the ability to configure the encoding of the log file; in which situation would switching it to something else than UTF-8 be useful? Thanks! I look forward to see this in mainline. Maxim