From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id SIy1Kn113GUvWQAAe85BDQ:P1 (envelope-from ) for ; Mon, 26 Feb 2024 12:26:53 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id SIy1Kn113GUvWQAAe85BDQ (envelope-from ) for ; Mon, 26 Feb 2024 12:26:53 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1708946813; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=RmBb2s/lmOyb+NSLQI1EpeZDbcLFsILZOtNjEfQtWqE=; b=hxQdDUn9mM/dyH//fDHDjGituqpS2lEn+ywLWxOXIz8+WYvLtdPZP6ENNOx26QiOFj7ts4 Bf3EPSMKo4VpRMTGDJ3CQN2P8iZrXLieCc69q7m0mhbcdxvTN1emwpLAb9rz+J06/7OU54 m0Nost7VXXY91LUrLMdED150n46JxgMi9cC9VzzDs9jyF0QYGKxnqpBQzyuC4Dn0pXFEXI tHHMPGwlUWT9OXkd2UBHQvk+WDr7N99mMwjDeDwwIFQgUkqGra9+8YINy2qlx+D2QhfyqL UX+FVPJdYUI7Z9vzs5di8YhBkvte+qw5Ixy+SHmJAXgpAZ6RgQpKYC91EEq+/Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1708946813; a=rsa-sha256; cv=none; b=vBQPLlHnRLZcSil8BwjkI0mbCLGYDYEDE4cMhM7b4Q/BOdO7P5QIcIf5mpl5ArKZfaJ7gc YhJ7DOWp9Cw/dxkJybf4FsTZ9WSLSuYTuzCtFuK1iAwpPqjwuneKbSfBZpzlQLTC2KL5oF dsTFV9m0juJdnWJNFzLtEpEyFo+UNdvJzOjGQb87RLVidNnD4ouqkh+vaTgpIS7o/P1XV1 W8BBtq5sWq2vcorwDWxbIbr/85keUsPUy/mkcmuGG95M/DilIb7dpgRLEPVINEmzUtkI4d AtlIeWeg+/8wT3L5GdSX46UvXn7B2s8LafDDnhJ97XKRAF8v3Kp4u4ccxRBpYg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9AAB642AC8 for ; Mon, 26 Feb 2024 12:26:53 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1reZ7e-0006NC-L8; Mon, 26 Feb 2024 06:26:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1reZ7c-0006MV-Fv for guix-devel@gnu.org; Mon, 26 Feb 2024 06:26:12 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1reZ7M-0002OV-UV; Mon, 26 Feb 2024 06:26:12 -0500 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id 2A4F427BBE2; Mon, 26 Feb 2024 11:25:53 +0000 (GMT) Received: from felis (localhost.lan [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 1ab072e0; Mon, 26 Feb 2024 11:25:52 +0000 (UTC) References: <87cysqn2ko.fsf@cbaines.net> <87ttlx6c71.fsf@gnu.org> User-agent: mu4e 1.10.8; emacs 29.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Subject: Re: February update on the Guile guix-daemon Date: Mon, 26 Feb 2024 11:05:23 +0000 In-reply-to: <87ttlx6c71.fsf@gnu.org> Message-ID: <87wmqr79he.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -6.05 X-Spam-Score: -6.05 X-Migadu-Queue-Id: 9AAB642AC8 X-Migadu-Scanner: mx13.migadu.com X-TUID: IX1PfuAXNs0r --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> - Then there's the big areas to work on next: >> >> - I think I'm going to need to use thread pools for SQLite operations >> in the daemon, as the build coordinator does. > > I think we should refrain from using POSIX threads directly and instead > use Fibers to its full extent. In this case, I=E2=80=99d use a resource = pool as > done in Cuirass (and in the Coordinator too, no? maybe that=E2=80=99s wha= t you > meant?). > >> - There's the low level work of setting up the build environment, the >> work on the guile-daemon branch helps a lot with this, but as >> pointed out by Ludo, there might be some issues with fork and >> similar operations in a Guile program using threads. > > Right. I think one of the things we discussed in Brussels is that, > since SQLite operations might block for a while, the daemon will have to > be multithreaded. Which means no fork/clone, which in turn probably > means delegating forking/cloning to a separate helper process. (That=E2= =80=99s > also what bubblewrap does, IIUC.) I've had a bit more of a think and have thought of some more things we can try, but I'm still not sure if it can be made to work. We can call (sleep 0) after every sqlite-step to allow other fibers to run if they're able to [1] and if we find places where sqlite-step takes too long, maybe we can use a short lived thread for this, but then make sure these threads have stopped before doing anything where they would cause problems (e.g. fork/clone). 1: https://github.com/wingo/fibers/wiki/Manual#31-blocking --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmXcdT1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdR8g//Xv43J9uYsdNhgJwVLK1TPmCECp2EGOPA sjcn/2fh9dqTZfDyqRzhtxZPKzyaFqyYi4P/9Dbga25lTK7CFRMpwN/SR3jCyu/Z aMx022WedKVXzT2NuBlF2DPU396e3mqcINWTKe9pAZj9gs1tuaJYOF4YhyUBDmI0 wU3VzmgZNuAW3fOyZ/dnRANLaroHW2cXFRaD7qtBOlIGtz60Q90PKTZYkDHByE+o d5JlECocxVT0roGOhEQF5ZkkAgKbN7MGeyu29YnGEbG8kEKTbKfyp/sHVl1iFNSU ucGKZdPP3qvbVMsGygeCw9L8+W3d1xfptn6SnFmXPPEDQR+lWjGbj/YIH6hkpLkt Uor4TTg6xsKTAZEpa4X7YC5pPKshIzPwoAw3BtDiZPEwPqQKVeOg9IuvnZtRvIND QI/HloC44dFA02tmnUYYsEvWEYFz1a6ispO1fJwVn/nbuAnd5W1V+lwZ/pndDslU IBVRcvdlrercQJGvtdUuSclqbDdrpShbny5kZSJW5y1z75hg+kiMxJoC56r+H/6T YdRXvi3WOOxXpAVDGaykvfEH6Y/mhdrYsLPywABKoxe4MXeknM2SCHqeSDmLv8+X 5WqUXOnEVo3hLMSR3lBH0r4y9xi4pK125WwYANIb07pY0D2JhBlYatx3Mh5cgW7e 1ETCC0vHmR4= =kVGI -----END PGP SIGNATURE----- --=-=-=--