From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id SFC1Amekb2dRhAAA62LTzQ:P1 (envelope-from ) for ; Sat, 28 Dec 2024 07:10:31 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id SFC1Amekb2dRhAAA62LTzQ (envelope-from ) for ; Sat, 28 Dec 2024 08:10:31 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=N0GZ7XKj; dkim=fail ("headers eddsa verify failed") header.d=russelstein.xyz header.s=ed25519 header.b=z9n1BMyr; dkim=fail ("headers rsa verify failed") header.d=russelstein.xyz header.s=rsa header.b="aez1yC/9"; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1735369827; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=ldcmPgyfQeG6UV21zmaHPxO9on3G7ugUGyD0nfNF3FQ=; b=ER5l2f31EFFDia29QnkLEbasDHqmyL14NX5/59Ajmbm/ArsGs49ROCSuDCqICuUTo9dqRM SnMk1j8/YvDkCRKpzmJUqyFojivUZiMMrmpn/8NjkNn2uVaKi/W65HpNBwKUmwM+CPAvnL FY6Is2ti2CWgQQR+VRCxsAH42Oimh1eFsNQqNTAIUkO1R6AXixPbFMNwyNnS8OrIJS+F37 G4scVqTHma9htCIYAJ1IVPvbhtkvttUAc5xbE9xPUfOwUVIVua0EtACiiTkw34VxDl0lV8 eo3cUSRiqtgWNz7nqNyzreadXxkanWpYRytP/BBuSKxjCeQlMzh3XEyKxEoXEw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=N0GZ7XKj; dkim=fail ("headers eddsa verify failed") header.d=russelstein.xyz header.s=ed25519 header.b=z9n1BMyr; dkim=fail ("headers rsa verify failed") header.d=russelstein.xyz header.s=rsa header.b="aez1yC/9"; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1735369827; a=rsa-sha256; cv=none; b=VDLa4cGWMhZXP8gXerirpK5MXdgLK1sADVgC0trBQ5Xkfty1jDwrVEptGCvNNEJxFeMr52 bVOCDrWZkOI+X2+cAkdZ5XbF/UUEusJnLJRzCLVfkzMFllyaiLoKzWp+fZrGn4CYO58XZX cBKW/0YHN5ei8ldchYWpHmmNwNtuzdLyENUubFHhMtDjsk35S6TrKMZ0UM45gKED8p9TDk z+5QqoRKUq/h1ooiG7vw7JicwG7759wzZhrzFyNW8lzJnI/vQ3nQtv/8bEcYTI6xxJA1zH QryAhQuwVsXAlJ2/BBYsZFt34D1WSEkK+fBsTo+VrvYOZ/Jxgd8qnZXbYF2NZw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D87071DFE5 for ; Sat, 28 Dec 2024 08:10:26 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tRQxb-0000G3-M9; Sat, 28 Dec 2024 02:10:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tRQxX-0000FO-6Q for bug-guix@gnu.org; Sat, 28 Dec 2024 02:10:03 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tRQxW-0007xD-Qv for bug-guix@gnu.org; Sat, 28 Dec 2024 02:10:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:References:In-Reply-To:From:To:Subject; bh=ldcmPgyfQeG6UV21zmaHPxO9on3G7ugUGyD0nfNF3FQ=; b=N0GZ7XKjUhwV5UN4w2h6ZNI9QZG65Iw4Kax0Z0/SJPPIGc8R9hM7FRuoiflgr4LpMfiUDEVFgtPOUZ1/Q8c09sBygNEnwXXNtyO9nI/UjMHXikWYv7wFZ1CX6zQL2wAYS+lNv/XJPu2xhk14SNJ7ttcPC60S9snc0I+3h7XUg0K/Wr0LJkZ9LsN0aPEhWaUH0Hl5R+g70v1D6FCN1b9K8BCzE3hyjAUZbUjq+lDbLTCjVbvfdk/mk00tQzBh4pe2PlwYTSXtU47UK3iYwXERCnSLdx9JkVQt931vFKWeJvFo7nEe+ElH8iMvmNLmoQwDan1s1NUprpXRxXXDwmiM+w==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tRQxW-00044t-Lc for bug-guix@gnu.org; Sat, 28 Dec 2024 02:10:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#31785: Multiple client 'build-paths' RPCs can lead to daemon deadlock Resent-From: Reepca Russelstein Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 28 Dec 2024 07:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 31785 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Christopher Baines , 31785@debbugs.gnu.org Received: via spool by 31785-submit@debbugs.gnu.org id=B31785.173536978415639 (code B ref 31785); Sat, 28 Dec 2024 07:10:02 +0000 Received: (at 31785) by debbugs.gnu.org; 28 Dec 2024 07:09:44 +0000 Received: from localhost ([127.0.0.1]:48542 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tRQxE-00044A-09 for submit@debbugs.gnu.org; Sat, 28 Dec 2024 02:09:44 -0500 Received: from mailout.russelstein.xyz ([209.141.47.21]:34444) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tRQxA-00043s-1S for 31785@debbugs.gnu.org; Sat, 28 Dec 2024 02:09:42 -0500 DKIM-Signature: v=1; a=ed25519-sha256; q=dns/txt; c=relaxed/relaxed; d=russelstein.xyz; s=ed25519; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ldcmPgyfQeG6UV21zmaHPxO9on3G7ugUGyD0nfNF3FQ=; b=z9n1BMyrJ18cYXggMXTDzC+PGZ u0QiwRzi0zGgd10xNykTBezsSUyHeq/FkHXNkzNr20DV4lS+im/w5VPK3SCw==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=russelstein.xyz; s=rsa; h=Content-Type:MIME-Version:Message-ID:Date: References:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ldcmPgyfQeG6UV21zmaHPxO9on3G7ugUGyD0nfNF3FQ=; b=aez1yC/9RkHSEUOE9CDaRUhFbb tSiIK54dnEp2fKSxBmtHAoqYCdLaSnyWjRyNZ7T0mP8O5HX2F+k/4gwuNunEPzp2tIlUEjnSTKtor Pv1RKmFR+frIi3PxwfugMdilR2/VD//M5Vow1z8lPMsqMBDahc6f+zdBg6DLeDayPbfDJBH28RGpj L42uOjWznSHl9QYWsp1OuQFPFTUL6Lp5sT3Z079AsQJNsVosKtmpZ/h1Yrbha0v2/qmiPlqNyW3p7 npTLL9MZcUeSjtB8xUc9odtlocLzVyqOWuyKwPldPjdH2LcTQKahqhF6rFG+nPIesRWX6ZWBX7MiF 4HQtbhqW8eVaZBLjqWAbB2ntQm+rA/yl+nw0ApKj+TxaiOs8VGd5rVvMH62IrQqYL6yx8bOMse49P +XsqMBb3oF1/vXqzPoCtT+Q7Ei/Ph5ZNBgheUguGRUf5s+yhp3ZcL/AQBLBKDRfdPtmtXH3FPhXzH tnZ289cvT170SvfH0jS0hf7RVtWQJT5x+XbjPuej1nC6FSzgJOgwlGsIhxlKXf5Bc03kIlQdIiJ5N DrhG2kIan+JAx5sL8qFWrKU9MCMFIEw8P1SFFTasZ6pDZUdCSRJIHEIZ/ucK6wDb2F0P8ITFhVWCn BebDSCjy1QxtiC58o3nx0/M4I+ZOGJRLc+36txqQ8=; Received: by russelstein.xyz with esmtpsa (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.98) (envelope-from ) id 1tRQv1-000000002ly-287S; Sat, 28 Dec 2024 01:07:28 -0600 In-Reply-To: <87bjwwyccp.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 27 Dec 2024 23:45:10 +0100") References: <87602ph0yv.fsf@gnu.org> <878qs9gg5k.fsf@gnu.org> <874j2xge0y.fsf@gnu.org> <87bjx16xej.fsf@russelstein.xyz> <87bjwwyccp.fsf@gnu.org> Date: Sat, 28 Dec 2024 01:06:52 -0600 Message-ID: <874j2o70c3.fsf@russelstein.xyz> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Reepca Russelstein From: Reepca Russelstein via Bug reports for GNU Guix Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -8.60 X-Spam-Score: -8.60 X-Migadu-Queue-Id: D87071DFE5 X-Migadu-Scanner: mx10.migadu.com X-TUID: X97fi/HlGm+X --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > I=E2=80=99m very much convinced by the patch. Yet it bothers me that I c= annot > reproduce the problem. I tried first with this test, which attempts to > reproduce what you describe in the commit log above: > But there=E2=80=99s no deadlock, and I think that=E2=80=99s because the p= roblem we=E2=80=99re > seeing has to do with substitute goals, and there=E2=80=99s no such goal = here. The problem we've been seeing in the wild has to do with substitute goals, yes, but the same problem also exists for derivation goals. Starting the 3 builds at basically the same time seems to be a bit too "nice", given that some effort has already been made to make the scheduler more deterministic, and that they all use the exact same mechanism for deciding how long to wait before retrying the locks. Also, with only 3 inputs, even if the assignment of processes to output path roles (e.g. builds, sits-on, blocked) were completely random, only 2 out of 8 possible assignments to the latter two roles result in a deadlock. While 3 inputs makes for a nice, simple demonstration of the problem, for reliably recreating it, we're probably going to want more. Also, if I understand correctly, the issue with destructors not being run in a timely manner is only for top-level goals. I've managed to create a deadlock using a derivation with 10 inputs, passing all of the inputs and the dependent derivation as top-level derivations to build-things. I've also changed the duration of each input derivation build from 3 seconds to 4 and added a 1-second sleep between starting each thread. I have only seen this arrangement fail to create a deadlock once, but for good measure, I've subsequently bumped up the number of inputs to 15. Note that this means that this test will require 23 seconds to pass (3 daemon processes, 1 job per daemon process, 4 seconds per build round, 3 jobs per build round, 15 jobs in 5 rounds =3D 20 seconds, plus 3 seconds for the dependent derivation). I also had to set #:verbosity 3 instead of 4 because I kept getting encoding errors that killed one of my threads. These are most puzzling because they occur even after changing everything to just use (current-error-port) and making sure to run (set-port-conversion-strategy! (current-error-port) 'escape) I suspect there may be some underlying bug in guix or guile. Also, I don't have make-custom-textual-output-port here; it appears to only be in guile-next. Here is the reproducer: =2D-8<---------------cut here---------------start------------->8--- ;; https://issues.guix.gnu.org/31785 (use-modules (guix) ((gnu packages) #:select (specification->package)) (srfi srfi-1) (ice-9 threads) (ice-9 match) (ice-9 textual-ports)) (define (nonce) (logxor (car (gettimeofday)) (cdr (gettimeofday)) (getpid))) (define input-drvs (map (lambda (n) (computed-file (string-append "drv" (number->string n)) #~(begin #$(nonce) (sleep 4) (mkdir #$output)))) (iota 15))) (define top-drv (computed-file "top-drv" #~(begin #$(nonce) (sleep 3) (pk 'deps: #$@input-drvs) (mkdir #$output)))) (%graft? #f) (let* ((drvs (cons top-drv input-drvs)) (builder (lambda (name lst) (call-with-new-thread (lambda () (with-store store (set-build-options store #:verbosity 3 #:max-build-jobs 1) (run-with-store store (mlet %store-monad ((lst (mapm %store-monad lower-object lst))) (built-derivations lst)))))))) (thread1 (begin (sleep 1) (builder 'thread1 drvs))) (thread2 (begin (sleep 1) (builder 'thread2 drvs))) (thread3 (begin (sleep 1) (builder 'thread3 drvs)))) (join-thread thread1) (join-thread thread2) (join-thread thread3)) =2D-8<---------------cut here---------------end--------------->8--- P.S: If in attempting to turn this into a proper test, you try using the timeout argument to join-thread, be aware that a second attempt at calling join-thread on the same thread will fail with "In procedure lock-mutex: mutex already locked by thread". This is because join-thread in (ice-9 threads) has a bug in it: unlock-mutex is not called in the "else" case of the cond. I am mentioning this here in case I forget to make a proper report of it. =2D reepca --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFLBAEBCAA1FiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAmdvo40XHHJlZXBjYUBy dXNzZWxzdGVpbi54eXoACgkQwWaqSV9/GJyiwAf+PIPwF7wE0fVHmD5KNzNsbN7V H7eOPm8VJJlKcowpGHM93qqP9vOqcP0r+8x/oYvzwkhDTTAscEQRdm1Nybd04vNj mnCbAksiJ2ULFaqQW4z6h8+UccCdm/o0BWsgtl+mJVaQ1Lc40L4RtScXELwDQoeF tamFGHCNeQwAbHEeBeYqmOobKv4wqKN9C0g1JMDPljJL0UXSWbXmL3tvmudcGYL3 D6rB8HBo7C9gWkKwrtO4X3OjI0mm9oaq6VBVbct63PQaAapbZxL+ZJt2mWsSDko7 EGnJyzy5bFZxHWAtfoLsYUk0swS1autVT8HNxq33/mkahw6bDRvczJUcK7abRQ== =RHfM -----END PGP SIGNATURE----- --=-=-=--