From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id ML9XJwjizl4+FwAA0tVLHw (envelope-from ) for ; Wed, 27 May 2020 21:56:24 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 2N9lIwjizl42SAAA1q6Kng (envelope-from ) for ; Wed, 27 May 2020 21:56:24 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 26DA594030A for ; Wed, 27 May 2020 21:56:24 +0000 (UTC) Received: from localhost ([::1]:54728 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1je429-0005uL-R3 for larch@yhetil.org; Wed, 27 May 2020 17:56:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48340) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1je40s-00042q-QT for guix-patches@gnu.org; Wed, 27 May 2020 17:55:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:38677) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1je40s-00022a-Gu for guix-patches@gnu.org; Wed, 27 May 2020 17:55:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1je40s-0002SW-Gh for guix-patches@gnu.org; Wed, 27 May 2020 17:55:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#41119] [PATCH] fix some issues with (guix nar) Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 27 May 2020 21:55:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41119 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Caleb Ristvedt Cc: guix-sysadmin@gnu.org, 41119@debbugs.gnu.org Received: via spool by 41119-submit@debbugs.gnu.org id=B41119.15906164919432 (code B ref 41119); Wed, 27 May 2020 21:55:02 +0000 Received: (at 41119) by debbugs.gnu.org; 27 May 2020 21:54:51 +0000 Received: from localhost ([127.0.0.1]:50223 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1je40g-0002S3-Jm for submit@debbugs.gnu.org; Wed, 27 May 2020 17:54:51 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40870) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1je40e-0002Rp-6I for 41119@debbugs.gnu.org; Wed, 27 May 2020 17:54:48 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46627) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1je40Y-0001uT-LL; Wed, 27 May 2020 17:54:42 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37222 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1je40W-0003n9-9Q; Wed, 27 May 2020 17:54:42 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87h7wsqu50.fsf@cune.org> Date: Wed, 27 May 2020 23:54:38 +0200 In-Reply-To: <87h7wsqu50.fsf@cune.org> (Caleb Ristvedt's message of "Wed, 06 May 2020 22:52:11 -0500") Message-ID: <87k10xgi1t.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 0.49 X-TUID: IisHRJIPk8E5 Hi, Caleb Ristvedt skribis: > From 43ee61b405b01038b3e7c84aba64521ab8a62236 Mon Sep 17 00:00:00 2001 > From: Caleb Ristvedt > Date: Wed, 6 May 2020 11:52:16 -0500 > Subject: [PATCH 2/2] nar: 'with-temporary-store-file' uses a single conne= ction > > Previously the 'with-store' form was entered every time a different tempo= rary > file was tried. This caused there to be as many simultaneous open connec= tions > as there were attempts, and prevented the (loop ...) call from being a ta= il > call. This change fixes that. > > * guix/nar.scm (with-temporary-store-file): open connection once prior to > entering the loop. > --- > guix/nar.scm | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/guix/nar.scm b/guix/nar.scm > index f91af72879..404cef8b97 100644 > --- a/guix/nar.scm > +++ b/guix/nar.scm > @@ -126,8 +126,8 @@ held." > (define-syntax-rule (with-temporary-store-file name body ...) > "Evaluate BODY with NAME bound to the file name of a temporary store i= tem > protected from GC." > - (let loop ((name (temporary-store-file))) > - (with-store store > + (with-store store > + (let loop ((name (temporary-store-file))) > ;; Add NAME to the current process' roots. (Opening this connecti= on to > ;; the daemon allows us to reuse its code that deals with the > ;; per-process roots file.) This change had an undesirable effect: the connection would be kept for the body of =E2=80=98with-temporary-store-file=E2=80=99, during which we=E2= =80=99d call: finalize-store-file -> register-path which accesses the database. At this point, for each =E2=80=98guix offload= =E2=80=99 process, we=E2=80=99d thus have the database open twice: once for the sessi= on=E2=80=99s guix-daemon, and once for that =E2=80=98register-path=E2=80=99 call. On berlin, the effect is that we see many =E2=80=98guix offload=E2=80=99 pr= ocesses stalled because the SQLite database is busy: --8<---------------cut here---------------start------------->8--- ludo@berlin ~$ guix processes |grep ^SessionPID|wc -l 104 ludo@berlin ~$ guix processes |recsel -e 'ClientCommand ~ "offload"'|grep ^= SessionPID |wc -l 69 ludo@berlin ~$ guix processes |recsel -e 'ClientCommand ~ "offload"'|head=20 SessionPID: 10916 ClientPID: 7408 ClientCommand: /gnu/store/18hp7flyb3yid3yp49i6qcdq0sbi5l1n-guile-3.0.2/bin/= guile \ /gnu/store/abiva5ivq99x30r2s9pa3jj0pv9g16sv-guix-1.1.0-4.bdc801e/bi= n/.guix-real offload x86_64-linux 3600 1 21600 SessionPID: 11333 ClientPID: 9505 ClientCommand: /gnu/store/18hp7flyb3yid3yp49i6qcdq0sbi5l1n-guile-3.0.2/bin/= guile \ /gnu/store/abiva5ivq99x30r2s9pa3jj0pv9g16sv-guix-1.1.0-4.bdc801e/bi= n/.guix-real offload x86_64-linux 3600 1 21600 SessionPID: 16277 ClientPID: 9179 ludo@berlin ~$ sudo strace -p 7408 strace: Process 7408 attached restart_syscall(<... resuming interrupted read ...>) =3D 0 fcntl(19, F_SETLK, {l_type=3DF_RDLCK, l_whence=3DSEEK_SET, l_start=3D125, l= _len=3D1}) =3D 0 fcntl(19, F_SETLK, {l_type=3DF_WRLCK, l_whence=3DSEEK_SET, l_start=3D120, l= _len=3D1}) =3D -1 EAGAIN (Resource temporarily unavailable) fcntl(19, F_SETLK, {l_type=3DF_UNLCK, l_whence=3DSEEK_SET, l_start=3D125, l= _len=3D1}) =3D 0 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=3D0, tv_nsec=3D100000000}, NULL)= =3D 0 fcntl(19, F_SETLK, {l_type=3DF_RDLCK, l_whence=3DSEEK_SET, l_start=3D125, l= _len=3D1}) =3D 0 fcntl(19, F_SETLK, {l_type=3DF_WRLCK, l_whence=3DSEEK_SET, l_start=3D120, l= _len=3D1}) =3D -1 EAGAIN (Resource temporarily unavailable) fcntl(19, F_SETLK, {l_type=3DF_UNLCK, l_whence=3DSEEK_SET, l_start=3D125, l= _len=3D1}) =3D 0 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=3D0, tv_nsec=3D100000000}, NULL)= =3D 0 fcntl(19, F_SETLK, {l_type=3DF_RDLCK, l_whence=3DSEEK_SET, l_start=3D125, l= _len=3D1}) =3D 0 fcntl(19, F_SETLK, {l_type=3DF_WRLCK, l_whence=3DSEEK_SET, l_start=3D120, l= _len=3D1}) =3D -1 EAGAIN (Resource temporarily unavailable) fcntl(19, F_SETLK, {l_type=3DF_UNLCK, l_whence=3DSEEK_SET, l_start=3D125, l= _len=3D1}) =3D 0 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=3D0, tv_nsec=3D100000000}, ^Cstr= ace: Process 7408 detached ludo@berlin ~$ sudo gdb -p 7408 [=E2=80=A6] (gdb) bt #0 0x00007f2e2aa327a1 in clock_nanosleep@GLIBC_2.2.5 () from target:/gnu/s= tore/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/libc.so.6 #1 0x00007f2e2aa37c03 in nanosleep () from target:/gnu/store/fa6wj5bxkj5ll= 1d7292a70knmyl7a0cr-glibc-2.31/lib/libc.so.6 #2 0x00007f2e2aa611a4 in usleep () from target:/gnu/store/fa6wj5bxkj5ll1d7= 292a70knmyl7a0cr-glibc-2.31/lib/libc.so.6 #3 0x00007f2e1e8245ea in unixSleep () from target:/gnu/store/807c6g9xqrxdj= yhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #4 0x00007f2e1e81f56e in sqliteDefaultBusyCallback () from target:/gnu/sto= re/807c6g9xqrxdjyhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #5 0x00007f2e1e81f5d9 in sqlite3InvokeBusyHandler () from target:/gnu/stor= e/807c6g9xqrxdjyhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #6 0x00007f2e1e877ec1 in sqlite3BtreeBeginTrans () from target:/gnu/store/= 807c6g9xqrxdjyhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #7 0x00007f2e1e89fc64 in sqlite3VdbeExec () from target:/gnu/store/807c6g9= xqrxdjyhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #8 0x00007f2e1e8a6d09 in sqlite3_step () from target:/gnu/store/807c6g9xqr= xdjyhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #9 0x00007f2e1e8a7add in sqlite3_exec () from target:/gnu/store/807c6g9xqr= xdjyhm8wm1r6jjjmc8q4vs-sqlite-3.31.1/lib/libsqlite3.so.0 #10 0x00007f2e2af0466d in ffi_call_unix64 () from target:/gnu/store/bw15z9k= h9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7 #11 0x00007f2e2af02ac0 in ffi_call_int () from target:/gnu/store/bw15z9kh9c= 65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7 #12 0x00007f2e2aff148e in scm_i_foreign_call () from target:/gnu/store/18hp= 7flyb3yid3yp49i6qcdq0sbi5l1n-guile-3.0.2/lib/libguile-3.0.so.1 --8<---------------cut here---------------end--------------->8--- They loop pretty much indefinitely on this and nothing (or very little) happens on the system. I=E2=80=99ll revert this patch but I=E2=80=99m happy to hear what you think= , Caleb. Another reason to implement temp roots in Scheme, as it would allow us to not open a connection to the daemon from =E2=80=98guix offload=E2=80=99! Thanks, Ludo=E2=80=99.