From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eLNFKukpnWHOMgEAgWs5BA (envelope-from ) for ; Tue, 23 Nov 2021 18:50:33 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2MfaJekpnWFvEwAAbx9fmQ (envelope-from ) for ; Tue, 23 Nov 2021 17:50:33 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0FDC1E476 for ; Tue, 23 Nov 2021 18:50:33 +0100 (CET) Received: from localhost ([::1]:39294 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mpZw8-0004mT-50 for larch@yhetil.org; Tue, 23 Nov 2021 12:50:32 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54356) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mpZui-000375-L6 for bug-guix@gnu.org; Tue, 23 Nov 2021 12:49:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:40665) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mpZug-0006If-Ln for bug-guix@gnu.org; Tue, 23 Nov 2021 12:49:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mpZug-0006eY-Fi for bug-guix@gnu.org; Tue, 23 Nov 2021 12:49:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: GC takes more than 9 hours on berlin Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Nov 2021 17:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Mathieu Othacehe Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.163768970025514 (code B ref 51787); Tue, 23 Nov 2021 17:49:02 +0000 Received: (at 51787) by debbugs.gnu.org; 23 Nov 2021 17:48:20 +0000 Received: from localhost ([127.0.0.1]:52211 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mpZu0-0006dS-9X for submit@debbugs.gnu.org; Tue, 23 Nov 2021 12:48:20 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53382) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mpZtw-0006dA-GB for 51787@debbugs.gnu.org; Tue, 23 Nov 2021 12:48:18 -0500 Received: from [2001:470:142:3::e] (port=56160 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mpZtr-0006D3-0U; Tue, 23 Nov 2021 12:48:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=D4+/pOhc4lcyyNqqozBds24SMdzwsCpWvMZZbi4l/uE=; b=WJCHv03maHI18x9v8FU8 GwW2B9/+ubBRJfRxV4wCZyznKqHXoZdxUiPVsXCQfM8COgAsaOXZaMFTbIwP4+u6zRN99Wte3KydD l69PKn0Kkbec9DwSYhbxFruA8Rr9trvqsLj9f2TOFJKeGsSLPYS3OzBxb9sgAEW4fBxOZ63ok2DDH yaBmmXHKUQodSI4DGuCPnzhU5+6wbOluwCzZvhBlkyigXUldkq3gI/DJ5HMJcwXk0B+VFYdKnNuoL jnBHDu3c6KJfJRyUWzwz5gT8Vt4RBbu9WcXuRhH4YDe91U/GIZiGiotvcfbfZ7/yq2qfU1cXaniDK dKmK1nIB9icDCA==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:51615 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mpZtq-0007WN-K0; Tue, 23 Nov 2021 12:48:10 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87o86pegr3.fsf@gnu.org> Date: Tue, 23 Nov 2021 18:48:08 +0100 In-Reply-To: <87o86pegr3.fsf@gnu.org> (Mathieu Othacehe's message of "Fri, 12 Nov 2021 11:49:04 +0000") Message-ID: <87zgpuhig7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 51787@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1637689833; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=D4+/pOhc4lcyyNqqozBds24SMdzwsCpWvMZZbi4l/uE=; b=IgdW/mWA7hV25ZYoggM8r0fOHL2+v/M0eSoJenXY6zc3VtV40aeu1ntmYFleIMUjuAGh++ HDCVVNMDh7A+O+1or7z+W5k1QwK1GrdcF/pmZ2TJQIdle6LiKgCmuAnh+JL4snAtMquQt5 bvdjv+POujZ2SKO6fqY1Jor1Y8RDSLrzt7r/EAWV0716uZAE2goFAzZH14kYGPJ1mCYpF1 1ccFNadX/9fokfgD4z7Yiy/I74ysUks94syiHNmMnWC2lNfUNw39RlZVr+KPqkG3K9KxKk QjEyZjbU2S2ppXS5FDwP+0nL4etyHIn4BWQ4sBhbmCdoieGY6ShrCadkBd0PeA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1637689833; a=rsa-sha256; cv=none; b=JXF5R5rIU69TWjAsl8LHwFI+EZ0TeJ4MoRW5mimJtD0uERP5Lpoj9eHoylHRNYKpp2Ew0F XLT8kKygs14NGtyDfKfQZ1brBccY4H9QIkAtEqErbeta4sFvx1xLKq7pLU5rOzhvsdHPUA vcyWRxSVDyb5Pcs6pLzVLgPKZxIPB+/8N8WsakLmXrjXar5bHJwfe8DJzDbiLLF7qbpRn7 l267Jv6exAeBu1QHhy22vrl7PG5kueLenMrf+kTdypDRrYHMxLR7WC9U6sPvmkLHHSdxWi OZIO0sz7ddr3IgIdcVObfc7jb8WzHU+psLpg3rRTXlyiZ2PBJ+fOSmCUMyrHZw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=WJCHv03m; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.59 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=WJCHv03m; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 0FDC1E476 X-Spam-Score: -3.59 X-Migadu-Scanner: scn0.migadu.com X-TUID: /a8SsivadCtH Hello! Mathieu Othacehe skribis: > On berlin, the daily GC command is still running whereas it was started > 9 hours ago. Some data points: =E2=80=A2 I deployed on berlin the new daemon featuring the faster =E2=80= =9Cdeleting unused links=E2=80=9D phase from on Nov. 20. However, that part runs after the GC lock has been released, so it=E2= =80=99s not really relevant (but it is relevant to I/O load and GC efficiency.) =E2=80=A2 When discussing together with Chris Baines yesterday on IRC, we found that the sqlite WAL file was 8=C2=A0GiB. I later ran: PRAGMA wal_checkpoint(TRUNCATE); which emptied it immediately. However, GC time wasn=E2=80=99t particul= arly different today. =E2=80=A2 =E2=80=98db.sqlite=E2=80=99 weighs in at 19=C2=A0GiB (!) so per= haps there=E2=80=99s something to do, like the =E2=80=9CVACUUM=E2=80=9D thing maybe. Chris? =E2=80=A2 Stracing the session=E2=80=99s guix-daemon process during GC su= ggests that most of the time goes into I/O from =E2=80=98db.sqlite=E2=80=99. It=E2= =80=99s not surprising because that GC phase is basically about browsing the database, but it does seem to take a little too long for each store item. =E2=80=A2 I haven=E2=80=99t checked recently but I recall that =E2=80=98g= uix gc --list-roots=E2=80=99 (or its C++ counterpart, =E2=80=98findRoots=E2=80=99) would take ages o= n berlin because of all the GC roots Cuirass registers. It may be that an hour or so goes into enumerating GC roots. Collecting garbage, Ludo=E2=80=99.