From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id gEWCOs0RJmX8agEA9RJhRA:P1 (envelope-from ) for ; Wed, 11 Oct 2023 05:09:02 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id gEWCOs0RJmX8agEA9RJhRA (envelope-from ) for ; Wed, 11 Oct 2023 05:09:02 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 823896820B for ; Wed, 11 Oct 2023 05:09:01 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=cYPq4qyS; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1696993741; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=7embXUi8ltgGfUVF8tM7ff61ZEFkLwGfqPtxhrsSbGk=; b=tYQ7dCrf1cNyGTha9DOTSPdtCH2drTDhV1KafDO6qtK7qSUwZONCMVQQodI2u0eagf2i7M 7XBWFlaZuFhGv8xqKrYv6WqeEbql5D+Td9tCU3UW0IuMHk69LEAWnNp79mo/HDEhOy3A/D kP8IcWkNgbsyJ4jFrqS1iw7PLFapJNrheLDr3VQQ53FWRSJlnmtYSJrdQh6xFM7E4CiG+m Q1GCRjnNehGHXA9/oOdfdoHMaYIAJ1jbliz0DGrhGucg7cK/E7AKpdKUnGcsEoUtUx/Cel lINc5QTgGrlNrkqmLGZPjHOKwAI/NQ/TcTDhsl96F+/gJTZJZj2GAKy+Fi1ZFw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1696993741; a=rsa-sha256; cv=none; b=C4XGBFsrtdthY1w4rXY3J+WV1wMCKE0GMjwUOO0cV55B5FXAMxHDkGwZH6g4oNcwhO7GeW IStgWBvESrwXLt9yH1DL0h0b7KpTqyoir3wGcjqPXqWdjtGMxOviXbTs/S4hgkUR7gMM2+ QSV0YTMmCCbdEtUX3VLapK8WJKfeUqDVl7T8wEaB5FoYS+T4aKqXtoeSsToIQecdT/DpxY KU7xx6UO4BzsyXLu5un0rGwSpzC/kumfLQ/vSgZU39W3gYsLXIVQnjrrvKfCwnKYXWH4NH 3qxiOBa5zRFz0Z0bCKo33LOS0vlmU56wtaxtQEwiXc0W3oKKSotQXHzqGiBuGg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=cYPq4qyS; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qqPaU-0006HL-5d; Tue, 10 Oct 2023 23:08:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qqPaS-0006H6-SJ for bug-guix@gnu.org; Tue, 10 Oct 2023 23:08:40 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qqPaS-0002LT-KV for bug-guix@gnu.org; Tue, 10 Oct 2023 23:08:40 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qqPan-0004cz-Q2 for bug-guix@gnu.org; Tue, 10 Oct 2023 23:09:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#54447: cuirass: missing derivation error Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 11 Oct 2023 03:09:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54447 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Mathieu Othacehe , guix-sysadmin , 54447@debbugs.gnu.org Received: via spool by 54447-submit@debbugs.gnu.org id=B54447.169699372717763 (code B ref 54447); Wed, 11 Oct 2023 03:09:01 +0000 Received: (at 54447) by debbugs.gnu.org; 11 Oct 2023 03:08:47 +0000 Received: from localhost ([127.0.0.1]:37272 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qqPaY-0004cR-PZ for submit@debbugs.gnu.org; Tue, 10 Oct 2023 23:08:47 -0400 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]:55709) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qqPaT-0004c8-Qo for 54447@debbugs.gnu.org; Tue, 10 Oct 2023 23:08:45 -0400 Received: by mail-qk1-x732.google.com with SMTP id af79cd13be357-774105e8c37so417035385a.3 for <54447@debbugs.gnu.org>; Tue, 10 Oct 2023 20:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696993694; x=1697598494; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7embXUi8ltgGfUVF8tM7ff61ZEFkLwGfqPtxhrsSbGk=; b=cYPq4qySscl2j1vD1Pmq8flmCQROFYNa2RwFcMatEd8SCaw2r9cRUmc8NBesUvWhMa sEK8KgkvWOIubhtbSdnAtNC8pztJa75OVZYq54gXzbZmwI6nXOnfz4jt5GXw6bdqJS7e c52IsiuQkeCvTpjzWk1XdJYqnrER/YD7GiaMlV+HwpdLLzdkczWg2XQdylrzRxkgZa5G necQSnrv6R3erMYvM/P4l+Gs7xY8mHaCsVsN8jikc5ZCXMcoVv0h8hhn4Ppr//h3TlPJ 52hftBjzUJHGZHbSwmpi/y/3LDbenFHXcSV2bDQjoo0ScPbha3+NfFkHjfLMWdOUtYpW 1TOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696993694; x=1697598494; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=7embXUi8ltgGfUVF8tM7ff61ZEFkLwGfqPtxhrsSbGk=; b=XkwiljdYa/jUzC42i1Bs5/P+F8L9ZfHLNcjshRmrVUnbts89WeZt8FWOfBwJ70E9Fi rjvlAPOeIeHHM85/cxSzWfoqY/TUBrPbO6TlkmXuQ/xbtXxtCSYRzcMUUbIC05odEAfr LKpZuS9mmrlyZNptAPWKMlf9v2t+ZneYxjxf5rTgrDXHlvFWhDjOqfuSpXDPs8tQfjPR +hbtbhv/gJpniTVN8zqYtuOkyVF/eZO3VnM5Y9OB1A0X3hTsFtblzoiyLuNHmWbFMLLQ FcWwMt21vBl01PwWvnMoztJLysmKTkfcIDl+IhzdNT80HSzjVHIA3jp8RerB+sxB8IFf v0Wg== X-Gm-Message-State: AOJu0Yzsad2i1muK90xATOu8JiKyF2S11fQU4fMy8ajLRflVzbWnfmmN c7+qsYffQHY6bGdN5vgzbgwQ31mQipYEgw== X-Google-Smtp-Source: AGHT+IHgf9b5lNIIhw/VPYl9f16YyZPdUVsAHZP45AwMp8qoNb+8vJmMdn4KznkkdTQKGUknjrqZzg== X-Received: by 2002:a05:620a:47b6:b0:765:9391:13e7 with SMTP id dt54-20020a05620a47b600b00765939113e7mr19980335qkb.49.1696993694557; Tue, 10 Oct 2023 20:08:14 -0700 (PDT) Received: from hurd (dsl-10-149-16.b2b2c.ca. [72.10.149.16]) by smtp.gmail.com with ESMTPSA id c27-20020a05620a165b00b0076cc0a6e127sm4838216qko.116.2023.10.10.20.08.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:08:13 -0700 (PDT) From: Maxim Cournoyer In-Reply-To: <87r0m2v5ih.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 10 Oct 2023 17:52:54 +0200") References: <877d8r4etz.fsf@gnu.org> <87r0m2v5ih.fsf@gnu.org> Date: Tue, 10 Oct 2023 23:08:12 -0400 Message-ID: <87mswpeu03.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Spam-Score: 5.73 X-Migadu-Queue-Id: 823896820B X-Migadu-Scanner: mx0.migadu.com X-Migadu-Spam-Score: 5.73 X-TUID: b4KjQQNBGbrB Hi Ludovic, Ludovic Court=C3=A8s writes: > Hello! > > Mathieu Othacehe skribis: > >> A lot of builds, among them ~20 system tests[1], are failing with: >> "cannot build missing derivation >> ?/gnu/store/hs6kp1lqgymhyp3jndc0dsp0pn4psgv0-gui-installed-desktop-os-en= crypted.drv?" >> errors. > > I have a disappointingly simple hypothesis for this. Remember that > =E2=80=9Cmissing derivation=E2=80=9D errors happen primarily for system t= ests. > > Turns out that =E2=80=98cleanup-cuirass-roots=E2=80=99 in maintenance.git= , used as an > mcron job, explicitly removes GC roots for things like *-os-encrypted > once they=E2=80=99re more than two days old, as well as GC roots for the > corresponding .drv. > > I think this was increasing the likelihood that a .drv would be GC=E2=80= =99d by > the time we run the test: under high load=C2=B9, it=E2=80=99s plausible t= hat a system > test wouldn=E2=80=99t be built within two days after it=E2=80=99s been qu= eued. > > I=E2=80=99m proposing the change below to address this; I don=E2=80=99t t= hink we need > =E2=80=98--gc-keep-outputs --gc-keep-derivations=E2=80=99 anymore now tha= t we keep > things in =E2=80=98guix publish=E2=80=99 cache first and foremost. > > Thoughts? Ah, so that mcron job is kind of a hack to hasten garbage collecting only *some* items faster than the default policy of 30 days? And we'd now avoid deleting selected .drv files while still deleting their outputs, so in the case something that needs it took more than 2 days to build, it could lead to having to rebuild the garbage collected outputs? I'm not sure if we need such a fancy hack with the 100 TiB of data we now have, but your fix seems reasonable (LGTM!) > In addition to the mcron job, Cuirass=E2=80=99s own =E2=80=98register-gc-= roots=E2=80=99 > procedure periodically deletes GC roots older than =E2=80=98%gc-roots-ttl= =E2=80=99 (30 > days in practice). That=E2=80=99s okay, except that it would be safer to= delete > GC roots for a .drv if and only if it=E2=80=99s been built already. Hm. I wonder if this could explain the other cases we've seen. It could be that building a derivation was interrupted or canceled for some reason, then 30 days elapsed, then was garbage collected, and after which it doesn't get recreated and we get the error of the missing .drv? --=20 Thanks, Maxim