From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id EI/WEcXAWmF7LQEAgWs5BA (envelope-from ) for ; Mon, 04 Oct 2021 10:52:21 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id CBRjDcXAWmEJBQAAbx9fmQ (envelope-from ) for ; Mon, 04 Oct 2021 08:52:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0677721BF1 for ; Mon, 4 Oct 2021 10:52:21 +0200 (CEST) Received: from localhost ([::1]:34086 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mXJhs-0007nN-14 for larch@yhetil.org; Mon, 04 Oct 2021 04:52:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50700) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mXJDa-0001Vd-Hn for guix-patches@gnu.org; Mon, 04 Oct 2021 04:21:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:52262) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mXJDZ-0000a9-MH for guix-patches@gnu.org; Mon, 04 Oct 2021 04:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mXJDZ-0007d7-J8 for guix-patches@gnu.org; Mon, 04 Oct 2021 04:21:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50960] [PATCH 10/10] shell: Maintain a profile cache. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 04 Oct 2021 08:21:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 50960@debbugs.gnu.org Received: via spool by 50960-submit@debbugs.gnu.org id=B50960.163333560629237 (code B ref 50960); Mon, 04 Oct 2021 08:21:01 +0000 Received: (at 50960) by debbugs.gnu.org; 4 Oct 2021 08:20:06 +0000 Received: from localhost ([127.0.0.1]:35575 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mXJCX-0007ap-GO for submit@debbugs.gnu.org; Mon, 04 Oct 2021 04:20:06 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45276) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mXJCV-0007ab-CG for 50960@debbugs.gnu.org; Mon, 04 Oct 2021 04:19:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:44292) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mXJCP-0006gF-8i; Mon, 04 Oct 2021 04:19:49 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=60526 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mXJCJ-0003H1-De; Mon, 04 Oct 2021 04:19:49 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20211002102240.27815-1-ludo@gnu.org> <20211002102240.27815-10-ludo@gnu.org> <91012c32e4f8b04beae71378d44787a81055391a.camel@telenet.be> <87ilyfy0q4.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 13 =?UTF-8?Q?Vend=C3=A9miaire?= an 230 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 04 Oct 2021 10:19:41 +0200 In-Reply-To: (Maxime Devos's message of "Sat, 02 Oct 2021 16:47:20 +0200") Message-ID: <87lf39ryle.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1633337541; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=mlYZqTeBhBZex1tFGNcKw2z6DpAd9IJJFXu5cO3z78A=; b=r0Ztar7DloowVZpzvVwyaW7JbhsaSzOtey2L212rP1wMhzHppUvjk0HavW/b0Hin6bvgjI 6k6PRwhnCcHwpVIIvgcS9l2ze512Cty0DnwCmo/UKYLeStdwt7oBhSBHyKFVlzbn5cInc2 eo1e60VWriwLm/uGKkPfks+lQckqBFGztkbaFcuZYvZTLU6jNRyLK0Kv9S8b/xqIAJUyRj 8OuXmftiYhJIe6CGjiJa2nmNKcDdCpzRM7uDACJCP9fFQ1Vh3zf+FvHcl5VYrRleRCXo2Q jTOL6eja8+qe6W0cYXtf4HFqzotonN+GfqK2ZBEQNiyhhhCsVWawDHZ/D+shTA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1633337541; a=rsa-sha256; cv=none; b=V38C+ZtywvnOjJUngFtO+aeaGnCCCmoNKaZwiToTIQL5rOpPA2fJlWg+Phx9jOkcTAG9mw AYYOMAh5UmvOzui7dCxYPbxx8cZ71sa2FU9BSJAtLH7D/C2nKwjGy/lPduxXRXwLyi+9bf UH/L2Cu7Fbqs7AgWqSLLdTxCwbbmNqI4vFiNv2BRwtmyg96JKrn0WIxqjbSUN5YLJR4Ozu X9fnhnoukC1PhF0wCGaqYdzrIDp1fer13FWyysjoa9s8kPYWd7lZzY/F5FL00CP6XrjO83 MGXgWW7Y3gVTSrnEv8QuPqZMuMtmUraaZeFvmj4fbfXK5r0/oAFm6C+xkIJ5dw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.91 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 0677721BF1 X-Spam-Score: -2.91 X-Migadu-Scanner: scn0.migadu.com X-TUID: lqSz7sxxz+t6 Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op za 02-10-2021 om 16:12 [+0200]: > [reordered] >> > So I think the cache should also check if these dependencies have been= modified. >> > To keep track of the dependencies, something like the =E2=80=98compile= -all,compile: >> > Keep track of dependencies of compiled modules.=E2=80=99 patch from >> > could be used, though the 'include= ', 'load', >> > 'include-from-path' and maybe 'use-modules' (if something like "guix s= hell -Lextra-modules-directory ..." >> > is done) macros would need to be replaced. >>=20 >> Problem is that any attempt to keep track of dependencies is always an >> approximation because macros can do anything=E2=80=94I can have my own m= acro >> that reads files at expansion time. So I=E2=80=99m inclined to not even= try. > > I expect most people use 'include', 'load', 'include-from-path' or 'load-= from-path' > instead of writing their own macro reading files at expansion time, and i= f they do > write their own macro, I expect it would be implemented in terms of the f= ormer anyway, > so I expect replacing these macros and 'use-modules' to be sufficient, es= pecially > if the flag proposed below is available as =E2=80=98escape hatch=E2=80=99= for when the dependency > tracking is insufficient. > > This is not merely an expansion-time problem, files loaded at 'load' or '= eval' time > are important as well. E.g., consider the case where the manifest is som= ething like > > (list > (package > (inherit stuff) > (source (local-file "stuff" #:recursive? #t))) > other-packages ...) In a =E2=80=98guix.scm=E2=80=99 file, =E2=80=98source=E2=80=99 doesn=E2=80= =99t matter. It does matter in a manifest, but that sounds a bit less common. > then all files inside 'stuff' are important as well. I don't see how dep= endencies > could be tracked here without an excessive amount of 'stat' calls, maybe = guix should > ignore these dependencies (possibly with a warning, and a reference to th= e manual > documenting which dependencies are tracked and which are not?).=20 [...] > A documented flag to always consider the cache stale seems good, though I= think > at least the dependencies made with the common macros and procedures 'inc= lude', > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recu= rsive > 'local-file' could be tracked, though this could be left as a TODO for la= ter > I suppose. Tracking those uses reliably is impossible: there could be same-named bindings that do other things, there could be custom macros, there could be =E2=80=9Cdynamic arguments=E2=80=9D (whose value is not known statically= ), etc. You have to expand + evaluate the code to get better results, and even then, there might be different paths in the code so you can=E2=80=99t be sure you= got it right. We could get an approximation for common uses by recognizing special forms as you suggest. But it=E2=80=99s just that, an approximation. In such situations, I err on the side of not even trying. The added complexity for a flaky result doesn=E2=80=99t pay off to me. I prefer to be upfront, document limitations, and let users handle them as they see fit. WDYT? A similar problem occurs with system provenance tracking, which saves =E2=80=98configuration.scm=E2=80=99 but leaves it up to the user to preserv= e additional files if needed (info "(guix) Service Reference"). Thanks, Ludo=E2=80=99.