From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GJXKI+InW2F0lQAAgWs5BA (envelope-from ) for ; Mon, 04 Oct 2021 18:12:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id AAN+H+InW2E+fwAA1q6Kng (envelope-from ) for ; Mon, 04 Oct 2021 16:12:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 02F9E3441F for ; Mon, 4 Oct 2021 18:12:18 +0200 (CEST) Received: from localhost ([::1]:37152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mXQZd-0004cs-5L for larch@yhetil.org; Mon, 04 Oct 2021 12:12:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47506) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mXQNn-0007bm-D1 for guix-patches@gnu.org; Mon, 04 Oct 2021 12:00:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:54865) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mXQNn-0001t8-2O for guix-patches@gnu.org; Mon, 04 Oct 2021 12:00:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mXQNm-0008EH-Vv for guix-patches@gnu.org; Mon, 04 Oct 2021 12:00:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50960] [PATCH 10/10] shell: Maintain a profile cache. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 04 Oct 2021 16:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 50960@debbugs.gnu.org Received: via spool by 50960-submit@debbugs.gnu.org id=B50960.163336315231505 (code B ref 50960); Mon, 04 Oct 2021 16:00:02 +0000 Received: (at 50960) by debbugs.gnu.org; 4 Oct 2021 15:59:12 +0000 Received: from localhost ([127.0.0.1]:38173 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mXQMy-0008C4-5J for submit@debbugs.gnu.org; Mon, 04 Oct 2021 11:59:12 -0400 Received: from albert.telenet-ops.be ([195.130.137.90]:45848) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mXQMs-0008Bq-1b for 50960@debbugs.gnu.org; Mon, 04 Oct 2021 11:59:10 -0400 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by albert.telenet-ops.be with bizsmtp id 1rz32600j0mfAB406rz37c; Mon, 04 Oct 2021 17:59:04 +0200 Message-ID: From: Maxime Devos Date: Mon, 04 Oct 2021 17:58:41 +0200 In-Reply-To: <87lf39ryle.fsf@gnu.org> References: <20211002102240.27815-1-ludo@gnu.org> <20211002102240.27815-10-ludo@gnu.org> <91012c32e4f8b04beae71378d44787a81055391a.camel@telenet.be> <87ilyfy0q4.fsf@gnu.org> <87lf39ryle.fsf@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-IsBlfaMsDAaUWMCzUWr2" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1633363144; bh=uTQ3FZMUyPXWdnAOipUp4VqFlLv/ZRHbFeenCFJLLu4=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=gzXXyt700gfYpf8qrCuY7E0EZxD1PWyPB1ZDnAocc6Cfq+NJd+8xeGYLsjj19wS7O Usjlp4lzWA/m5ASrOsi446WETLoA0skGdrSntIwjmtpcuWaKZ5j/dbvB7LkD9aVJ4Y uStw7rV3ZptzoxY6UisLjyuFXS2ntgSJw7TcFTJnNqXJacaRmisen/fvSzCh8+OMtb Lst+Bwo4Gwq5BiHRfNw7IamhESkHXaeUiBVhNDNMYK5pm2Puzkhpx4WuoZSKEggj0k QZTGBcDBkBibn4hsufY29D5VHYzHL7OxmwpHl7Qt1tUOoNGG1IPrSSjmArijuiwm8p ay3DdGZD6Qqcg== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1633363938; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=uTQ3FZMUyPXWdnAOipUp4VqFlLv/ZRHbFeenCFJLLu4=; b=hnY3g3aOdOBp6/DnF+c6RZvLgRCr+kROceOICPlAxXmgtG9AyVo6owQSCkHKDcOa+L0gK5 I87DbglvIOJYsoTFt1VcfYUGM8/39t8eruSbWmc4cjM+nMr+JJt87wp5fObKFQ4tbpJFhF xQgkOpFUnkVUScN2427NkNTfmwxYyPJYv+FKd5rcjxR5z/C6h8xTf6gOyL9aIuDb0LQAAh qVTQQAdxvb/xVVtt7YRN1cWDh2kENcpN6LlQ1IxtUl8odQdIZjgTX/921aF1oxoqpmgBbI /79gAzfMSuFgmYI53SycRP+E5KJkA0QJ5YzQAiwmGX/deLc8Og5JDhvcCQ3l3w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1633363938; a=rsa-sha256; cv=none; b=oAr88z0YPCF2aUyN1pa9XklxZ1/HJEFXPrtHALyhysIv3hz9HnTPPLTFw+BeQJKkHRyrCe jrg2JDwvMuYRD/L/s7WespzVkqnB0UF1MTw17EU9Ljee1fze7C0Y/PuRNi8lAVc+QjESH1 Q9aF+nctVKUrlkwjbha0KpBHGn66HH4xYyV+99U/S54iWifSHNEj2i9/0D0pqzfeu/hvl1 WB+miyn1U96eAcj5SPvju1MMJxZxJh1cH5j1SJVSfOR7coEw31GcOpbnpmi5AghJbhhsN/ gaxn27mVdkEk8J0oGmC/vMWOibi5LLCSGHY7a/aHGC/Q+I/8Gq0rNGM8hkvZoQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=gzXXyt70; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -3.91 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=gzXXyt70; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 02F9E3441F X-Spam-Score: -3.91 X-Migadu-Scanner: scn1.migadu.com X-TUID: KxRQH1KUL+o4 --=-IsBlfaMsDAaUWMCzUWr2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op ma 04-10-2021 om 10:19 [+0200]: > > A documented flag to always consider the cache stale seems good, though= I think > > at least the dependencies made with the common macros and procedures 'i= nclude', > > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-re= cursive > > 'local-file' could be tracked, though this could be left as a TODO for = later > > I suppose. >=20 > Tracking those uses reliably is impossible: there could be same-named > bindings that do other things, there could be custom macros, there could > be =E2=80=9Cdynamic arguments=E2=80=9D (whose value is not known statical= ly), etc. You > have to expand + evaluate the code to get better results, and even then, > there might be different paths in the code so you can=E2=80=99t be sure y= ou got > it right. I think there's a miscommunication here. From what I'm reading, what you h= ave in mind is that, to determine the dependency information, "guix shell" woul= d open "guix.scm", read it with the procedure 'read' and sniff for 'load', 'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form -- something like 'module-file-dependencies' and 'extract-dependencies', bu= t more general. However, my idea is to replace these macros, such that, when "guix shell" loads "guix.scm" or "manifest.scm", these macros inform "guix shell" that "guix.scm" or "manifest.scm" depend on certain files referred to by 'load', 'include-from-path', etc. forms, using a mechanism like the 'notice-dependency' defined in . Then, when "guix shell" puts the resulting profile in the cache, it includes the generated list of files. And when "guix shell" finds an entry in the cache, it will check if the files in this list (and guix.sc= m or manifest.scm of course) have been modified. If some are (or the forcing flag is set), guix.scm needs to be loaded and a new profile needs to be generated. If they are all unchanged, guix.scm will _not_ be read: the cac= hed profile can be reused. It's not 100% reliable (e.g. the list of packages in the manifest could depend on the phase of the moon if (current-time) is used) (is that what you mean by =E2=80=98different paths=E2=80=99 and =E2=80=98dynamic argument= s=E2=80=99?), but it should cover the vast majority of cases. I don't know a non-artifical situation where =E2=80=98custom macros=E2=80= =99 are a problem -- do you know an example in the wild where this dependency tracking scheme would fail to work? > We could get an approximation for common uses by recognizing special > forms as you suggest. But it=E2=80=99s just that, an approximation. It's an approximation, sure, but it seems to be a quite accurate approximat= ion to me. And it's not really recognising special forms that I'm suggesting, but rather modifying the macros behind these forms to inform "guix shell" of what the dependencies are. > In such situations, I err on the side of not even trying. The added > complexity for a flaky result doesn=E2=80=99t pay off to me. I prefer to= be > upfront, document limitations, and let users handle them as they see > fit. About complexity: there's some extra code, sure, but it doesn't seem comple= x to me. To track dependencies in , I only needed to add 'notice-dependency' and some other code to (guix build= compile), and some code to build-aux/compile-all.scm to save/load the dependency info= rmation to/from the builddir and to also check the mtime of dependencies. Does it still seem flaky to you, after my explanation on how the dependency information would be determined? Being upfront, documenting limitations, and having a =E2=80=98force rebuild= flag=E2=80=99 (is that what you mean by =E2=80=98letting users handle them as they see fi= t=E2=80=99?) (possibly also documenting 'notice-dependency') is not incompatible with the automatic dependency tracking. More abstractly, this seems more like a =E2=80=98Perfect is the enemy of th= e good=E2=80=99 situation. While I would prefer 'perfect' above 'good', and the automatic dependency t= racking certainly isn't 'perfect', it does seem 'good' to me, and perfection appear= s to be impossible and there's the =E2=80=98force rebuild flag=E2=80=99 as an es= cape hatch, so I believe 'good-but-not-perfect' to be acceptable here, as long as it is documented i= n the manual that there are some limitation on what dependencies are automatically track= ed. Greetings, Maxime. --=-IsBlfaMsDAaUWMCzUWr2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYVskshccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7o9RAQCMHGDE6veWG0GNgcj9MUOlBs/9 G5xDCZkVsqfM7KsIpQEA5QWvEXT6O9lcGYZUU6VByqjg/T2t45wdRTDSOeThzAc= =dE78 -----END PGP SIGNATURE----- --=-IsBlfaMsDAaUWMCzUWr2--