From: "Ludovic Courtès" <ludo@gnu.org>
To: Maxime Devos <maximedevos@telenet.be>
Cc: 50960@debbugs.gnu.org
Subject: [bug#50960] [PATCH 10/10] shell: Maintain a profile cache.
Date: Fri, 08 Oct 2021 09:37:55 +0200 [thread overview]
Message-ID: <8735pc6k6k.fsf@gnu.org> (raw)
In-Reply-To: <f821bfcc64304e467575a26c4473ddc6863eb42e.camel@telenet.be> (Maxime Devos's message of "Mon, 04 Oct 2021 17:58:41 +0200")
Hi Maxime,
Maxime Devos <maximedevos@telenet.be> skribis:
> Ludovic Courtès schreef op ma 04-10-2021 om 10:19 [+0200]:
>> > A documented flag to always consider the cache stale seems good, though I think
>> > at least the dependencies made with the common macros and procedures 'include',
>> > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
>> > 'local-file' could be tracked, though this could be left as a TODO for later
>> > I suppose.
>>
>> Tracking those uses reliably is impossible: there could be same-named
>> bindings that do other things, there could be custom macros, there could
>> be “dynamic arguments” (whose value is not known statically), etc. You
>> have to expand + evaluate the code to get better results, and even then,
>> there might be different paths in the code so you can’t be sure you got
>> it right.
>
> I think there's a miscommunication here. From what I'm reading, what you have
> in mind is that, to determine the dependency information, "guix shell" would
> open "guix.scm", read it with the procedure 'read' and sniff for 'load',
> 'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form
> -- something like 'module-file-dependencies' and 'extract-dependencies', but
> more general.
Yes, that’s roughly what I thought you had in mind, sorry!
> However, my idea is to replace these macros, such that, when "guix shell"
> loads "guix.scm" or "manifest.scm", these macros inform "guix shell"
> that "guix.scm" or "manifest.scm" depend on certain files referred to
> by 'load', 'include-from-path', etc. forms, using a mechanism like the
> 'notice-dependency' defined in <https://issues.guix.gnu.org/50384>.
>
> Then, when "guix shell" puts the resulting profile in the cache,
> it includes the generated list of files. And when "guix shell" finds
> an entry in the cache, it will check if the files in this list (and guix.scm
> or manifest.scm of course) have been modified. If some are (or the forcing
> flag is set), guix.scm needs to be loaded and a new profile needs to be
> generated. If they are all unchanged, guix.scm will _not_ be read: the cached
> profile can be reused.
>
> It's not 100% reliable (e.g. the list of packages in the manifest could
> depend on the phase of the moon if (current-time) is used) (is that what
> you mean by ‘different paths’ and ‘dynamic arguments’?), but it should
> cover the vast majority of cases.
>
> I don't know a non-artifical situation where ‘custom macros’ are a problem
> -- do you know an example in the wild where this dependency tracking scheme
> would fail to work?
I think we have to look at the cost and at the benefits. The cost:
either we spread ‘notice-dependency’ calls all over the place, or we
interpose on ‘open-file’ altogether, assuming that even works. The
benefit: we’d have an 80% solution. There would still be edge cases not
handled correctly; a realistic example is a ‘guix.scm’ whose execution
differs based on the presence of a file or environment variable. People
would have to know about this pitfall and pass the right flag.
[...]
>> In such situations, I err on the side of not even trying. The added
>> complexity for a flaky result doesn’t pay off to me. I prefer to be
>> upfront, document limitations, and let users handle them as they see
>> fit.
>
> About complexity: there's some extra code, sure, but it doesn't seem complex
> to me. To track dependencies in <https://issues.guix.gnu.org/50384>,
> I only needed to add 'notice-dependency' and some other code to (guix build compile),
> and some code to build-aux/compile-all.scm to save/load the dependency information
> to/from the builddir and to also check the mtime of dependencies.
>
> Does it still seem flaky to you, after my explanation on how the dependency
> information would be determined?
>
> Being upfront, documenting limitations, and having a ‘force rebuild flag’
> (is that what you mean by ‘letting users handle them as they see fit’?)
> (possibly also documenting 'notice-dependency') is not incompatible with
> the automatic dependency tracking.
>
> More abstractly, this seems more like a ‘Perfect is the enemy of the good’ situation.
> While I would prefer 'perfect' above 'good', and the automatic dependency tracking
> certainly isn't 'perfect', it does seem 'good' to me, and perfection appears to
> be impossible and there's the ‘force rebuild flag’ as an escape hatch, so I believe
> 'good-but-not-perfect' to be acceptable here, as long as it is documented in the manual
> that there are some limitation on what dependencies are automatically tracked.
Yeah, I can relate to that sentiment, even if I’m all too often on the
“perfectionist” side of things. ;-)
I guess I could live with an 80% dependency tracking solution. However,
my criteria for it would be that (1) it should be orthogonal (e.g., not
requiring explicit calls to ‘notice-dependency’ in many places), and (2)
it should be self-contained and reasonably small. Perhaps having
‘load*’ or similar instrument ‘open-file’ or similar could help achieve
that?
I would rather not block ‘guix shell’ on that though.
WDYT?
Thanks for taking the time to clarify what you had in mind!
Ludo’.
next prev parent reply other threads:[~2021-10-08 7:39 UTC|newest]
Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-02 10:21 [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 01/10] packages: Add 'package-development-inputs' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 02/10] profiles: Add 'package->development-manifest' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 03/10] DRAFT Add 'guix shell' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file Ludovic Courtès
2021-10-02 11:52 ` Liliana Marie Prikler
2021-10-02 13:43 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Ludovic Courtès
2021-10-05 7:50 ` Maxime Devos
2021-10-08 7:44 ` Ludovic Courtès
2021-10-02 14:15 ` [bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file Maxime Devos
2021-10-04 8:07 ` Ludovic Courtès
2021-10-05 7:51 ` Maxime Devos
2021-10-02 10:22 ` [bug#50960] [PATCH 05/10] environment: Add tests for '--profile' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 06/10] environment: Skip derivation computation when '--profile' is used Ludovic Courtès
2021-10-02 11:39 ` Liliana Marie Prikler
2021-10-02 13:46 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 07/10] environment: Do not connect to the daemon when '--profile' is used Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 08/10] environment: Autoload some modules Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 09/10] cache: Gracefully handle non-existent cache Ludovic Courtès
2021-10-02 13:28 ` Maxime Devos
2021-10-02 10:22 ` [bug#50960] [PATCH 10/10] shell: Maintain a profile cache Ludovic Courtès
2021-10-02 13:43 ` Maxime Devos
2021-10-02 14:12 ` Ludovic Courtès
2021-10-02 14:47 ` Maxime Devos
2021-10-04 8:19 ` Ludovic Courtès
2021-10-04 14:20 ` zimoun
2021-10-04 15:58 ` Maxime Devos
2021-10-08 7:37 ` Ludovic Courtès [this message]
2021-10-02 13:52 ` Maxime Devos
2021-10-02 14:14 ` Ludovic Courtès
2021-10-02 14:22 ` Maxime Devos
2021-10-04 8:08 ` Ludovic Courtès
2021-10-02 10:50 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Jelle Licht
2021-10-02 13:52 ` Ludovic Courtès
2021-10-02 12:10 ` pelzflorian (Florian Pelz)
2021-10-02 13:40 ` Ludovic Courtès
2021-10-02 15:08 ` pelzflorian (Florian Pelz)
2021-10-04 8:22 ` Ludovic Courtès
2021-10-04 9:23 ` pelzflorian (Florian Pelz)
2021-10-04 16:50 ` Maxime Devos
2021-10-02 13:03 ` Christine Lemmer-Webber
2021-10-02 14:00 ` [bug#50960] ‘guix shell’ shebangs Ludovic Courtès
2021-10-03 22:50 ` Katherine Cox-Buday
2021-10-02 23:57 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Vagrant Cascadian
2021-10-03 8:36 ` Nicolò Balzarotti
2021-10-04 8:34 ` Ludovic Courtès
2021-10-04 17:12 ` Maxime Devos
2021-10-04 6:56 ` zimoun
2021-10-04 8:39 ` Ludovic Courtès
2021-10-04 10:40 ` zimoun
2021-10-04 12:23 ` Ludovic Courtès
2021-10-04 13:42 ` zimoun
2021-10-04 17:38 ` Leo Famulari
2021-10-08 7:43 ` Ludovic Courtès
2021-10-08 16:16 ` Leo Famulari
2021-10-09 13:38 ` Ludovic Courtès
2021-10-11 0:29 ` Leo Famulari
2021-10-04 21:29 ` [bug#50960] [EXT] " Thompson, David
2021-10-07 9:26 ` Ludovic Courtès
2021-10-07 10:52 ` pelzflorian (Florian Pelz)
2021-10-07 11:17 ` [bug#50960] [EXT] " Thompson, David
2021-10-07 12:01 ` pelzflorian (Florian Pelz)
2021-10-08 14:24 ` Katherine Cox-Buday
2021-10-11 9:13 ` zimoun
2021-10-06 8:12 ` Konrad Hinsen
2021-10-07 8:34 ` Ludovic Courtès
2021-10-07 9:15 ` Liliana Marie Prikler
2021-10-08 15:45 ` Konrad Hinsen
2021-10-09 7:45 ` Liliana Marie Prikler
2021-10-11 8:32 ` Ludovic Courtès
2021-10-09 8:07 ` Stefan
2021-10-11 21:37 ` [bug#50960] [PATCH v2 00/11] 'guix shell' strikes again Ludovic Courtès
2021-10-11 21:37 ` [bug#50960] [PATCH v2 01/11] packages: Add 'package-development-inputs' Ludovic Courtès
2021-10-12 6:39 ` zimoun
2021-10-12 9:54 ` Ludovic Courtès
2021-10-12 11:52 ` zimoun
2021-10-11 21:38 ` [bug#50960] [PATCH v2 02/11] profiles: Add 'package->development-manifest' Ludovic Courtès
2021-10-12 6:43 ` zimoun
2021-10-12 9:27 ` Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 03/11] Add 'guix shell' Ludovic Courtès
2021-10-13 16:51 ` pelzflorian (Florian Pelz)
2021-10-11 21:38 ` [bug#50960] [PATCH v2 04/11] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 05/11] DRAFT shell: Honor in ~/.config/guix/shell-authorized-directories Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 06/11] environment: Add tests for '--profile' Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 07/11] environment: Skip derivation computation when '--profile' is used Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 08/11] environment: Do not connect to the daemon " Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 09/11] environment: Autoload some modules Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 10/11] cache: Gracefully handle non-existent cache Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 11/11] shell: Maintain a profile cache Ludovic Courtès
2021-10-12 8:53 ` [bug#50960] [PATCH v2 00/11] 'guix shell' strikes again pelzflorian (Florian Pelz)
2021-10-12 8:57 ` pelzflorian (Florian Pelz)
2021-10-12 9:55 ` Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 00/10] Adding 'guix shell': last call! Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 01/10] packages: Add 'package-development-inputs' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 02/10] profiles: Add 'package->development-manifest' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 03/10] Add 'guix shell' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 04/10] shell: By default load the local 'guix.scm' or 'manifest.scm' file Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 05/10] environment: Add tests for '--profile' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 06/10] environment: Skip derivation computation when '--profile' is used Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 07/10] environment: Do not connect to the daemon " Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 08/10] environment: Autoload some modules Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 09/10] cache: Gracefully handle non-existent cache Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 10/10] shell: Maintain a profile cache Ludovic Courtès
2021-10-19 8:43 ` [bug#50960] [PATCH v3 00/10] Adding 'guix shell': last call! zimoun
2021-10-25 13:41 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' zimoun
2021-10-25 18:19 ` Ludovic Courtès
2021-10-25 19:45 ` zimoun
2021-10-25 18:25 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8735pc6k6k.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=50960@debbugs.gnu.org \
--cc=maximedevos@telenet.be \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).