[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Maxime Devos <maximedevos@telenet.be>]

[-- Attachment #1: Type: text/plain, Size: 4429 bytes --]

Ludovic Courtès schreef op ma 04-10-2021 om 10:19 [+0200]:
> > A documented flag to always consider the cache stale seems good, though I think
> > at least the dependencies made with the common macros and procedures 'include',
> > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
> > 'local-file' could be tracked, though this could be left as a TODO for later
> > I suppose.
> 
> Tracking those uses reliably is impossible: there could be same-named
> bindings that do other things, there could be custom macros, there could
> be “dynamic arguments” (whose value is not known statically), etc.  You
> have to expand + evaluate the code to get better results, and even then,
> there might be different paths in the code so you can’t be sure you got
> it right.

I think there's a miscommunication here.  From what I'm reading, what you have
in mind is that, to determine the dependency information, "guix shell" would
open "guix.scm", read it with the procedure 'read' and sniff for 'load',
'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form
-- something like 'module-file-dependencies' and 'extract-dependencies', but
more general.

However, my idea is to replace these macros, such that, when "guix shell"
loads "guix.scm" or "manifest.scm", these macros inform "guix shell"
that "guix.scm" or "manifest.scm" depend on certain files referred to
by 'load', 'include-from-path', etc. forms, using a mechanism like the
'notice-dependency' defined in <https://issues.guix.gnu.org/50384>.

Then, when "guix shell" puts the resulting profile in the cache,
it includes the generated list of files.  And when "guix shell" finds
an entry in the cache, it will check if the files in this list (and guix.scm
or manifest.scm of course) have been modified.  If some are (or the forcing
flag is set), guix.scm needs to be loaded and a new profile needs to be
generated.  If they are all unchanged, guix.scm will _not_ be read: the cached
profile can be reused.

It's not 100% reliable (e.g. the list of packages in the manifest could
depend on the phase of the moon if (current-time) is used) (is that what
you mean by ‘different paths’ and ‘dynamic arguments’?), but it should
cover the vast majority of cases.

I don't know a non-artifical situation where ‘custom macros’ are a problem
-- do you know an example in the wild where this dependency tracking scheme
would fail to work?

> We could get an approximation for common uses by recognizing special
> forms as you suggest.  But it’s just that, an approximation.

It's an approximation, sure, but it seems to be a quite accurate approximation
to me.  And it's not really recognising special forms that I'm suggesting,
but rather modifying the macros behind these forms to inform "guix shell"
of what the dependencies are.

> In such situations, I err on the side of not even trying.  The added
> complexity for a flaky result doesn’t pay off to me.  I prefer to be
> upfront, document limitations, and let users handle them as they see
> fit.

About complexity: there's some extra code, sure, but it doesn't seem complex
to me.  To track dependencies in <https://issues.guix.gnu.org/50384>,
I only needed to add 'notice-dependency' and some other code to (guix build compile),
and some code to build-aux/compile-all.scm to save/load the dependency information
to/from the builddir and to also check the mtime of dependencies.

Does it still seem flaky to you, after my explanation on how the dependency
information would be determined?

Being upfront, documenting limitations, and having a ‘force rebuild flag’
(is that what you mean by ‘letting users handle them as they see fit’?)
(possibly also documenting 'notice-dependency') is not incompatible with
the automatic dependency tracking.

More abstractly, this seems more like a ‘Perfect is the enemy of the good’ situation.
While I would prefer 'perfect' above 'good', and the automatic dependency tracking
certainly isn't 'perfect', it does seem 'good' to me, and perfection appears to
be impossible and there's the ‘force rebuild flag’ as an escape hatch, so I believe
'good-but-not-perfect' to be acceptable here, as long as it is documented in the manual
that there are some limitation on what dependencies are automatically tracked.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]