bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Stefan Kangas]

Severity: wishlist

Please consider adding a function and/or command to clear the eln
cache of redundant eln cache directories.

The exact details of this would of course need to be discussed, but as
a starting point perhaps we could imagine two modes of operation:

1.  Remove cache directories for versions older than the version
    specified by a new user option that defaults to
    (- emacs-major-version 2) or something.

    Remove cache directories for the same version as the current one
    but with a different hash.

    Remove duplicate directories for any given version, except the
    most recently added one.

2.  Remove all eln cache directories except the one currently in use.

I imagine that casual users only ever use one version of Emacs at the
same time (i.e. because they simply install Emacs using their package
manager), so perhaps the first one could be the default mode of
operation.

We could add a number of options to ensure a user can tweak this to
their own liking.

One idea for a command name is `native-compile-cache-clear'.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Thu, 29 Apr 2021 16:59:44 -0500
> Cc: Andrea Corallo <akrl@sdf.org>
> 
> 1.  Remove cache directories for versions older than the version
>     specified by a new user option that defaults to
>     (- emacs-major-version 2) or something.
> 
>     Remove cache directories for the same version as the current one
>     but with a different hash.
> 
>     Remove duplicate directories for any given version, except the
>     most recently added one.
> 
> 2.  Remove all eln cache directories except the one currently in use.

This is somewhat tricky, so I suggest that you (or someone else) comes
up with a convenient UI for such a feature and describes it in enough
detail to see whether it's feasible.  Namely, how would the user
indicate which *.eln files can be deleted and which should stay?

Some thought points:

  . the .eln hashes depend not only on the version but also on other
    internal details, like the ABI version and more
  . it isn't clear how the user will be able to come up with "the
    hash" (especially since there are several hashes involved, not
    just one)

> I imagine that casual users only ever use one version of Emacs at the
> same time (i.e. because they simply install Emacs using their package
> manager), so perhaps the first one could be the default mode of
> operation.

I don't know what you mean by "casual users", but I generally have all
the versions since Emacs 21 installed on my machines.

Btw, we might go look what others do with similar caches, for example
Guile.  Do they even bother to clean these caches up?

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> The exact details of this would of course need to be discussed, but as
> a starting point perhaps we could imagine two modes of operation:

[...]

> 2.  Remove all eln cache directories except the one currently in use.
>
> I imagine that casual users only ever use one version of Emacs at the
> same time (i.e. because they simply install Emacs using their package
> manager), so perhaps the first one could be the default mode of
> operation.
>
> We could add a number of options to ensure a user can tweak this to
> their own liking.
>
> One idea for a command name is `native-compile-cache-clear'.

I think 2) would be sufficient for most people, and, yes, having a
command for this would be nice.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> Cc: 48108@debbugs.gnu.org, Andrea Corallo <akrl@sdf.org>
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 01 Jul 2022 13:11:03 +0200
> 
> Stefan Kangas <stefan@marxist.se> writes:
> 
> > The exact details of this would of course need to be discussed, but as
> > a starting point perhaps we could imagine two modes of operation:
> 
> [...]
> 
> > 2.  Remove all eln cache directories except the one currently in use.
> >
> > I imagine that casual users only ever use one version of Emacs at the
> > same time (i.e. because they simply install Emacs using their package
> > manager), so perhaps the first one could be the default mode of
> > operation.
> >
> > We could add a number of options to ensure a user can tweak this to
> > their own liking.
> >
> > One idea for a command name is `native-compile-cache-clear'.
> 
> I think 2) would be sufficient for most people, and, yes, having a
> command for this would be nice.

It shouldn't be hard to implement, but do we want to delete such a
directory only under ~/.emacs.d/eln-cache/, or under all the
directories in native-comp-eln-load-path?

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Rudolf Schlatte]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 48108@debbugs.gnu.org, Andrea Corallo <akrl@sdf.org>
>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Fri, 01 Jul 2022 13:11:03 +0200
>> 
>> Stefan Kangas <stefan@marxist.se> writes:
>> 
>> > The exact details of this would of course need to be discussed, but as
>> > a starting point perhaps we could imagine two modes of operation:
>> 
>> [...]
>> 
>> > 2.  Remove all eln cache directories except the one currently in use.
>> >
>> > I imagine that casual users only ever use one version of Emacs at the
>> > same time (i.e. because they simply install Emacs using their package
>> > manager), so perhaps the first one could be the default mode of
>> > operation.
>> >
>> > We could add a number of options to ensure a user can tweak this to
>> > their own liking.
>> >
>> > One idea for a command name is `native-compile-cache-clear'.
>> 
>> I think 2) would be sufficient for most people, and, yes, having a
>> command for this would be nice.
>
> It shouldn't be hard to implement, but do we want to delete such a
> directory only under ~/.emacs.d/eln-cache/, or under all the
> directories in native-comp-eln-load-path?

How about only deleting .eln files below these directories, and removing
the directories only when they are empty afterwards?  I'm suddenly
imagining users who put other files into such directories, or add their
home directory to native-comp-eln-load-path, or ...

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Rudolf Schlatte <rudi@constantly.at>
> Date: Fri, 01 Jul 2022 18:18:16 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It shouldn't be hard to implement, but do we want to delete such a
> > directory only under ~/.emacs.d/eln-cache/, or under all the
> > directories in native-comp-eln-load-path?
> 
> How about only deleting .eln files below these directories, and removing
> the directories only when they are empty afterwards?  I'm suddenly
> imagining users who put other files into such directories, or add their
> home directory to native-comp-eln-load-path, or ...

IMO, that makes no sense.  These directories are for the *.eln files
only.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> It shouldn't be hard to implement, but do we want to delete such a
> directory only under ~/.emacs.d/eln-cache/, or under all the
> directories in native-comp-eln-load-path?

The last element in that variable is supposed to be the system
directory...  but perhaps it'd make sense for this command to prune .eln
directories in all directories in native-comp-eln-load-path where the
user has write access?  (That will normally, by default, just be
~/.emacs.d/eln-cache/.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefan@marxist.se,  48108@debbugs.gnu.org,  akrl@sdf.org
> Date: Sat, 02 Jul 2022 12:40:05 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It shouldn't be hard to implement, but do we want to delete such a
> > directory only under ~/.emacs.d/eln-cache/, or under all the
> > directories in native-comp-eln-load-path?
> 
> The last element in that variable is supposed to be the system
> directory...  but perhaps it'd make sense for this command to prune .eln
> directories in all directories in native-comp-eln-load-path where the
> user has write access?

That's what I'm asking, basically.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> The last element in that variable is supposed to be the system
>> directory...  but perhaps it'd make sense for this command to prune .eln
>> directories in all directories in native-comp-eln-load-path where the
>> user has write access?
>
> That's what I'm asking, basically.

I've now added this to Emacs 29 under the name
`native-compile-prune-cache'.  (Feel free to adjust the name -- I felt
that -clear-cache was misleading, since we're not removing the cache for
the current invocation.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48108: Fwd: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

`native-compile-prune-cache' does not remove the preloaded/ subdir (or its
contents) of the native-lisp/ directories, and consequently does not remove
the directories themselves.

For example, after running it I still have *.eln files in the following
directories (comp-native-version-dir = "29.0.50-c0f9fe8e"):

~/.emacs.d/.eln-cache/29.0.50-c0f9fe8e
D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-57ea32b2\preloaded
D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-7c1dd663\preloaded
D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-a22e1c10\preloaded
D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-c0f9fe8e\preloaded

which means the command deleted the stale cache in ~/.emacs.d/, but not the
ones in the source tree.

Is that intended?

[-- Attachment #2: Type: text/html, Size: 1071 bytes --]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Mon, 28 Nov 2022 09:39:14 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Kangas <stefankangas@gmail.com>, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> `native-compile-prune-cache' does not remove the preloaded/ subdir (or its contents) of the native-lisp/
> directories, and consequently does not remove the directories themselves.
> 
> For example, after running it I still have *.eln files in the following directories (comp-native-version-dir =
> "29.0.50-c0f9fe8e"):
> 
> ~/.emacs.d/.eln-cache/29.0.50-c0f9fe8e
> D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-57ea32b2\preloaded
> D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-7c1dd663\preloaded
> D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-a22e1c10\preloaded
> D:\Devel\emacs\repo\trunk\native-lisp\29.0.50-c0f9fe8e\preloaded
> 
> which means the command deleted the stale cache in ~/.emacs.d/, but not the ones in the source tree.
> 
> Is that intended?

AFAIR, yes.  The command is intended to help users to clean up the eln-cache
of files _they_ compile, but it isn't supposed to touch *.eln files
installed as part of Emacs.  That's for sysadmins.  And besides, on many
systems the preloaded *.eln files are not writable by "normal" users.

See bug#48108, where this was discussed.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

Some of the weird things I see are perhaps the consequence of running Emacs
in place from the build tree.

It is assumed that normal users have write access to
native-lisp/VERSION-xxxxxxxx/, but not to
native-lisp/VERSION-xxxxxxx/preloaded?

The function already deletes any *.eln file in a stale native
VERSION-xxxxxxx directory, just not the ones in preload/

[-- Attachment #2: Type: text/html, Size: 543 bytes --]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]

Also, if you set a relative path in native-comp-eln-load-path (directly,
not via `startup-redirect-eln-cache'), it defaults to
`invocation-directory', which is (in an in-place run) the src/ subdir.

Which is weird:

(push "./test-native" native-comp-eln-load-path)

and then the test-native/ subdir is created both under
`user-emacs-directory' and `invocation-directory' (the .eln files go to the
second one).

[-- Attachment #2: Type: text/html, Size: 561 bytes --]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Mon, 28 Nov 2022 14:09:42 +0100
> Cc: larsi@gnus.org, 48108@debbugs.gnu.org, stefankangas@gmail.com
> 
> Some of the weird things I see are perhaps the consequence of running Emacs in place from the build tree.
> 
> It is assumed that normal users have write access to native-lisp/VERSION-xxxxxxxx/, but not to
> native-lisp/VERSION-xxxxxxx/preloaded?

No, to both.

> The function already deletes any *.eln file in a stale native VERSION-xxxxxxx directory, just not the ones in
> preload/

That's a bug, IMO.  The last directory in native-comp-eln-load-path should
not be emptied.  On a multi-user system, some other user could run an Emacs
that uses the files in the native-lisp/VERSION-xxxxxxx/ directory.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Mon, 28 Nov 2022 14:22:05 +0100
> Cc: larsi@gnus.org, 48108@debbugs.gnu.org, stefankangas@gmail.com
> 
> Also, if you set a relative path in native-comp-eln-load-path (directly, not via `startup-redirect-eln-cache'), it
> defaults to `invocation-directory', which is (in an in-place run) the src/ subdir. 
> 
> Which is weird:
> 
> (push "./test-native" native-comp-eln-load-path)
> 
> and then the test-native/ subdir is created both under `user-emacs-directory' and `invocation-directory' (the
> .eln files go to the second one).

Why is this weird?  These files are needed early at startup, when Emacs is
still unable to do anything fancy, so how is it "weird" that relative file
names are interpreted as relative to where Emacs runs?  Are you aware of any
other programs that interpret relative file names differently?

If the effect surprises you, don't add relative file names to the variable.
You aren't supposed to anyway; AFAIR this option is only used during the
build, if at all.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

On Mon, Nov 28, 2022 at 2:38 PM Eli Zaretskii <eliz@gnu.org> wrote:

That's a bug, IMO.  The last directory in native-comp-eln-load-path should
> not be emptied.  On a multi-user system, some other user could run an Emacs
> that uses the files in the native-lisp/VERSION-xxxxxxx/ directory.
>

I'll create a bug report for that, then.

[-- Attachment #2: Type: text/html, Size: 833 bytes --]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Mon, Nov 28, 2022 at 2:42 PM Eli Zaretskii <eliz@gnu.org> wrote:

Why is this weird?  These files are needed early at startup, when Emacs is
> still unable to do anything fancy, so how is it "weird" that relative file
> names are interpreted as relative to where Emacs runs?  Are you aware of
> any
> other programs that interpret relative file names differently?
>

It is used also for files that are not required early at startup (unless
you consider "early at startup" native-compiling the .el/.elc files
required or loaded in init.el).

If the effect surprises you, don't add relative file names to the variable.
>

For sure.


> You aren't supposed to anyway; AFAIR this option is only used during the
> build, if at all.
>

 I don't disagree, but that's not evident reading
native-comp-eln-load-path's docstring.

Anyway, the fact that if you set a relative dir ./test-compile in that
variable both ~/.emacs.d/test-compile and src/test-compile directories are
created is also a bug.

[-- Attachment #2: Type: text/html, Size: 1988 bytes --]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Mon, 28 Nov 2022 14:54:32 +0100
> Cc: larsi@gnus.org, 48108@debbugs.gnu.org, stefankangas@gmail.com
> 
> On Mon, Nov 28, 2022 at 2:42 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  Why is this weird?  These files are needed early at startup, when Emacs is
>  still unable to do anything fancy, so how is it "weird" that relative file
>  names are interpreted as relative to where Emacs runs?  Are you aware of any
>  other programs that interpret relative file names differently?
> 
> It is used also for files that are not required early at startup (unless you consider "early at startup"
> native-compiling the .el/.elc files required or loaded in init.el).

I do.

>  You aren't supposed to anyway; AFAIR this option is only used during the
>  build, if at all.
> 
>  I don't disagree, but that's not evident reading native-comp-eln-load-path's docstring.

This is Emacs: you get enough rope to hang yourself.  We trust you that you
either know what you are doing, or won't do it.

> Anyway, the fact that if you set a relative dir ./test-compile in that variable both ~/.emacs.d/test-compile and
> src/test-compile directories are created is also a bug.

Why do you think it's a bug?

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

On Mon, Nov 28, 2022, 15:43 Eli Zaretskii <eliz@gnu.org> wrote:

>
> This is Emacs: you get enough rope to hang yourself.  We trust you that you
> either know what you are doing, or won't do it.
>

Still, a brief reminder/warning in the docstring would be nice. We warn,
occasionally.

>
> Why do you think it's a bug?
>

Because the ~/. emacs.d/ one is created but left empty.

>

[-- Attachment #2: Type: text/html, Size: 1098 bytes --]

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Mon, 28 Nov 2022 16:14:27 +0100
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>, 48108@debbugs.gnu.org, stefankangas@gmail.com
> 
> On Mon, Nov 28, 2022, 15:43 Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  This is Emacs: you get enough rope to hang yourself.  We trust you that you
>  either know what you are doing, or won't do it.
> 
> Still, a brief reminder/warning in the docstring would be nice. We warn, occasionally. 
> 
>  Why do you think it's a bug?
> 
> Because the ~/. emacs.d/ one is created but left empty. 

I still don't understand why would that constitute a bug.  It is created so
that future compilation wouldn't have to worry about its existence if and
when they need to deposit *.eln files there.

bug#48108: Wishlist: Command to remove redundant eln cache directories [native-comp]

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

On Mon, Nov 28, 2022 at 5:48 PM Eli Zaretskii <eliz@gnu.org> wrote:

I still don't understand why would that constitute a bug.  It is created so
> that future compilation wouldn't have to worry about its existence if and
> when they need to deposit *.eln files there.
>

Because preventively creating subdirs that will, or will not, ever be used
is untidy.

Also, because if someone has taken the effort of setting the variable, it's
quite possible they don't want .eln files in ~/.emacs.d/

Yes, they should use an absolute path, and/or `startup-redirect-eln-cache'
(preferably).

[-- Attachment #2: Type: text/html, Size: 1204 bytes --]