bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Alan Mackenzie]

Hello, Emacs and Andrea.

With a native compiled recent master branch, do:

    emacs -Q
    M-x locate-library <RET> files <RET>

The result in the echo area is something like:


    Library is file ~/emacs/emacs.git/sub-master-5/lisp/files.el
                                                             ^^^

..  This is wrong.  The Emacs is _NOT_ running uncompiled Lisp, it's
running native compiled files.  The reply should be a file name ending
in ".eln".

This problem is surely also in the emacs-28 branch.

#########################################################################

The cause of the problem is in `locate-file' in files.el.  There, if a
..eln file is found, it's corresponding .el source is looked up in a hash
table, and the .el name returned instead.

This mechanism was inserted by the following patch:

commit 3224a443060a5f21bb910064fc06fe4432810355
Author: Andrea Corallo <akrl@sdf.org>
Date:   Sun Jul 19 10:46:24 2020 +0200

    Move eln files into dedicated cache directories

    When loading a elc file search for a corresponding eln one into
    `comp-eln-load-path' directories and load it if available.
    `comp-eln-load-path' contains by default two directory (user and
    system one).

It seems there is some use case for which returning the .el file name in
place of the .eln is appropriate.  It isn't appropriate for
locate-library.

I would suggest moving the hash table mechanism out of locate-file, and
into the other function which calls locate-file, where the substitution
is wanted.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Eli Zaretskii]

> Date: Fri, 24 Dec 2021 11:22:48 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> The cause of the problem is in `locate-file' in files.el.  There, if a
> ..eln file is found, it's corresponding .el source is looked up in a hash
> table, and the .el name returned instead.
> 
> This mechanism was inserted by the following patch:
> 
> commit 3224a443060a5f21bb910064fc06fe4432810355
> Author: Andrea Corallo <akrl@sdf.org>
> Date:   Sun Jul 19 10:46:24 2020 +0200
> 
>     Move eln files into dedicated cache directories
> 
>     When loading a elc file search for a corresponding eln one into
>     `comp-eln-load-path' directories and load it if available.
>     `comp-eln-load-path' contains by default two directory (user and
>     system one).
> 
> It seems there is some use case for which returning the .el file name in
> place of the .eln is appropriate.  It isn't appropriate for
> locate-library.
> 
> I would suggest moving the hash table mechanism out of locate-file, and
> into the other function which calls locate-file, where the substitution
> is wanted.

If you want this to be fixed on the emacs-28 branch, the above
proposal will not fly.  So either we will need a special, safer,
solution for the release branch, or we will have this solved only on
master.  (I prefer the former, FWIW.)

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> The cause of the problem is in `locate-file' in files.el.  There, if a
> ..eln file is found, it's corresponding .el source is looked up in a hash
> table, and the .el name returned instead.

This also breaks loadhist, but in a different way -- see bug#51308.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Andrea Corallo]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Emacs and Andrea.
>
> With a native compiled recent master branch, do:
>
>     emacs -Q
>     M-x locate-library <RET> files <RET>
>
> The result in the echo area is something like:
>
>
>     Library is file ~/emacs/emacs.git/sub-master-5/lisp/files.el
>                                                              ^^^
>
> ..  This is wrong.  The Emacs is _NOT_ running uncompiled Lisp, it's
> running native compiled files.  The reply should be a file name ending
> in ".eln".
>
> This problem is surely also in the emacs-28 branch.
>
> #########################################################################
>
> The cause of the problem is in `locate-file' in files.el.  There, if a
> ..eln file is found, it's corresponding .el source is looked up in a hash
> table, and the .el name returned instead.
>
> This mechanism was inserted by the following patch:
>
> commit 3224a443060a5f21bb910064fc06fe4432810355
> Author: Andrea Corallo <akrl@sdf.org>
> Date:   Sun Jul 19 10:46:24 2020 +0200
>
>     Move eln files into dedicated cache directories
>
>     When loading a elc file search for a corresponding eln one into
>     `comp-eln-load-path' directories and load it if available.
>     `comp-eln-load-path' contains by default two directory (user and
>     system one).
>
> It seems there is some use case for which returning the .el file name in
> place of the .eln is appropriate.  It isn't appropriate for
> locate-library.
>
> I would suggest moving the hash table mechanism out of locate-file, and
> into the other function which calls locate-file, where the substitution
> is wanted.

[resending as the mail was rejected by bug-gnu-emacs@debbugs.gnu.org]

Hi Alan,

The trouble in this area is that `locate-library' might be used by
existing programs to obtain the source file (.elc files are tipically in
the same directory where the corresponding .el are).

This was the big trouble also for `load-file-name' and that's why we had
to add `load-true-file-name', all other solution caused unacceptable
incompatibilities with existing software.

AFAIR `locate-library' had the same exact issue.

Best Regards

  Andrea

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Alan Mackenzie]

Hello, Andrea.

On Fri, Dec 24, 2021 at 13:18:13 +0000, Andrea Corallo wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Emacs and Andrea.

> > With a native compiled recent master branch, do:

> >     emacs -Q
> >     M-x locate-library <RET> files <RET>

> > The result in the echo area is something like:


> >     Library is file ~/emacs/emacs.git/sub-master-5/lisp/files.el
> >                                                              ^^^

> > ..  This is wrong.  The Emacs is _NOT_ running uncompiled Lisp, it's
> > running native compiled files.  The reply should be a file name ending
> > in ".eln".

> > This problem is surely also in the emacs-28 branch.

> > #########################################################################

> > The cause of the problem is in `locate-file' in files.el.  There, if a
> > ..eln file is found, it's corresponding .el source is looked up in a hash
> > table, and the .el name returned instead.

> > This mechanism was inserted by the following patch:

> > commit 3224a443060a5f21bb910064fc06fe4432810355
> > Author: Andrea Corallo <akrl@sdf.org>
> > Date:   Sun Jul 19 10:46:24 2020 +0200

> >     Move eln files into dedicated cache directories

> >     When loading a elc file search for a corresponding eln one into
> >     `comp-eln-load-path' directories and load it if available.
> >     `comp-eln-load-path' contains by default two directory (user and
> >     system one).

> > It seems there is some use case for which returning the .el file name in
> > place of the .eln is appropriate.  It isn't appropriate for
> > locate-library.

> > I would suggest moving the hash table mechanism out of locate-file, and
> > into the other function which calls locate-file, where the substitution
> > is wanted.

> [resending as the mail was rejected by bug-gnu-emacs@debbugs.gnu.org]

Apologies for the non-sensical address.

> Hi Alan,

> The trouble in this area is that `locate-library' might be used by
> existing programs to obtain the source file (.elc files are tipically in
> the same directory where the corresponding .el are).

Are you saying that programs obtain the path of the .elc file, then
expect to get the source file by removing the "c" at the end?  Possibly
something similar for .eln files?

> This was the big trouble also for `load-file-name' and that's why we had
> to add `load-true-file-name', all other solution caused unacceptable
> incompatibilities with existing software.

But it's broken locate-library.  The doc string states clearly that this
function "show[s] the PRECISE file name of Emacs library LIBRARY".
Currently, the function doesn't do this.

This was a nuisance for me yesterday and today, when I was trying to find
out whether a particular Lisp file was .el, .elc, or .eln.  I needed to
use the workaround of C-h v on load-history.

> AFAIR `locate-library' had the same exact issue.

> Best Regards

>   Andrea

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Andrea Corallo]

Alan Mackenzie <acm@muc.de> writes:

[...]

>> Hi Alan,
>
>> The trouble in this area is that `locate-library' might be used by
>> existing programs to obtain the source file (.elc files are tipically in
>> the same directory where the corresponding .el are).
>
> Are you saying that programs obtain the path of the .elc file, then
> expect to get the source file by removing the "c" at the end?

Precisely.

Despite what the docstring says we have to deal with the existing
software on the field.  IIRC at the time after a number of bug reports
was deemed that minimizing incompatibilities and making the
native-compiler as much transparent as possible was priority.

One option would be to update the docstring and add a second function to
return the true value similarly to what we did with
`load-true-file-name'.

The other option is of course what you've suggested.

Thanks!

  Andrea

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Alan Mackenzie]

Hello, Andrea.

On Sat, Dec 25, 2021 at 10:43:51 +0000, Andrea Corallo wrote:
> Alan Mackenzie <acm@muc.de> writes:

> [...]

> >> Hi Alan,

> >> The trouble in this area is that `locate-library' might be used by
> >> existing programs to obtain the source file (.elc files are tipically in
> >> the same directory where the corresponding .el are).

> > Are you saying that programs obtain the path of the .elc file, then
> > expect to get the source file by removing the "c" at the end?

> Precisely.

> Despite what the docstring says we have to deal with the existing
> software on the field.  IIRC at the time after a number of bug reports
> was deemed that minimizing incompatibilities and making the
> native-compiler as much transparent as possible was priority.

> One option would be to update the docstring and add a second function to
> return the true value similarly to what we did with
> `load-true-file-name'.

> The other option is of course what you've suggested.

Another option would be to print the true library name when
INTERACTIVE-CALL (an argument) is non-nil.  Or we could add yet another
&optional argument.

The doc string needs fixing whatever we do, I think.

None of this feels very satisfactory, but Emacs is nearly 50 years old.
;-)

> Thanks!

>   Andrea

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#52773: Master and (probably) emacs-28: locate-library hides .eln files.

[Eli Zaretskii]

> Date: Sat, 25 Dec 2021 11:25:19 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: 52773@debbugs.gnu.org, acm@muc.de
> 
> Another option would be to print the true library name when
> INTERACTIVE-CALL (an argument) is non-nil.  Or we could add yet another
> &optional argument.

Better do both: add another argument that the interactive call will
set.