How does one find out what file a library has been loaded from?

How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Emacs.

Forgive me if I've asked this before, but how does one determine what
file a library has been loaded from?  In particular, whether that file
is a source file, a byte compiled file or a native compiled file.

I've tried looking at load-history, and it tells me cc-engine was loaded
from 

  "/home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc"

..  I think that's a lie.  I suspect I'm dealing with a native compiled
cc-engine, here.  In fact, the string ".eln" doesn't occur anywhere in
load-history, and I _know_ some of the files are native compiled - my
build was configured with-native-compilation.

So, how do I find this out?

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Tue, 19 Jul 2022 10:52:08 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Hello, Emacs.
> 
> Forgive me if I've asked this before, but how does one determine what
> file a library has been loaded from?  In particular, whether that file
> is a source file, a byte compiled file or a native compiled file.
> 
> I've tried looking at load-history, and it tells me cc-engine was loaded
> from 
> 
>   "/home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc"
> 
> ..  I think that's a lie.  I suspect I'm dealing with a native compiled
> cc-engine, here.  In fact, the string ".eln" doesn't occur anywhere in
> load-history, and I _know_ some of the files are native compiled - my
> build was configured with-native-compilation.
> 
> So, how do I find this out?

What do you like to find? whether it was loaded from cc-engine.elc or
cc-engine-XXXXXX.eln?  If so, why does it matter?

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Tue, Jul 19, 2022 at 15:39:11 +0300, Eli Zaretskii wrote:
> > Date: Tue, 19 Jul 2022 10:52:08 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > Hello, Emacs.

> > Forgive me if I've asked this before, but how does one determine what
> > file a library has been loaded from?  In particular, whether that file
> > is a source file, a byte compiled file or a native compiled file.

> > I've tried looking at load-history, and it tells me cc-engine was loaded
> > from 

> >   "/home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc"

> > ..  I think that's a lie.  I suspect I'm dealing with a native compiled
> > cc-engine, here.  In fact, the string ".eln" doesn't occur anywhere in
> > load-history, and I _know_ some of the files are native compiled - my
> > build was configured with-native-compilation.

> > So, how do I find this out?

> What do you like to find? whether it was loaded from cc-engine.elc or
> cc-engine-XXXXXX.eln?

Exactly that, yes.

> If so, why does it matter?

It matters a great deal.  There's the purely philosophical point that one
should be able to control and understand ones own Emacs.  A further point
is that Emacs should not deceive its users.

There's the point that if you're doing benchmark timings, the results are
meaningless if you don't know what you're timing.

The reason I asked was I was seeing a bug in what I suspected to be a
..eln, but didn't see it in .elc's, and thus suspected a native compiler
bug.  I've seen other things since, and don't suspect that any more.

But I'd still like to be able to know what file I'm executing.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Andrea Corallo]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Eli.
>
> On Tue, Jul 19, 2022 at 15:39:11 +0300, Eli Zaretskii wrote:
>> > Date: Tue, 19 Jul 2022 10:52:08 +0000
>> > From: Alan Mackenzie <acm@muc.de>
>
>> > Hello, Emacs.
>
>> > Forgive me if I've asked this before, but how does one determine what
>> > file a library has been loaded from?  In particular, whether that file
>> > is a source file, a byte compiled file or a native compiled file.
>
>> > I've tried looking at load-history, and it tells me cc-engine was loaded
>> > from 
>
>> >   "/home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc"
>
>> > ..  I think that's a lie.  I suspect I'm dealing with a native compiled
>> > cc-engine, here.  In fact, the string ".eln" doesn't occur anywhere in
>> > load-history, and I _know_ some of the files are native compiled - my
>> > build was configured with-native-compilation.
>
>> > So, how do I find this out?
>
>> What do you like to find? whether it was loaded from cc-engine.elc or
>> cc-engine-XXXXXX.eln?
>
> Exactly that, yes.
>
>> If so, why does it matter?
>
> It matters a great deal.  There's the purely philosophical point that one
> should be able to control and understand ones own Emacs.  A further point
> is that Emacs should not deceive its users.
>
> There's the point that if you're doing benchmark timings, the results are
> meaningless if you don't know what you're timing.
>
> The reason I asked was I was seeing a bug in what I suspected to be a
> ..eln, but didn't see it in .elc's, and thus suspected a native compiler
> bug.  I've seen other things since, and don't suspect that any more.
>
> But I'd still like to be able to know what file I'm executing.

We coudln't change the `load-history' content as unfortunatelly it was
breaking many packages (tested).

If you have a native compiled function (say `find-file') to get the eln
file containing it one can use.

(native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))

Hope it helps

  Andrea

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Tue, 19 Jul 2022 15:01:12 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > What do you like to find? whether it was loaded from cc-engine.elc or
> > cc-engine-XXXXXX.eln?
> 
> Exactly that, yes.
> 
> > If so, why does it matter?
> 
> It matters a great deal.  There's the purely philosophical point that one
> should be able to control and understand ones own Emacs.

I didn't major in philosophy, so I cannot help you here, sorry.

> A further point is that Emacs should not deceive its users.

It doesn't.

> There's the point that if you're doing benchmark timings, the results are
> meaningless if you don't know what you're timing.

You are timing compiled Lisp code.  How exactly was it compiled
shouldn't matter _in_principle_, exactly as you don't have any easy
way of knowing what version of byte-compiler and byte-optimizer was
used to produce a .elc file -- and the differences can be significant.

If you do need to make this kind of distinctions, you have to use
"different means".  Like disassembly of the byte-code or looking if
cc-engine-XXXXXX.eln file is in your eln-cache, or using "C-h f" on
some function from cc-engine, where Emacs will tell you whether it's
byte-compiled ot natively-compiled (you can also use
subr-native-elisp-p directly if you want).

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

> There's the point that if you're doing benchmark timings, the results are
> meaningless if you don't know what you're timing.

`C-h o <foo> RET` should tell you whether that function is "native"
or not.  Maybe we should tweak the language to say "byte compiled" for
the non-native compiled functions, tho.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Tue, Jul 19, 2022 at 18:50:00 +0300, Eli Zaretskii wrote:
> > Date: Tue, 19 Jul 2022 15:01:12 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > What do you like to find? whether it was loaded from cc-engine.elc or
> > > cc-engine-XXXXXX.eln?

> > Exactly that, yes.

> > > If so, why does it matter?

[ .... ]

> > A further point is that Emacs should not deceive its users.

> It doesn't.

It most assuredly does.  The doc string for load-history says that 

    FILE-NAME is the name of a file that has been loaded into Emacs.

This is untrue.  What is true is that it is the "NOTIONAL" file that has
been loaded into Emacs.... or that it is the name of the source file
with a "c" appended.

When the function of load-history was changed, why was its doc string
not amended?  (Yes, I am volunteering to take on this task.)

> > There's the point that if you're doing benchmark timings, the
> > results are meaningless if you don't know what you're timing.

> You are timing compiled Lisp code.  How exactly was it compiled
> shouldn't matter _in_principle_, ....

You might well want to compare the speed of byte compiled code with the
same source native compiled, as many of us have already attempted to do.

> .... exactly as you don't have any easy way of knowing what version of
> byte-compiler and byte-optimizer was used to produce a .elc file --
> and the differences can be significant.

Eli, that's .....  No, I won't write it.

> If you do need to make this kind of distinctions, you have to use
> "different means".

Or do you mean "difficult means"?  Let me propose that there should be
an easy way of finding this out.

> Like disassembly of the byte-code or looking if cc-engine-XXXXXX.eln
> file is in your eln-cache, or using "C-h f" on some function from
> cc-engine, where Emacs will tell you whether it's byte-compiled or
> natively-compiled (you can also use subr-native-elisp-p directly if
> you want).

I don't want this.

It is clear that load-history no longer supports all its use cases.
Andrea has reported that trying to update it lead to too many problems.

So, how about a new additional variable called something like
load-file-history which would be like load-history, just it would store
the name of the source file (if known) as well as the name of the loaded
file?  In time (?10 years), load-history could be deprecated and
eventually superseded.

Or, less dramatically, an alist with the notional name in entries' cars,
and the actual loaded file name in the cdrs?  With this, it would be
trivial to find the loaded file name from the information in
load-history.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Tue, 19 Jul 2022 17:07:09 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > A further point is that Emacs should not deceive its users.
> 
> > It doesn't.
> 
> It most assuredly does.  The doc string for load-history says that 
> 
>     FILE-NAME is the name of a file that has been loaded into Emacs.
> 
> This is untrue.

Not really (please take a good look at what the code actually does).
But if you are bothered by that detail, I'm okay with having a note
there regarding *.eln files.  (Somehow, I'm not sure you will settle
for that.)

> > You are timing compiled Lisp code.  How exactly was it compiled
> > shouldn't matter _in_principle_, ....
> 
> You might well want to compare the speed of byte compiled code with the
> same source native compiled, as many of us have already attempted to do.

If you want to do that, just knowing what was actually loaded won't
help you, because you will have to actually _prevent_ Emacs from
loading the .eln files, and that's not easy and currently not really
supported on the user level, at least not conveniently.  So you will
have to rename directories and stuff, and once you are there,
load-history is the last thing you will worry about, because you will
know in advance what Emacs loads, as you force it to do that yourself.

> Or do you mean "difficult means"?  Let me propose that there should be
> an easy way of finding this out.

Andrea gave you one way; I gave another.  None of them is difficult,
please don't exaggerate.

> It is clear that load-history no longer supports all its use cases.
> Andrea has reported that trying to update it lead to too many problems.

Yes, and therefore we won't change load-history any time soon.  Please
use the other ways that were proposed, even if you for some reason I
cannot understand don't like them.

> So, how about a new additional variable called something like
> load-file-history which would be like load-history, just it would store
> the name of the source file (if known) as well as the name of the loaded
> file?

No, we won't have that.  It isn't needed, from my POV, and having yet
another load-related path list will complicate the part of Emacs that
is already mind-boggling.

Again, you have been pointed to two ways of getting the information
you want, and that is more than enough for a corner use case such as
this one.

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Tue, Jul 19, 2022 at 22:13:53 +0300, Eli Zaretskii wrote:
> > Date: Tue, 19 Jul 2022 17:07:09 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > A further point is that Emacs should not deceive its users.

> > > It doesn't.

> > It most assuredly does.  The doc string for load-history says that 

> >     FILE-NAME is the name of a file that has been loaded into Emacs.

> > This is untrue.

> Not really (please take a good look at what the code actually does).

If it loads a .eln file, and says it has loaded a .elc file, that is an
untruth.  Not a "sort of not quite true", but a blatant untruth.  I had a
look at the relevant code in lread.c some while ago.

> But if you are bothered by that detail, I'm okay with having a note
> there regarding *.eln files.  (Somehow, I'm not sure you will settle
> for that.)

I will write a patch for the doc string and another for the Elisp manual.
I'm not happy with the state of things, but will probably have to accept
it.

> > > You are timing compiled Lisp code.  How exactly was it compiled
> > > shouldn't matter _in_principle_, ....

> > You might well want to compare the speed of byte compiled code with
> > the same source native compiled, as many of us have already attempted
> > to do.

> If you want to do that, just knowing what was actually loaded won't
> help you, because you will have to actually _prevent_ Emacs from
> loading the .eln files, and that's not easy and currently not really
> supported on the user level, at least not conveniently.  So you will
> have to rename directories and stuff, and once you are there,
> load-history is the last thing you will worry about, because you will
> know in advance what Emacs loads, as you force it to do that yourself.

Yes.  I don't think this is good.  But I suspect it's my own fault for
not paying attention to the emacs-devel threads about these things a year
or two ago.  That you input M-x load-file RET ~/foo.elc RET, and get a
different file loaded instead doesn't strike me as at all good.

> > Or do you mean "difficult means"?  Let me propose that there should be
> > an easy way of finding this out.

> Andrea gave you one way; I gave another.  None of them is difficult,
> please don't exaggerate.

There's nothing particularly difficult in any of Emacs if you're prepared
to put in the time and energy to find out about it.  The method Andrea
gave is not easy to remember, and (having looked for it) is not to be
found in the Elisp manual.  It involves the obscure undocumented
abstraction "native compilation unit".  But it is certainly a lot better
than no method at all.

> > It is clear that load-history no longer supports all its use cases.
> > Andrea has reported that trying to update it lead to too many problems.

> Yes, and therefore we won't change load-history any time soon.  Please
> use the other ways that were proposed, even if you for some reason I
> cannot understand don't like them.

These other ways jar horribly with what used to be the philosophy (I know
you don't like the word) of Emacs, of being open and honest with users.
I shouldn't have to use obscure workarounds to discover what should be
open and obvious.

> > So, how about a new additional variable called something like
> > load-file-history which would be like load-history, just it would
> > store the name of the source file (if known) as well as the name of
> > the loaded file?

> No, we won't have that.  It isn't needed, from my POV, and having yet
> another load-related path list will complicate the part of Emacs that
> is already mind-boggling.

> Again, you have been pointed to two ways of getting the information
> you want, and that is more than enough for a corner use case such as
> this one.

I will set about amending the doc string and manual entry for
load-history.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

>> If you want to do that, just knowing what was actually loaded won't
>> help you, because you will have to actually _prevent_ Emacs from
>> loading the .eln files, and that's not easy and currently not really
>> supported on the user level, at least not conveniently.  So you will
>> have to rename directories and stuff, and once you are there,
>> load-history is the last thing you will worry about, because you will
>> know in advance what Emacs loads, as you force it to do that yourself.
>
> Yes.  I don't think this is good.

AFAIK we all agree on that.  Complaining about it won't help.

>> Andrea gave you one way; I gave another.  None of them is difficult,
>> please don't exaggerate.
>
> There's nothing particularly difficult in any of Emacs if you're prepared
> to put in the time and energy to find out about it.  The method Andrea
> gave is not easy to remember, and (having looked for it) is not to be
> found in the Elisp manual.  It involves the obscure undocumented
> abstraction "native compilation unit".  But it is certainly a lot better
> than no method at all.

Designing a better method will need to know what is the intended use.
IIUC your main purpose is to check whether you're running source code,
byte-code, or native code, and for that you shouldn't look for a file
name and infer the result from the file name (which could be
misleading: a .elc can contain non-byte code, and a .eln can contain
byte-code (and source code as well)) but instead you're better off
checking with `byte-code-function-p` and `subrp` (or
`subr-native-elisp-p`).

> These other ways jar horribly with what used to be the philosophy (I know
> you don't like the word) of Emacs, of being open and honest with users.

Hi Alan.  I'm Stefan, and this is Eli.  The same guys you've known for,
what, 20 years?  It's not like we changed philosophy along the way.
As the email to which you replied explained it's just that maintaining
backward compatibility prevented us from making the information
available in "the obvious way".

The info is still very much available out in the open, tho.
And we're still interested in figuring out *how* to make it more
easily available.  Andrea did a great job integrating the native
compiler while trying to preserve compatibility.  But it's not like this
is the end of the story: we can and should definitely improve it (maybe
starting with `describe-function`).

> I shouldn't have to use obscure workarounds to discover what should be
> open and obvious.

So, please help us.  You're venting your frustration, but we suffer from
those issues (rather than benefit from them) just like you do.

> I will set about amending the doc string and manual entry for
> load-history.

Thanks,


        Stefan

Re: How does one find out what file a library has been loaded from?

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There's the point that if you're doing benchmark timings, the results are
>> meaningless if you don't know what you're timing.
>
> `C-h o <foo> RET` should tell you whether that function is "native"
> or not.  Maybe we should tweak the language to say "byte compiled" for
> the non-native compiled functions, tho.

(For what is worth) I agree with that.

  Andrea

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Wed, Jul 20, 2022 at 11:47:11 +0000, Alan Mackenzie wrote:

> On Tue, Jul 19, 2022 at 22:13:53 +0300, Eli Zaretskii wrote:
> > > Date: Tue, 19 Jul 2022 17:07:09 +0000
> > > Cc: emacs-devel@gnu.org
> > > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> If it loads a .eln file, and says it has loaded a .elc file, that is an
> untruth.  Not a "sort of not quite true", but a blatant untruth.  I had
> a look at the relevant code in lread.c some while ago.

> > But if you are bothered by that detail, I'm okay with having a note
> > there regarding *.eln files.  (Somehow, I'm not sure you will settle
> > for that.)

> I will write a patch for the doc string and another for the Elisp manual.
> I'm not happy with the state of things, but will probably have to accept
> it.

[ .... ]

Here's a first preliminary effort at amending loading.texi:



diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 54fc16ec9f..32ab08fcfd 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -1033,11 +1033,11 @@ Where Defined
 @cindex where was a symbol defined
 
 @defun symbol-file symbol &optional type
-This function returns the name of the file that defined @var{symbol}.
-If @var{type} is @code{nil}, then any kind of definition is acceptable.
-If @var{type} is @code{defun}, @code{defvar}, or @code{defface}, that
-specifies function definition, variable definition, or face definition
-only.
+This function returns a file name associated with the file that
+defined @var{symbol} (@pxref{eln files}).  If @var{type} is
+@code{nil}, then any kind of definition is acceptable.  If @var{type}
+is @code{defun}, @code{defvar}, or @code{defface}, that specifies
+function definition, variable definition, or face definition only.
 
 The value is normally an absolute file name.  It can also be @code{nil},
 if the definition is not associated with any file.  If @var{symbol}
@@ -1049,14 +1049,15 @@ Where Defined
 @code{load-history}.
 
 @defvar load-history
-The value of this variable is an alist that associates the names of
-loaded library files with the names of the functions and variables
-they defined, as well as the features they provided or required.
+The value of this variable is an alist that associates names
+associated with loaded library files (@pxref{eln files}) with the
+names of the functions and variables the files defined, as well as the
+features they provided or required.
 
 Each element in this alist describes one loaded library (including
 libraries that are preloaded at startup).  It is a list whose @sc{car}
-is the absolute file name of the library (a string).  The rest of the
-list elements have these forms:
+is an absolute file name associated with the library (a string)
+(@pxref{eln files}).  The rest of the list elements have these forms:
 
 @table @code
 @item @var{var}
@@ -1083,9 +1084,30 @@ Where Defined
 @code{eval-buffer} on a buffer that is not visiting a file.
 @end defvar
 
-  The command @code{eval-region} updates @code{load-history}, but does so
-by adding the symbols defined to the element for the file being visited,
-rather than replacing that element.  @xref{Eval}.
+@anchor{eln files} For backwards compatibility, @code{load-history}
+stores and @code{symbol-file} returns the name of a notional byte
+compiled @file{.elc} file in the same directory as its source file
+when the real file loaded from is a natively compiled file elsewhere.
+This @file{.elc} file may or may not actually exist.  For other files,
+their absolute file names are used.  If you want to find the actual
+file loaded from, and you suspect if may really be a native compiled
+file, something like the following should help.  You need to know the
+name of a function which hasn't been advised, say @var{foo}, defined
+in the suspected native compiled file.  Then
+
+@lisp
+(let ((foo-fun (symbol-function #'FOO)))
+       (and foo-fun (subr-native-elisp-p foo-fun)
+            (native-comp-unit-file (subr-native-comp-unit foo-fun))))
+@end lisp
+
+@noindent
+will return either the name of the native compiled file defining
+@var{foo}, or @code{nil} if there is no such file.
+
+The command @code{eval-region} updates @code{load-history}, but does
+so by adding the symbols defined to the element for the file being
+visited, rather than replacing that element.  @xref{Eval}.
 
 @kindex function-history @r{(function symbol property)}
 In addition to @code{load-history}, every function keeps track of its


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Wed, 20 Jul 2022 20:34:05 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Here's a first preliminary effort at amending loading.texi:

Thanks, but in "needs work"(TM).

>  @defun symbol-file symbol &optional type
> -This function returns the name of the file that defined @var{symbol}.
> -If @var{type} is @code{nil}, then any kind of definition is acceptable.
> -If @var{type} is @code{defun}, @code{defvar}, or @code{defface}, that
> -specifies function definition, variable definition, or face definition
> -only.
> +This function returns a file name associated with the file that
> +defined @var{symbol} (@pxref{eln files}).  If @var{type} is
> +@code{nil}, then any kind of definition is acceptable.  If @var{type}
> +is @code{defun}, @code{defvar}, or @code{defface}, that specifies
> +function definition, variable definition, or face definition only.

This change is for the worse: it introduces a vague and confusing
notion of "file name associated with the file that defines" a symbol.
This should be removed from the patch, as it doesn't add any useful
information, just muddies the waters.

>  @defvar load-history
> -The value of this variable is an alist that associates the names of
> -loaded library files with the names of the functions and variables
> -they defined, as well as the features they provided or required.
> +The value of this variable is an alist that associates names
> +associated with loaded library files (@pxref{eln files}) with the
> +names of the functions and variables the files defined, as well as the
> +features they provided or required.

Likewise.

>  Each element in this alist describes one loaded library (including
>  libraries that are preloaded at startup).  It is a list whose @sc{car}
> -is the absolute file name of the library (a string).  The rest of the
> -list elements have these forms:
> +is an absolute file name associated with the library (a string)
> +(@pxref{eln files}).  The rest of the list elements have these forms:

Likewise.

> -  The command @code{eval-region} updates @code{load-history}, but does so
> -by adding the symbols defined to the element for the file being visited,
> -rather than replacing that element.  @xref{Eval}.
> +@anchor{eln files} For backwards compatibility, @code{load-history}
> +stores and @code{symbol-file} returns the name of a notional byte
> +compiled @file{.elc} file in the same directory as its source file
> +when the real file loaded from is a natively compiled file elsewhere.
> +This @file{.elc} file may or may not actually exist.  For other files,
> +their absolute file names are used.

This last sentence is "out of the blue": what "other files"?  The text
should also have a cross-reference to where native compilation is
described in the manual.

>                                        If you want to find the actual
> +file loaded from, and you suspect if may really be a native compiled
> +file, something like the following should help.  You need to know the
> +name of a function which hasn't been advised, say @var{foo}, defined
> +in the suspected native compiled file.  Then
> +
> +@lisp
> +(let ((foo-fun (symbol-function #'FOO)))
> +       (and foo-fun (subr-native-elisp-p foo-fun)
> +            (native-comp-unit-file (subr-native-comp-unit foo-fun))))
> +@end lisp
> +
> +@noindent
> +will return either the name of the native compiled file defining
> +@var{foo}, or @code{nil} if there is no such file.

This is not a good way of documenting some technique in this manual.
The way we describe such stuff is by documenting the functions a
program needs to use, not by giving a random example which calls the
functions without any documentation of the functions themselves.

Also, native-comp-unit-file doesn't exist in a build without native
compilation support, so some feature test is missing.

Finally, "FOO" is not how we refer to a meta-syntactic variable in the
manual: we use @var{foo} instead.

> +The command @code{eval-region} updates @code{load-history}, but does
> +so by adding the symbols defined to the element for the file being
> +visited, rather than replacing that element.  @xref{Eval}.

This part should be before the text which explains the issues with
loading *.eln files.

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Alan Mackenzie <acm@muc.de>, Eli Zaretskii <eliz@gnu.org>,
>         emacs-devel@gnu.org
> Date: Wed, 20 Jul 2022 18:36:24 +0000
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> There's the point that if you're doing benchmark timings, the results are
> >> meaningless if you don't know what you're timing.
> >
> > `C-h o <foo> RET` should tell you whether that function is "native"
> > or not.  Maybe we should tweak the language to say "byte compiled" for
> > the non-native compiled functions, tho.
> 
> (For what is worth) I agree with that.

I've now made this change on the release branch.

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Thu, Jul 21, 2022 at 09:13:03 +0300, Eli Zaretskii wrote:
> > Date: Wed, 20 Jul 2022 20:34:05 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > Here's a first preliminary effort at amending loading.texi:

> Thanks, but it "needs work"(TM).

> >  @defun symbol-file symbol &optional type
> > -This function returns the name of the file that defined @var{symbol}.
> > -If @var{type} is @code{nil}, then any kind of definition is acceptable.
> > -If @var{type} is @code{defun}, @code{defvar}, or @code{defface}, that
> > -specifies function definition, variable definition, or face definition
> > -only.
> > +This function returns a file name associated with the file that
> > +defined @var{symbol} (@pxref{eln files}).  If @var{type} is
> > +@code{nil}, then any kind of definition is acceptable.  If @var{type}
> > +is @code{defun}, @code{defvar}, or @code{defface}, that specifies
> > +function definition, variable definition, or face definition only.

> This change is for the worse: it introduces a vague and confusing
> notion of "file name associated with the file that defines" a symbol.
> This should be removed from the patch, as it doesn't add any useful
> information, just muddies the waters.

It's accurate, though.  The current text is not accurate.  The situation
it is describing is vague and confusing.

Would this strategy be an improvement: "This function returns a file
name.  When the file from which the function was loaded was a source
file or byte compiled file .......  When that file was a native compiled
file ......."?

[ .... ]

> > -  The command @code{eval-region} updates @code{load-history}, but does so
> > -by adding the symbols defined to the element for the file being visited,
> > -rather than replacing that element.  @xref{Eval}.
> > +@anchor{eln files} For backwards compatibility, @code{load-history}
> > +stores and @code{symbol-file} returns the name of a notional byte
> > +compiled @file{.elc} file in the same directory as its source file
> > +when the real file loaded from is a natively compiled file elsewhere.
> > +This @file{.elc} file may or may not actually exist.  For other files,
> > +their absolute file names are used.

> This last sentence is "out of the blue": what "other files"?

Files other than "a natively compiled file elsewhere".  But I'll admit it
doesn't read well, yet.

> The text should also have a cross-reference to where native
> compilation is described in the manual.

OK.

> >                                        If you want to find the actual
> > +file loaded from, and you suspect if may really be a native compiled
> > +file, something like the following should help.  You need to know the
> > +name of a function which hasn't been advised, say @var{foo}, defined
> > +in the suspected native compiled file.  Then
> > +
> > +@lisp
> > +(let ((foo-fun (symbol-function #'FOO)))
> > +       (and foo-fun (subr-native-elisp-p foo-fun)
> > +            (native-comp-unit-file (subr-native-comp-unit foo-fun))))
> > +@end lisp
> > +
> > +@noindent
> > +will return either the name of the native compiled file defining
> > +@var{foo}, or @code{nil} if there is no such file.

> This is not a good way of documenting some technique in this manual.
> The way we describe such stuff is by documenting the functions a
> program needs to use, not by giving a random example which calls the
> functions without any documentation of the functions themselves.

OK.  But I think here could be an exception.  Describing the functions
separately on their own page will not help users to get the loaded file
name without a great deal of research.  I've tried out this recipe and
it works, but I don't yet know what these native-comp-unit functions are
for, what they do in any detail, or even what a compilation-unit is.
The functions are not already in the Elisp manual, and their doc strings
are somewhat terse.

I still think it would be a good thing to be able to get the name of an
actual load file from the .elc name stored in load-history without
having to go through the intermediate step of knowing a function name
defined by it.

> Also, native-comp-unit-file doesn't exist in a build without native
> compilation support, so some feature test is missing.

Do you mean a test in the TexInfo sources which would test whether it's
necessary to include that example in the finished manual?

> Finally, "FOO" is not how we refer to a meta-syntactic variable in the
> manual: we use @var{foo} instead.

Sorry.  I thought that @var{FOO} would not work in @lisp, but I tried it
out and it does work.  I've already corrected it.

> > +The command @code{eval-region} updates @code{load-history}, but does
> > +so by adding the symbols defined to the element for the file being
> > +visited, rather than replacing that element.  @xref{Eval}.

> This part should be before the text which explains the issues with
> loading *.eln files.

OK, that's easily fixed.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

> It's accurate, though.  The current text is not accurate.  The situation
> it is describing is vague and confusing.
>
> Would this strategy be an improvement: "This function returns a file
> name.  When the file from which the function was loaded was a source
> file or byte compiled file .......  When that file was a native compiled
> file ......."?

BTW, the design of the native code compiler is that it aims to be
transparent, a bit like a JIT compiler would be transparent: whether the
native code comes from a .eln file or was generated on-the-fly or
combined from several .eln files (or any other option you can come
up with) should ideally not matter (ELisp code shouldn't need to know
and end-users shouldn't need to know either (barring bugs or curiosity
of course)).

Clearly we do want to make that info visible somehow (e.g. for debugging
purposes, but also because we can never make something really
transparent), but it's very different from `load-history` which keeps
track of information that is relevant: whether /usr/share/.../org.el, or
usr/share/.../org.elc, or ~/.emacs.d/elpa/.../org.el, or ... was loaded
can result in different outcomes even if there's no bug anywhere.
Those 4 files do not have the same semantics., e.g. because it's not the
same version of Org, or because of the `eval-when-compile` in `org.el`.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Thu, 21 Jul 2022 17:37:19 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > +This function returns a file name associated with the file that
> > > +defined @var{symbol} (@pxref{eln files}).  If @var{type} is
> > > +@code{nil}, then any kind of definition is acceptable.  If @var{type}
> > > +is @code{defun}, @code{defvar}, or @code{defface}, that specifies
> > > +function definition, variable definition, or face definition only.
> 
> > This change is for the worse: it introduces a vague and confusing
> > notion of "file name associated with the file that defines" a symbol.
> > This should be removed from the patch, as it doesn't add any useful
> > information, just muddies the waters.
> 
> It's accurate, though.

No, it isn't accurate, because it doesn't say anything definitive.

What exactly did you want to say here, and why?  (See, I didn't even
understand you intention, from reading that text.)

> > >                                        If you want to find the actual
> > > +file loaded from, and you suspect if may really be a native compiled
> > > +file, something like the following should help.  You need to know the
> > > +name of a function which hasn't been advised, say @var{foo}, defined
> > > +in the suspected native compiled file.  Then
> > > +
> > > +@lisp
> > > +(let ((foo-fun (symbol-function #'FOO)))
> > > +       (and foo-fun (subr-native-elisp-p foo-fun)
> > > +            (native-comp-unit-file (subr-native-comp-unit foo-fun))))
> > > +@end lisp
> > > +
> > > +@noindent
> > > +will return either the name of the native compiled file defining
> > > +@var{foo}, or @code{nil} if there is no such file.
> 
> > This is not a good way of documenting some technique in this manual.
> > The way we describe such stuff is by documenting the functions a
> > program needs to use, not by giving a random example which calls the
> > functions without any documentation of the functions themselves.
> 
> OK.  But I think here could be an exception.  Describing the functions
> separately on their own page will not help users to get the loaded file
> name without a great deal of research.

You can describe them, and then show the example.  Or fill in the
blanks as part of the functions' description.

>                                         I've tried out this recipe and
> it works, but I don't yet know what these native-comp-unit functions are
> for, what they do in any detail, or even what a compilation-unit is.
> The functions are not already in the Elisp manual, and their doc strings
> are somewhat terse.

If you cannot figure it out from the code, feel free to ask questions.

> I still think it would be a good thing to be able to get the name of an
> actual load file from the .elc name stored in load-history without
> having to go through the intermediate step of knowing a function name
> defined by it.

Did you try comp-el-to-eln-filename?

> > Also, native-comp-unit-file doesn't exist in a build without native
> > compilation support, so some feature test is missing.
> 
> Do you mean a test in the TexInfo sources which would test whether it's
> necessary to include that example in the finished manual?

No, I mean a test in the @example code.  You want that snippet to be
complete, so that readers could copy it into their programs, right?

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Stefan.

On Thu, Jul 21, 2022 at 13:52:43 -0400, Stefan Monnier wrote:
> > It's accurate, though.  The current text is not accurate.  The situation
> > it is describing is vague and confusing.

> > Would this strategy be an improvement: "This function returns a file
> > name.  When the file from which the function was loaded was a source
> > file or byte compiled file .......  When that file was a native compiled
> > file ......."?

> BTW, the design of the native code compiler is that it aims to be
> transparent, a bit like a JIT compiler would be transparent:

I think the word you should be using is not "transparent", but "opaque".

> whether the native code comes from a .eln file or was generated
> on-the-fly or combined from several .eln files (or any other option
> you can come up with) should ideally not matterd.

That is a gross departure from long valued Emacs philosophy - that there
should be nothing "transparent" (i.e. opaque) to the user, and that it's
not for us maintainers to predict what users want to do in their hacking
(and thus restrict Emacs to just doing those predicted things).

For crying out loud, you can even get the position of the gap in Emacs
Lisp.  That is surely a candidate for being "transparent" (i.e. opaque)
if ever there were one.

> (ELisp code shouldn't need to know and end-users shouldn't need to
> know either (barring bugs or curiosity of course))

For those reasons, amongs others, there shouldn't be these transparent (i.e.
opaque) things.

> Clearly we do want to make that info visible somehow (e.g. for debugging
> purposes, but also because we can never make something really
> transparent), but it's very different from `load-history` which keeps
> track of information that is relevant: whether /usr/share/.../org.el, or
> usr/share/.../org.elc, or ~/.emacs.d/elpa/.../org.el, or ... was loaded
> can result in different outcomes even if there's no bug anywhere.
> Those 4 files do not have the same semantics., e.g. because it's not the
> same version of Org, or because of the `eval-when-compile` in `org.el`.

I think we've violently in agreement, here.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

>> (ELisp code shouldn't need to know and end-users shouldn't need to
>> know either (barring bugs or curiosity of course))
> For those reasons, amongs others, there shouldn't be these transparent (i.e.
> opaque) things.

You're confusing "transparent" and "opaque".  Opaque means users
can't know, transparent means they don't need to know.  Very different.

In any case, my point is that this design decision to make the native
code compilation transparent should be reflected in the docs and should
inform how we document the features.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli. 

On Thu, Jul 21, 2022 at 20:53:28 +0300, Eli Zaretskii wrote:
> > Date: Thu, 21 Jul 2022 17:37:19 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > +This function returns a file name associated with the file that
> > > > +defined @var{symbol} (@pxref{eln files}).  If @var{type} is
> > > > +@code{nil}, then any kind of definition is acceptable.  If @var{type}
> > > > +is @code{defun}, @code{defvar}, or @code{defface}, that specifies
> > > > +function definition, variable definition, or face definition only.

> > > This change is for the worse: it introduces a vague and confusing
> > > notion of "file name associated with the file that defines" a symbol.
> > > This should be removed from the patch, as it doesn't add any useful
> > > information, just muddies the waters.

> > It's accurate, though.

> No, it isn't accurate, because it doesn't say anything definitive.

It says (or implies) there is nothing definitive to say.

I think it says as much as you can say about the connection between the
name of the loaded file and the file name recorded in load-history in a
single sentence.

> What exactly did you want to say here, and why?  (See, I didn't even
> understand you intention, from reading that text.)

That there exists such a relationship between the file and the recorded
file name, but avoiding the falsehood that the file name is (in general)
the name of that file.  As an example there is a relationship between

    (i)
    /home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc,
    the file name recorded in load-history;

and
  
    (ii)
    "/home/acm/.emacs.d/eln-cache/29.0.50-850ec122/cc-engine-fae36ae5-5d7a60de.eln",
    the loaded file

..  



> > > >                                        If you want to find the actual
> > > > +file loaded from, and you suspect if may really be a native compiled
> > > > +file, something like the following should help.  You need to know the
> > > > +name of a function which hasn't been advised, say @var{foo}, defined
> > > > +in the suspected native compiled file.  Then
> > > > +
> > > > +@lisp
> > > > +(let ((foo-fun (symbol-function #'FOO)))
> > > > +       (and foo-fun (subr-native-elisp-p foo-fun)
> > > > +            (native-comp-unit-file (subr-native-comp-unit foo-fun))))
> > > > +@end lisp
> > > > +
> > > > +@noindent
> > > > +will return either the name of the native compiled file defining
> > > > +@var{foo}, or @code{nil} if there is no such file.

> > > This is not a good way of documenting some technique in this manual.
> > > The way we describe such stuff is by documenting the functions a
> > > program needs to use, not by giving a random example which calls the
> > > functions without any documentation of the functions themselves.

> > OK.  But I think here could be an exception.  Describing the functions
> > separately on their own page will not help users to get the loaded file
> > name without a great deal of research.

> You can describe them, and then show the example.  Or fill in the
> blanks as part of the functions' description.

Why is giving the code snippet, as I proposed, not a good thing?  Would
it be better to write a new function incorporating the procedure, and
document that?

> >                                         I've tried out this recipe and
> > it works, but I don't yet know what these native-comp-unit functions are
> > for, what they do in any detail, or even what a compilation-unit is.
> > The functions are not already in the Elisp manual, and their doc strings
> > are somewhat terse.

> If you cannot figure it out from the code, feel free to ask questions.

I can figure out just about anything from Emacs's code (apart from the
philosophical things), but there are only so many hours in a day.

> > I still think it would be a good thing to be able to get the name of an
> > actual load file from the .elc name stored in load-history without
> > having to go through the intermediate step of knowing a function name
> > defined by it.

> Did you try comp-el-to-eln-filename?

No.  How could I have known that such a function exists?  It generates
file names which might not name existing files.  It doesn't seem ideal
for the purpose.

> > > Also, native-comp-unit-file doesn't exist in a build without native
> > > compilation support, so some feature test is missing.

> > Do you mean a test in the TexInfo sources which would test whether it's
> > necessary to include that example in the finished manual?

> No, I mean a test in the @example code.  You want that snippet to be
> complete, so that readers could copy it into their programs, right?

Ah, I see.  Yes, you're right.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Stefan.

On Thu, Jul 21, 2022 at 14:37:06 -0400, Stefan Monnier wrote:
> >> (ELisp code shouldn't need to know and end-users shouldn't need to
> >> know either (barring bugs or curiosity of course))
> > For those reasons, amongs others, there shouldn't be these transparent (i.e.
> > opaque) things.

> You're confusing "transparent" and "opaque".  Opaque means users
> can't know, transparent means they don't need to know.  Very different.

Let's put it this way, if the Emacs maintainers had decided to block as
far as possible users' access to the actual file code has been loaded
from, the result would look pretty much like what we've got.

> In any case, my point is that this design decision to make the native
> code compilation transparent should be reflected in the docs and should
> inform how we document the features.

If you are right in your distinction between "transparent" and "opaque"
the matters we are talking about fall into the category "opaque".  This
is a massive philosophical departure, before which the philosopy was to
make NOTHING in Emacs opaque to the user.  If there was any debate about
this, then I must have missed it.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

> Let's put it this way, if the Emacs maintainers had decided to block as
> far as possible users' access to the actual file code has been loaded
> from, the result would look pretty much like what we've got.

I find this statement rather offensive.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Thu, 21 Jul 2022 20:39:48 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > > > +This function returns a file name associated with the file that
> > > > > +defined @var{symbol} (@pxref{eln files}).  If @var{type} is
> > > > > +@code{nil}, then any kind of definition is acceptable.  If @var{type}
> > > > > +is @code{defun}, @code{defvar}, or @code{defface}, that specifies
> > > > > +function definition, variable definition, or face definition only.
> 
> > > > This change is for the worse: it introduces a vague and confusing
> > > > notion of "file name associated with the file that defines" a symbol.
> > > > This should be removed from the patch, as it doesn't add any useful
> > > > information, just muddies the waters.
> 
> > > It's accurate, though.
> 
> > No, it isn't accurate, because it doesn't say anything definitive.
> 
> It says (or implies) there is nothing definitive to say.

But that is not true.  The original text says something definitive and
useful.  Adding information to it should NOT lose any of the
information that was there to begin with, because that information
still correct and not obsolete.

> I think it says as much as you can say about the connection between the
> name of the loaded file and the file name recorded in load-history in a
> single sentence.

It is never a good thing to say "as much as you can say" if it leaves
the reader less wise.

> > What exactly did you want to say here, and why?  (See, I didn't even
> > understand you intention, from reading that text.)
> 
> That there exists such a relationship between the file and the recorded
> file name, but avoiding the falsehood that the file name is (in general)
> the name of that file.

Then TRT is to say what happens normally, and then add the description
of what happens in exceptional cases, including the description of
when the exceptions happen.

> As an example there is a relationship between
> 
>     (i)
>     /home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc,
>     the file name recorded in load-history;
> 
> and
>   
>     (ii)
>     "/home/acm/.emacs.d/eln-cache/29.0.50-850ec122/cc-engine-fae36ae5-5d7a60de.eln",
>     the loaded file

What does symbol-file has to do with any such "relationship"?  All you
want to _add_ is that in an Emacs with native-compilation, loading a
.elc file can eventually load the corresponding .eln file instead.  So
why not just say that?

> > You can describe them, and then show the example.  Or fill in the
> > blanks as part of the functions' description.
> 
> Why is giving the code snippet, as I proposed, not a good thing?

Because it uses functions not described in the manual.

> Would it be better to write a new function incorporating the
> procedure, and document that?

In principle, yes.  But we need to discuss first what would that
function be.  See below.

> > >                                         I've tried out this recipe and
> > > it works, but I don't yet know what these native-comp-unit functions are
> > > for, what they do in any detail, or even what a compilation-unit is.
> > > The functions are not already in the Elisp manual, and their doc strings
> > > are somewhat terse.
> 
> > If you cannot figure it out from the code, feel free to ask questions.
> 
> I can figure out just about anything from Emacs's code (apart from the
> philosophical things), but there are only so many hours in a day.

You are splitting hair, but let me rephrase: if you cannot figure it
out in some reasonable amount of time, feel free to ask.

> > > I still think it would be a good thing to be able to get the name of an
> > > actual load file from the .elc name stored in load-history without
> > > having to go through the intermediate step of knowing a function name
> > > defined by it.
> 
> > Did you try comp-el-to-eln-filename?
> 
> No.  How could I have known that such a function exists?

I just told you about it.  I told you about it not as an accusation,
but as a way to help you find the best way of solving your problem.

> It generates file names which might not name existing files.  It
> doesn't seem ideal for the purpose.

Then I think you should describe the purpose better and in more
detail.  What exactly are you trying to accomplish and why?  What is
the data from which you start and what is the data you want to obtain
as result?  In particular, is the starting point a function or a file
name? or something else entirely?

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Sat, Jul 23, 2022 at 13:11:54 +0300, Eli Zaretskii wrote:
> > Date: Thu, 21 Jul 2022 20:39:48 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > > > +This function returns a file name associated with the file that
> > > > > > +defined @var{symbol} (@pxref{eln files}).  If @var{type} is
> > > > > > +@code{nil}, then any kind of definition is acceptable.  If @var{type}
> > > > > > +is @code{defun}, @code{defvar}, or @code{defface}, that specifies
> > > > > > +function definition, variable definition, or face definition only.

> > > > > This change is for the worse: it introduces a vague and confusing
> > > > > notion of "file name associated with the file that defines" a symbol.
> > > > > This should be removed from the patch, as it doesn't add any useful
> > > > > information, just muddies the waters.

> > > > It's accurate, though.

> > > No, it isn't accurate, because it doesn't say anything definitive.

> > It says (or implies) there is nothing definitive to say.

> But that is not true.  The original text says something definitive and
> useful.  Adding information to it should NOT lose any of the
> information that was there to begin with, because that information
> still correct and not obsolete.

I just don't understand you saying that the information is still correct.
In about "half" the cases, when a .eln file has been loaded, the current
text in the Elisp manual and load-history doc string is no longer
correct.  The file given in load-history is not the file that was loaded
into Emacs.

> > I think it says as much as you can say about the connection between the
> > name of the loaded file and the file name recorded in load-history in a
> > single sentence.

> It is never a good thing to say "as much as you can say" if it leaves
> the reader less wise.

It is proving difficult to amend that text to make it accurate, while not
confusing its reader.

> > > What exactly did you want to say here, and why?  (See, I didn't even
> > > understand you intention, from reading that text.)

> > That there exists such a relationship between the file and the recorded
> > file name, but avoiding the falsehood that the file name is (in general)
> > the name of that file.

> Then TRT is to say what happens normally, and then add the description
> of what happens in exceptional cases, including the description of
> when the exceptions happen.

We don't have exceptional cases here.  We have two normal cases, one
where we load a .elc file, the other where we load a .eln file.  I think
both somehow have to be described as normal cases.

> > As an example there is a relationship between

> >     (i)
> >     /home/acm/emacs/emacs.git/sub-master-5/lisp/progmodes/cc-engine.elc,
> >     the file name recorded in load-history;

> > and

> >     (ii)
> >     "/home/acm/.emacs.d/eln-cache/29.0.50-850ec122/cc-engine-fae36ae5-5d7a60de.eln",
> >     the loaded file

> What does symbol-file has to do with any such "relationship"?  All you
> want to _add_ is that in an Emacs with native-compilation, loading a
> .elc file can eventually load the corresponding .eln file instead.  So
> why not just say that?

Maybe I'll try that.  One reason not to say it is that we intend the .eln
case to become the more common case, don't we?  Also, we really ought to
describe what /home/acm/..../cc-engine.elc _is_ in the documentation.
There's no guarantee that that file exists.

> > > You can describe them, and then show the example.  Or fill in the
> > > blanks as part of the functions' description.

> > Why is giving the code snippet, as I proposed, not a good thing?

> Because it uses functions not described in the manual.

OK.

> > Would it be better to write a new function incorporating the
> > procedure, and document that?

> In principle, yes.  But we need to discuss first what would that
> function be.  See below.

> > > > I've tried out this recipe and it works, but I don't yet know
> > > > what these native-comp-unit functions are for, what they do in
> > > > any detail, or even what a compilation-unit is.  The functions
> > > > are not already in the Elisp manual, and their doc strings are
> > > > somewhat terse.

> > I can figure out just about anything from Emacs's code (apart from the
> > philosophical things), but there are only so many hours in a day.

> You are splitting hair, but let me rephrase: if you cannot figure it
> out in some reasonable amount of time, feel free to ask.

All right.  I think it would be useful to tie down the abstraction
"native compilation unit".  It is probably something quite simple, yet
there is no "unit" in the Elisp manual index.  What is a "native
compilation unit"?

> > > > I still think it would be a good thing to be able to get the name
> > > > of an actual load file from the .elc name stored in load-history
> > > > without having to go through the intermediate step of knowing a
> > > > function name defined by it.

> > > Did you try comp-el-to-eln-filename?

> > No.  How could I have known that such a function exists?

> I just told you about it.  I told you about it not as an accusation,
> but as a way to help you find the best way of solving your problem.

OK, thanks.  But I still don't think I could have found out the existence
of this function without asing "are there any relevant
functions/variables to what I'm trying to do?".

> > It generates file names which might not name existing files.  It
> > doesn't seem ideal for the purpose.

> Then I think you should describe the purpose better and in more
> detail.

I simply wish to know the file from which a function has been loaded, or
the loaded file corresponding to some source file.  I would like to know
whether this file is a source file, a .elc, or a .eln.

> What exactly are you trying to accomplish and why?

There are lots of reasons I might want to know the loaded file, some of
which have already come up in the thread.  I might want to be sure I've
built Emacs with native compilation.  I might be interested in
benchmarking or RAM occupancy.  I might wish to firm up my mental model
of my Emacs (this indeed has been the case for me).

> What is the data from which you start and what is the data you want to
> obtain as result?

The name of a source file or function, and the name of the matching file
which was loaded.

> In particular, is the starting point a function or a file name? or
> something else entirely?

Either a function or a file name.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Sun, 24 Jul 2022 11:27:56 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > But that is not true.  The original text says something definitive and
> > useful.  Adding information to it should NOT lose any of the
> > information that was there to begin with, because that information
> > still correct and not obsolete.
> 
> I just don't understand you saying that the information is still correct.
> In about "half" the cases, when a .eln file has been loaded, the current
> text in the Elisp manual and load-history doc string is no longer
> correct.  The file given in load-history is not the file that was loaded
> into Emacs.

Natively-compiled Lisp packages are still an exception; people are
definitely less familiar with them than with the byte-compiled ones..
They might not be one day, at which time we might need to change the
text, but for now they are.  So describing the simple case of .elc
first and the more complex .eln case afterwards sounds TRT to me.  As
a nice benefit, it will allow simple and clear description in both
cases, with a simple logic that will not confuse the reader.

> It is proving difficult to amend that text to make it accurate, while not
> confusing its reader.

That's my point: you don't need to amend.  That text is still correct,
as long as no *.eln files are involved.  Just _add_ to it the
description of what happens when *.eln files _are_ around.  This is
good methodology in manuals: describe the simple case first and the
more complex one after it.  Trying to describe both together usually
leads to a much more complex and harder to read text.

If you are still unconvinced, just leave it alone, and I will suggest
the change to that text.

> > Then TRT is to say what happens normally, and then add the description
> > of what happens in exceptional cases, including the description of
> > when the exceptions happen.
> 
> We don't have exceptional cases here.  We have two normal cases, one
> where we load a .elc file, the other where we load a .eln file.  I think
> both somehow have to be described as normal cases.

See above: "normal" and "exception" have slightly different meanings
here.

> One reason not to say it is that we intend the .eln case to become
> the more common case, don't we?

No need to worry about that, not yet.

> Also, we really ought to describe what /home/acm/..../cc-engine.elc
> _is_ in the documentation.  There's no guarantee that that file
> exists.

You are saying that if the .elc file doesn't exist, load-history still
names it and not the .el file instead?  That's not what I see here: I
see the .el files where .elc don't exist.

> All right.  I think it would be useful to tie down the abstraction
> "native compilation unit".  It is probably something quite simple, yet
> there is no "unit" in the Elisp manual index.  What is a "native
> compilation unit"?

Let's get back to that if we decide that the code proposed by Andrea
is really the best solution for this.  If we decide that the best
solution doesn't involve any functions that have "compilation unit" in
their names, we don't need to define that term.

> > > > Did you try comp-el-to-eln-filename?
> 
> > > No.  How could I have known that such a function exists?
> 
> > I just told you about it.  I told you about it not as an accusation,
> > but as a way to help you find the best way of solving your problem.
> 
> OK, thanks.  But I still don't think I could have found out the existence
> of this function without asing "are there any relevant
> functions/variables to what I'm trying to do?".
> 
> > > It generates file names which might not name existing files.  It
> > > doesn't seem ideal for the purpose.
> 
> > Then I think you should describe the purpose better and in more
> > detail.
> 
> I simply wish to know the file from which a function has been loaded, or
> the loaded file corresponding to some source file.  I would like to know
> whether this file is a source file, a .elc, or a .eln.

That is still too general.  I suggest to describe several specific use
cases in more detail, because the solutions might be different.

In general, you can use load-history or symbol-file to know which file
was loaded.  If load-history says it was a .elc file, and the build
supports native compilation, you need to see if the function was
natively-compiled, using

    (subr-native-elisp-p (symbol-function FUNC))

where FUNC is a function's symbol.  If the above returns non-nil, then
comp-el-to-eln-rel-filename will produce the base name of the .eln
file corresponding to a .el file.

The above are the building blocks; what's left is to use them in a
particular case.

> I might want to be sure I've built Emacs with native compilation.

Use native-comp-available-p for this.

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

> I just don't understand you saying that the information is still correct.
> In about "half" the cases, when a .eln file has been loaded, the current
> text in the Elisp manual and load-history doc string is no longer
> correct.  The file given in load-history is not the file that was loaded
> into Emacs.

Because of the design decision to make native code compilation
transparent, `load-history` does not mean "this is exactly the file that
was loaded directly" but rather something like: "this is the file from
which we loaded the definitions, tho the path to loading them may
include turning the code into some other representation such a native
code, while being (hopefully) faithful to the file's semantics".

So if you actually need to know more precisely where the code comes
from, you need to look at supplemental information (such as the one
Andrea gave you).

>> > > Did you try comp-el-to-eln-filename?
>> > No.  How could I have known that such a function exists?
>> I just told you about it.  I told you about it not as an accusation,
>> but as a way to help you find the best way of solving your problem.
> OK, thanks.  But I still don't think I could have found out the existence
> of this function without asing "are there any relevant
> functions/variables to what I'm trying to do?".

Which is why your effort to try and document this is worthwhile, indeed.
[ It may also inform improvements to the API to make it easier to find,
  but documenting the status quo is an indispensable first step.  ]

>> Then I think you should describe the purpose better and in more detail.
> I simply wish to know the file from which a function has been loaded, or
> the loaded file corresponding to some source file.

"I" as in "I the human being", and "simply" as in "out of pure curiosity"?
What are you going to do with that knowledge?  Would you be satisfied
with knowing only if it's native code rather than byte code (rather
than see the gory file name with all its hashes and stuff)?

> I would like to know whether this file is a source file, a .elc, or
> a .eln.

`C-h o <function> RET` should tell you that without needing any knowledge
of ELisp.  Admittedly, it doesn't quite satisfy your requirements since
the input is now a function name rather than a file name.

>> What exactly are you trying to accomplish and why?
> There are lots of reasons I might want to know the loaded file, some of
> which have already come up in the thread.  I might want to be sure I've
> built Emacs with native compilation.

I think the canonical way to test this is

    (featurep 'native-compile)

> I might be interested in benchmarking or RAM occupancy.

I don't see the connection, but `subrp` or `byte-code-function-p` or
just printing the output of `symbol-function` on a relevant function
should tell you how your code was compiled.  No need for any file name.

>> What is the data from which you start and what is the data you want to
>> obtain as result?
> The name of a source file or function, and the name of the matching file
> which was loaded.

Wanting a "matching file" is IMO the result of your mental model not
firmed up enough yet, then :-)

After all, even if the answer is a nasty `.../blabla.eln` you still
won't know whether your function is implemented as native code since
that `.eln` can contain (and generate) byte code.  And that's not
just theory, it does happen in practice.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Sun, 24 Jul 2022 15:16:46 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> In general, you can use load-history or symbol-file to know which file
> was loaded.  If load-history says it was a .elc file, and the build
> supports native compilation, you need to see if the function was
> natively-compiled, using
> 
>     (subr-native-elisp-p (symbol-function FUNC))
> 
> where FUNC is a function's symbol.  If the above returns non-nil, then
> comp-el-to-eln-rel-filename will produce the base name of the .eln
> file corresponding to a .el file.
> 
> The above are the building blocks; what's left is to use them in a
> particular case.

Btw, we could extend symbol-file to support a new kind of TYPE arg,
to ask about a natively-compiled symbol, and then we could return the
.eln file instead of the .elc file, when appropriate.

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Sun, 24 Jul 2022 18:37:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> CC: emacs-devel@gnu.org
> 
> Btw, we could extend symbol-file to support a new kind of TYPE arg,
> to ask about a natively-compiled symbol, and then we could return the
> .eln file instead of the .elc file, when appropriate.

Or maybe add a new optional argument native-p, which, if non-nil,
would report the .eln file, if any, from which the symbol was loaded.

Re: How does one find out what file a library has been loaded from?

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 3818 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

In that context, how does find-function work?

>> I just don't understand you saying that the information is still correct.
>> In about "half" the cases, when a .eln file has been loaded, the current
>> text in the Elisp manual and load-history doc string is no longer
>> correct.  The file given in load-history is not the file that was loaded
>> into Emacs.
>
> Because of the design decision to make native code compilation
> transparent, `load-history` does not mean "this is exactly the file that
> was loaded directly" but rather something like: "this is the file from
> which we loaded the definitions, tho the path to loading them may
> include turning the code into some other representation such a native
> code, while being (hopefully) faithful to the file's semantics".
>
> So if you actually need to know more precisely where the code comes
> from, you need to look at supplemental information (such as the one
> Andrea gave you).
>
>>> > > Did you try comp-el-to-eln-filename?
>>> > No.  How could I have known that such a function exists?
>>> I just told you about it.  I told you about it not as an accusation,
>>> but as a way to help you find the best way of solving your problem.
>> OK, thanks.  But I still don't think I could have found out the existence
>> of this function without asing "are there any relevant
>> functions/variables to what I'm trying to do?".
>
> Which is why your effort to try and document this is worthwhile, indeed.
> [ It may also inform improvements to the API to make it easier to find,
>   but documenting the status quo is an indispensable first step.  ]
>
>>> Then I think you should describe the purpose better and in more detail.
>> I simply wish to know the file from which a function has been loaded, or
>> the loaded file corresponding to some source file.
>
> "I" as in "I the human being", and "simply" as in "out of pure curiosity"?
> What are you going to do with that knowledge?  Would you be satisfied
> with knowing only if it's native code rather than byte code (rather
> than see the gory file name with all its hashes and stuff)?
>
>> I would like to know whether this file is a source file, a .elc, or
>> a .eln.
>
> `C-h o <function> RET` should tell you that without needing any knowledge
> of ELisp.  Admittedly, it doesn't quite satisfy your requirements since
> the input is now a function name rather than a file name.
>
>>> What exactly are you trying to accomplish and why?
>> There are lots of reasons I might want to know the loaded file, some of
>> which have already come up in the thread.  I might want to be sure I've
>> built Emacs with native compilation.
>
> I think the canonical way to test this is
>
>     (featurep 'native-compile)
>
>> I might be interested in benchmarking or RAM occupancy.
>
> I don't see the connection, but `subrp` or `byte-code-function-p` or
> just printing the output of `symbol-function` on a relevant function
> should tell you how your code was compiled.  No need for any file name.
>
>>> What is the data from which you start and what is the data you want to
>>> obtain as result?
>> The name of a source file or function, and the name of the matching file
>> which was loaded.
>
> Wanting a "matching file" is IMO the result of your mental model not
> firmed up enough yet, then :-)
>
> After all, even if the answer is a nasty `.../blabla.eln` you still
> won't know whether your function is implemented as native code since
> that `.eln` can contain (and generate) byte code.  And that's not
> just theory, it does happen in practice.
>
>
>         Stefan
>
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Tue, 19 Jul 2022 15:32:40 +0000
> 
> If you have a native compiled function (say `find-file') to get the eln
> file containing it one can use.
> 
> (native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))

Andrea, does anything similar to subr-native-comp-unit exists for
other types of symbols accepted by symbol-file: defvar and defface?

If not, then the only way to produce the same information for them
would be to generate the base name of the .eln file with
comp-el-to-eln-rel-filename, and then look for that file along
native-comp-eln-load-path, right?

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

T.V Raman [2022-07-24 08:49:51] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> In that context, how does find-function work?

I don't understand the question: it works the same as it did.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Sun, 24 Jul 2022 08:49:51 -0700
> 
> In that context, how does find-function work?

find-function finds the source file, not the compiled file.

Re: How does one find out what file a library has been loaded from?

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> Date: Tue, 19 Jul 2022 15:32:40 +0000
>> 
>> If you have a native compiled function (say `find-file') to get the eln
>> file containing it one can use.
>> 
>> (native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))
>
> Andrea, does anything similar to subr-native-comp-unit exists for
> other types of symbols accepted by symbol-file: defvar and defface?

No it does not exists.

The reason why the compilation unit concept was introduced is to have
the GC capable of unloading the eln when possible, this mechanism is
only related to functions that are indeed memory mapped from the loaded
shared libraries.  Variables etc are just regular variables (tipically
defined at eln load time) so they didn't required any new mechanism.

> If not, then the only way to produce the same information for them
> would be to generate the base name of the .eln file with
> comp-el-to-eln-rel-filename, and then look for that file along
> native-comp-eln-load-path, right?

Yes I think so.

  Andrea

Re: How does one find out what file a library has been loaded from?

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 630 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


Reason I asked:
if find-function knows where the function is, shouldn't it be possible
to then tell what file it came from, and so consequently doesn't that
answer the more general question of what  library a file came from?
> T.V Raman [2022-07-24 08:49:51] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> In that context, how does find-function work?
>
> I don't understand the question: it works the same as it did.
>
>
>         Stefan
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: How does one find out what file a library has been loaded from?

[Stefan Monnier]

> if find-function knows where the function is, shouldn't it be possible
> to then tell what file it came from, and so consequently doesn't that
> answer the more general question of what  library a file came from?

There are different notions of origin at play here.  `find-function`
tries to find the source code origin.  Alan seems to want to know the
origin of the code in a shape as close as possible to that present in
the Emacs session.


        Stefan

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Sun, 24 Jul 2022 17:46:02 +0000
> 
> >> (native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))
> >
> > Andrea, does anything similar to subr-native-comp-unit exists for
> > other types of symbols accepted by symbol-file: defvar and defface?
> 
> No it does not exists.
> 
> The reason why the compilation unit concept was introduced is to have
> the GC capable of unloading the eln when possible, this mechanism is
> only related to functions that are indeed memory mapped from the loaded
> shared libraries.  Variables etc are just regular variables (tipically
> defined at eln load time) so they didn't required any new mechanism.
> 
> > If not, then the only way to produce the same information for them
> > would be to generate the base name of the .eln file with
> > comp-el-to-eln-rel-filename, and then look for that file along
> > native-comp-eln-load-path, right?
> 
> Yes I think so.

Andrea and Alan, please review and test the version of symbol-file
below that I intend to install soon.  TIA.

(defun locate-eln-file (eln-file)
  "Locate a natively-compiled ELN-FILE by searching its load path.
This function looks in directories named by `native-comp-eln-load-path'."
  (or (locate-file-internal (concat comp-native-version-dir "/" eln-file)
		   native-comp-eln-load-path)
      (locate-file-internal
       ;; Preloaded *.eln files live in the preloaded/ subdirectory of
       ;; the last entry in `native-comp-eln-load-path'.
       (concat comp-native-version-dir "/preloaded/" eln-file)
       (last native-comp-eln-load-path))))

(defun symbol-file (symbol &optional type native-p)
  "Return the name of the file that defined SYMBOL.
The value is normally an absolute file name.  It can also be nil,
if the definition is not associated with any file.  If SYMBOL
specifies an autoloaded function, the value can be a relative
file name without extension.

If TYPE is nil, then any kind of definition is acceptable.  If
TYPE is `defun', `defvar', or `defface', that specifies function
definition, variable definition, or face definition only.
Otherwise TYPE is assumed to be a symbol property.

If NATIVE-P is nil, and SYMBOL was loaded from a .eln file, this
function will return the absolute file name of that .eln file,
if found.

This function only works for symbols defined in Lisp files.  For
symbols that are defined in C files, use `help-C-file-name'
instead."
  (if (and (or (null type) (eq type 'defun))
	   (symbolp symbol)
	   (autoloadp (symbol-function symbol)))
      (nth 1 (symbol-function symbol))
    (if (and native-p (or (null type) (eq type 'defun))
	     (symbolp symbol)
	     (subr-native-elisp-p (symbol-function symbol)))
	;; native-comp-unit-file returns unnormalized file names.
	(expand-file-name (native-comp-unit-file (subr-native-comp-unit
						  (symbol-function symbol))))
      (let ((elc-file
	     (catch 'found
	       (pcase-dolist (`(,file . ,elems) load-history)
		 (when (if type
			   (if (eq type 'defvar)
			       ;; Variables are present just as their
			       ;; names.
			       (member symbol elems)
			     ;; Many other types are represented as
			     ;; (TYPE . NAME).
			     (or (member (cons type symbol) elems)
				 (memq
				  symbol
				  (alist-get type
					     (alist-get 'define-symbol-props
							elems)))))
			 ;; We accept all types, so look for variable def
			 ;; and then for any other kind.
			 (or (member symbol elems)
			     (let ((match (rassq symbol elems)))
			       (and match
				    (not (eq 'require (car match)))))))
		   (throw 'found file))))))
	;; If they asked for the .eln file, try to find it.
	(or (and elc-file
		 native-p
		 (let* ((sans-ext (file-name-sans-extension elc-file))
			(el-file
			 (and (fboundp 'zlib-available-p)
			      (zlib-available-p)
			      (concat sans-ext ".el.gz")))
			(el-file-backup (concat sans-ext ".el")))
		   (or (and el-file (file-exists-p el-file))
		       (and (file-exists-p el-file-backup)
			    (setq el-file el-file-backup))
		       (setq el-file nil))
		   (if (stringp el-file)
		       (locate-eln-file
			(comp-el-to-eln-rel-filename el-file)))))
	    elc-file)))))

Re: How does one find out what file a library has been loaded from?

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: acm@muc.de, emacs-devel@gnu.org
>> Date: Sun, 24 Jul 2022 17:46:02 +0000
>> 
>> >> (native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))
>> >
>> > Andrea, does anything similar to subr-native-comp-unit exists for
>> > other types of symbols accepted by symbol-file: defvar and defface?
>> 
>> No it does not exists.
>> 
>> The reason why the compilation unit concept was introduced is to have
>> the GC capable of unloading the eln when possible, this mechanism is
>> only related to functions that are indeed memory mapped from the loaded
>> shared libraries.  Variables etc are just regular variables (tipically
>> defined at eln load time) so they didn't required any new mechanism.
>> 
>> > If not, then the only way to produce the same information for them
>> > would be to generate the base name of the .eln file with
>> > comp-el-to-eln-rel-filename, and then look for that file along
>> > native-comp-eln-load-path, right?
>> 
>> Yes I think so.
>
> Andrea and Alan, please review and test the version of symbol-file
> below that I intend to install soon.  TIA.
>
> (defun locate-eln-file (eln-file)
>   "Locate a natively-compiled ELN-FILE by searching its load path.
> This function looks in directories named by `native-comp-eln-load-path'."
>   (or (locate-file-internal (concat comp-native-version-dir "/" eln-file)
> 		   native-comp-eln-load-path)
>       (locate-file-internal
>        ;; Preloaded *.eln files live in the preloaded/ subdirectory of
>        ;; the last entry in `native-comp-eln-load-path'.
>        (concat comp-native-version-dir "/preloaded/" eln-file)
>        (last native-comp-eln-load-path))))
>
> (defun symbol-file (symbol &optional type native-p)
>   "Return the name of the file that defined SYMBOL.
> The value is normally an absolute file name.  It can also be nil,
> if the definition is not associated with any file.  If SYMBOL
> specifies an autoloaded function, the value can be a relative
> file name without extension.
>
> If TYPE is nil, then any kind of definition is acceptable.  If
> TYPE is `defun', `defvar', or `defface', that specifies function
> definition, variable definition, or face definition only.
> Otherwise TYPE is assumed to be a symbol property.
>
> If NATIVE-P is nil, and SYMBOL was loaded from a .eln file, this
> function will return the absolute file name of that .eln file,
> if found.
>
> This function only works for symbols defined in Lisp files.  For
> symbols that are defined in C files, use `help-C-file-name'
> instead."
>   (if (and (or (null type) (eq type 'defun))
> 	   (symbolp symbol)
> 	   (autoloadp (symbol-function symbol)))
>       (nth 1 (symbol-function symbol))
>     (if (and native-p (or (null type) (eq type 'defun))
> 	     (symbolp symbol)
> 	     (subr-native-elisp-p (symbol-function symbol)))
> 	;; native-comp-unit-file returns unnormalized file names.
> 	(expand-file-name (native-comp-unit-file (subr-native-comp-unit
> 						  (symbol-function symbol))))
>       (let ((elc-file
> 	     (catch 'found
> 	       (pcase-dolist (`(,file . ,elems) load-history)
> 		 (when (if type
> 			   (if (eq type 'defvar)
> 			       ;; Variables are present just as their
> 			       ;; names.
> 			       (member symbol elems)
> 			     ;; Many other types are represented as
> 			     ;; (TYPE . NAME).
> 			     (or (member (cons type symbol) elems)
> 				 (memq
> 				  symbol
> 				  (alist-get type
> 					     (alist-get 'define-symbol-props
> 							elems)))))
> 			 ;; We accept all types, so look for variable def
> 			 ;; and then for any other kind.
> 			 (or (member symbol elems)
> 			     (let ((match (rassq symbol elems)))
> 			       (and match
> 				    (not (eq 'require (car match)))))))
> 		   (throw 'found file))))))
> 	;; If they asked for the .eln file, try to find it.
> 	(or (and elc-file
> 		 native-p
> 		 (let* ((sans-ext (file-name-sans-extension elc-file))
> 			(el-file
> 			 (and (fboundp 'zlib-available-p)
> 			      (zlib-available-p)
> 			      (concat sans-ext ".el.gz")))
> 			(el-file-backup (concat sans-ext ".el")))
> 		   (or (and el-file (file-exists-p el-file))
> 		       (and (file-exists-p el-file-backup)
> 			    (setq el-file el-file-backup))
> 		       (setq el-file nil))
> 		   (if (stringp el-file)
> 		       (locate-eln-file
> 			(comp-el-to-eln-rel-filename el-file)))))
> 	    elc-file)))))

Hi Eli,

I think this has some overlap with the code we have in
'maybe_swap_for_eln', but it does not look trivial to decouple it, so
I'm not sure is it worth the pain.

Other than that only note I've is that (in 'maybe_swap_for_eln1') we do
reject the .eln file if it's younger than the corresponding .elc one.  I
think we should mimic this as well here no?

Otherwise LGTM.

Thanks

  Andrea

Re: How does one find out what file a library has been loaded from?

[Alan Mackenzie]

Hello, Eli.

On Sun, Jul 31, 2022 at 15:52:33 +0300, Eli Zaretskii wrote:
> > From: Andrea Corallo <akrl@sdf.org>
> > Cc: acm@muc.de, emacs-devel@gnu.org
> > Date: Sun, 24 Jul 2022 17:46:02 +0000

> > >> (native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))

> > > Andrea, does anything similar to subr-native-comp-unit exists for
> > > other types of symbols accepted by symbol-file: defvar and defface?

> > No it does not exists.

> > The reason why the compilation unit concept was introduced is to have
> > the GC capable of unloading the eln when possible, this mechanism is
> > only related to functions that are indeed memory mapped from the loaded
> > shared libraries.  Variables etc are just regular variables (tipically
> > defined at eln load time) so they didn't required any new mechanism.

> > > If not, then the only way to produce the same information for them
> > > would be to generate the base name of the .eln file with
> > > comp-el-to-eln-rel-filename, and then look for that file along
> > > native-comp-eln-load-path, right?

> > Yes I think so.

> Andrea and Alan, please review and test the version of symbol-file
> below that I intend to install soon.  TIA.

Sorry to have taken so long.  I've deliberately avoided reading Andrea's
response.

This is brilliant!  It seems to work for me flawlessly.  There's just one
slight mistake:

[ .... ]

> If NATIVE-P is nil, and SYMBOL was loaded from a .eln file, this
> function will return the absolute file name of that .eln file,
> if found.

That should surely read "If NATIVE-P is non-nil, ....".
                                        ^^^^

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How does one find out what file a library has been loaded from?

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>> Cc: acm@muc.de, emacs-devel@gnu.org
>>> Date: Sun, 24 Jul 2022 17:46:02 +0000
>>> 
>>> >> (native-comp-unit-file (subr-native-comp-unit (symbol-function #'find-file)))
>>> >
>>> > Andrea, does anything similar to subr-native-comp-unit exists for
>>> > other types of symbols accepted by symbol-file: defvar and defface?
>>> 
>>> No it does not exists.
>>> 
>>> The reason why the compilation unit concept was introduced is to have
>>> the GC capable of unloading the eln when possible, this mechanism is
>>> only related to functions that are indeed memory mapped from the loaded
>>> shared libraries.  Variables etc are just regular variables (tipically
>>> defined at eln load time) so they didn't required any new mechanism.
>>> 
>>> > If not, then the only way to produce the same information for them
>>> > would be to generate the base name of the .eln file with
>>> > comp-el-to-eln-rel-filename, and then look for that file along
>>> > native-comp-eln-load-path, right?
>>> 
>>> Yes I think so.
>>
>> Andrea and Alan, please review and test the version of symbol-file
>> below that I intend to install soon.  TIA.
>>
>> (defun locate-eln-file (eln-file)
>>   "Locate a natively-compiled ELN-FILE by searching its load path.
>> This function looks in directories named by `native-comp-eln-load-path'."
>>   (or (locate-file-internal (concat comp-native-version-dir "/" eln-file)
>> 		   native-comp-eln-load-path)
>>       (locate-file-internal
>>        ;; Preloaded *.eln files live in the preloaded/ subdirectory of
>>        ;; the last entry in `native-comp-eln-load-path'.
>>        (concat comp-native-version-dir "/preloaded/" eln-file)
>>        (last native-comp-eln-load-path))))
>>
>> (defun symbol-file (symbol &optional type native-p)
>>   "Return the name of the file that defined SYMBOL.
>> The value is normally an absolute file name.  It can also be nil,
>> if the definition is not associated with any file.  If SYMBOL
>> specifies an autoloaded function, the value can be a relative
>> file name without extension.
>>
>> If TYPE is nil, then any kind of definition is acceptable.  If
>> TYPE is `defun', `defvar', or `defface', that specifies function
>> definition, variable definition, or face definition only.
>> Otherwise TYPE is assumed to be a symbol property.
>>
>> If NATIVE-P is nil, and SYMBOL was loaded from a .eln file, this
>> function will return the absolute file name of that .eln file,
>> if found.
>>
>> This function only works for symbols defined in Lisp files.  For
>> symbols that are defined in C files, use `help-C-file-name'
>> instead."
>>   (if (and (or (null type) (eq type 'defun))
>> 	   (symbolp symbol)
>> 	   (autoloadp (symbol-function symbol)))
>>       (nth 1 (symbol-function symbol))
>>     (if (and native-p (or (null type) (eq type 'defun))
>> 	     (symbolp symbol)
>> 	     (subr-native-elisp-p (symbol-function symbol)))
>> 	;; native-comp-unit-file returns unnormalized file names.
>> 	(expand-file-name (native-comp-unit-file (subr-native-comp-unit
>> 						  (symbol-function symbol))))
>>       (let ((elc-file
>> 	     (catch 'found
>> 	       (pcase-dolist (`(,file . ,elems) load-history)
>> 		 (when (if type
>> 			   (if (eq type 'defvar)
>> 			       ;; Variables are present just as their
>> 			       ;; names.
>> 			       (member symbol elems)
>> 			     ;; Many other types are represented as
>> 			     ;; (TYPE . NAME).
>> 			     (or (member (cons type symbol) elems)
>> 				 (memq
>> 				  symbol
>> 				  (alist-get type
>> 					     (alist-get 'define-symbol-props
>> 							elems)))))
>> 			 ;; We accept all types, so look for variable def
>> 			 ;; and then for any other kind.
>> 			 (or (member symbol elems)
>> 			     (let ((match (rassq symbol elems)))
>> 			       (and match
>> 				    (not (eq 'require (car match)))))))
>> 		   (throw 'found file))))))
>> 	;; If they asked for the .eln file, try to find it.
>> 	(or (and elc-file
>> 		 native-p
>> 		 (let* ((sans-ext (file-name-sans-extension elc-file))
>> 			(el-file
>> 			 (and (fboundp 'zlib-available-p)
>> 			      (zlib-available-p)
>> 			      (concat sans-ext ".el.gz")))
>> 			(el-file-backup (concat sans-ext ".el")))
>> 		   (or (and el-file (file-exists-p el-file))
>> 		       (and (file-exists-p el-file-backup)
>> 			    (setq el-file el-file-backup))
>> 		       (setq el-file nil))
>> 		   (if (stringp el-file)
>> 		       (locate-eln-file
>> 			(comp-el-to-eln-rel-filename el-file)))))
>> 	    elc-file)))))
>
> Hi Eli,
>
> I think this has some overlap with the code we have in
> 'maybe_swap_for_eln', but it does not look trivial to decouple it, so
> I'm not sure is it worth the pain.
>
> Other than that only note I've is that (in 'maybe_swap_for_eln1') we do
> reject the .eln file if it's younger than the corresponding .elc one.  I
> think we should mimic this as well here no?

I do realize now this is probably not a real case as: if (in
'maybe_swap_for_eln1') we reject the eln while loading because older
than the elc we give up entirely on loading native code, even if might
be available in another directory down in `native-comp-eln-load-path'.
So your code should be just fine in this respect.

Thanks

  Andrea

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 02 Aug 2022 08:43:53 +0000
> 
> > Other than that only note I've is that (in 'maybe_swap_for_eln1') we do
> > reject the .eln file if it's younger than the corresponding .elc one.  I
> > think we should mimic this as well here no?
> 
> I do realize now this is probably not a real case as: if (in
> 'maybe_swap_for_eln1') we reject the eln while loading because older
> than the elc we give up entirely on loading native code, even if might
> be available in another directory down in `native-comp-eln-load-path'.
> So your code should be just fine in this respect.

I'm not sure I understand: in what sense do we "give up entirely on
loading native code"?  I don't see in maybe_swap_for_eln1 or in
maybe_swap_for_eln anything that would mean something like that.  All
I see is that we continue using the .elc file if the timestamp test
fails.  So in that case, my changes would return the .eln file even
though it was not actually loaded.  We could document this corner use
case, and say that we return the .eln file even if it's outdated, or
we could add the timestamp test you suggested, and return the .elc
file if the test fails.

Or what am I missing?

Thanks.

Re: How does one find out what file a library has been loaded from?

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: acm@muc.de, emacs-devel@gnu.org
>> Date: Tue, 02 Aug 2022 08:43:53 +0000
>> 
>> > Other than that only note I've is that (in 'maybe_swap_for_eln1') we do
>> > reject the .eln file if it's younger than the corresponding .elc one.  I
>> > think we should mimic this as well here no?
>> 
>> I do realize now this is probably not a real case as: if (in
>> 'maybe_swap_for_eln1') we reject the eln while loading because older
>> than the elc we give up entirely on loading native code, even if might
>> be available in another directory down in `native-comp-eln-load-path'.
>> So your code should be just fine in this respect.
>
> I'm not sure I understand: in what sense do we "give up entirely on
> loading native code"?  I don't see in maybe_swap_for_eln1 or in
> maybe_swap_for_eln anything that would mean something like that.  All
> I see is that we continue using the .elc file if the timestamp test
> fails.

Yes correct, that's what I meant, we give up for this specific .elc file
instead of keep on searching.

> So in that case, my changes would return the .eln file even
> though it was not actually loaded.

Ops that's correct, I miss-read your code sorry.

> We could document this corner use case, and say that we return the
> .eln file even if it's outdated, or we could add the timestamp test
> you suggested, and return the .elc file if the test fails.

I think would be nicer to have the timestamp test.

Still if the filesystem state changes the result could be incorrect, ex
one could even remove the .eln file and even if the definition is coming
from it we'd return the .elc.  Perhaps we should document this?

Thanks

  Andrea

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> Date: Mon, 1 Aug 2022 19:31:47 +0000
> Cc: Andrea Corallo <akrl@sdf.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> This is brilliant!  It seems to work for me flawlessly.  There's just one
> slight mistake:
> 
> [ .... ]
> 
> > If NATIVE-P is nil, and SYMBOL was loaded from a .eln file, this
> > function will return the absolute file name of that .eln file,
> > if found.
> 
> That should surely read "If NATIVE-P is non-nil, ....".
>                                         ^^^^

Yes, of course.  Fixed.

With this change installed, feel free to review your proposed
documentation changes in this regard.

Re: How does one find out what file a library has been loaded from?

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 02 Aug 2022 14:13:58 +0000
> 
> > So in that case, my changes would return the .eln file even
> > though it was not actually loaded.
> 
> Ops that's correct, I miss-read your code sorry.
> 
> > We could document this corner use case, and say that we return the
> > .eln file even if it's outdated, or we could add the timestamp test
> > you suggested, and return the .elc file if the test fails.
> 
> I think would be nicer to have the timestamp test.
> 
> Still if the filesystem state changes the result could be incorrect, ex
> one could even remove the .eln file and even if the definition is coming
> from it we'd return the .elc.  Perhaps we should document this?

OK, thanks.  I added the timestamp test, fixed the typo spotted by
Alan, and installed the changes together with their documentation.