bug#6716: 23.2; Setting `find-function-source-path' has no effect.

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

Because `find-library-name' receives as its LIBRARY argument the full
path, but doesn't strip the directory part, setting
`find-function-source-path' has no effect on symbol finding -- Emacs
still tries the path guessed according to load path (which is not
correct in case you have the Elisp sources in directory different from
the compiled files).

This simple change seems to fix it for me:

diff --git a/lisp/emacs-lisp/find-func.el b/lisp/emacs-lisp/find-func.el
index 216d91b..f704c63 100644
--- a/lisp/emacs-lisp/find-func.el
+++ b/lisp/emacs-lisp/find-func.el
@@ -150,10 +150,10 @@ (defun find-library-name (library)
   (if (string-match "\\.el\\(c\\(\\..*\\)?\\)\\'" library)
       (setq library (replace-match "" t t library)))
   (or 
-   (locate-file library
+   (locate-file (file-name-nondirectory library)
 		(or find-function-source-path load-path)
 		(find-library-suffixes))
-   (locate-file library
+   (locate-file (file-name-nondirectory library)
 		(or find-function-source-path load-path)
 		load-file-rep-suffixes)
    (error "Can't find library %s" library)))


Regards,

  Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Stefan Monnier]

> Because `find-library-name' receives as its LIBRARY argument the full
> path, but doesn't strip the directory part, setting
> `find-function-source-path' has no effect on symbol finding -- Emacs
> still tries the path guessed according to load path (which is not
> correct in case you have the Elisp sources in directory different from
> the compiled files).

The file name has been changed to a full absolute name to avoid showing
the wrong file when there's some shadowing going on.  So your patch
would re-introduce this problem.

Also, it may not work for files that were loaded as "foo/bar".  I think
the load-history needs to be changed to keep track of both the absolute
file name and the name used to load the file
(i.e. "/bla/bla/foo/bar.elc" and "foo/bar").  Then in
find-function-source-path we will first try for /bla/bla/foo/bar.el and
when that fails we can fall back on searching for foo/bar.el.


        Stefan


> This simple change seems to fix it for me:

> diff --git a/lisp/emacs-lisp/find-func.el b/lisp/emacs-lisp/find-func.el
> index 216d91b..f704c63 100644
> --- a/lisp/emacs-lisp/find-func.el
> +++ b/lisp/emacs-lisp/find-func.el
> @@ -150,10 +150,10 @@ (defun find-library-name (library)
>    (if (string-match "\\.el\\(c\\(\\..*\\)?\\)\\'" library)
>        (setq library (replace-match "" t t library)))
>    (or 
> -   (locate-file library
> +   (locate-file (file-name-nondirectory library)
>  		(or find-function-source-path load-path)
>  		(find-library-suffixes))
> -   (locate-file library
> +   (locate-file (file-name-nondirectory library)
>  		(or find-function-source-path load-path)
>  		load-file-rep-suffixes)
>     (error "Can't find library %s" library)))


> Regards,

>   Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Because `find-library-name' receives as its LIBRARY argument the full
>> path, but doesn't strip the directory part, setting
>> `find-function-source-path' has no effect on symbol finding -- Emacs
>> still tries the path guessed according to load path (which is not
>> correct in case you have the Elisp sources in directory different from
>> the compiled files).
>
> The file name has been changed to a full absolute name to avoid showing
> the wrong file when there's some shadowing going on.  So your patch
> would re-introduce this problem.

I don't think I understand -- setting `f-f-s-p' should provide means to
override the normal load-path search. I don't care if the source I jump
to is really "equivalent" to the version I have loaded (it might not be
anyway -- the symbol definition is often loaded from a byte-compiled file,
whereas the equivalent source file could have been changed since then).
The only thing I care about is to jump to the/a source *I want* (that's
why I set `f-f-s-p') for a symbol I search for. The current behaviour
is broken for any purpose I can think of (other than when the absolute
path is right, which is normally never the case with `f-f-s-p' set).

When you look at the current `find-library-name' definition, it's
_obviously_ wrong to pass an absolute file name to it. It just makes no
sense.

> Also, it may not work for files that were loaded as "foo/bar".  I think
> the load-history needs to be changed to keep track of both the absolute
> file name and the name used to load the file
> (i.e. "/bla/bla/foo/bar.elc" and "foo/bar").  Then in
> find-function-source-path we will first try for /bla/bla/foo/bar.el and
> when that fails we can fall back on searching for foo/bar.el.

I assume you meant `find-library-name', not `find-function-source-path'
in the last sentence.

I don't know if my fix is the best way to do it, but it fixes the bug
for me now and from your reply I'm not able to understand why it should
not be installed (for now, at least) or what concrete alternatives you
suggest (changing `load-history' implementation does not sound like
something that would be done any time soon -- or are you (or anybody)
going to do it?).


  Štěpán

>
>
>         Stefan
>
>
>> This simple change seems to fix it for me:
>
>> diff --git a/lisp/emacs-lisp/find-func.el b/lisp/emacs-lisp/find-func.el
>> index 216d91b..f704c63 100644
>> --- a/lisp/emacs-lisp/find-func.el
>> +++ b/lisp/emacs-lisp/find-func.el
>> @@ -150,10 +150,10 @@ (defun find-library-name (library)
>>    (if (string-match "\\.el\\(c\\(\\..*\\)?\\)\\'" library)
>>        (setq library (replace-match "" t t library)))
>>    (or 
>> -   (locate-file library
>> +   (locate-file (file-name-nondirectory library)
>>  		(or find-function-source-path load-path)
>>  		(find-library-suffixes))
>> -   (locate-file library
>> +   (locate-file (file-name-nondirectory library)
>>  		(or find-function-source-path load-path)
>>  		load-file-rep-suffixes)
>>     (error "Can't find library %s" library)))
>
>
>> Regards,
>
>>   Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

Štěpán Němec <stepnem@gmail.com> writes:

> I don't know if my fix is the best way to do it, but it fixes the bug
> for me now and from your reply I'm not able to understand why it should
> not be installed (for now, at least) [...]

One problem I stumbled upon just now is that my "fix" appears to break
finding symbols in loaded files not in `f-f-s-p' or `load-path', such as
the init file.

Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Stefan Monnier]

> I don't care if the source I jump to is really "equivalent" to the
> version I have loaded

There lies the problem.  Other people do.  So the fix to your problem
will need to satisfy both cases.


        Stefan

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I don't care if the source I jump to is really "equivalent" to the
>> version I have loaded
>
> There lies the problem.  Other people do.  So the fix to your problem
> will need to satisfy both cases.

But if you care about that, you don't need `find-function-source-path'
at all, no? (And actually, I don't see a reliable way to jump to the
"right" source of a byte-compiled function in general (as I already
pointed out in the previous mail).)

Also, you replied to none of my other questions, notably -- do you
really (_really_) plan to reimplement `load-history', or was that just a
"would be nice to have"? If the latter, could you propose a better
solution that would improve the current situation? (I'm sorry, but as I
also already wrote, I didn't really understand the point(s) you were
making.)

Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Stefan Monnier]

>> There lies the problem.  Other people do.  So the fix to your problem
>> will need to satisfy both cases.
> But if you care about that, you don't need `find-function-source-path'
> at all, no?

But your patch also affects the case where find-function-source-path is nil.

> (And actually, I don't see a reliable way to jump to the
> "right" source of a byte-compiled function in general (as I already
> pointed out in the previous mail).)

It doesn't have to work right when it's not possible.  But in the normal
case where the .el and the .elc files are in the same directory and the
.elc is the byte-compiled version of the .el file, it should work right.

> Also, you replied to none of my other questions, notably -- do you
> really (_really_) plan to reimplement `load-history', or was that just a
> "would be nice to have"?

I didn't say "reimplement".  Just that it needs to be tweaked with more
info.  We've changed it several times in the past, there's nothing
particularly tricky about that.

> If the latter, could you propose a better solution that would improve
> the current situation? (I'm sorry, but as I also already wrote,
> I didn't really understand the point(s) you were making.)

Some directories are not in the load-path, because the files therein are
expected to be loaded via something like (require 'semantic/sort) or
(load "term/vt100"), so if you see /blib/blob/semantic/sort.elc in the
load-history, you can't just take "sort.elc" and look for "sort.el" on
load-path because you'll find a completely unrelated file.


        Stefan

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> There lies the problem.  Other people do.  So the fix to your problem
>>> will need to satisfy both cases.
>> But if you care about that, you don't need `find-function-source-path'
>> at all, no?
>
> But your patch also affects the case where find-function-source-path is nil.

Yes. (And the current code ignores it even when it is non-nil. Actually,
the current code ignores everything -- it simply tries the one single
file name it guesses is right and either finds it or errors out.)

>> (And actually, I don't see a reliable way to jump to the
>> "right" source of a byte-compiled function in general (as I already
>> pointed out in the previous mail).)
>
> It doesn't have to work right when it's not possible.  But in the normal
> case where the .el and the .elc files are in the same directory and the
> .elc is the byte-compiled version of the .el file, it should work right.
>
>> Also, you replied to none of my other questions, notably -- do you
>> really (_really_) plan to reimplement `load-history', or was that just a
>> "would be nice to have"?
>
> I didn't say "reimplement".  Just that it needs to be tweaked with more
> info.  We've changed it several times in the past, there's nothing
> particularly tricky about that.
>
>> If the latter, could you propose a better solution that would improve
>> the current situation? (I'm sorry, but as I also already wrote,
>> I didn't really understand the point(s) you were making.)
>
> Some directories are not in the load-path, because the files therein are
> expected to be loaded via something like (require 'semantic/sort) or
> (load "term/vt100"), so if you see /blib/blob/semantic/sort.elc in the
> load-history, you can't just take "sort.elc" and look for "sort.el" on
> load-path because you'll find a completely unrelated file.

OK, I think I get it, thank you very much for the explanation. Honestly,
this kind of loading seems pathologic to me -- it never crossed my mind
to load/require "path/file" instead of first adding "path" to
`load-path' and then `load'ing "file". What use does this have? Or
rather, is it necessary (I imagine people might be trying to use this as
a poor man's module system)?

In any case, I'm obviously still missing a lot (such as how the case
with symbols defined outside load-path is handled or how the Help cross
references come into play -- they still don't work even with my "fix");
I'll try to look at the code some more.

  Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Stefan Monnier]

>> But your patch also affects the case where find-function-source-path is nil.

> Yes. (And the current code ignores it even when it is non-nil. Actually,
> the current code ignores everything -- it simply tries the one single
> file name it guesses is right and either finds it or errors out.)

Indeed, the current code's exclusive use of absolute file names is not
correct either.  It's just the result of uncoordinated changes.

> OK, I think I get it, thank you very much for the explanation. Honestly,
> this kind of loading seems pathologic to me -- it never crossed my mind
> to load/require "path/file" instead of first adding "path" to
> `load-path' and then `load'ing "file". What use does this have? Or
> rather, is it necessary (I imagine people might be trying to use this as
> a poor man's module system)?

Given the fact that we need to make sure every file name's first 8 chars
are unique (for the MSDOS build), it can be sometimes very convenient.
Semantic uses it extensively.  It's not used as a module system, tho
it's somewhat used as a way to specify "packages" of modules.
It also has the advantage of limiting the growth of the load-path
directory, which might improve performance somewhat (probably
negligible, tho, on current machines).

> In any case, I'm obviously still missing a lot (such as how the case
> with symbols defined outside load-path is handled or how the Help cross
> references come into play -- they still don't work even with my "fix");
> I'll try to look at the code some more.


        Stefan

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[MON KEY]

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

,---- SM worte:
| Also, it may not work for files that were loaded as "foo/bar".  I
| think the load-history needs to be changed to keep track of both the
| absolute file name and the name used to load the file
| (i.e. "/bla/bla/foo/bar.elc" and "foo/bar").
|
| Then in find-function-source-path we will first try for
| /bla/bla/foo/bar.el and when that fails we can fall back on searching
| for foo/bar.el.
`----

,---- SN wrote:
| When you look at the current `find-library-name' definition, it's
| _obviously_ wrong to pass an absolute file name to it. It just makes no
| sense.
`----

IMHO this type of problem has less to do w/ the path handling of
`find-library-name' and `locate-file's and more to do w/ how these
(and related procedures) find/frob file-name suffixes on a given path
e.g. the +real+ problem is with how Emacs plays with these:

`find-library-suffixes', `get-load-suffixes', `load-file-rep-suffixes'

FWIW Štěpán it may be useful to look at this thread here:

:SEE (URL `http://lists.gnu.org/archive/html/emacs-devel/2010-01/msg01060.html')

My impression is that Stefan's file completion regime is brittle w/re
this type of stuff... hence his (understandable) hesitancy to make any
dramatic changes/modifications/tweaks.

WIBN if Emacs lisp could treat namestrings as "Lisp Type" objects as
opposed to "Unix/C Type" objects? I.e. something like Common Lisp's more
CLOS centric handlers:

`namestring'
`directory-namestring'
`host-namestring'
`parse-namestring'
`file-namestring'
`pathname'
`pathname-name'
`pathname-type'
`pathname-directory'
`truename'
`make-pathname'
`merge-pathnames'
`enough-namestring'

:SEE (URL `http://www.lispworks.com/documentation/lw50/CLHS/Body/19_ab.htm')

Stefan, how reasaonable/welcome would it be to dovetail the EIEIO features
with C primitives to accomplish something like what CL offers ITR?

--
/s_P\

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

MON KEY <monkey@sandpframing.com> writes:

> bug#6716: 23.2; Setting `find-function-source-path' has no effect.
>
> ,---- SM worte:
> | Also, it may not work for files that were loaded as "foo/bar".  I
> | think the load-history needs to be changed to keep track of both the
> | absolute file name and the name used to load the file
> | (i.e. "/bla/bla/foo/bar.elc" and "foo/bar").
> |
> | Then in find-function-source-path we will first try for
> | /bla/bla/foo/bar.el and when that fails we can fall back on searching
> | for foo/bar.el.
> `----
>
> ,---- SN wrote:
> | When you look at the current `find-library-name' definition, it's
> | _obviously_ wrong to pass an absolute file name to it. It just makes no
> | sense.
> `----
>
> IMHO this type of problem has less to do w/ the path handling of
> `find-library-name' and `locate-file's and more to do w/ how these
> (and related procedures) find/frob file-name suffixes on a given path
> e.g. the +real+ problem is with how Emacs plays with these:
>
> `find-library-suffixes', `get-load-suffixes', `load-file-rep-suffixes'
>
> FWIW Štěpán it may be useful to look at this thread here:
>
> :SEE
> (URL `http://lists.gnu.org/archive/html/emacs-devel/2010-01/msg01060.html')

Thank you, but I think you misunderstood. The problem at hand (in this
bug report) _is_ the problem with how `find-library-name' handles paths;
I don't see any problem with the suffixes as in your bug report.

As I wrote in the mail you quote above, `find-library-name' passes an
absolute pathname to `locate-file', which makes no sense. That's the
basic symptom of what needs to be cured (although as Stefan points out,
simply removing the path component is probably not the right solution);
it has nothing to do with {find-library,get-load,load-file-rep}-suffixes
or file name completion.

> My impression is that Stefan's file completion regime is brittle w/re
> this type of stuff... hence his (understandable) hesitancy to make any
> dramatic changes/modifications/tweaks.
>
> WIBN if Emacs lisp could treat namestrings as "Lisp Type" objects as
> opposed to "Unix/C Type" objects? I.e. something like Common Lisp's more
> CLOS centric handlers:
>
> `namestring'
> `directory-namestring'
> `host-namestring'
> `parse-namestring'
> `file-namestring'
> `pathname'
> `pathname-name'
> `pathname-type'
> `pathname-directory'
> `truename'
> `make-pathname'
> `merge-pathnames'
> `enough-namestring'
>
> :SEE (URL `http://www.lispworks.com/documentation/lw50/CLHS/Body/19_ab.htm')
>
> Stefan, how reasaonable/welcome would it be to dovetail the EIEIO features
> with C primitives to accomplish something like what CL offers ITR?

Wouldn't it be better to discuss this on emacs-devel or file another bug
report? I really don't see any relation here.

  Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[MON KEY]

On Tue, Jul 27, 2010 at 5:39 AM, Štěpán Němec <stepnem@gmail.com> wrote:
,----
| As I wrote in the mail you quote above, `find-library-name' passes an
| absolute pathname to `locate-file', which makes no sense.
`----

Maybe, but but only if/when viewed out of the context of other closely
related (and often tightly coupled) features.

Sorry, but you can't "fix" find-library-name w/out tweaking
`locate-file' and that _can not_ happen without accounting for Emacs'
reliance on the `*-suffixes' vars/fncs.

,----
| basic symptom of what needs to be cured (although as Stefan points out,
| simply removing the path component is probably not the right solution);
`----

Damn Skippy. This is because the problem is also elsewhere (-:

,----
| it has nothing to do with {find-library,get-load,load-file-rep}-suffixes
| or file name completion.
`----

Well, assuming there can be a portable and non-backward-incompatible
fix for the symptoms you describe, I seriously doubt you will find it
w/out careful consideration of how the fix will affect these things.

Its not a simple matter of changing _only_ how pathnames (absolute or
otherwise) are handled.

here's an overview of the affected callers:

 find-library-name -> locate-file -> locate-file-internal

Profile these for a bit.

These are fundamental procedures. They do a fair degree of the primary
heavy lifting for a number of reliant satellite routines. Sooner or
later you'll find snags and weird corner cases where dependent
features will fail if they frob file namestrings in `load-history'
having extensions which rely on the return values of one or more of
the following:

`load-file-rep-suffixes'
`jka-compr-load-suffixes'
`load-suffixes'
`get-load-suffixes'
`load-history-regexp'

Of itself this isn't a problem _BUT_ it invariably becomes one as
soon as one attempts a meta-view towards fixing Emacs' approach to
file/dir traversal/searches. This is esp. so when one considers that
in the wild Emacsen are running on multiple different platforms.

Hence my reference to CL's file-system abstractions for filenames,
pathnames, namestrings, etc. for both logical and physical pathnames
as these were formulated with these types of problems in mind.

Right now Emacs is storing/accessing file "types" and "names" as well
as directories and pathnames (both logical and physical) across a
bewildering number of variables, constancts, custom-forms, functions,
subrs, and C primitives. A good majority of these objects are either
duplicate (and/or replicate with minor adaptation) an existing return
value or were otherwise implemented as a kludge to accommodate a
platform dependent file-system issue.

Many of these redundancies could/should be abstracted away by
establishing a more object-centric approach which +specifically+
address file-system frobbing and which relies less on string splitting
and regexps to manipulate file-system data-structures. Or, we could
just keep piling on more and more AWKish duct-tape...


>> Stefan, how reasaonable/welcome would it be to dovetail the EIEIO features
>> with C primitives to accomplish something like what CL offers ITR?

,----
|  Wouldn't it be better to discuss this on emacs-devel or file another bug
`----

Nah, they're too busy over there discussing how to improve Emacs'
learning curve.

,----
| I really don't see any relation here.
`----

There is one and it is more than tangential.

FWIW A good deal of Emacs' file-system fncns are Unix'ized versions
of CLTLv1 given a non-CLTLv1 name.

That no one really seems to acknowledge (or remember this) doesn't
mean it isn't so.

This is too bad because it blinds many to the reality of Greenspun's 10th:

 "Any sufficiently complicated C or Fortran program contains an ad hoc,
  informally-specified, bug-ridden, slow implementation of half of
  Common Lisp."

:SEE (URL `http://en.wikipedia.org/wiki/Greenspun%27s_Tenth_Rule')

>  Štěpán

--
/s_P\

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Lars Ingebrigtsen]

Štěpán Němec <stepnem@gmail.com> writes:

> Because `find-library-name' receives as its LIBRARY argument the full
> path, but doesn't strip the directory part, setting
> `find-function-source-path' has no effect on symbol finding -- Emacs
> still tries the path guessed according to load path (which is not
> correct in case you have the Elisp sources in directory different from
> the compiled files).

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

`find-function-source-path' is now obsolete (`find-library-source-path'
is used instead), but there wasn't a case to reproduce the problem here.
Are there still problems in this area in recent Emacs versions?  If so,
do you have a recipe to reproduce the problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Štěpán Němec]

On Sun, 13 Feb 2022 11:02:33 +0100
Lars Ingebrigtsen wrote:

> Štěpán Němec <stepnem@gmail.com> writes:
>
>> Because `find-library-name' receives as its LIBRARY argument the full
>> path, but doesn't strip the directory part, setting
>> `find-function-source-path' has no effect on symbol finding -- Emacs
>> still tries the path guessed according to load path (which is not
>> correct in case you have the Elisp sources in directory different from
>> the compiled files).
>
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
>
> `find-function-source-path' is now obsolete (`find-library-source-path'
> is used instead), but there wasn't a case to reproduce the problem here.
> Are there still problems in this area in recent Emacs versions?  If so,
> do you have a recipe to reproduce the problem?

It was the same issue as later reported in bug#50508 (which does include
a simple reproducer), and from a quick look at the current source code
it seems that the description you quote above still applies, too.

My specific use case was jumping (via e.g. `find-function') to source
files (typically a VCS repository checkout) when running an Emacs
without the sources installed along the compiled files.

Looking at bug#50508, it seems you have decided to not try to fix this,
and instead adjust the docs and rename/obsolete the option, so I guess
you'll want to close this one, too.

BTW, while grepping now I noticed `package--reload-previously-loaded'
still trying to use `find-function-source-path'; I don't know if or how
it should be updated.

-- 
Štěpán

bug#6716: 23.2; Setting `find-function-source-path' has no effect.

[Lars Ingebrigtsen]

Štěpán Němec <stepnem@gmail.com> writes:

> Looking at bug#50508, it seems you have decided to not try to fix this,
> and instead adjust the docs and rename/obsolete the option, so I guess
> you'll want to close this one, too.

OK; closing.

> BTW, while grepping now I noticed `package--reload-previously-loaded'
> still trying to use `find-function-source-path'; I don't know if or how
> it should be updated.

We usually respect variables that have been marked as obsolete until
they're removed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no