why is site-lisp placed before the default load path?

why is site-lisp placed before the default load path?

[Alex Dunn]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

Right now, `init_lread` in lread.c creates the initial load path by
concatenating PATH_SITELOADSEARCH and PATH_LOADSEARCH, in that order.
Since subdirs.el searches recursively and adds everything to the load-path,
if I install (for example), realgud into site-lisp, things break because a
number of Emacs' own libraries are shadowed by realgud's language and
utility modules:
- https://github.com/realgud/realgud/tree/master/realgud/common
- https://github.com/realgud/realgud/tree/master/realgud/lang

Looking through the code history, this order has been deliberately
maintained for well over a decade, but what's the reason for letting core
libraries be overridden like this?  Unless I'm missing something it seems
safer to ensure that Emacs' own stuff is always at the front of the
load-path at launch.  If I want, say, a newer version of Org, I can always
manually prepend a different location to the load-path.

Would the maintainers be open to considering changing the order of the
load-path?

[-- Attachment #2: Type: text/html, Size: 1270 bytes --]

Re: why is site-lisp placed before the default load path?

[Eli Zaretskii]

> From: Alex Dunn <dunn.alex@gmail.com>
> Date: Mon, 1 Aug 2016 09:24:34 -0700
> 
> Right now, `init_lread` in lread.c creates the initial load path by concatenating PATH_SITELOADSEARCH and
> PATH_LOADSEARCH, in that order. Since subdirs.el searches recursively and adds everything to the
> load-path, if I install (for example), realgud into site-lisp, things break because a number of Emacs' own
> libraries are shadowed by realgud's language and utility modules:
> - https://github.com/realgud/realgud/tree/master/realgud/common
> - https://github.com/realgud/realgud/tree/master/realgud/lang
> 
> Looking through the code history, this order has been deliberately maintained for well over a decade, but
> what's the reason for letting core libraries be overridden like this?

It's precisely so that users could have their local versions loaded in
preference to the bundled ones, I think.

If realgud somehow causes trouble due to this, the problem should be
solved there, IMO.

Re: why is site-lisp placed before the default load path?

[Alex Dunn]

[-- Attachment #1: Type: text/plain, Size: 2211 bytes --]

> It's precisely so that users could have their local versions loaded in
> preference to the bundled ones, I think.

But there are plenty of other ways to do that.   This makes it a little too
easy to override important core libraries, IMO.

realgud is just an example.  It uses names like js.el and info.el because
they are supposed to be loaded via `load-relative`, rather than required
directly, but subdirs.el placing them at the front of the load path is
what's causing trouble.  Rocky could add 'realgud-' to the beginning of
every elisp file in the application, but `load-relative` and the directory
structure makes that unnecessary.

You might argue that every elisp file in an application should be named to
avoid conflicts with core libraries, but the only reason I see for that
requirement is the current ordering of the load path.

And beyond the inconvenience caused by programs using `load-relative`, or
programs just accidentally using the same name as a core library, it seems
possible that a malicious developer could tuck their own `url.el` into an
otherwise innocuous package and cause some mayhem.

On Mon, Aug 1, 2016 at 9:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Alex Dunn <dunn.alex@gmail.com>
> > Date: Mon, 1 Aug 2016 09:24:34 -0700
> >
> > Right now, `init_lread` in lread.c creates the initial load path by
> concatenating PATH_SITELOADSEARCH and
> > PATH_LOADSEARCH, in that order. Since subdirs.el searches recursively
> and adds everything to the
> > load-path, if I install (for example), realgud into site-lisp, things
> break because a number of Emacs' own
> > libraries are shadowed by realgud's language and utility modules:
> > - https://github.com/realgud/realgud/tree/master/realgud/common
> > - https://github.com/realgud/realgud/tree/master/realgud/lang
> >
> > Looking through the code history, this order has been deliberately
> maintained for well over a decade, but
> > what's the reason for letting core libraries be overridden like this?
>
> It's precisely so that users could have their local versions loaded in
> preference to the bundled ones, I think.
>
> If realgud somehow causes trouble due to this, the problem should be
> solved there, IMO.
>

[-- Attachment #2: Type: text/html, Size: 3376 bytes --]

Re: why is site-lisp placed before the default load path?

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 1997 bytes --]

On Mon, Aug 1, 2016 at 1:19 PM, Alex Dunn <dunn.alex@gmail.com> wrote:

> But there are plenty of other ways to do that.   This makes it a little
> too easy to override important core libraries, IMO.
>

​It should be easy for a site or a user to override default behavior in
Emacs.  Sometimes this means having modifications loaded prior to
particular local initialization files running and changing load-path.
​


>
> realgud is just an example.  It uses names like js.el and info.el because
> they are supposed to be loaded via `load-relative`, rather than required
> directly, but subdirs.el placing them at the front of the load path is
> what's causing trouble.
>

​Given all the existing tools that look for Elisp files by filename,
find-library being just one, it is not a good assumption that the filename
will always be uniquely identified by adding a directory name.

  Rocky could add 'realgud-' to the beginning of every elisp file in the
> application,
>

​That would be much better and safer and more generally useful.
​


> but `load-relative` and the directory structure makes that unnecessary.
>
> You might argue that every elisp file in an application should be named to
> avoid conflicts with core libraries, but the only reason I see for that
> requirement is the current ordering of the load path.
>

​There are others.
​


>
> And beyond the inconvenience caused by programs using `load-relative`, or
> programs just accidentally using the same name as a core library, it seems
> possible that a malicious developer could tuck their own `url.el` into an
> otherwise innocuous package and cause some mayhem.
>

​Yes, Emacs is a pretty open environment that relies on a lot of trust
among users and developers.  The community helps keep bad actors from
acting badly.  Emacs has existed for decades without a major issue like you
describe and there are very likely good reasons for that.

​Bob
​
​

[-- Attachment #2: Type: text/html, Size: 4510 bytes --]

Re: why is site-lisp placed before the default load path?

[Alex Dunn]

[-- Attachment #1: Type: text/plain, Size: 2379 bytes --]

OK.  I'm still not convinced this is good default behavior, but I'll defer
to y'all.  Realgud solved the issue in their case by adding .nosearch files
in the subdirectories, which I didn't realize was an available tool.

On Mon, Aug 1, 2016 at 10:54 AM, Robert Weiner <rsw@gnu.org> wrote:

>
> On Mon, Aug 1, 2016 at 1:19 PM, Alex Dunn <dunn.alex@gmail.com> wrote:
>
>> But there are plenty of other ways to do that.   This makes it a little
>> too easy to override important core libraries, IMO.
>>
>
> ​It should be easy for a site or a user to override default behavior in
> Emacs.  Sometimes this means having modifications loaded prior to
> particular local initialization files running and changing load-path.
> ​
>
>
>>
>> realgud is just an example.  It uses names like js.el and info.el because
>> they are supposed to be loaded via `load-relative`, rather than required
>> directly, but subdirs.el placing them at the front of the load path is
>> what's causing trouble.
>>
>
> ​Given all the existing tools that look for Elisp files by filename,
> find-library being just one, it is not a good assumption that the filename
> will always be uniquely identified by adding a directory name.
>
>   Rocky could add 'realgud-' to the beginning of every elisp file in the
>> application,
>>
>
> ​That would be much better and safer and more generally useful.
> ​
>
>
>> but `load-relative` and the directory structure makes that unnecessary.
>>
>> You might argue that every elisp file in an application should be named
>> to avoid conflicts with core libraries, but the only reason I see for that
>> requirement is the current ordering of the load path.
>>
>
> ​There are others.
> ​
>
>
>>
>> And beyond the inconvenience caused by programs using `load-relative`, or
>> programs just accidentally using the same name as a core library, it seems
>> possible that a malicious developer could tuck their own `url.el` into an
>> otherwise innocuous package and cause some mayhem.
>>
>
> ​Yes, Emacs is a pretty open environment that relies on a lot of trust
> among users and developers.  The community helps keep bad actors from
> acting badly.  Emacs has existed for decades without a major issue like you
> describe and there are very likely good reasons for that.
>
> ​Bob
> ​
> ​
>
>

[-- Attachment #2: Type: text/html, Size: 5214 bytes --]

Re: why is site-lisp placed before the default load path?

[James Cloos]

Having site-lisp first goes back probably as long as site-lisp has
existed.

It worked infinitely lot better before subdirectoried were automatically
included.

That is what broke things.  (As I've complained before.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: why is site-lisp placed before the default load path?

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

On Tue, Aug 2, 2016 at 1:35 PM, James Cloos <cloos@jhcloos.com> wrote:

> That is what broke things.  (As I've complained before.)


​If a package unintentionally shadows/overrides a standard/core Emacs
library, then it is the package that is broken and needs to be fixed.  If
instead the behavior is intentional then that is what the current load-path
order is designed to allow.

Bob

[-- Attachment #2: Type: text/html, Size: 976 bytes --]

Re: why is site-lisp placed before the default load path?

[James Cloos]

>>>>> "RW" == Robert Weiner <rsw@gnu.org> writes:

RW> ​If a package unintentionally shadows/overrides a standard/core Emacs
RW> library, then it is the package that is broken and needs to be fixed.  If
RW> instead the behavior is intentional then that is what the current load-path
RW> order is designed to allow.

The point is that if sub directories are nto automatically included, it is
easier to  manage things.  Especially when using one's dist's packaging.

I had to make substantial changes to my .emacs when the auto sub dirs
was added to fix all of the damage that caused.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: why is site-lisp placed before the default load path?

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On Tue, Aug 2, 2016 at 4:07 PM, James Cloos <cloos@jhcloos.com> wrote:

> >>>>> "RW" == Robert Weiner <rsw@gnu.org> writes:
>
> RW> ​If a package unintentionally shadows/overrides a standard/core Emacs
> RW> library, then it is the package that is broken and needs to be fixed.
> If
> RW> instead the behavior is intentional then that is what the current
> load-path
> RW> order is designed to allow.
>
> The point is that if sub directories are nto automatically included, it is
> easier to  manage things.  Especially when using one's dist's packaging.
>

​I can see where you might have test subdirectories or others that you
would not want included in load-path.  If you want to justify a change
though, you should give one or more clear examples of where the default
behavior is the major cause of a problem.
​


>
> I had to make substantial changes to my .emacs when the auto sub dirs
> was added to fix all of the damage that caused.


​Be explicit.  Don't just say damage.  Give precise examples of what
happened and what was required to fix it so everyone can understand what
you are talking​

​about.

Bob
​

[-- Attachment #2: Type: text/html, Size: 2573 bytes --]

Re: why is site-lisp placed before the default load path?

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 904 bytes --]

On 2016-08-01 13:54, Robert Weiner wrote:
>> realgud is just an example.  It uses names like js.el and info.el
>> because they are supposed to be loaded via `load-relative`, rather
>> than required directly, but subdirs.el placing them at the front of
>> the load path is what's causing trouble.
> 
> Given all the existing tools that look for Elisp files by filename,
> find-library being just one, it is not a good assumption that the
> filename will always be uniquely identified by adding a directory
> name.

I'm not sure about this. The only bit of code that is supposed to load these internal files loads them using an absolute path, so that assumption if fine AFAICT.

>> Rocky could add 'realgud-' to the beginning of every elisp file in
>> the application,
>  
> ​That would be much better and safer and more generally useful.

Can you explain why?

Thanks!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: why is site-lisp placed before the default load path?

[Stefan Monnier]

>> Rocky could add 'realgud-' to the beginning of every elisp file in the
>> application,
> ​That would be much better and safer and more generally useful.

Ever since CEDET was merged into Emacs, we've started to improve
support for the situation where files are in subdirs which aren't
themselves in load-path.  So you can (require 'dir/file).  There are
still various problems with such situations but I think it's not
a practice we should discourage.


        Stefan


PS: Obviously, this didn't reach ELPA yet ;-)