Question regarding load-path handling for ELPA packages

Question regarding load-path handling for ELPA packages

[Xiyue Deng]

(I'm not sure whether this is the right mailing list for this kind of
question.  Please help redirect if possible.)

Hi,

I would like to understand how package.el handles `load-path' of
installed ELPA packages.  As I observe from load-path after ELPA
packages are initialized, it looks like only the package installation
root path is added to `load-path', but not any of the nested directory.

For example, if an addon `elpafoo` has a nested `elpabar' directory, as
in the following example:

,----
| ~/.config/emacs/elpa/elpafoo/
| ~/.config/emacs/elpa/elpafoo/elpafoo.el
| ~/.config/emacs/elpa/elpafoo/elpabar
| ~/.config/emacs/elpa/elpafoo/elpabar/elpabar.el
`----

For such a package, both `elpafoo.el' and `elpabar/elpabar.el' will be
byte-compiled, but only `~/.config/emacs/elpa/elpafoo' is added to
`load-path', not `~/.config/emacs/elpa/elpafoo/elpabar'.

I was trying to find in package.el source where this is handled but
cannot seem to have a definite answer.  So I wonder whether my current
understanding is correct?  Is this going to be the case forward?  And if
possible which code path is handling this logic?

To give a little more background (probably long), I was studying dh-elpa
which is Debian's tool that handles Emacs addons in a similar fashion as
how ELPA packages are handled, which among other things handles the byte
compiling of the source files to generate *.elc files.  Up to now,
dh-elpa doesn't support nested directories, so if an addon has *.el in a
nest directory they won't be byte-compiled.  This has been working well
up-to-now, but I noticed that some ELPA packages (e.g. auctex) have
nested directories and handled like I mentioned above, which dh-elpa
ignores.  I tried to add nested directory handling, which handles the
package installation under `/usr/share/emacs/site-lisp/elpa' and somehow
all nested directories are added to `load-path' as well (it's adding
this directory to package-directory-list but I'm not sure whether this
is the reason of the nested directories being added to `load-path').
Normally this should not be a problem, but for the case of auctex, it
has some source files under `style/' named `url.el', `array.el', etc.,
which collide with system packages and they don't have any `(provide
'foo)' declared in the source files.  So in my Emacs installation, in
which I use eglot-ensure in all prog-mode, eglot was broken as it tries
to load those alternative url.el, array.el and so on instead of the ones
shipped in Emacs, and as they don't have the corresponding `provide', it
causes error and Emacs will stop processing further.

Sorry this is getting a lot longer that it was supposed to be.  So in
the end I have the following questions:

* For ELPA package installed through package.el, should only the
installation root path be added to `load-path'?
  - Or recursively add nested directories will also be considered in the
  future?

* If it happens to be the latter case, is there any guidelines for
packages to avoid potential name collision?

* Would someone kindly provide some pointer to how `load-path' is being
handled as my rusty Elisp is preventing me to fully understand this.

Thanks for reading this unexpectedly (maybe unnecessarily) long post,
and looking forward to any insights.

-- 
Xiyue Deng

Re: Question regarding load-path handling for ELPA packages

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> (I'm not sure whether this is the right mailing list for this kind of
> question.  Please help redirect if possible.)
>
> Hi,
>
> I would like to understand how package.el handles `load-path' of
> installed ELPA packages.  As I observe from load-path after ELPA
> packages are initialized, it looks like only the package installation
> root path is added to `load-path', but not any of the nested directory.
>
> For example, if an addon `elpafoo` has a nested `elpabar' directory, as
> in the following example:
>
> ,----
> | ~/.config/emacs/elpa/elpafoo/
> | ~/.config/emacs/elpa/elpafoo/elpafoo.el
> | ~/.config/emacs/elpa/elpafoo/elpabar
> | ~/.config/emacs/elpa/elpafoo/elpabar/elpabar.el
> `----
>
> For such a package, both `elpafoo.el' and `elpabar/elpabar.el' will be
> byte-compiled, but only `~/.config/emacs/elpa/elpafoo' is added to
> `load-path', not `~/.config/emacs/elpa/elpafoo/elpabar'.
>
> I was trying to find in package.el source where this is handled but
> cannot seem to have a definite answer.  So I wonder whether my current
> understanding is correct?  Is this going to be the case forward?  And if
> possible which code path is handling this logic?
>
> To give a little more background (probably long), I was studying dh-elpa
> which is Debian's tool that handles Emacs addons in a similar fashion as
> how ELPA packages are handled, which among other things handles the byte
> compiling of the source files to generate *.elc files.  Up to now,
> dh-elpa doesn't support nested directories, so if an addon has *.el in a
> nest directory they won't be byte-compiled.  This has been working well
> up-to-now, but I noticed that some ELPA packages (e.g. auctex) have
> nested directories and handled like I mentioned above, which dh-elpa
> ignores.  I tried to add nested directory handling, which handles the
> package installation under `/usr/share/emacs/site-lisp/elpa' and somehow
> all nested directories are added to `load-path' as well (it's adding
> this directory to package-directory-list but I'm not sure whether this
> is the reason of the nested directories being added to `load-path').
> Normally this should not be a problem, but for the case of auctex, it
> has some source files under `style/' named `url.el', `array.el', etc.,
> which collide with system packages and they don't have any `(provide
> 'foo)' declared in the source files.  So in my Emacs installation, in
> which I use eglot-ensure in all prog-mode, eglot was broken as it tries
> to load those alternative url.el, array.el and so on instead of the ones
> shipped in Emacs, and as they don't have the corresponding `provide', it
> causes error and Emacs will stop processing further.
>
> Sorry this is getting a lot longer that it was supposed to be.  So in
> the end I have the following questions:
>
> * For ELPA package installed through package.el, should only the
> installation root path be added to `load-path'?
>   - Or recursively add nested directories will also be considered in the
>   future?
>
> * If it happens to be the latter case, is there any guidelines for
> packages to avoid potential name collision?
>
> * Would someone kindly provide some pointer to how `load-path' is being
> handled as my rusty Elisp is preventing me to fully understand this.
>
> Thanks for reading this unexpectedly (maybe unnecessarily) long post,
> and looking forward to any insights.

Friendly ping.  If this mailing list is not a good place for this
question, can someone point me to a better one?  Thanks in advance!

-- 
Xiyue Deng

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen via Emacs development discussions.]

Xiyue Deng <manphiz@gmail.com> writes:

> > * Would someone kindly provide some pointer to how `load-path' is being
> > handled as my rusty Elisp is preventing me to fully understand this.

Can only answer this one: AFAIU this is done using the autoload
mechanism - see `package-generate-autoloads'.

In effect, the parent directory of the autoloads file is added to
`load-path' when this file is loaded - typically when activating
packages.

An analog trick should allow to add subdirectories.


Michael.

Re: Question regarding load-path handling for ELPA packages

[Xiyue Deng]

Michael Heerdegen via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> > * Would someone kindly provide some pointer to how `load-path' is being
>> > handled as my rusty Elisp is preventing me to fully understand this.
>
> Can only answer this one: AFAIU this is done using the autoload
> mechanism - see `package-generate-autoloads'.
>
> In effect, the parent directory of the autoloads file is added to
> `load-path' when this file is loaded - typically when activating
> packages.
>

Thanks a lot for the pointer, Michael!  Will take a look.

> An analog trick should allow to add subdirectories.
>

Not sure about this as it may potentially break things (e.g. auctex).
Would still like to hear what the developers think.

>
> Michael.
>
>

-- 
Xiyue Deng

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen via Emacs development discussions.]

Xiyue Deng <manphiz@gmail.com> writes:

> > * For ELPA package installed through package.el, should only the
> > installation root path be added to `load-path'?
> >   - Or recursively add nested directories will also be considered in the
> >   future?

I'm not involved much in package.el, but I guess given that such
additions to `load-path' are be possible manually, and that changing
this now could introduce potential breakage in existing packages, I would
not expect this to change.  And I don't remember discussions about this.

Michael.

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen via Emacs development discussions.]

Xiyue Deng <manphiz@gmail.com> writes:

> > An analog trick should allow to add subdirectories.
>
> Not sure about this as it may potentially break things (e.g. auctex).

In case of AucTex I think you obviously should not do that.  But you
asked.

What are the concrete questions left you want to answer?  How dh-elpa
could be improved or fixed to not omit the byte compilation of
subdirectories?


Michael.

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen via Emacs development discussions.]

Michael Heerdegen via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> In case of AucTex I think you obviously should not do that.  But you
> asked.
>
> What are the concrete questions left you want to answer?  How dh-elpa
> could be improved or fixed to not omit the byte compilation of
> subdirectories?

Or, if you want to know how compilation when installing packages works:
see `package-unpack' which calls `package--compile'.  That function
compiles all ELisp files including those in subdirectories (using
`byte-recompile-directory') minus those matching one of the patterns
specified in the packages ".elpaignore" file when it exists.

Michael.

Re: Question regarding load-path handling for ELPA packages

[Xiyue Deng]

Hi Michael,

Michael Heerdegen via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> > An analog trick should allow to add subdirectories.
>>
>> Not sure about this as it may potentially break things (e.g. auctex).
>
> In case of AucTex I think you obviously should not do that.  But you
> asked.
>

Ack.

> What are the concrete questions left you want to answer?  How dh-elpa
> could be improved or fixed to not omit the byte compilation of
> subdirectories?

Actually I'd like to make dh-elpa handle `load-path' of its package
installation directory (by default it's
`/usr/share/emacs/site-lisp/elpa/foo') the same as how package.el
handles any ELPA package, a.k.a. only add the installation path to
`load-path' but not its subdirectories recursively.  Hopefully with your
pointer I can figure this out, which make take a while.  The byte
compilation of subdirectories is not really important as long as I can
make `load-path' sane, AIUI.

>
>
> Michael.
>
>

-- 
Xiyue Deng

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen via Emacs development discussions.]

Xiyue Deng <manphiz@gmail.com> writes:

> Actually I'd like to make dh-elpa handle `load-path' of its package
> installation directory (by default it's
> `/usr/share/emacs/site-lisp/elpa/foo') the same as how package.el
> handles any ELPA package, a.k.a. only add the installation path to
> `load-path' but not its subdirectories recursively.  Hopefully with your
> pointer I can figure this out, which make take a while.

Does dh-elpa generate autoloads?  Can that process maybe even just call
`package-generate-autoloads'?

Anyway, all the magic is the addition of this (constant!) expression:

#+begin_src emacs-lisp
(add-to-list 'load-path
             (or (and load-file-name
                      (directory-file-name
                       (file-name-directory load-file-name)))
                 (car load-path)))
#+end_src

to the generated loaddefs files.  "Constant" means: file names are not
substituted in that expression - i.e. really that expression literally.

This works because when the according autoloads file is loaded the
variable `load-file-name' then contains the according current file name
and ... etc, you get it?

Anyway, ask as many question as you need to understand the process.

If you do want to ask a developer more involved you can for example CC
Stefan Monnier <monnier@iro.umontreal.ca> though, I think so far I
answered your questions correctly.


Michael.

Re: Question regarding load-path handling for ELPA packages

[Xiyue Deng]

Hi Michael,

Michael Heerdegen via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Actually I'd like to make dh-elpa handle `load-path' of its package
>> installation directory (by default it's
>> `/usr/share/emacs/site-lisp/elpa/foo') the same as how package.el
>> handles any ELPA package, a.k.a. only add the installation path to
>> `load-path' but not its subdirectories recursively.  Hopefully with your
>> pointer I can figure this out, which make take a while.
>
> Does dh-elpa generate autoloads?  Can that process maybe even just call
> `package-generate-autoloads'?
>
> Anyway, all the magic is the addition of this (constant!) expression:
>
> #+begin_src emacs-lisp
> (add-to-list 'load-path
>              (or (and load-file-name
>                       (directory-file-name
>                        (file-name-directory load-file-name)))
>                  (car load-path)))
> #+end_src
>
> to the generated loaddefs files.  "Constant" means: file names are not
> substituted in that expression - i.e. really that expression literally.
>
> This works because when the according autoloads file is loaded the
> variable `load-file-name' then contains the according current file name
> and ... etc, you get it?
>
> Anyway, ask as many question as you need to understand the process.
>
> If you do want to ask a developer more involved you can for example CC
> Stefan Monnier <monnier@iro.umontreal.ca> though, I think so far I
> answered your questions correctly.
>
>
> Michael.
>
>

Thanks for your expanded explanations and code pointers!  It helped me a
lot in understanding how package.el handles `load-path'.  After some
more investigation, it turns out the behavior I observe in dh-elpa
installation path is actually a result of the existence of subdirs.el in
its parent directory.  I have added an extended explanations in the
Debian tracking bug[1].

I think one last question I'd like to ask is the direction of
subdirectory handling in ELPA packages.  Though currently the case of
auctex may cause breakage, this is in the minority and hence fixing it
can be localized to just one package, while there may be benefits to do
so for other cases.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069210#15

-- 
Xiyue Deng

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen]

Xiyue Deng <manphiz@gmail.com> writes:

> Thanks for your expanded explanations and code pointers!  It helped me a
> lot in understanding how package.el handles `load-path'.  After some
> more investigation, it turns out the behavior I observe in dh-elpa
> installation path is actually a result of the existence of subdirs.el in
> its parent directory.  I have added an extended explanations in the
> Debian tracking bug[1].

I didn't investigate that much - but AFAIU
`normal-top-level-add-to-load-path' doesn't add subdirectories
recursively.  Or does it?

And - we are lucky and I'm on Debian, so I can have a look.

On my stable branch system, an emacs installation with package "auctex"
installed will not have subdirectories of auctex stuff installed.  After
startup I only see "/usr/share/emacs/site-lisp/auctex" and
"/usr/share/auctex" in `load-path'.  Or does your problem occur in
experimental only?

> I think one last question I'd like to ask is the direction of
> subdirectory handling in ELPA packages.  Though currently the case of
> auctex may cause breakage, this is in the minority and hence fixing it
> can be localized to just one package, while there may be benefits to do
> so for other cases.

What would those benefits be?

Note that subdirectories may contain random stuff like test files not to
be intended to be loaded for normal functioning at all.  It's not a
problem to keep the "normal" source code files in the top-level
directory - there is room enough there...  And keeping load-path small
makes lookup faster.  We may add a lot of unnecessary crap.  Things
would also get harder to investigate with load-path complicated in such
a way.  Any package still can add subdirectories explicitly.  In sum I
see more disadvantages.

Michael.

Re: Question regarding load-path handling for ELPA packages

[Xiyue Deng]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Thanks for your expanded explanations and code pointers!  It helped me a
>> lot in understanding how package.el handles `load-path'.  After some
>> more investigation, it turns out the behavior I observe in dh-elpa
>> installation path is actually a result of the existence of subdirs.el in
>> its parent directory.  I have added an extended explanations in the
>> Debian tracking bug[1].
>
> I didn't investigate that much - but AFAIU
> `normal-top-level-add-to-load-path' doesn't add subdirectories
> recursively.  Or does it?
>

This is the content of subdirs.el:

,----
| (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
|     (normal-top-level-add-subdirs-to-load-path))
`----

So looks like it does :)

> And - we are lucky and I'm on Debian, so I can have a look.
>
> On my stable branch system, an emacs installation with package "auctex"
> installed will not have subdirectories of auctex stuff installed.  After
> startup I only see "/usr/share/emacs/site-lisp/auctex" and
> "/usr/share/auctex" in `load-path'.  Or does your problem occur in
> experimental only?
>

(This part is mostly Debian centric.)

Sorry I wasn't very clear about the auctex status.  As a matter of fact,
the current auctex is not yet Elpafied, so while it's usable, its
current installation is not detected by package.el so that if you use
`(use-package auctex :ensure t)' it will install a separate auctex from
ELPA (see tracking bug[1]).  On the other hand, I am experimenting an
attempt to Elpafy auctex[2] based on the recursive byte-compiling which
I'm also experimenting on dh-elpa and is mostly ready[3], with your help
here.  With this support dh-elpa can keep the same directory layout as
ELPA for package with source files in nested directories as well.

>> I think one last question I'd like to ask is the direction of
>> subdirectory handling in ELPA packages.  Though currently the case of
>> auctex may cause breakage, this is in the minority and hence fixing it
>> can be localized to just one package, while there may be benefits to do
>> so for other cases.
>
> What would those benefits be?
>
> Note that subdirectories may contain random stuff like test files not to
> be intended to be loaded for normal functioning at all.  It's not a
> problem to keep the "normal" source code files in the top-level
> directory - there is room enough there...  And keeping load-path small
> makes lookup faster.  We may add a lot of unnecessary crap.  Things
> would also get harder to investigate with load-path complicated in such
> a way.  Any package still can add subdirectories explicitly.  In sum I
> see more disadvantages.

I agree with you arguments here.  My initial thoughts for adding
subdirectories are mainly to give developer some flexibilities to
organize the source code, but if subdirectories can be added explicitly
then it's fine.  Do you know where how that's done is documented for
ELPA?

>
> Michael.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056939
[2] https://salsa.debian.org/manphiz/auctex/-/tree/elpafy-new?ref_type=heads
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069210

-- 
Xiyue Deng

Re: Question regarding load-path handling for ELPA packages

[Michael Heerdegen]

Xiyue Deng <manphiz@gmail.com> writes:

> > [...] doesn't add subdirectories recursively.  Or does it?
> This is the content of subdirs.el:
>
> ,----
> | (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
> |     (normal-top-level-add-subdirs-to-load-path))
> `----

Ok indeed - `normal-top-level-add-to-load-path'
vs. `normal-top-level-add-subdirs-to-load-path', I see.


> [...] With this support dh-elpa can keep the same directory layout as
> ELPA for package with source files in nested directories as well.

I see.  Then I think the more you mimic (or call) what package.el does
the saver you are.

> > Any package still can add subdirectories explicitly.  In sum I see
> > more disadvantages.
>
> I agree with you arguments here.  My initial thoughts for adding
> subdirectories are mainly to give developer some flexibilities to
> organize the source code, but if subdirectories can be added explicitly
> then it's fine.  Do you know where how that's done is documented for
> ELPA?

Sorry, no, and I didn't find anything.  Could be that it's undocumented.
At least I did not find an explicit description of how subdirectories of
the content directory are handled.


Michael.

Re: Question regarding load-path handling for ELPA packages

[Xiyue Deng]

Hi Michael,

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> > [...] doesn't add subdirectories recursively.  Or does it?
>> This is the content of subdirs.el:
>>
>> ,----
>> | (if (fboundp 'normal-top-level-add-subdirs-to-load-path)
>> |     (normal-top-level-add-subdirs-to-load-path))
>> `----
>
> Ok indeed - `normal-top-level-add-to-load-path'
> vs. `normal-top-level-add-subdirs-to-load-path', I see.
>
>
>> [...] With this support dh-elpa can keep the same directory layout as
>> ELPA for package with source files in nested directories as well.
>
> I see.  Then I think the more you mimic (or call) what package.el does
> the saver you are.
>
>> > Any package still can add subdirectories explicitly.  In sum I see
>> > more disadvantages.
>>
>> I agree with you arguments here.  My initial thoughts for adding
>> subdirectories are mainly to give developer some flexibilities to
>> organize the source code, but if subdirectories can be added explicitly
>> then it's fine.  Do you know where how that's done is documented for
>> ELPA?
>
> Sorry, no, and I didn't find anything.  Could be that it's undocumented.
> At least I did not find an explicit description of how subdirectories of
> the content directory are handled.

Thanks for help checking!  It's not urgent as we don't have an actual
use case yet.  Will revisit should the need comes up.

>
>
> Michael.

-- 
Xiyue Deng