Where should dynamic modules be installed?

Where should dynamic modules be installed?

[Ulrich Mueller]

We are going to include the first package with a dynamically loaded
module (emacs-libvterm) in Gentoo. The question arose where the .so file
should be installed? Since it is a system dependent binary, installing
it under /usr/share/emacs/ looks wrong. I cannot find anything about a
recommended install location in the documentation.

Would /usr/lib{,64}/emacs/site-modules/ be a reasonable place for this?

Also, wouldn't it be cleaner if Emacs had a separate variable (analog to
image-load-path, custom-theme-load-path, etc.) for loading of dynamic
modules, instead of reusing load-path?

Re: Where should dynamic modules be installed?

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Date: Mon, 21 Oct 2019 12:45:02 +0200
> Cc: gnu-emacs@gentoo.org
> 
> We are going to include the first package with a dynamically loaded
> module (emacs-libvterm) in Gentoo. The question arose where the .so file
> should be installed? Since it is a system dependent binary, installing
> it under /usr/share/emacs/ looks wrong. I cannot find anything about a
> recommended install location in the documentation.

Why do you think /usr/share/emacs/ is wrong?  That's where the
site-lisp directory lives.

> Also, wouldn't it be cleaner if Emacs had a separate variable (analog to
> image-load-path, custom-theme-load-path, etc.) for loading of dynamic
> modules, instead of reusing load-path?

The idea with reusing load-path was that modules will almost always be
accompanied by some .el file(s), and having those separately from the
module binaries would be a complication.

Re: Where should dynamic modules be installed?

[Ulrich Mueller]

>>>>> On Mon, 21 Oct 2019, Eli Zaretskii wrote:

>> We are going to include the first package with a dynamically loaded
>> module (emacs-libvterm) in Gentoo. The question arose where the .so
>> file should be installed? Since it is a system dependent binary,
>> installing it under /usr/share/emacs/ looks wrong. I cannot find
>> anything about a recommended install location in the documentation.

> Why do you think /usr/share/emacs/ is wrong?  That's where the
> site-lisp directory lives.

Yes, but modules aren't lisp files but dynamic libraries, which are
architecture dependent. The FHS [1] says that /usr/share is for data
files that are architecture *independent*.

>> Also, wouldn't it be cleaner if Emacs had a separate variable (analog
>> to image-load-path, custom-theme-load-path, etc.) for loading of
>> dynamic modules, instead of reusing load-path?

> The idea with reusing load-path was that modules will almost always be
> accompanied by some .el file(s), and having those separately from the
> module binaries would be a complication.

If the installation follows FHS guidelines, then *.el{,c} will go into
/usr/share/, while modules will go into /usr/lib/ or /usr/lib64/. Which
means that load paths for elisp files and modules will be disjoint, so
it looks a bit unnatural to use the same variable for them.

[1] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html

Re: Where should dynamic modules be installed?

[Paul Eggert]

On 10/21/19 3:45 AM, Ulrich Mueller wrote:
> We are going to include the first package with a dynamically loaded
> module (emacs-libvterm) in Gentoo. The question arose where the .so file
> should be installed? Since it is a system dependent binary, installing
> it under /usr/share/emacs/ looks wrong. I cannot find anything about a
> recommended install location in the documentation.

There isn't one. We haven't thought through the issues here. Thanks for 
bringing up the topic.

> Would /usr/lib{,64}/emacs/site-modules/ be a reasonable place for this?

Where do you put executables like hexl, movemail, and profile? I suggest 
puting .so files near there. In Fedora 30, this directory is 
/usr/libexec/emacs/26.2/x86_64-redhat-linux-gnu (which is the value of 
the exec-directory variable). Since modules can be version-specific, it 
makes sense to have the Emacs version number somewhere in the directory 
name. Do you do that with other Emacs directories?

Also, why have "site-" in the name? Do you have "site-" in other 
Emacs-related directory names?

> Also, wouldn't it be cleaner if Emacs had a separate variable (analog to
> image-load-path, custom-theme-load-path, etc.) for loading of dynamic
> modules, instead of reusing load-path?

Emacs should do that, yes. How about if we use a variable named 
'module-directory' for that? It would work like exec-directory, except 
for modules.

I expect that the main reason Emacs doesn't have a variable like 
module-directory now, is that packagers haven't yet encountered the 
Emacs module system (it has been off by default, though that will change 
in Emacs 27) and so haven't run into problems like the one you mention.

Re: Where should dynamic modules be installed?

[Andy Moreton]

On Wed 23 Oct 2019, Paul Eggert wrote:

> On 10/21/19 3:45 AM, Ulrich Mueller wrote:
>> We are going to include the first package with a dynamically loaded
>> module (emacs-libvterm) in Gentoo. The question arose where the .so file
>> should be installed? Since it is a system dependent binary, installing
>> it under /usr/share/emacs/ looks wrong. I cannot find anything about a
>> recommended install location in the documentation.
>
> There isn't one. We haven't thought through the issues here. Thanks for
> bringing up the topic.
>
>> Would /usr/lib{,64}/emacs/site-modules/ be a reasonable place for this?
>
> Where do you put executables like hexl, movemail, and profile? I suggest
> puting .so files near there. In Fedora 30, this directory is 
> /usr/libexec/emacs/26.2/x86_64-redhat-linux-gnu (which is the value of the
> exec-directory variable). Since modules can be version-specific, it makes
> sense to have the Emacs version number somewhere in the directory name. Do you
> do that with other Emacs directories?
>
> Also, why have "site-" in the name? Do you have "site-" in other Emacs-related
> directory names?
>
>> Also, wouldn't it be cleaner if Emacs had a separate variable (analog to
>> image-load-path, custom-theme-load-path, etc.) for loading of dynamic
>> modules, instead of reusing load-path?
>
> Emacs should do that, yes. How about if we use a variable named
> 'module-directory' for that? It would work like exec-directory, except 
> for modules.

`exec-directory' is documented as being for executables installed from
the emacs distribution, whereas modules may be installed from other
sources.

Why not use `module-load-path` to allow for system/site/user locations
for dynamic modules ?

> I expect that the main reason Emacs doesn't have a variable like
> module-directory now, is that packagers haven't yet encountered the 
> Emacs module system (it has been off by default, though that will change in
> Emacs 27) and so haven't run into problems like the one you mention.

Indeed.

    AndyM

Re: Where should dynamic modules be installed?

[Stefan Monnier]

>> I expect that the main reason Emacs doesn't have a variable like
>> module-directory now, is that packagers haven't yet encountered the 
>> Emacs module system (it has been off by default, though that will change in
>> Emacs 27) and so haven't run into problems like the one you mention.
>
> Indeed.

Not only that, but also sharing filesystems between different machines
has been rather rare with current cheap large disks, so storing
architecture-specific files alongside architecture-agnostic files hasn't
been problematic in practice.  Maybe instead of looking for <foo>.so
we should be looking for <foo>.<arch>.so to make it easy to share
a directory between machines of different architectures?

If Debian doesn't want to put those architecture-specific files into
/usr/share/emacs/site-lisp, then it can tweak the `load-path` to include
/usr/lib/<arch>/emacs/site-modules or something like that.


        Stefan

Re: Where should dynamic modules be installed?

[Eli Zaretskii]

> > Also, wouldn't it be cleaner if Emacs had a separate variable (analog to
> > image-load-path, custom-theme-load-path, etc.) for loading of dynamic
> > modules, instead of reusing load-path?
> 
> Emacs should do that, yes. How about if we use a variable named 
> 'module-directory' for that? It would work like exec-directory, except 
> for modules.

I think a single directory would be too restrictive, we should have a
path.  And we should use load-path as fallback, because we used that
for several releases.

Re: Where should dynamic modules be installed?

[Ulrich Mueller]

>>>>> On Thu, 24 Oct 2019, Paul Eggert wrote:

> On 10/21/19 3:45 AM, Ulrich Mueller wrote:
>> We are going to include the first package with a dynamically loaded
>> module (emacs-libvterm) in Gentoo. The question arose where the .so
>> file should be installed? Since it is a system dependent binary,
>> installing it under /usr/share/emacs/ looks wrong. I cannot find
>> anything about a recommended install location in the documentation.

> There isn't one. We haven't thought through the issues here. Thanks
> for bringing up the topic.

>> Would /usr/lib{,64}/emacs/site-modules/ be a reasonable place for
>> this?

> Where do you put executables like hexl, movemail, and profile? I
> suggest puting .so files near there.

The FHS says that /usr/libexec/ is for "binaries that are not intended
to be executed directly by users or shell scripts".
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html

IMHO shared objects don't belong there, but in /usr/lib/.

> In Fedora 30, this directory is
> /usr/libexec/emacs/26.2/x86_64-redhat-linux-gnu (which is the value of
> the exec-directory variable). Since modules can be version-specific,
> it makes sense to have the Emacs version number somewhere in the
> directory name. Do you do that with other Emacs directories?

Layout of the relevant directories in Gentoo is as follows (using 26.3
as example):

GNU Emacs proper:
   /usr/libexec/emacs/26.3/x86_64-pc-linux-gnu/ - aux executables
   /usr/share/emacs/26.3/lisp/ - *.el *.elc
   /usr/share/emacs/26.3/etc/ - misc files

Add-on packages (Gentoo app-emacs category):
   /usr/share/emacs/site-lisp/<package_name>/ - *.el *.elc
   /usr/share/emacs/etc/<package_name>/ - misc files

> Also, why have "site-" in the name? Do you have "site-" in other
> Emacs-related directory names?

It would follow the precedent of "site-lisp". I don't have a strong
opinion about it though, either of "site-modules" or "modules" would be
fine.

>> Also, wouldn't it be cleaner if Emacs had a separate variable (analog
>> to image-load-path, custom-theme-load-path, etc.) for loading of
>> dynamic modules, instead of reusing load-path?

> Emacs should do that, yes. How about if we use a variable named
> 'module-directory' for that? It would work like exec-directory, except
> for modules.

> I expect that the main reason Emacs doesn't have a variable like
> module-directory now, is that packagers haven't yet encountered the
> Emacs module system (it has been off by default, though that will
> change in Emacs 27) and so haven't run into problems like the one you
> mention.

If we (Gentoo) follow the same scheme as we do for elisp-files, then we
would keep modules of different packages in separate directories (and
also separate from any modules installed by Emacs proper). So it should
be a path variable (similar to load-path), rather than a single
directory.

Re: Where should dynamic modules be installed?

[Ulrich Mueller]

>>>>> On Thu, 24 Oct 2019, Stefan Monnier wrote:

> If Debian doesn't want to put those architecture-specific files into
> /usr/share/emacs/site-lisp, then it can tweak the `load-path` to
> include /usr/lib/<arch>/emacs/site-modules or something like that.

I wonder why the intermediate <arch> dir would be needed? /usr/lib (like
/usr/bin) is architecture dependent by definition and cannot be shared.

Re: Where should dynamic modules be installed?

[Stefan Monnier]

> I wonder why the intermediate <arch> dir would be needed? /usr/lib (like
> /usr/bin) is architecture dependent by definition and cannot be shared.

I assumed it might be useful for multiarch installs,


        Stefan