How to ship native modules?

How to ship native modules?

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

As I am working on making the Gnus-GSS support ready for general
consumption, I need to know how this thing should be distributed.

Right now there are two separate issues, with the second dependent on the
first:

The first issue is that I have no idea how to actually distribute the
native GSSAPI module. It's just a single, reasonably small, C file that
needs to be compiled with the libgssapi_krb5.so library. If this is bundled
in Emacs proper, making it a part of the build system isn't too
complicated, although I don't believe this has been done before.

If it is to be shipped as part of ELPA, then a different question emerges:
Should I simply add code to the Elisp files that checks if it can find the
native library and if not, compile it on the fly? It's doable, but I'd hard
to reinvent several wheels to do so. Essentially I'd have to rebuild part
of autoconf in Elisp.

The second issue is how to ship my changes to Gnus. My changes are limited
to a changes to nnimap.el, but this new feature will now try to require the
‘gss’ feature when gssapi authentication is enabled. If the gssapi library
is shipped as part of ELPA, then we end up with the interesting situation
where the packaged version of Gnus depends on a package in ELPA.

How should I continue with this?

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1504 bytes --]

Re: How to ship native modules?

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Mon, 20 Feb 2017 18:00:24 +0800
> 
> The first issue is that I have no idea how to actually distribute the native GSSAPI module. It's just a single,
> reasonably small, C file that needs to be compiled with the libgssapi_krb5.so library. If this is bundled in
> Emacs proper, making it a part of the build system isn't too complicated, although I don't believe this has been
> done before.

IMO, it makes little sense to distribute loadable modules with Emacs
itself: if they are part of the standard distribution, they should
just be "normal" source files in src/, and don't have to go through
all the trouble of using the emacs-modules machinery.

> If it is to be shipped as part of ELPA, then a different question emerges: Should I simply add code to the Elisp
> files that checks if it can find the native library and if not, compile it on the fly? It's doable, but I'd hard to
> reinvent several wheels to do so. Essentially I'd have to rebuild part of autoconf in Elisp.

I'm not sure I see the reason for "rebuilding part of autoconf".  Your
Lisp code should just (load "foo"), and leave it to the user and
package.el to put the compiled module where Emacs will find it (along
load-path).  Am I missing something?

> The second issue is how to ship my changes to Gnus. My changes are limited to a changes to nnimap.el, but
> this new feature will now try to require the ‘gss’ feature when gssapi authentication is enabled. If the gssapi
> library is shipped as part of ELPA, then we end up with the interesting situation where the packaged version
> of Gnus depends on a package in ELPA.

If the package is on ELPA, IMO it should provide a separate .el file
with a feature that Gnus will 'require', given some defcustom.

Thanks.

Re: How to ship native modules?

[Aurélien Aptel]

On Mon, Feb 20, 2017 at 11:00 AM, Elias Mårtenson <lokedhs@gmail.com> wrote:
> How should I continue with this?

As you probably already know, there are no official ways to ship
native modules at the moment (see previous discussions here [1] and
here [2]).

1: http://lists.gnu.org/archive/html/emacs-devel/2017-02/msg00245.html
2: http://lists.gnu.org/archive/html/emacs-devel/2017-02/msg00572.html

Re: How to ship native modules?

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 2668 bytes --]

On 20 February 2017 at 23:27, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Elias Mårtenson <lokedhs@gmail.com>
> > Date: Mon, 20 Feb 2017 18:00:24 +0800
> >
> > The first issue is that I have no idea how to actually distribute the
> native GSSAPI module. It's just a single,
> > reasonably small, C file that needs to be compiled with the
> libgssapi_krb5.so library. If this is bundled in
> > Emacs proper, making it a part of the build system isn't too
> complicated, although I don't believe this has been
> > done before.
>
> IMO, it makes little sense to distribute loadable modules with Emacs
> itself: if they are part of the standard distribution, they should
> just be "normal" source files in src/, and don't have to go through
> all the trouble of using the emacs-modules machinery.
>

Are you suggesting that I port the GSS module to use the native Emacs
internal API's? If I do that, would you accept that into Emacs proper?

Personally, I'd definitely prefer to not have to rely on a loadable module
for this kind of functionality. I guess the only drawback is that most
distributions won't ship Emacs with GSS support, since that required the
gssapi libraries to be available.

What is the best solution here?

> If it is to be shipped as part of ELPA, then a different question
> emerges: Should I simply add code to the Elisp
> > files that checks if it can find the native library and if not, compile
> it on the fly? It's doable, but I'd hard to
> > reinvent several wheels to do so. Essentially I'd have to rebuild part
> of autoconf in Elisp.
>
> I'm not sure I see the reason for "rebuilding part of autoconf".  Your
> Lisp code should just (load "foo"), and leave it to the user and
> package.el to put the compiled module where Emacs will find it (along
> load-path).  Am I missing something?
>

I'd expect the user to run M-x package-installl gss, and then have working
GSSAPI support in Gnus. Currently, ELPA doesn't have any facilities to
support compilation of native modules as part of ELPA package installation.

Are you suggesting that the user manually download the C part of the "gss"
package and compile it, and then run "emacs -L path/to/custom/libs" when
they start Emacs? That would work, but that doesn't seem to be the most
ideal user interface.


> If the package is on ELPA, IMO it should provide a separate .el file
> with a feature that Gnus will 'require', given some defcustom.
>

Fair enough. That can be done with a relatively minor patch to nnimap.el.
However, it still leaves the issue of how to deal with the native module
part of this.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 3601 bytes --]

Re: How to ship native modules?

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Tue, 21 Feb 2017 00:01:26 +0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
>  IMO, it makes little sense to distribute loadable modules with Emacs
>  itself: if they are part of the standard distribution, they should
>  just be "normal" source files in src/, and don't have to go through
>  all the trouble of using the emacs-modules machinery.
> 
> Are you suggesting that I port the GSS module to use the native Emacs internal API's?

If the module is to come with Emacs out of the box, yes.

> If I do that, would you accept that into Emacs proper?

I don't know, I'd have to look at the sources first.

> Personally, I'd definitely prefer to not have to rely on a loadable module for this kind of functionality. I guess the
> only drawback is that most distributions won't ship Emacs with GSS support, since that required the gssapi
> libraries to be available.

Is it different from any other optional library in any significant
way?  E.g., GPM or the image libraries?  They all require to have the
library installed for Emacs to be able to be built with its support.

>  I'm not sure I see the reason for "rebuilding part of autoconf". Your
>  Lisp code should just (load "foo"), and leave it to the user and
>  package.el to put the compiled module where Emacs will find it (along
>  load-path). Am I missing something?
> 
> I'd expect the user to run M-x package-installl gss, and then have working GSSAPI support in Gnus. Currently,
> ELPA doesn't have any facilities to support compilation of native modules as part of ELPA package
> installation.

You can provide a small Makefile that the user will have to invoke in
order to compile the C source.  Same as in modules/mod-test/ in the
Emacs sources.

> Are you suggesting that the user manually download the C part of the "gss" package and compile it, and then
> run "emacs -L path/to/custom/libs" when they start Emacs? That would work, but that doesn't seem to be the
> most ideal user interface.

Compile: yes (except that I see no reason for downloading manually, it
can all come to the end-user machine when the package is installed).
But I see no reason to run Emacs specially: if the Makefile has an
'install' target that puts the compiled module in site-lisp, Emacs
will find it when some Lisp attempts to load it.

>  If the package is on ELPA, IMO it should provide a separate .el file
>  with a feature that Gnus will 'require', given some defcustom.
> 
> Fair enough. That can be done with a relatively minor patch to nnimap.el. However, it still leaves the issue of
> how to deal with the native module part of this.

The module should be installed in site-lisp, IMO.

Re: How to ship native modules?

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 3525 bytes --]

On 21 February 2017 at 00:30, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Elias Mårtenson <lokedhs@gmail.com>
> > Date: Tue, 21 Feb 2017 00:01:26 +0800
> > Cc: emacs-devel <emacs-devel@gnu.org>
> >
> >  IMO, it makes little sense to distribute loadable modules with Emacs
> >  itself: if they are part of the standard distribution, they should
> >  just be "normal" source files in src/, and don't have to go through
> >  all the trouble of using the emacs-modules machinery.
> >
> > Are you suggesting that I port the GSS module to use the native Emacs
> internal API's?
>
> If the module is to come with Emacs out of the box, yes.
>

Since the main beneficiary of this feature is Gnus, it does make sense.


> > If I do that, would you accept that into Emacs proper?
>
> I don't know, I'd have to look at the sources first.
>

It's not that much work to implement the C part of this. I'll have to get
acquainted with the internal Emacs API's and then do the implementation.
Hopefully it should be a non-controversial thing to include.


> > Personally, I'd definitely prefer to not have to rely on a loadable
> module for this kind of functionality. I guess the
> > only drawback is that most distributions won't ship Emacs with GSS
> support, since that required the gssapi
> > libraries to be available.
>
> Is it different from any other optional library in any significant
> way?  E.g., GPM or the image libraries?  They all require to have the
> library installed for Emacs to be able to be built with its support.
>

Technically, it's the same thing. The difference is that most people don't
have the krb5 packages installed, and enabling the GSS feature would
require packagers to add another dependency to Emacs, giving them two
alternatives: 1) Not include GSS in their prebuilt version of Emacs, or 2)
add another dependency to Emacs, resulting in another package being
installed with it.

Now, krb5 is not a big library, so option 2 above is IMHO the right way to
go, but I worry that packagers won't see it that way and we'll end up with
a feature that users would have to rebuild their Emacs to use.

However, there is an upside. Most people who need this feature are in
corporate environments, usually on Windows (Kerberos is the standard
authentication mechanism in Active Directory) and the runtime libraries are
a standard part of Windows, so at on that platform there is no issue.


> > I'd expect the user to run M-x package-installl gss, and then have
> working GSSAPI support in Gnus. Currently,
> > ELPA doesn't have any facilities to support compilation of native
> modules as part of ELPA package
> > installation.
>
> You can provide a small Makefile that the user will have to invoke in
> order to compile the C source.  Same as in modules/mod-test/ in the
> Emacs sources.
>

I could, and I'd have to. But it would be very nice if this could be part
of the M‍-‍x package‍-‍install process.

If it was, then going the ELPA route is definitely the best way.


> Compile: yes (except that I see no reason for downloading manually, it
> can all come to the end-user machine when the package is installed).
> But I see no reason to run Emacs specially: if the Makefile has an
> 'install' target that puts the compiled module in site-lisp, Emacs
> will find it when some Lisp attempts to load it.
>

Installing in site‍-‍lisp requires root. Can I put it somewhere in
$HOME/.emacs.d?

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 4965 bytes --]

Re: How to ship native modules?

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Tue, 21 Feb 2017 10:48:03 +0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
>  Is it different from any other optional library in any significant
>  way? E.g., GPM or the image libraries? They all require to have the
>  library installed for Emacs to be able to be built with its support.
> 
> Technically, it's the same thing. The difference is that most people don't have the krb5 packages installed, and
> enabling the GSS feature would require packagers to add another dependency to Emacs, giving them two
> alternatives: 1) Not include GSS in their prebuilt version of Emacs, or 2) add another dependency to Emacs,
> resulting in another package being installed with it.
> 
> Now, krb5 is not a big library, so option 2 above is IMHO the right way to go, but I worry that packagers won't
> see it that way and we'll end up with a feature that users would have to rebuild their Emacs to use.
> 
> However, there is an upside. Most people who need this feature are in corporate environments, usually on
> Windows (Kerberos is the standard authentication mechanism in Active Directory) and the runtime libraries
> are a standard part of Windows, so at on that platform there is no issue.

Really?  I was under the impression that one would need to install the
Kerberos libraries from http://web.mit.edu/kerberos/dist/, and then
have the GSSAPI library built for Windows.  These all don't seem to be
available out of the box on MS-Windows.  Am I missing something?

>  You can provide a small Makefile that the user will have to invoke in
>  order to compile the C source. Same as in modules/mod-test/ in the
>  Emacs sources.
> 
> I could, and I'd have to. But it would be very nice if this could be part of the M‍-‍x package‍-‍install process.

I think that making package.el be able to compile programs in
languages like C is an ambitious goal that is best left to end users
at this stage.  For simple enough modules, the Makefile can be made
small and portable, so IMO it isn't worth the complexity to do more
than that, at least not yet.

> Installing in site‍-‍lisp requires root. Can I put it somewhere in $HOME/.emacs.d?

Yes, like we do with Lisp packages.  A module can live anywhere a Lisp
package can live, in order for Emacs to find it.

Re: How to ship native modules?

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1734 bytes --]

On 21 February 2017 at 11:41, Eli Zaretskii <eliz@gnu.org> wrote:


> > However, there is an upside. Most people who need this feature are in
> corporate environments, usually on
> > Windows (Kerberos is the standard authentication mechanism in Active
> Directory) and the runtime libraries
> > are a standard part of Windows, so at on that platform there is no issue.
>
> Really?  I was under the impression that one would need to install the
> Kerberos libraries from http://web.mit.edu/kerberos/dist/, and then
> have the GSSAPI library built for Windows.  These all don't seem to be
> available out of the box on MS-Windows.  Am I missing something?
>

Yes, I was a bit hasty. If you want to use the standard C API, then you
need those libraries. However, you could use the Windows API, SSPI instead.
It implements the same functionality using a different API (because of
course they do).

Note that I don't have any intention of implementing SSPI support myself,
but implementing an SSPI backend would not be terribly difficult.


> I think that making package.el be able to compile programs in
> languages like C is an ambitious goal that is best left to end users
> at this stage.  For simple enough modules, the Makefile can be made
> small and portable, so IMO it isn't worth the complexity to do more
> than that, at least not yet.
>

Fair enough.

That leave me with only one question: With the main user of this
functionality being Gnus, should GSS support be implemented as part of
Emacs proper, or shipped using ELPA?

The pros and cons of each approach should be well understood now, and I'm
willing to implement it either way. So we're up to a policy decision on the
part of the Emacs maintainers now.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 2476 bytes --]

Re: How to ship native modules?

[Andreas Politz]

Elias Mårtenson <lokedhs@gmail.com> writes:

> If it is to be shipped as part of ELPA, then a different question
> emerges: Should I simply add code to the Elisp files that checks if it
> can find the native library and if not, compile it on the fly? It's
> doable, but I'd hard to reinvent several wheels to do so. Essentially
> I'd have to rebuild part of autoconf in Elisp.

Hi, I am the author of pdf-tools and I have a very similar, if not the
same, problem.

I think your analogy to autoconf is not the right one.  What's needed is
more like an abstraction over the various systems including package
manager and possibly naming schemes.  For example on Arch you need to
install the package krb5 with pacman, while on Debian its probably
libkrb5-dev and libkrb5 using aptitude and on Windows you'd download
some installer or whatever.

There are various other details to consider. For example make is called
gmake on BSD, or you need to setup an environment variable
(e.g. PKG_CONFIG_PATH on macos).

This could be implemented in a neat, small package, your package depends
on and calls into as part of the installation or first-time-running
process.

Adding a new system-configuration should be fairly easy.

I've been pondering to write such a thing,

what do you think ?

-ap

Re: How to ship native modules?

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

On 21 February 2017 at 12:50, Andreas Politz <politza@hochschule-trier.de>
wrote:

> Elias Mårtenson <lokedhs@gmail.com> writes:
>
> > If it is to be shipped as part of ELPA, then a different question
> > emerges: Should I simply add code to the Elisp files that checks if it
> > can find the native library and if not, compile it on the fly? It's
> > doable, but I'd hard to reinvent several wheels to do so. Essentially
> > I'd have to rebuild part of autoconf in Elisp.
>
> Hi, I am the author of pdf-tools and I have a very similar, if not the
> same, problem.
>

I use pdf-tools, and I have definitely seen the same issue there.


> I think your analogy to autoconf is not the right one.  What's needed is
> more like an abstraction over the various systems including package
> manager and possibly naming schemes.  For example on Arch you need to
> install the package krb5 with pacman, while on Debian its probably
> libkrb5-dev and libkrb5 using aptitude and on Windows you'd download
> some installer or whatever.
>

Indeed. The MIT and Heimdal implementations also have some minor
differences that has to be taken into account, making the Makefile more
complex.


> There are various other details to consider. For example make is called
> gmake on BSD, or you need to setup an environment variable
> (e.g. PKG_CONFIG_PATH on macos).
>

I don't use OSX much anymore, so I wasn't aware of that.


> This could be implemented in a neat, small package, your package depends
> on and calls into as part of the installation or first-time-running
> process.
>
> Adding a new system-configuration should be fairly easy.
>
> I've been pondering to write such a thing,
>
> what do you think ?
>

I think that sounds like a great idea. If you build a prototype I'd be
happy to provide whatever assistance I can.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 2828 bytes --]

Re: How to ship native modules?

[Andreas Politz]

Elias Mårtenson <lokedhs@gmail.com> writes:

> I think that sounds like a great idea. If you build a prototype I'd be happy to provide whatever assistance I can.

Great, I'll get back here, if I have something working.

-ap

Re: How to ship native modules?

[Stefan Monnier]

> You can provide a small Makefile that the user will have to invoke in
> order to compile the C source.

The user doesn't even have to invoke it: package-install can invoke it
(indirectly) by having something akin to

    (eval-when-compile (shell-command "make"))

in one of the Elisp files.


        Stefan

Re: How to ship native modules?

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

On 21 Feb 2017 11:24 PM, "Stefan Monnier" <monnier@iro.umontreal.ca> wrote:

> You can provide a small Makefile that the user will have to invoke in
> order to compile the C source.

The user doesn't even have to invoke it: package-install can invoke it
(indirectly) by having something akin to

    (eval-when-compile (shell-command "make"))

in one of the Elisp files.


I thought ELPA packages could only contain el files? (granted, even if that
is the case, I could always encode the C code in elisp code, but that's a
bit ugly)

[-- Attachment #2: Type: text/html, Size: 1215 bytes --]

Re: How to ship native modules?

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Tue, 21 Feb 2017 12:13:44 +0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> That leave me with only one question: With the main user of this functionality being Gnus, should GSS
> support be implemented as part of Emacs proper, or shipped using ELPA?
> 
> The pros and cons of each approach should be well understood now, and I'm willing to implement it either
> way. So we're up to a policy decision on the part of the Emacs maintainers now.

I looked at the C source.  It's quite small, and if reimplemented as
part of Emacs, will be even smaller (no need to reinvent XCAR, intern,
etc.).  So I see no problems with adding this to the Emacs sources,
probably as a separate C file.

John?

Re: How to ship native modules?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 21 Feb 2017 09:44:39 -0500
> 
> > You can provide a small Makefile that the user will have to invoke in
> > order to compile the C source.
> 
> The user doesn't even have to invoke it: package-install can invoke it
> (indirectly) by having something akin to
> 
>     (eval-when-compile (shell-command "make"))
> 
> in one of the Elisp files.

Yes, of course.  Provided that the 'make' doesn't need any
command-line options that only human can deduce (depends on how clever
and reliable the Makefile is).

Re: How to ship native modules?

[Stefan Monnier]

> I thought ELPA packages could only contain el files? (granted, even if that
> is the case, I could always encode the C code in elisp code, but that's a
> bit ugly)

No, they can contain any file you like (they're just tarballs).
Some commonly used non-Elisp files are image files and .info files, but
C files will work just as well (tho package.el knows nothing about them).


        Stefan

Re: How to ship native modules?

[Göktuğ Kayaalp]

> (eval-when-compile (shell-command "make"))

Mind that this can cause problems in a system where ‘make’ points to
sth. else than GNU Make, e.g. on BSDs it's bmake.  I'd rather have

(defconst my-module-C-extensions-compiled-p
  (= 0
     (eval-when-compile (shell-command (or (executable-find "gmake")
                                           "make")))))

but maybe it would be better to introduce some standard machinery for
native modules (at least for those in C) early on that later the
community won't have to deal with fragmentation over different ad-hoc
systems.  At least a standard way to define how to build the module
would be nice:

(define-dynamic-module example
  :path "/path/to/module"
  :compile-method gnu-make)

;; (build-dynamic-module 'example)

That can be included in NAME-pkg.el for packaging and distributing
modules (or maybe define-package may be extended for building dynamic
modules of a package).  Certain make recipes like ‘all’ and ‘test’ may
be required.

Regards,
-gk.

Re: How to ship native modules?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I looked at the C source. It's quite small, and if reimplemented as part of
> Emacs, will be even smaller (no need to reinvent XCAR, intern, etc.). So I
> see no problems with adding this to the Emacs sources, probably as a
> separate C file.

> John?

Sure, I'm fine with that being in core.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2