Wrapping an Emacs dynamic module in a package

Wrapping an Emacs dynamic module in a package

[Joris Vankerschaver]

Hi all,

I have an emacs dynamic module that I would like install as part of a
multi-file package. That is, I have a shared object file _mymod.so, and
some Elisp wrappers mymod-pkg.el and mymod.el that do various bits of
initialization.

When Emacs installs my package, things go wrong when generating the
autoloads for _mymod.so. I get a

  (error "_mymod.so:0:0: error: scan-error: (Containing expression ends
prematurely 238020 238021))

This happens during autoload-generate-file-autoloads for _mymod.so.

Two questions:

1) Is there any way to avoid autoload processing shared object files?

2) Is there an example of an Emacs package that contains a dynamic module,
so I can take a look at how this works?

Please let me know if you need more info! I'm more an Emacs user than a
developer, but I'm happy to dig into the Emacs internals to get a handle on
this.

With best wishes,
Joris

Re: Wrapping an Emacs dynamic module in a package

[Eli Zaretskii]

> From: Joris Vankerschaver <joris.vankerschaver@gmail.com>
> Date: Sun, 16 Oct 2016 18:37:54 +0100
> 
> I have an emacs dynamic module that I would like install as part of a
> multi-file package. That is, I have a shared object file _mymod.so, and
> some Elisp wrappers mymod-pkg.el and mymod.el that do various bits of
> initialization.
> 
> When Emacs installs my package, things go wrong when generating the
> autoloads for _mymod.so. I get a
> 
>   (error "_mymod.so:0:0: error: scan-error: (Containing expression ends
> prematurely 238020 238021))
> 
> This happens during autoload-generate-file-autoloads for _mymod.so.

What do you mean by "Emacs installs", and why does that cause Emacs to
try to generate autoloads for _mymod.so?

Re: Wrapping an Emacs dynamic module in a package

[Joris Vankerschaver]

Hi Eli,

Sorry, I should have been more specific. Here's what I did:

1) I wrote an emacs module _mymod.so, as well as wrappers mymod.el (with
some local setup) and mymod-pkg.el (which only has a call to
define-package).

2) I tarred these together to form mymod-0.1.tar

3) I installed the resulting package using M-x package-install-file

In my messages buffer, I see that Emacs tries to generate autoloads for the
.so, and fails with the error indicated in the original email ("_mymod.so:0:0:
error: scan-error: (Containing expression ends prematurely 238020
238021))").

I'm pretty confident that the emacs module is correctly built (I can load
it, and run the functions in it). Also, if I rename _mymod.so to something
else, e.g. _mymod.blablabla, the rest of the package installs correctly and
can be loaded. So it really seems as if Emacs is choking on trying to
generate autoloads for _mymod.so.

Maybe backtracking a little bit:

1) Should Emacs be able to generate autoloads for an .so?

2) Does anybody have any experience with embedding an emacs module in an
emacs package? Any best practices I should be aware of?

Thanks,
Joris.





On Mon, Oct 17, 2016 at 6:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Joris Vankerschaver <joris.vankerschaver@gmail.com>
> > Date: Sun, 16 Oct 2016 18:37:54 +0100
> >
> > I have an emacs dynamic module that I would like install as part of a
> > multi-file package. That is, I have a shared object file _mymod.so, and
> > some Elisp wrappers mymod-pkg.el and mymod.el that do various bits of
> > initialization.
> >
> > When Emacs installs my package, things go wrong when generating the
> > autoloads for _mymod.so. I get a
> >
> >   (error "_mymod.so:0:0: error: scan-error: (Containing expression ends
> > prematurely 238020 238021))
> >
> > This happens during autoload-generate-file-autoloads for _mymod.so.
>
> What do you mean by "Emacs installs", and why does that cause Emacs to
> try to generate autoloads for _mymod.so?
>
>

Re: Wrapping an Emacs dynamic module in a package

[Eli Zaretskii]

> From: Joris Vankerschaver <joris.vankerschaver@gmail.com>
> Date: Mon, 17 Oct 2016 09:34:25 +0100
> Cc: help-gnu-emacs@gnu.org
> 
> 1) I wrote an emacs module _mymod.so, as well as wrappers mymod.el (with
> some local setup) and mymod-pkg.el (which only has a call to
> define-package).
> 
> 2) I tarred these together to form mymod-0.1.tar
> 
> 3) I installed the resulting package using M-x package-install-file
> 
> In my messages buffer, I see that Emacs tries to generate autoloads for the
> .so, and fails with the error indicated in the original email ("_mymod.so:0:0:
> error: scan-error: (Containing expression ends prematurely 238020
> 238021))").
> 
> I'm pretty confident that the emacs module is correctly built (I can load
> it, and run the functions in it). Also, if I rename _mymod.so to something
> else, e.g. _mymod.blablabla, the rest of the package installs correctly and
> can be loaded. So it really seems as if Emacs is choking on trying to
> generate autoloads for _mymod.so.
> 
> Maybe backtracking a little bit:
> 
> 1) Should Emacs be able to generate autoloads for an .so?

No, I don't think so.  I think you should have these autoloads in the
Lisp part of your package.

So I suggest to review your definition of the package that package.el
uses to decide which files to scan for autoloads, and perhaps change
it to not imply that _mymod.so should be scanned.

Caveat: I have never prepared any package using the package.el
infrastructure, so I could be wrong in what I write above.  Or it
could be a bug in package.el, in which case I suggest to report it,
including all the details, using "M-x report-emacs-bug RET".

Thanks.

Re: Wrapping an Emacs dynamic module in a package

[Stefan Monnier]

> 1) I wrote an emacs module _mymod.so, as well as wrappers mymod.el (with
> some local setup) and mymod-pkg.el (which only has a call to
> define-package).

FWIW, I doubt you wrote _mymod.so.  Most likely you wrote something like
_mymod.c and then you asked a C compiler and a linker turn that into
a _mycmd.so.

Yes, it's a nitpick, but when you send your bug-report, please try to
use precise wording, because it's often surprisingly easy to introduce
major confusion otherwise.

> In my messages buffer, I see that Emacs tries to generate autoloads for the
> .so, and fails with the error indicated in the original email ("_mymod.so:0:0:
> error: scan-error: (Containing expression ends prematurely 238020
> 238021))").

In your bug-report, please show the corresponding backtrace.

> 1) Should Emacs be able to generate autoloads for an .so?

No.  It shouldn't scan those files for autoloads.

> 2) Does anybody have any experience with embedding an emacs module in an
> emacs package? Any best practices I should be aware of?

You might very well be the first to try it out.

Note, tho, that the ELPA format is a tarball containing .el files,
i.e. *source* files.  Similarly, it should hold _mycmd.c rather than
_mycmd.so.  Then again, package.el doesn't know how to turn that
_mycmd.c into _mycmd.so.   You can probably work around that problem by
adding something like

    (eval-when-compile
      (call-process "cc" nil nil nil <args>))

in your mymod.el (so that when the .el file gets byte-compiled, it
triggers compilation and linking of the C module).

But yes, including the .so *should* happen work (barring bugs) as long
as you're lucky enough to use the package on the right OS with the right
Emacs version, on the right architecture.


        Stefan

Re: Wrapping an Emacs dynamic module in a package

[Joris Vankerschaver]

Hi Stefan,

Thanks for your reply. I will create a bug report as you suggested, with
the necessary info (clear description, backtrace, etc).

For the record, I think I know what the issue is: update-directory-autoloads
in autoload.el will look at get-load-suffixes to build a list of files that
need autoloads. It removes .elc from the list of extensions to take into
consideration when building this list, and I think that also removing .so
will do trick. This at least allows my package to install cleanly (and be
usable afterwards).

You raise a point that I hadn't thought about: including the shared object
file in my package will make it of course platform-dependent.

Also, thanks for correcting me on my verbiage regarding _mymod.SO vs.
_mymod.C. It's always good not to take things for granted.

All the best,
Joris



> Date: Mon, 17 Oct 2016 09:27:16 -0400
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> To: help-gnu-emacs@gnu.org
> Subject: Re: Wrapping an Emacs dynamic module in a package
> Message-ID: <jwvbmyjdshz.fsf-monnier+gmane.emacs.help@gnu.org>
> Content-Type: text/plain
>
> > 1) I wrote an emacs module _mymod.so, as well as wrappers mymod.el (with
> > some local setup) and mymod-pkg.el (which only has a call to
> > define-package).
>
> FWIW, I doubt you wrote _mymod.so.  Most likely you wrote something like
> _mymod.c and then you asked a C compiler and a linker turn that into
> a _mycmd.so.
>
> Yes, it's a nitpick, but when you send your bug-report, please try to
> use precise wording, because it's often surprisingly easy to introduce
> major confusion otherwise.
>
> > In my messages buffer, I see that Emacs tries to generate autoloads for
> the
> > .so, and fails with the error indicated in the original email
> ("_mymod.so:0:0:
> > error: scan-error: (Containing expression ends prematurely 238020
> > 238021))").
>
> In your bug-report, please show the corresponding backtrace.
>
> > 1) Should Emacs be able to generate autoloads for an .so?
>
> No.  It shouldn't scan those files for autoloads.
>
> > 2) Does anybody have any experience with embedding an emacs module in an
> > emacs package? Any best practices I should be aware of?
>
> You might very well be the first to try it out.
>
> Note, tho, that the ELPA format is a tarball containing .el files,
> i.e. *source* files.  Similarly, it should hold _mycmd.c rather than
> _mycmd.so.  Then again, package.el doesn't know how to turn that
> _mycmd.c into _mycmd.so.   You can probably work around that problem by
> adding something like
>
>     (eval-when-compile
>       (call-process "cc" nil nil nil <args>))
>
> in your mymod.el (so that when the .el file gets byte-compiled, it
> triggers compilation and linking of the C module).
>
> But yes, including the .so *should* happen work (barring bugs) as long
> as you're lucky enough to use the package on the right OS with the right
> Emacs version, on the right architecture.
>
>

Re: Wrapping an Emacs dynamic module in a package

[Stefan Monnier]

> For the record, I think I know what the issue is: update-directory-autoloads
> in autoload.el will look at get-load-suffixes to build a list of files that
> need autoloads.

Duh!  Indeed, that's obvious now that you say it.  Hopefully that can be
fixed in Emacs-25.2.


        Stefan