good examples of Emacs modules?

good examples of Emacs modules?

[Paul Eggert]

I am thinking of assigning a programming problem for students to use the 
new Emacs module facility. Are there good examples of how module-load 
can be used, that I can point people at? Ideally the examples would work 
on RHEL 7, since that's the standard instructional environment around 
here. Thanks for any pointers you can provide.

Re: good examples of Emacs modules?

[Aurélien Aptel]

On Thu, Mar 31, 2016 at 1:17 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> I am thinking of assigning a programming problem for students to use the new
> Emacs module facility. Are there good examples of how module-load can be
> used, that I can point people at? Ideally the examples would work on RHEL 7,
> since that's the standard instructional environment around here. Thanks for
> any pointers you can provide.

I'm in the process of writing an introduction. You can find a draft here:

http://diobla.info/blog-archive/modules-tut.html

It should get them started. Everything else is documented in the
emacs-module.h header.

Re: good examples of Emacs modules?

[Óscar Fuentes]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> I'm in the process of writing an introduction. You can find a draft here:
>
> http://diobla.info/blog-archive/modules-tut.html
>
> It should get them started. 

Thanks for working on this. I'm considering a module for using libgit2
from Emacs, but passing data structures around and the creation, storage
and disposal of long-lived data (sessions, etc) looks a bit daunting.
Some documentation would help a lot.

> Everything else is documented in the emacs-module.h header.

The documentation on that file is scant, to say it lightly. We need high
level descriptions of the APIs, with examples, if possible. Then, a
chapter on techniques and recommended strategies for creating and using
modules that reflect common C API designs. Whithout that, the real job
is not creating a module for libfoo, but figuring out how emacs modules
are supposed to work.

Re: good examples of Emacs modules?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 31 Mar 2016 17:34:03 +0200
> 
> > Everything else is documented in the emacs-module.h header.
> 
> The documentation on that file is scant, to say it lightly. We need high
> level descriptions of the APIs, with examples, if possible. Then, a
> chapter on techniques and recommended strategies for creating and using
> modules that reflect common C API designs. Whithout that, the real job
> is not creating a module for libfoo, but figuring out how emacs modules
> are supposed to work.

You can also look in modules/mod-test/, if you haven't already.

Re: good examples of Emacs modules?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Everything else is documented in the emacs-module.h header.
>> 
>> The documentation on that file is scant, to say it lightly. We need high
>> level descriptions of the APIs, with examples, if possible. Then, a
>> chapter on techniques and recommended strategies for creating and using
>> modules that reflect common C API designs. Whithout that, the real job
>> is not creating a module for libfoo, but figuring out how emacs modules
>> are supposed to work.
>
> You can also look in modules/mod-test/, if you haven't already.

Thank you, that file acts as a nice cookbook. Still need to figure out
how to best handle a long-lived session with an external API.

I'll like to see the high level document that Aurélien started to be
expanded, though, and eventually incorporated into the official Emacs
documentation.

Re: good examples of Emacs modules?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 31 Mar 2016 19:07:29 +0200
> 
> > You can also look in modules/mod-test/, if you haven't already.
> 
> Thank you, that file acts as a nice cookbook. Still need to figure out
> how to best handle a long-lived session with an external API.

Not sure what you mean by "a long-lived session with an external
API".  Can you elaborate?

> I'll like to see the high level document that Aurélien started to be
> expanded, though, and eventually incorporated into the official Emacs
> documentation.

I think Aurélien writes a tutorial, whereas the manual needs a
reference.  But I agree that a tutorial would be a tremendously
helpful resource in this matter.

Re: good examples of Emacs modules?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Thank you, that file acts as a nice cookbook. Still need to figure out
>> how to best handle a long-lived session with an external API.
>
> Not sure what you mean by "a long-lived session with an external
> API".  Can you elaborate?

Some APIs are session-based. Those give the user an object that lives
through the session and needs to be disposed at the end. On the case of
VC or Magit, Emacs would request from libgit2 to work with a given repo,
operate on it and close the session. That means that if the user kills
the magit-status buffer that was visiting a repo, the associated session
object must be closed (this is a simplistic scenario, in practice things
are more complicated.) It would be nice to explain how to handle this
use case on a robust way (the failure to properly close a session,
transaction, whatever on some packages can produce very annoying
consequences.) How objects created by the module system interact with
garbage collection is an start.

[snip]

Re: good examples of Emacs modules?

[Aurélien Aptel]

Hi all,

On Thu, Mar 31, 2016 at 10:23 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> consequences.) How objects created by the module system interact with
> garbage collection is an start.

Every emacs_value allocated inside a Lisp-exposed function becomes
invalid when control of the function ends (when it exits) unless you
return that value or you mark it global with the the
env->make_global_ref() API call.

As I was googling for a browseable emacs git repo to look quickly at
emacs-module.h I've noticed that Syohei Yoshida (cc'ed) has figured a
lot of things out by himself (kudos to you!) and has already made some
simple and not-so-simple modules:

- dead simple module that exposes an ioctl syscall to Lisp
  https://github.com/syohex/emacs-eject

- json parser/encoder
  https://github.com/syohex/emacs-parson
  this one is interesting because he has done some benchmarks!

- libGeoIP binding
  https://github.com/syohex/emacs-geoip

- libbarcode binding
  https://github.com/syohex/emacs-barcode

- libmemcached binding
  https://github.com/syohex/emacs-memcached

- libyaml binding
  https://github.com/syohex/emacs-libyaml

- libqrencode binding
  https://github.com/syohex/emacs-qrencode

- embedded Lua interpreter
  https://github.com/syohex/emacs-lua

- embedded Ruby interpreter
  https://github.com/syohex/emacs-mruby-test

- wrapper that lets you implement modules in Nim
  https://github.com/yuutayamada/nim-emacs-module

I've looked briefly at most of them and I think they are good
examples. Especially how he only implements low-level functionality of
package xyz in a xyz-core module which is exactly how I envisioned it.

- Tom Tromey has also made a binding on libffi which means you can
call C stuff from Lisp without writing a module.
  https://github.com/tromey/emacs-ffi

It seems the japanese-speaking Emacs community has picked up on the
modules feature quite fast. I found several article/blog about it. Too
bad I can't read it, it looks interesting :)

- article on how the emacs-eject module was done
  http://syohex.hatenablog.com/entry/2015/12/16/185458
- article on the module feature with *benchmarks* and side-by-side
code comparison. Very cool!
  http://coldnew.github.io/blog/2015/01/03_emacsdynamic/

Syohei, I really like what you did and I would be interested in your
feedback on what could be improved, what should we add&optimize in the
module API, etc.

Thanks!

Re: good examples of Emacs modules?

[Óscar Fuentes]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> Hi all,
>
> On Thu, Mar 31, 2016 at 10:23 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
>> consequences.) How objects created by the module system interact with
>> garbage collection is an start.
>
> Every emacs_value allocated inside a Lisp-exposed function becomes
> invalid when control of the function ends (when it exits) unless you
> return that value or you mark it global with the the
> env->make_global_ref() API call.

Ok.

> As I was googling for a browseable emacs git repo to look quickly at
> emacs-module.h I've noticed that Syohei Yoshida (cc'ed) has figured a
> lot of things out by himself (kudos to you!) and has already made some
> simple and not-so-simple modules:

Thank you. The more examples, the better.

I'll like to encourage you to keep expanding the introduction, though.
The type on information it contains saves a lot of head scratching.

BTW, it is necessary to explicitly export the symbols on Windows (and on
GNU/Linux too depending on the arguments used):

int plugin_is_GPL_compatible;

should be

int __declspec(dllexport) plugin_is_GPL_compatible;

(Windows)

int __attribute__ ((visibility("default"))) plugin_is_GPL_compatible;

(GNU/Linux, when you compile your module with -fvisibility=hidden, which
is a Good Thing.)

This is usually implemented with a macro.

More info:

https://gcc.gnu.org/wiki/Visibility

That page mentions C++ but it applies to C too.

[snip]

Re: good examples of Emacs modules?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 31 Mar 2016 23:59:36 +0200
> 
> BTW, it is necessary to explicitly export the symbols on Windows (and on
> GNU/Linux too depending on the arguments used):
> 
> int plugin_is_GPL_compatible;
> 
> should be
> 
> int __declspec(dllexport) plugin_is_GPL_compatible;
> 
> (Windows)
> 
> int __attribute__ ((visibility("default"))) plugin_is_GPL_compatible;

I don't need any such Windows-specific attributes, so I'm unsure why
you think you do.  The test in modules/mod-test compiles and passes
the tests just fine without that.

> (GNU/Linux, when you compile your module with -fvisibility=hidden, which
> is a Good Thing.)

If someone uses -fvisibility=hidden on a shared object, they know what
they are doing, and they need the attribute on every exported symbol,
not just on plugin_is_GPL_compatible.

Re: good examples of Emacs modules?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 31 Mar 2016 23:59:36 +0200
>> 
>> BTW, it is necessary to explicitly export the symbols on Windows (and on
>> GNU/Linux too depending on the arguments used):
>> 
>> int plugin_is_GPL_compatible;
>> 
>> should be
>> 
>> int __declspec(dllexport) plugin_is_GPL_compatible;
>> 
>> (Windows)
>> 
>> int __attribute__ ((visibility("default"))) plugin_is_GPL_compatible;
>
> I don't need any such Windows-specific attributes, so I'm unsure why
> you think you do.  The test in modules/mod-test compiles and passes
> the tests just fine without that.

This is because MinGW defaults to "export everything" (as soon as the
compiler sees a dllexport, it disables the "export everything" feature),
same as gcc on GNU/Linux does. On Windows this is not a good practice if
you have multiple dlls (modules) with potentially identical symbols
exported. Not a big deal on the case of Emacs modules, but having to
explicitly exporting symbols is something a non-MinGW-immersed Windows
developer assumes as a fact.

>> (GNU/Linux, when you compile your module with -fvisibility=hidden, which
>> is a Good Thing.)
>
> If someone uses -fvisibility=hidden on a shared object, they know what
> they are doing, and they need the attribute on every exported symbol,
> not just on plugin_is_GPL_compatible.

Yes, that's the idea. Explicitly exporting symbols is a recommended
practice (by GCC) on GNU/Linux and a requirement on Windows, except on
the case of MinGW which implements a hack, which is problematic, and the
MinGW developers (the old ones such as Danny, who knew the binutils and
gcc internals) advised against using it.

Anyway, it is possibly a good idea to left things how they are now as
most modules will be developed on GNU/Linux and decorating
`plugin_is_GPL_compatible' may cause confusion, both to module writers
on GNU/Linux and builders on Windows.

So let's forget the issue for now.

Re: good examples of Emacs modules?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 01 Apr 2016 15:20:42 +0200
> 
> >> int __declspec(dllexport) plugin_is_GPL_compatible;
> >> 
> >> (Windows)
> >> 
> >> int __attribute__ ((visibility("default"))) plugin_is_GPL_compatible;
> >
> > I don't need any such Windows-specific attributes, so I'm unsure why
> > you think you do.  The test in modules/mod-test compiles and passes
> > the tests just fine without that.
> 
> This is because MinGW defaults to "export everything" (as soon as the
> compiler sees a dllexport, it disables the "export everything" feature),
> same as gcc on GNU/Linux does.

Yes, I know.  But this is not unique to Emacs, so I don't see why we
need to do something that no one else does.

> On Windows this is not a good practice if you have multiple dlls
> (modules) with potentially identical symbols exported.

If this program exists, it will preclude programs using such DLLs from
compiling.

Re: good examples of Emacs modules?

[Syohei Yoshida]

Hi,

>
> Syohei, I really like what you did and I would be interested in your
> feedback on what could be improved, what should we add&optimize in the
> module API, etc.
>

I want vector(or list) and hash table API(there are some vector API in
emacs_env).
I know that we can use them via env->funcall, however we need a lot of code for
using them(intern function name, set arguments to emacs_value array,
env->funcall).
I think it is useful if we can handle vector, list, hash table easily.

Regards

-- 
Syohei YOSHIDA(syohex@gmail.com)

Re: good examples of Emacs modules?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Syohei Yoshida <syohex@gmail.com> schrieb am Sa., 2. Apr. 2016 um 04:10 Uhr:

> Hi,
>
> >
> > Syohei, I really like what you did and I would be interested in your
> > feedback on what could be improved, what should we add&optimize in the
> > module API, etc.
> >
>
> I want vector(or list) and hash table API(there are some vector API in
> emacs_env).
> I know that we can use them via env->funcall, however we need a lot of
> code for
> using them(intern function name, set arguments to emacs_value array,
> env->funcall).
> I think it is useful if we can handle vector, list, hash table easily.
>

I think the API should be kept as small as possible. Every environment
member has to be designed, maintained, etc. If we add vector, list, and
hash table functions, why stop there? Why not add buffer and string
functions? Or process and network functions? I think new functions should
only be added if they either solve a problem that could not be solved
otherwise, or the replacement code would be very awkward, or there is a
proven significant efficiency gain.

[-- Attachment #2: Type: text/html, Size: 1397 bytes --]

Re: good examples of Emacs modules?

[Stefan Monnier]

> I think the API should be kept as small as possible. Every environment
> member has to be designed, maintained, etc. If we add vector, list, and
> hash table functions, why stop there? Why not add buffer and string
> functions? Or process and network functions? I think new functions should
> only be added if they either solve a problem that could not be solved
> otherwise, or the replacement code would be very awkward, or there is a
> proven significant efficiency gain.

FWIW, I completely agree with keeping the API minimalist.  I think we
can accommodate the request "on the other side", e.g. with a library (or
several of them) that the module can link with (probably statically)
which will do the dance of interning the function name and such.


        Stefan

Re: good examples of Emacs modules?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 31 Mar 2016 22:23:56 +0200
> 
> Some APIs are session-based. Those give the user an object that lives
> through the session and needs to be disposed at the end. On the case of
> VC or Magit, Emacs would request from libgit2 to work with a given repo,
> operate on it and close the session. That means that if the user kills
> the magit-status buffer that was visiting a repo, the associated session
> object must be closed (this is a simplistic scenario, in practice things
> are more complicated.) It would be nice to explain how to handle this
> use case on a robust way (the failure to properly close a session,
> transaction, whatever on some packages can produce very annoying
> consequences.) How objects created by the module system interact with
> garbage collection is an start.

I think you want to wrap some kind of "handle" that is meaningful to
libgit2 into a user-ptr object (see the existing docs about modules
for details about these).  A user-ptr object has a finalizer, a
function that is called by GC when the object goes out of scope.  The
finalizer function should do whatever libgit2 needs to close the
session and free whatever resources the session used.

You have a demo if using a user-ptr in modules/mod-test/.

Re: good examples of Emacs modules?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> I think you want to wrap some kind of "handle" that is meaningful to
> libgit2 into a user-ptr object (see the existing docs about modules
> for details about these).  A user-ptr object has a finalizer, a
> function that is called by GC when the object goes out of scope.

This means that GCing the object is unpredictable, right? Also, if the
API uses a struct as the session object, you need to store it somewhere
and expose a pointer to it to the Elisp code; that is, you can't put
arbitrary data on the Elisp memory space.

> The
> finalizer function should do whatever libgit2 needs to close the
> session and free whatever resources the session used.
>
> You have a demo if using a user-ptr in modules/mod-test/.

Thanks.

Re: good examples of Emacs modules?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 01 Apr 2016 15:25:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think you want to wrap some kind of "handle" that is meaningful to
> > libgit2 into a user-ptr object (see the existing docs about modules
> > for details about these).  A user-ptr object has a finalizer, a
> > function that is called by GC when the object goes out of scope.
> 
> This means that GCing the object is unpredictable, right?

It's as predictable and as unpredictable as what you have with any
other Lisp object.  As long as the object is referenced, it will not
be GC'ed.

> Also, if the API uses a struct as the session object, you need to
> store it somewhere and expose a pointer to it to the Elisp code;
> that is, you can't put arbitrary data on the Elisp memory space.

The user-ptr object is precisely the way to produce a Lisp object that
holds such a pointer, so I'm not sure why you are describing this as
some kind of problem: it's a solution, not a problem.  You allocate
memory for the struct, store the info there, then wrap the resulting
pointer in user-ptr, and that's it.  (The finalizer should free the
memory and perform whatever other cleanup is needed.)  If this is
somehow problematic, please tell why.

Re: good examples of Emacs modules?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 31. März
2016 um 16:49 Uhr:

> On Thu, Mar 31, 2016 at 1:17 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> > I am thinking of assigning a programming problem for students to use the
> new
> > Emacs module facility. Are there good examples of how module-load can be
> > used, that I can point people at? Ideally the examples would work on
> RHEL 7,
> > since that's the standard instructional environment around here. Thanks
> for
> > any pointers you can provide.
>
> I'm in the process of writing an introduction. You can find a draft here:
>
> http://diobla.info/blog-archive/modules-tut.html
>
> It should get them started. Everything else is documented in the
> emacs-module.h header.
>
>
Thanks. I'd suggest the following additions:
1. Please discuss error handling early on. How errors are handled is
arguably the most subtle point of the API.
2. Please discuss the lifetime of environments and values. This is also
extremely subtle and can easily lead to undefined behavior.
3. Please add explicit checks for the size of structures. If the actual
structure passed from Emacs is smaller than the expected structure,
undefined behavior will happen if one of the "excess" members is accessed.
4. You should also mention how Emacs deals with stack overflow, because
that can lead to inconsistent data structures and undefined behavior as
well.

I'm writing a reference and a set of caveats for modules, unfortunately
progress is very slow on my part.

[-- Attachment #2: Type: text/html, Size: 2104 bytes --]