A plea for dynamically loadable extension modules

A plea for dynamically loadable extension modules

[Mario Lang]

Hello.

I'd like to re-raise discussion about this topic by explaining
my current situation and the problems arising from it.

I am writing bindings for the GNOME Assistive Technology Service
Provider Interface, a library which allows querying of the GNOME
desktop and extracting information about various objects on the desktop.
Emacs is my favourite User Interface, so I'd like to eventually
make those bindings available to elisp.

Currently, I wrote the bindings for Guile, so that I can call
the underlying C functions from Scheme.
I then used some code from the guile CVS which interfaces
Guile to Emacs Lisp by running a Guile process as external program.
However, this code has several problems:

1. It does not handle multiline strings correctly.  When a string
containing a \n is received, the parser fails on the elisp side.

2. Scheme does to my knowledge not have a way to produce
equivalents to lisp's keywords.  I had to introduce
a re-search-forward/replace-match loop which replaces
{#:.*} with :.*.  This way I can say #:tag in scheme, and get
:tag in elisp.

3. I do nto see a way to work with non-standard object types.
For instance, my bindings to GNOME's AT-SPI use a smob in Scheme
which holds the pointer to the actual objects in AT-SPI.
So, upon executing a function like (desktop 0) I get
#<Accessible 0xsomehexaddr role=unknown>
which is the printed representation of my internal smob.

However, I can not pass those objects to emacs lisp and vice versa,
which makes programming for these bindings in elisp directly
very hard.  What I'd need to do is create some kind of
cache on the scheme side which allows to reference those objects
by something else, like a id or something.  This is hackish, and
not something I particularily want to do.

However, there are several attempts to extend emacs
in a way which allows to load bindings to other C routines
dynamically.  I particularily remember Dave Love's patch,
which utilized libtool AFAIK.
However, I can only use those extentions to impelemnt what I want
if they are integrated into Emacs.  The whole point of
using such an extention is to avoid that the user has
to recompile Emacs.  Emacs is a large system, and many people
just do not want to mess with their existing Emacs installation.
So if I'd write my bindings to AT-SPI either as Emacs patch,
or as something which uses Dave's loader extention, in both
cases, the user would need to patch and recompile emacs, which
is not really good IMHO, because it would lower the
chance of people using my code.  Additionally, it also makes
it hard for distributions to include my code, since they'd need to
merge it with their emacs package, which is something they probably do not
want to do.

Is there any chance that we get the dynloading extention
for 21.4?  If not, could you please explain why?

Some people might think now: "Why use an editor to
do your project?"  Well, Emacs is an editor right, but
it is also an platform independent, and mostly user interface
independent programming environment.  Code I write in 
emacs lisp runs usually equally well either under X,
or under text interfaces.  This is a property that
NO OPEN SOURCE PROJECT KNOWN TO ME has.  And it is a 
wonderful property to me as a blind programmer, because programs I write
in Emacs Lisp do usually run under X Windows, even if
I am not even technically able to test them under X.

So I'd really love to see the posibility to extend Emacs
Lisp with certain functions, which need code at the C level.

P.S.: The latest released version of the project I am talking about
can be found at http://delysid.org/gspi.html in case you are
wondering.

-- 
Thanks,
  Mario

Re: A plea for dynamically loadable extension modules

[Nic]

Mario Lang <mlang@delysid.org> writes:

> Is there any chance that we get the dynloading extention 
> for 21.4?  If not, could you please explain why? 

I think rms made a statement on this subject a number of months
ago. The reason dynamic loading has not been included in emacs so far
is that there would then be the potential of breaking the terms of
the licence under which emacs is distributed.

The GPL requires all code linked to emacs to also be Free. If
dynamic linking was allowed that couldn't be gauranteed.

The reason guile can do it is that guile is licenced under the
GPL+exception which allows for linking to non-commercial code.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk

Re: A plea for dynamically loadable extension modules

[Kenichi Handa]

In article <871xw8i3tr.fsf@tapsellferrier.co.uk>, Nic <nferrier@tapsellferrier.co.uk> writes:

> Mario Lang <mlang@delysid.org> writes:
>>  Is there any chance that we get the dynloading extention 
>>  for 21.4?  If not, could you please explain why? 

> I think rms made a statement on this subject a number of months
> ago. The reason dynamic loading has not been included in emacs so far
> is that there would then be the potential of breaking the terms of
> the licence under which emacs is distributed.

> The GPL requires all code linked to emacs to also be Free. If
> dynamic linking was allowed that couldn't be gauranteed.

Just a note:

The latest X allows dynamic loading various objects for
XLC/XOM/XIM support.

For instance, see this file
    <http://ftp.x.org/pub/R6.6/xc/nls/XI18N_OBJS/en_US.UTF-8>

The content is:
# CATEGORY(XLC|XIM|OM)	SHARED_LIBRARY_NAME	FUNCTION_NAME
#
#	XI18N objects table for euro locales
#
XLC	common/xlcUTF-8		_XlcUnicodeLoader	# XLC_open
XOM	common/xomLTRTTB	_XomGenericOpenOM	# XOM_open
XIM	common/xiiimp		_SwitchOpenIM		# XIM_open
XIM	common/xiiimp		_XimpLocalOpenIM	# XIM_open

It is possible that any commercial programs come here.

So, as far as Emacs is linked with X (by regarding X library
as something fundamental one as libc), it's impossible to
guarantee the above condition.

---
Ken'ichi HANDA
handa@m17n.org

Re: A plea for dynamically loadable extension modules

[Mario Lang]

Nic <nferrier@tapsellferrier.co.uk> writes:

> Mario Lang <mlang@delysid.org> writes:
>
>> Is there any chance that we get the dynloading extention 
>> for 21.4?  If not, could you please explain why? 
>
> I think rms made a statement on this subject a number of months
> ago. The reason dynamic loading has not been included in emacs so far
> is that there would then be the potential of breaking the terms of
> the licence under which emacs is distributed.
>
> The GPL requires all code linked to emacs to also be Free. If
> dynamic linking was allowed that couldn't be gauranteed.

I do not fully understand this.  If the use of
non-free code in Emacs is explicitly forbidden, what is the problem then?
If someone would use a dynamic loader extention to integrate
commercial code with Emacs, he would just violate the license.

That is basicly the same thing you could do now already.
One could extend emacs with some code which links against
non-free libraries.  That would be a violation of the GPL,
and would be illegal.

> The reason guile can do it is that guile is licenced under the
> GPL+exception which allows for linking to non-commercial code.

Please understand me right, I do not want to link non-free
code into Emacs.  I want to link free code into Emacs.
So I do not understand why this is relevant.  I am not looking
for a loophole to allow linking of non-free code
into my favourite environment, what I would like to have
is the ability to interface with existing free libraries, like
the GNOME AT-SPI, without having to patch and recompile Emacs.

-- 
CYa,
  Mario | Debian Developer <URL:http://debian.org/>
        | Get my public key via finger mlang@db.debian.org
        | 1024D/7FC1A0854909BCCDBE6C102DDFFC022A6B113E44

Re: A plea for dynamically loadable extension modules

[Jason Rumney]

Mario Lang wrote:
> I do not fully understand this.  If the use of
> non-free code in Emacs is explicitly forbidden, what is the problem then?
> If someone would use a dynamic loader extention to integrate
> commercial code with Emacs, he would just violate the license.

Allowing dynamic linking makes it easier for the end-user to violate
the license, perhaps unwittingly, while the distributer of the linked
module might technically escape from any wrongdoing since they are not
doing the linking.

But I think the specific change that the objections were raised about
was a general dynamic linking mechanism that would allow Emacs to link
to any library. There is another type of dynamic linking, where
linked modules would need to conform to an Emacs-specific interface.
Then only Emacs-specific modules could be linked to Emacs. It would then
be much easier to tell developers of non-Free modules that linked to
Emacs to make their code Free, since they could not claim that their
module was exempt from the GPL because it was designed to link with 'Y',
not with Emacs.

Re: A plea for dynamically loadable extension modules

[Mario Lang]

Jason Rumney <jasonr@gnu.org> writes:

> Mario Lang wrote:
>> I do not fully understand this.  If the use of
>> non-free code in Emacs is explicitly forbidden, what is the problem then?
>> If someone would use a dynamic loader extention to integrate
>> commercial code with Emacs, he would just violate the license.
>
> Allowing dynamic linking makes it easier for the end-user to violate
> the license, perhaps unwittingly, while the distributer of the linked
> module might technically escape from any wrongdoing since they are not
> doing the linking.
>
> But I think the specific change that the objections were raised about
> was a general dynamic linking mechanism that would allow Emacs to link
> to any library. There is another type of dynamic linking, where
> linked modules would need to conform to an Emacs-specific interface.

I was not refering to a generic FFI, I was refering to
the second case you mentioned, an extention which required
linked modules to be specially written for Emacs.

Quoting Dave's mail to g.e.sources from 10 Apr 2002 23:22:27 +0100:

"This patch for Emacs 21.1 basically extends `load' to allow loading
compiled C modules and provides a script to build them."

It seems clear to me according to the attached REAMDE that
this extention does not provide a generic FFI.

So I think the point you raised above is not valid here.

> Then only Emacs-specific modules could be linked to Emacs. It would then
> be much easier to tell developers of non-Free modules that linked to
> Emacs to make their code Free, since they could not claim that their
> module was exempt from the GPL because it was designed to link with 'Y',
> not with Emacs.

Exactly.  The patch from Dave satisfies the mentioned
properties AFAICS.

-- 
CYa,
  Mario | Debian Developer <URL:http://debian.org/>
        | Get my public key via finger mlang@db.debian.org
        | 1024D/7FC1A0854909BCCDBE6C102DDFFC022A6B113E44

Re: A plea for dynamically loadable extension modules

[Paul Jarc]

Nic <nferrier@tapsellferrier.co.uk> wrote:
> Mario Lang <mlang@delysid.org> writes:
>> Is there any chance that we get the dynloading extention
>> for 21.4?  If not, could you please explain why?
>
> I think rms made a statement on this subject a number of months
> ago. The reason dynamic loading has not been included in emacs so far
> is that there would then be the potential of breaking the terms of
> the licence under which emacs is distributed.

That doesn't make sense.  Emacs could use the same exception in its
license that Guile uses.  The fact that the exception is used for
Guile shows that the FSF does not object to the exception in
principle.  There may be a specific reason it's not desirable for
Emacs, but then what is it?


paul

Re: A plea for dynamically loadable extension modules

[Kevin Rodgers]

Nic wrote:

> Mario Lang <mlang@delysid.org> writes:
>>Is there any chance that we get the dynloading extention 
>>for 21.4?  If not, could you please explain why? 
> 
> I think rms made a statement on this subject a number of months
> ago. The reason dynamic loading has not been included in emacs so far
> is that there would then be the potential of breaking the terms of
> the licence under which emacs is distributed.
> 
> The GPL requires all code linked to emacs to also be Free. If
> dynamic linking was allowed that couldn't be gauranteed.

But Emacs already has the ability, built into it, to run non-Free software:
the Lisp interpreter.

-- 
Kevin Rodgers

Re: A plea for dynamically loadable extension modules

[Stephen J. Turnbull]

>>>>> "Mario" == Mario Lang <mlang@delysid.org> writes:

    Mario> Emacs is my favourite User Interface, so I'd like to
    Mario> eventually make those bindings available to elisp.

XEmacs supports dynamically loadable extension modules (currently in
the trunk LDAP and PostgreSQL are normally built as modules), and the
facility is under active development.  We are currently working on
extending the facility to the Windows environment.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: A plea for dynamically loadable extension modules

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> >>>>> "Mario" == Mario Lang <mlang@delysid.org> writes:
> 
>     Mario> Emacs is my favourite User Interface, so I'd like to
>     Mario> eventually make those bindings available to elisp.
> 
> XEmacs supports dynamically loadable extension modules (currently in
> the trunk LDAP and PostgreSQL are normally built as modules), and
> the facility is under active development.  We are currently working
> on extending the facility to the Windows environment.

Should the model lend itself to Emacs easily, or are there
specialities involved with the memory organization/garbage
collection/call semantics and whatever else would sound like it was
heavily dependent on the specifics of internals?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: A plea for dynamically loadable extension modules

[Stephen J. Turnbull]

>>>>> "David" == David Kastrup <dak@gnu.org> writes:

    David> "Stephen J. Turnbull" <stephen@xemacs.org> writes:

    >> >>>>> "Mario" == Mario Lang <mlang@delysid.org> writes:

    Mario> Emacs is my favourite User Interface, so I'd like to
    Mario> eventually make those bindings available to elisp.

    >> XEmacs supports dynamically loadable extension modules

    David> Should the model lend itself to Emacs easily, or are there
    David> specialities involved with the memory organization/garbage
    David> collection/call semantics and whatever else would sound
    David> like it was heavily dependent on the specifics of
    David> internals?

AFAIK it's actually not very difficult on most modern Unix platforms,
including Mac OS X.

The main technical issues are about how to handle module unloading,
which is greatly desired by developer-users, and rather scary to those
of us who have to support the product.  AFAIK the main unsolved
problem is what to do about module-local Lisp data if the module gets
unloaded.

Anyway, Emacs has had dynamically loading support for several years,
it's just that the maintainers refused to apply the patch.  I suppose
that the patches for 20.x probably can be updated for 21.x.  Sorry, I
don't know where they are, and unfortunately Debian has apparently
removed its package versions, too.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: A plea for dynamically loadable extension modules

[Alex Schroeder]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Anyway, Emacs has had dynamically loading support for several years,
> it's just that the maintainers refused to apply the patch.  I suppose
> that the patches for 20.x probably can be updated for 21.x.  Sorry, I
> don't know where they are, and unfortunately Debian has apparently
> removed its package versions, too.

http://www.emacswiki.org/cgi-bin/emacs/DynamicallyExtendingEmacs says
that Steve Kemp was the author of one of these patches; I think his
patch for Emacs 20 was the first one.  I would like to see Dave's
patch in Emacs.  Perhaps the simple requirement of having an
Emacs-specific "initizalization" function available would be enough to
deter would-be license-breakers?

Alex.
-- 
http://www.emacswiki.org/cgi-bin/alex.pl
I was on holidays from 2003-07-01 to 2003-07-29 and have a lot of catching up to do.

Re: A plea for dynamically loadable extension modules

[Richard Stallman]

    So, as far as Emacs is linked with X (by regarding X library
    as something fundamental one as libc), it's impossible to
    guarantee the above condition.

I think that dynamic linking of these plug-ins that are meant
for use with X does not entail any violation of the GPL,
because these plug-ins do not relate specifically to Emacs
in any way.

If the plug-in relates specifically to Emacs, then our position
is that it constitutes an Emacs extension and has to be covered
by the GPL.  However, it would be nice not to have to go to court
about that, and if we don't have dynamic linking then we won't
have to.

Re: A plea for dynamically loadable extension modules

[Richard Stallman]

    That doesn't make sense.  Emacs could use the same exception in its
    license that Guile uses.  The fact that the exception is used for
    Guile shows that the FSF does not object to the exception in
    principle.  There may be a specific reason it's not desirable for
    Emacs, but then what is it?

That sort of exception is similar to use of the LGPL.  In most cases
it is a step backwards.  Only in special cases is it indirectly
beneficial.  See http://www.gnu.org/philosophy/why-not-lgpl.html
for an explanation of this issue.

Re: A plea for dynamically loadable extension modules

[Paul Jarc]

Richard Stallman <rms@gnu.org> wrote:
>     Emacs could use the same exception in its license that Guile
>     uses.
>
> That sort of exception is similar to use of the LGPL.

(I wonder why Guile doesn't simply use the LGPL, then.)  But aren't
there plans to make Emacs based on Guile with an elisp translator?
Then Emacs will get the exception (and the dynamic extension
capability) anyway, won't it?


paul

Re: A plea for dynamically loadable extension modules

[Richard Stallman]

    (I wonder why Guile doesn't simply use the LGPL, then.)  But aren't
    there plans to make Emacs based on Guile with an elisp translator?
    Then Emacs will get the exception (and the dynamic extension
    capability) anyway, won't it?

Only the Guile code would (perhaps) have this exception (if we didn't
remove it from the copy in Emacs).  The rest of Emacs certainly would
not.

Re: A plea for dynamically loadable extension modules

[Leo]

On 2003-07-30 13:42 +0100, Nic wrote:
> Mario Lang <mlang@delysid.org> writes:
>
>> Is there any chance that we get the dynloading extention 
>> for 21.4?  If not, could you please explain why? 
>
> I think rms made a statement on this subject a number of months
> ago. The reason dynamic loading has not been included in emacs so far
> is that there would then be the potential of breaking the terms of
> the licence under which emacs is distributed.
>
> The GPL requires all code linked to emacs to also be Free. If
> dynamic linking was allowed that couldn't be gauranteed.

Has this situation changed after adapting GPL3?

>
> The reason guile can do it is that guile is licenced under the
> GPL+exception which allows for linking to non-commercial code.

-- 
Leo <sdl.web AT gmail.com>                         (GPG Key: 9283AA3F)

         Gnus is one component of the Emacs operating system.

Re: A plea for dynamically loadable extension modules

[Richard Stallman]

    > The GPL requires all code linked to emacs to also be Free. If
    > dynamic linking was allowed that couldn't be gauranteed.

    Has this situation changed after adapting GPL3?

We have tried to make it more explicit that the GPL applies to code
designed to be dynamically linked into Emacs.  However, the question
is whether any free software license can make that requirement.
That is a question of copyright law.

    > The reason guile can do it is that guile is licenced under the
    > GPL+exception which allows for linking to non-commercial code.

I think that meanmt to say "non-GPL code". or "proprietary code".

(Let's not confuse commercial with proprietary!
See http://www.gnu.org/philosophy/words-to-avoid.html.)

Re: A plea for dynamically loadable extension modules

[Leo]

On 2007-08-19 01:45 +0100, Richard Stallman wrote:
>     > The GPL requires all code linked to emacs to also be Free. If
>     > dynamic linking was allowed that couldn't be gauranteed.
>
>     Has this situation changed after adapting GPL3?
>
> We have tried to make it more explicit that the GPL applies to code
> designed to be dynamically linked into Emacs.  However, the question
> is whether any free software license can make that requirement.
> That is a question of copyright law.

But is it OK to include this patch¹ in Emacs? It allows C DEFUN
functions to be placed in external libraries and dynamically loaded when
they were needed.

Footnotes: 
¹  http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

-- 
Leo <sdl.web AT gmail.com>                         (GPG Key: 9283AA3F)

         Gnus is one component of the Emacs operating system.

Re: A plea for dynamically loadable extension modules

[dhruva]

Hi,

On 8/19/07, Leo <sdl.web@gmail.com> wrote:
> But is it OK to include this patch¹ in Emacs? It allows C DEFUN
> functions to be placed in external libraries and dynamically loaded when
> they were needed.

I personally feel it would be an extremely nice feature to have the
above functionality in Emacs. If the concern is to make sure such
libraries are released under GPL, I propose the following method:

1. All dynamic libraries loaded dynamically (through explicit call to
LoadLibrary/dlopen) must expose a function 'IsGPLed'
2. In main Emacs, we load it and look for that function
(GetProcAdderess/dlsym) and execute if found. It could return a
'bool'.
3. The absence of the function or a false return can prevent loading
that extension library
4. We could cover the meaning and legal bindings of having 'IsGPLed'
somewhere in the license and it is the responsibility of the
library/owner to make sure it is completely covered under GPL if the
function is implemented to return 'true' and Emacs does not take any
responsibility beyond checking for the function and it's return value
and no more.

Would this be a feasible approach?

-dky

-- 
Dhruva Krishnamurthy
Contents reflect my personal views only!

Re: A plea for dynamically loadable extension modules

[Stephen J. Turnbull]

dhruva writes:

 > Would [restricting linakge to modules containing a specific
 > function] be a feasible approach?

I certainly hope not.  That's a link-time or run-time restriction.
The GPL only covers copying and distribution activity.

Re: A plea for dynamically loadable extension modules

[Thien-Thi Nguyen]

() dhruva <dhruvakm@gmail.com>
() Sun, 19 Aug 2007 08:52:29 +0530

   Would this be a feasible approach?

i don't think so.  the intent is that the action of linking GPL code to
non-free code should not be prohibited.  instead, doing so should extend
GPL protections to the non-free code, in essence "freeing" it.  i use
quotes because to realize the new state of the module's ("formerly"
non-free) code requires downstream action of (imputes responsibility to)
someone, possibly not the same person as the user.

here is my take on what this means, concretely:

A/ user does not redistribute anything.
=> module's status is irrelevant
   (GPL only pertains to *re*distribution)

B/ user redistributes w/in organization.
=> module's status is irrelevant
   (GPL only pertains when organization boundary is crossed)

C/ user redistributes outside organization.
=> GPL requires that to redistribute Emacs + module, source code for
   both Emacs and the module be also available; if source for the module
   cannot or will not be available, GPL prohibits such redistribution
   altogether.

to sum up, link time exception based on `(GPL-p module)' does not
address the intent (assuming i understand both the intent and your
proposal).

thi

Re: A plea for dynamically loadable extension modules

[David Hansen]

On Sun, 19 Aug 2007 08:52:29 +0530 dhruva wrote:

> On 8/19/07, Leo <sdl.web@gmail.com> wrote:
>> But is it OK to include this patch¹ in Emacs? It allows C DEFUN
>> functions to be placed in external libraries and dynamically loaded when
>> they were needed.
>
> I personally feel it would be an extremely nice feature to have the
> above functionality in Emacs. If the concern is to make sure such
> libraries are released under GPL, I propose the following method:
>
> 1. All dynamic libraries loaded dynamically (through explicit call to
> LoadLibrary/dlopen) must expose a function 'IsGPLed'
> 2. In main Emacs, we load it and look for that function
> (GetProcAdderess/dlsym) and execute if found. It could return a
> 'bool'.
> 3. The absence of the function or a false return can prevent loading
> that extension library
> 4. We could cover the meaning and legal bindings of having 'IsGPLed'
> somewhere in the license and it is the responsibility of the
> library/owner to make sure it is completely covered under GPL if the
> function is implemented to return 'true' and Emacs does not take any
> responsibility beyond checking for the function and it's return value
> and no more.
>
> Would this be a feasible approach?

Isn't that similar to what Linux does with kernel modules?  I think what
the FSF wants to avoid is exactly the current situation with Linux and
some non free kernel modules.

David

Re: A plea for dynamically loadable extension modules

[Richard Stallman]

Our legal position is that the GPL applies to any code _distributed_
to be used specifically as part of Emacs, whether they are combined
by dynamic linking or not.  The GPL also applies to any code that
Emacs is specifically designed to link with (except that in that case,
the responsibility to avoid license conflicts is ours).

This requirement would not apply to some general-purpose library, even
if it could be dynamically linked into Emacs, if some user decides to
link it in.

> 4. We could cover the meaning and legal bindings of having 'IsGPLed'
> somewhere in the license and it is the responsibility of the
> library/owner to make sure it is completely covered under GPL if the
> function is implemented to return 'true' and Emacs does not take any
> responsibility beyond checking for the function and it's return value
> and no more.

That seems to propose a change in the GPL.  It isn't feasible.

Re: A plea for dynamically loadable extension modules

[Richard Stallman]

    But is it OK to include this patch¹ in Emacs? It allows C DEFUN
    functions to be placed in external libraries and dynamically loaded when
    they were needed.

I will have to think about it some more.
Meanwhile, can you please email me (not the list) the patch?