Adding GnuTLS to the Windows port

Adding GnuTLS to the Windows port

[Juanma Barranquero]

I've been working on compiling Emacs on Windows with GnuTLS support.

It works, but there are a few questions that should be decided before
I dig deeper into it.

Currently I can link Emacs against the static library libgnutls.a, or
use libgnutls.dll.a and have Emacs depend on libgnutls-26.dll (which
in turn brings libgcrypt-11, libpgp-error-0 and libtasn1-3).

Options:

1) Static linking
    Pros: no failure because of missing DLLs
    Contras: 5+MB, which won't be used by most Windows users; Emacs
tied to a specific version of GnuTLS.

2) Dynamic linking
    Pros: update of GnuTLS possible without recompiling
    Contras: the executable depends on the DLLs, even when dumping, so
failure to load if they are unavailable.

3) Dynamic loading (the same system used on image.c for the image libraries)
    Pros: most flexible; Emacs can run without the DLLs.
    Contras: code changes in gnutls.c to use function pointers (like image.c).

IMO, 3 is the best option, but it does not make sense to implement it
until the GnuTLS interface has stabilized, which I think it's a bit
far off yet. Opinions?

An additional question related to 3: it would make sense to rename
`image-library-alist' to `dynamic-library-alist' (with an obsolete
alias `image-library-alist'), move it to some other suitable .c file
(perhaps emacs.c?) and rewrite its doc to turn it into a more generic
facility.  It certainly would be handy to use it for (gnutls .
"libgnutls-26.dll"), and seems like a logical step.

BTW, the same question will have to be answered sooner or later for
xml.c / libxml2 and dbus.

    Juanma

Re: Adding GnuTLS to the Windows port

[Juanma Barranquero]

On Sun, Oct 3, 2010 at 22:52, Juanma Barranquero <lekktu@gmail.com> wrote:

> An additional question related to 3: it would make sense to rename
> `image-library-alist' to `dynamic-library-alist' (with an obsolete
> alias `image-library-alist'), move it to some other suitable .c file
> (perhaps emacs.c?) and rewrite its doc to turn it into a more generic
> facility.

Patch attached.

Though the description of dynamic-library-alist, currently at
"(elisp)Image Formats" should be moved... where?

    Juanma


2010-10-05  Juanma Barranquero	<lekktu@gmail.com>

	* emacs.c (Vdynamic_library_alist)
	(syms_of_emacs) <dynamic-library-alist>: Move from image.c and rename.
	Doc fix.

	* lisp.h (Vdynamic_library_alist): Declare extern.

	* image.c (Vimage_library_alist)
	(syms_of_image) <image-library-alist>: Move to emacs.c and rename.
	(lookup_image_type): Use Vdynamic_library_alist.
	(Finit_image_library): Doc fix.

	* image.el (image-library-alist): Declare as obsolete alias.
	(image-type-available-p): Use `dynamic-library-alist'.

	* term/w32-win.el (dynamic-library-alist):
	Use instead of `image-library-alist'.

	* display.texi (Image Formats): Describe `dynamic-library-alist'.

	* NEWS: Mention `dynamic-library-alist'.

	* INSTALL: Refer to `dynamic-library-alist'.



=== modified file 'doc/lispref/display.texi'
--- doc/lispref/display.texi	2010-09-09 02:18:38 +0000
+++ doc/lispref/display.texi	2010-10-05 02:58:35 +0000
@@ -4056,7 +4056,7 @@
   Emacs can display a number of different image formats; some of them
 are supported only if particular support libraries are installed on
 your machine.  In some environments, Emacs can load image
-libraries on demand; if so, the variable @code{image-library-alist}
+libraries on demand; if so, the variable @code{dynamic-library-alist}
 can be used to modify the set of known names for these dynamic
 libraries (though it is not possible to add new image formats).

@@ -4082,21 +4082,25 @@
 @code{image-type-available-p}.
 @end defvar

-@defvar image-library-alist
-This in an alist of image types vs external libraries needed to
-display them.
-
-Each element is a list @code{(@var{image-type} @var{library}...)},
-where the car is a supported image format from @code{image-types}, and
-the rest are strings giving alternate filenames for the corresponding
-external libraries to load.
-
-Emacs tries to load the libraries in the order they appear on the
-list; if none is loaded, the running session of Emacs won't support
-the image type.  @code{pbm} and @code{xbm} don't need to be listed;
-they're always supported.
-
-This variable is ignored if the image libraries are statically linked
+@defvar dynamic-library-alist
+This is an alist of dynamic libraries vs external files implementing them.
+
+Each element is a list @code{(@var{library} @var{file}...)}, where the
+car is a symbol representing a supported external library, and the
+rest are strings giving alternate filenames for that library.
+
+Emacs tries to load the library from the files in the orden they
+appear on the list; if none is loaded, the running session of Emacs
+won't have access to that library.
+
+Note that image types @code{pbm} and @code{xbm} do not need entries in
+this variable because they do not depend on external libraries and are
+always available.
+
+Also note that this is not a generic facility for accessing external
+libraries; only those already known by Emacs will be loaded.
+
+This variable is ignored if the given libraries are statically linked
 into Emacs.
 @end defvar


=== modified file 'etc/NEWS'
--- etc/NEWS	2010-10-04 00:43:11 +0000
+++ etc/NEWS	2010-10-05 03:11:55 +0000
@@ -568,6 +568,10 @@
 \f
 * Lisp changes in Emacs 24.1

+** `image-library-alist' is renamed to `dynamic-library-alist'.
+The variable is now used to load all kind of supported dynamic libraries,
+not just image libraries.
+
 ** New variable syntax-propertize-function to set syntax-table properties.
 Replaces font-lock-syntactic-keywords which are now obsolete.
 This allows syntax-table properties to be set independently from font-lock:

=== modified file 'lisp/image.el'
--- lisp/image.el	2010-09-09 02:25:12 +0000
+++ lisp/image.el	2010-10-05 02:59:37 +0000
@@ -330,14 +330,16 @@
   type)


-(defvar image-library-alist)
+(define-obsolete-variable-alias
+    'image-library-alist
+    'dynamic-library-alist "24.1")

 ;;;###autoload
 (defun image-type-available-p (type)
   "Return non-nil if image type TYPE is available.
 Image types are symbols like `xbm' or `jpeg'."
   (and (fboundp 'init-image-library)
-       (init-image-library type image-library-alist)))
+       (init-image-library type dynamic-library-alist)))


 ;;;###autoload

=== modified file 'lisp/term/w32-win.el'
--- lisp/term/w32-win.el	2010-01-13 08:35:10 +0000
+++ lisp/term/w32-win.el	2010-10-05 02:27:57 +0000
@@ -196,10 +196,10 @@
   "Report an error when a suspend is attempted."
   (error "Suspending an Emacs running under W32 makes no sense"))

-(defvar image-library-alist)
+(defvar dynamic-library-alist)

-;;; Set default known names for image libraries
-(setq image-library-alist
+;;; Set default known names for external libraries
+(setq dynamic-library-alist
       '((xpm "libxpm.dll" "xpm4.dll" "libXpm-nox4.dll")
         (png "libpng12d.dll" "libpng12.dll" "libpng.dll"
 	 ;; these are libpng 1.2.8 from GTK+

=== modified file 'nt/INSTALL'
--- nt/INSTALL	2010-07-24 13:35:31 +0000
+++ nt/INSTALL	2010-10-05 02:50:31 +0000
@@ -258,7 +258,7 @@
   library is not an error; the associated image format will simply be
   unavailable.  Note that once Emacs has determined that a library can
   not be found, there's no way to force it to try again, other than
-  restarting.  See the variable `image-library-alist' to configure the
+  restarting.  See the variable `dynamic-library-alist' to configure the
   expected names of the libraries.

   Some image libraries have dependencies on one another, or on zlib.

=== modified file 'src/emacs.c'
--- src/emacs.c	2010-10-03 13:59:56 +0000
+++ src/emacs.c	2010-10-05 02:37:56 +0000
@@ -187,6 +187,9 @@
    Lisp code.  */
 Lisp_Object Vemacs_copyright, Vemacs_version;

+/* Alist of external libraries and files implementing them.  */
+Lisp_Object Vdynamic_library_alist;
+
 /* If non-zero, emacs should not attempt to use a window-specific code,
    but instead should use the virtual terminal under which it was started.  */
 int inhibit_window_system;
@@ -2507,6 +2510,24 @@
 	       doc: /* Version numbers of this version of Emacs.  */);
   Vemacs_version = build_string (emacs_version);

+  DEFVAR_LISP ("dynamic-library-alist", &Vdynamic_library_alist,
+    doc: /* Alist of dynamic libraries vs external files implementing them.
+Each element is a list (LIBRARY FILE...), where the car is a symbol
+representing a supported external library, and the rest are strings giving
+alternate filenames for that library.
+
+Emacs tries to load the library from the files in the orden they appear on
+the list; if none is loaded, the running session of Emacs won't have access
+to that library.
+
+Note that image types `pbm' and `xbm' do not need entries in this variable
+because they do not depend on external libraries and are always available.
+
+Also note that this is not a generic facility for accessing external
+libraries; only those already known by Emacs will be loaded.  */);
+  Vdynamic_library_alist = Qnil;
+  Fput (intern_c_string ("dynamic-library-alist"), Qrisky_local_variable, Qt);
+
   /* Make sure IS_DAEMON starts up as false.  */
   daemon_pipe[1] = 0;
 }

=== modified file 'src/image.c'
--- src/image.c	2010-10-01 12:25:21 +0000
+++ src/image.c	2010-10-05 02:24:30 +0000
@@ -567,10 +567,6 @@

 Lisp_Object Vimage_types;

-/* An alist of image types and libraries that implement the type.  */
-
-Lisp_Object Vimage_library_alist;
-
 /* Cache for delayed-loading image types.  */

 static Lisp_Object Vimage_type_cache;
@@ -645,7 +641,7 @@
   struct image_type *type;

   /* We must initialize the image-type if it hasn't been already.  */
-  if (NILP (Finit_image_library (symbol, Vimage_library_alist)))
+  if (NILP (Finit_image_library (symbol, Vdynamic_library_alist)))
     return 0;			/* unimplemented */

   for (type = image_types; type; type = type->next)
@@ -1923,10 +1919,11 @@
   }

 /* Load a DLL implementing an image type.
-   The `image-library-alist' variable associates a symbol,
-   identifying an image type, to a list of possible filenames.
+   The argument LIBRARIES is usually the variable
+   `dynamic-library-alist', which associates a symbol, identifying
+   an external DLL library, to a list of possible filenames.
    The function returns NULL if no library could be loaded for
-   the given image type, or if the library was previously loaded;
+   the given symbol, or if the library was previously loaded;
    else the handle of the DLL.  */
 static HMODULE
 w32_delayed_load (Lisp_Object libraries, Lisp_Object type)
@@ -8583,7 +8580,7 @@

 Image types pbm and xbm are prebuilt; other types are loaded here.
 Libraries to load are specified in alist LIBRARIES (usually, the value
-of `image-library-alist', which see).  */)
+of `dynamic-library-alist', which see).  */)
   (Lisp_Object type, Lisp_Object libraries)
 {
   Lisp_Object tested;
@@ -8659,20 +8656,6 @@
 To check whether it is really supported, use `image-type-available-p'.  */);
   Vimage_types = Qnil;

-  DEFVAR_LISP ("image-library-alist", &Vimage_library_alist,
-    doc: /* Alist of image types vs external libraries needed to display them.
-
-Each element is a list (IMAGE-TYPE LIBRARY...), where the car is a symbol
-representing a supported image type, and the rest are strings giving
-alternate filenames for the corresponding external libraries.
-
-Emacs tries to load the libraries in the order they appear on the
-list; if none is loaded, the running session of Emacs won't
-support the image type.  Types 'pbm and 'xbm don't need to be
-listed; they are always supported.  */);
-  Vimage_library_alist = Qnil;
-  Fput (intern_c_string ("image-library-alist"), Qrisky_local_variable, Qt);
-
   DEFVAR_LISP ("max-image-size", &Vmax_image_size,
     doc: /* Maximum size of images.
 Emacs will not load an image into memory if its pixel width or

=== modified file 'src/lisp.h'
--- src/lisp.h	2010-09-29 09:06:53 +0000
+++ src/lisp.h	2010-10-05 02:24:06 +0000
@@ -3300,6 +3300,7 @@
 extern Lisp_Object Vinstallation_directory;
 extern Lisp_Object empty_unibyte_string, empty_multibyte_string;
 extern Lisp_Object Qfile_name_handler_alist;
+extern Lisp_Object Vdynamic_library_alist;
 extern void (*fatal_error_signal_hook) (void);
 EXFUN (Fkill_emacs, 1) NO_RETURN;
 #if HAVE_SETLOCALE

Re: Adding GnuTLS to the Windows port

[Christoph]

On 10/4/2010 10:00 PM, Juanma Barranquero wrote:

Just a few spelling things...

> +Emacs tries to load the library from the files in the orden they
> +appear on the list;

Should be `order'.

> +(setq dynamic-library-alist
>         '((xpm "libxpm.dll" "xpm4.dll" "libXpm-nox4.dll")
>           (png "libpng12d.dll" "libpng12.dll" "libpng.dll"
>   	 ;; these are libpng 1.2.8 from GTK+

I am not sure if case matters, but I think the XPM library name supplied 
with the binary distribution is actualy `libXpm.dll'.

> +Emacs tries to load the library from the files in the orden they appear on
> +the list; if none is loaded, the running session of Emacs won't have access
> +to that library.

Again, should be `order'.

Christoph

Re: Adding GnuTLS to the Windows port

[Juanma Barranquero]

On Tue, Oct 5, 2010 at 06:16, Christoph <cschol2112@googlemail.com> wrote:

> Just a few spelling things...

Thanks.

> Should be `order'.

Not in Spanish ;-)

>> +(setq dynamic-library-alist
>>        '((xpm "libxpm.dll" "xpm4.dll" "libXpm-nox4.dll")
>>          (png "libpng12d.dll" "libpng12.dll" "libpng.dll"
>>         ;; these are libpng 1.2.8 from GTK+
>
> I am not sure if case matters, but I think the XPM library name supplied
> with the binary distribution is actualy `libXpm.dll'.

No, case does not matter. Note that I'm not changing the library
names, they have been like so for a while.

> Again, should be `order'.

I'm surprised there's so little Spanish in that docstring. Thanks,
I'll fix them.

    Juanma

Re: Adding GnuTLS to the Windows port

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Tue, 5 Oct 2010 06:00:10 +0200
> 
> Patch attached.

Thanks.

> Though the description of dynamic-library-alist, currently at
> "(elisp)Image Formats" should be moved... where?

How about os.texi, e.g. in the "Startup" node?

A few comments regarding the patch:

> +Each element is a list @code{(@var{library} @var{file}...)}, where the
                                                         ^^^
@dots instead of a literal "...".

> +car is a symbol representing a supported external library, and the
   ^^^
We use either @code{car} or @sc{car}.

> +Emacs tries to load the library from the files in the orden they
> +appear on the list; if none is loaded, the running session of Emacs
                       ^^^^^^^^^^^^^^^^^
"if none is found" sounds better.

> +won't have access to that library.

"...and the features that depend on that library will be unavailable."

> +** `image-library-alist' is renamed to `dynamic-library-alist'.
> +The variable is now used to load all kind of supported dynamic libraries,
> +not just image libraries.

Please document here the fact that `image-library-alist' is still
available as an obsolete alias.

Re: Adding GnuTLS to the Windows port

[Juanma Barranquero]

On Tue, Oct 5, 2010 at 13:46, Eli Zaretskii <eliz@gnu.org> wrote:

>> Though the description of dynamic-library-alist, currently at
>> "(elisp)Image Formats" should be moved... where?
>
> How about os.texi, e.g. in the "Startup" node?

AFAICS, there's a Starting Up node, which is a menu, and then

  Startup Summary
  Init File
  Terminal-Specific
  Command-Line Arguments

None of them seems entirely relevant.

> @dots instead of a literal "...".

OK, though there are hundred of instances of literal "..." on doc/**/*.texi

> We use either @code{car} or @sc{car}.
> "if none is found" sounds better.
> "...and the features that depend on that library will be unavailable."

Applied, thanks.

> Please document here the fact that `image-library-alist' is still
> available as an obsolete alias.

Done.

Now, if Stefan and Chong would speak...

    Juanma

Re: Adding GnuTLS to the Windows port

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Wed, 6 Oct 2010 12:30:21 +0200
> Cc: emacs-devel@gnu.org
> 
> On Tue, Oct 5, 2010 at 13:46, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >> Though the description of dynamic-library-alist, currently at
> >> "(elisp)Image Formats" should be moved... where?
> >
> > How about os.texi, e.g. in the "Startup" node?
> 
> AFAICS, there's a Starting Up node, which is a menu, and then
> 
>   Startup Summary
>   Init File
>   Terminal-Specific
>   Command-Line Arguments
> 
> None of them seems entirely relevant.

I meant "Startup Summary", but now that I think of it again, perhaps
an entirely new node, on the same level as "Starting Up", would be a
better idea.  Something like "Dynamic Libraries".

> > @dots instead of a literal "...".
> 
> OK, though there are hundred of instances of literal "..." on doc/**/*.texi

Corrections are welcome.  @dots{} and @enddots{} look much better in
print.

Re: Adding GnuTLS to the Windows port

[Stefan Monnier]

>> Please document here the fact that `image-library-alist' is still
>> available as an obsolete alias.

> Now, if Stefan and Chong would speak...

I'm not sure I understand the issue well enough to say much.  The only
point I care about, really, is that such a change be made keeping in
mind the likely future addition of a more general support for
dynamic loading.


        Stefan

Re: Adding GnuTLS to the Windows port

[Juanma Barranquero]

On Thu, Oct 7, 2010 at 09:52, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> I'm not sure I understand the issue well enough to say much.

The issue is quite simple:

- On the Windows port we compile support to use several libraries as
DLLs, but these libraries aren't linked to the executable (i.e.,
loading Emacs does not load them) but dynamically at first use, by
determining whether the library (the file containing it) can be found
and loaded; one reason being that our "official" binary must be able
to work whether they are available or not.

- Originally we hardcoded the filenames of the libraries, checking
each name in turn; for example, for xpm, we tried "libxmp.dll", then
"xmp4.dll", etc.

- At some point, we added an alist to associate libraries with their
candidate filenames, to make it cleaner and also user-configurable.

- For historical reasons (as all of these libraries were for image
support) the alist variable was called `image-library-alist'.

And now the issue is that it is quite likely that we'll end using the
same variable for other kinds of libraries, like libxml2, GnuTLS, and
perhaps D-BUS or others. So `image-library-alist' is no longer a good
name. All my patch does is renaming it to `dynamic-library-alist' and
keeping the old name as an obsolete alias.

Note, BTW, that all the above comes from the Windows port, but there's
nothing Windows-specific in (image|dynamic)-library-alist. Any port
that does delayed dynamic loading can benefit from it.

> The only
> point I care about, really, is that such a change be made keeping in
> mind the likely future addition of a more general support for
> dynamic loading.

I cannot comment on that, as I know nothing about the future "general
support for dynamic loading", other than the legal issues that were
commented here a while ago. Care to elaborate (or point to some old
thread I've missed)?

    Juanma

Re: Adding GnuTLS to the Windows port

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Thu, 7 Oct 2010 12:09:43 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> > The only
> > point I care about, really, is that such a change be made keeping in
> > mind the likely future addition of a more general support for
> > dynamic loading.
> 
> I cannot comment on that, as I know nothing about the future "general
> support for dynamic loading", other than the legal issues that were
> commented here a while ago.

I don't see how the machinery in question can get in the way of the
more general support for dynamic loading.  Having an alist of
libraries indexed by the features which need them is something we will
need to keep if we want true dynamic loading.

Re: Adding GnuTLS to the Windows port

[Stefan Monnier]

>> > The only point I care about, really, is that such a change be made
>> > keeping in mind the likely future addition of a more general
>> > support for dynamic loading.
>> 
>> I cannot comment on that, as I know nothing about the future "general
>> support for dynamic loading", other than the legal issues that were
>> commented here a while ago.

> I don't see how the machinery in question can get in the way of the
> more general support for dynamic loading.  Having an alist of
> libraries indexed by the features which need them is something we will
> need to keep if we want true dynamic loading.

Good, thanks.  Then please go ahead,


        Stefan