bug#8761: gnutls should use xmalloc not malloc

bug#8761: gnutls should use xmalloc not malloc

[Paul Eggert]

Here's a proposed patch for the problem.

* gnutls.c: Use Emacs's memory allocators.
Without this change, the gnutls library would invoke malloc etc.
directly, which causes problems on non-SYNC_INPUT hosts, and which
runs afoul of improving memory_full behavior.
(fn_gnutls_global_set_mem_functions): New macro or function pointer.
(emacs_gnutls_global_init): Use it to specify xmalloc, xrealloc,
xfree instead of the default malloc, realloc, free.
(Fgnutls_boot): No need to check for memory allocation failure,
since xmalloc does that for us.
=== modified file 'src/gnutls.c'
--- src/gnutls.c	2011-05-23 00:03:40 +0000
+++ src/gnutls.c	2011-05-30 07:53:40 +0000
@@ -110,6 +110,10 @@
 DEF_GNUTLS_FN (int, gnutls_global_init, (void));
 DEF_GNUTLS_FN (void, gnutls_global_set_log_function, (gnutls_log_func));
 DEF_GNUTLS_FN (void, gnutls_global_set_log_level, (int));
+DEF_GNUTLS_FN (void, gnutls_global_set_mem_functions,
+	       (gnutls_alloc_function, gnutls_alloc_function,
+		gnutls_is_secure_function, gnutls_realloc_function,
+		gnutls_free_function));
 DEF_GNUTLS_FN (int, gnutls_handshake, (gnutls_session_t));
 DEF_GNUTLS_FN (int, gnutls_init, (gnutls_session_t *, gnutls_connection_end_t));
 DEF_GNUTLS_FN (int, gnutls_priority_set_direct,
@@ -168,6 +172,7 @@
   LOAD_GNUTLS_FN (library, gnutls_global_init);
   LOAD_GNUTLS_FN (library, gnutls_global_set_log_function);
   LOAD_GNUTLS_FN (library, gnutls_global_set_log_level);
+  LOAD_GNUTLS_FN (library, gnutls_global_set_mem_functions);
   LOAD_GNUTLS_FN (library, gnutls_handshake);
   LOAD_GNUTLS_FN (library, gnutls_init);
   LOAD_GNUTLS_FN (library, gnutls_priority_set_direct);
@@ -213,6 +218,7 @@
 #define fn_gnutls_global_init			gnutls_global_init
 #define fn_gnutls_global_set_log_function	gnutls_global_set_log_function
 #define fn_gnutls_global_set_log_level		gnutls_global_set_log_level
+#define fn_gnutls_global_set_mem_functions	gnutls_global_set_mem_functions
 #define fn_gnutls_handshake			gnutls_handshake
 #define fn_gnutls_init				gnutls_init
 #define fn_gnutls_priority_set_direct		gnutls_priority_set_direct
@@ -582,7 +588,11 @@
   int ret = GNUTLS_E_SUCCESS;

   if (!gnutls_global_initialized)
-    ret = fn_gnutls_global_init ();
+    {
+      fn_gnutls_global_set_mem_functions (xmalloc, xmalloc, NULL,
+					  xrealloc, xfree);
+      ret = fn_gnutls_global_init ();
+    }
   gnutls_global_initialized = 1;

   return gnutls_make_error (ret);
@@ -768,8 +778,7 @@
     {
       GNUTLS_LOG (2, max_log_level, "allocating x509 credentials");
       x509_cred = XPROCESS (proc)->gnutls_x509_cred;
-      if (fn_gnutls_certificate_allocate_credentials (&x509_cred) < 0)
-        memory_full ();
+      fn_gnutls_certificate_allocate_credentials (&x509_cred);

       if (NUMBERP (verify_flags))
         {
@@ -792,8 +801,7 @@
     {
       GNUTLS_LOG (2, max_log_level, "allocating anon credentials");
       anon_cred = XPROCESS (proc)->gnutls_anon_cred;
-      if (fn_gnutls_anon_allocate_client_credentials (&anon_cred) < 0)
-        memory_full ();
+      fn_gnutls_anon_allocate_client_credentials (&anon_cred);
     }
   else
     {

bug#8761: gnutls should use xmalloc not malloc

[Ted Zlatanov]

On Mon, 30 May 2011 01:35:26 -0700 Paul Eggert <eggert@cs.ucla.edu> wrote: 

PE> Here's a proposed patch for the problem.
PE> * gnutls.c: Use Emacs's memory allocators.
PE> Without this change, the gnutls library would invoke malloc etc.
PE> directly, which causes problems on non-SYNC_INPUT hosts, and which
PE> runs afoul of improving memory_full behavior.
PE> (fn_gnutls_global_set_mem_functions): New macro or function pointer.
PE> (emacs_gnutls_global_init): Use it to specify xmalloc, xrealloc,
PE> xfree instead of the default malloc, realloc, free.
PE> (Fgnutls_boot): No need to check for memory allocation failure,
PE> since xmalloc does that for us.

This looks good.

PE> -      if (fn_gnutls_certificate_allocate_credentials (&x509_cred) < 0)
PE> -        memory_full ();
PE> +      fn_gnutls_certificate_allocate_credentials (&x509_cred);

PE> -      if (fn_gnutls_anon_allocate_client_credentials (&anon_cred) < 0)
PE> -        memory_full ();
PE> +      fn_gnutls_anon_allocate_client_credentials (&anon_cred);

I'm OK with the change, but how do I test that it behaves properly?

Ted

bug#8761: gnutls should use xmalloc not malloc

[Paul Eggert]

> I'm OK with the change, but how do I test that it behaves properly?

It's a bit hard to arrange for a test, since the thing you want
to try is to run gnutls when malloc fails.  Perhaps you can run
Emacs inside a debugger, and artificially arrange for malloc to
return NULL at exactly the right point.  But to be honest, I expect
that the old memory-failure code was never exercised, and that it's
low priority to test the new memory-failure code.