bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

In the thread around

http://permalink.gmane.org/gmane.emacs.devel/182469

Rich Felker announced that he had found that a small addition to
unexelf.c allowed emacs to extend its hybrid malloc implementation
(previously only used for cygwin) to musl libc, but without using
features specific to that malloc implementation.

A corresponding patch was tested in the "Alpine Linux" distribution, see

http://git.alpinelinux.org/cgit/aports/plain/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6

This patch series proposes the relevant part of that change and adds
some (rather trivial) groundwork to make it reliably work on all systems
which use unexelf.c for dumping.

Personally, I've been using hybrid malloc on top of the libc malloc
(jemalloc) on FreeBSD 10 for several months now without problems.

I also tested it with other malloc libraries, in particular some of
those derived from Doug Lea's malloc.  It also works on GNU/Linux, on
top of glibc.  This might be of some interest as, IIRC, some glibc
people were not too happy about keeping malloc_get_state and stuff.

If you get an error about exceeding "static heap size" during dumping
just bump STATIC_HEAP_SIZE in src/sheap.c a bit.

If the commit messages in the following patches don't explain enough,
please ask.  Also, please apply the patch from

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22085

beforehand.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Keep gmalloc symbols internal --]
[-- Type: text/x-diff, Size: 7149 bytes --]

From 0ee89d5aa9eae1bf72e5cc715d8b2e3afce37703 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Sun, 29 Nov 2015 17:39:56 +0100
Subject: [PATCH 2/5] For HYBRID_MALLOC, give most gmalloc symbols internal
 linkage.

This avoids clashes with symbols if the after-dump malloc is
derived from Doug Lea's implementation.

* src/gmalloc.c (emacs_abort, __morecore, __default_morecore):
Move declarations up.  For HYBRID_MALLOC, turn all `extern'
declarations below to `static' ones.
(aligned_alloc): Declare for !MSDOS as well.
(heapsize, _fraghead): Move resp. copy declaration downwards.
For HYBRID_MALLOC, conditionalize out the other definitions,
since the previous `static' declarations double as tentative
definitions, anyway.
(_malloc, _free, _realloc, __free_hook, _aligned_blocks)
(__realloc_hook, __memalign_hook): Conditionalize out.
(cfree, memalign, valloc): Ditto.
---
 src/gmalloc.c | 58 +++++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 41 insertions(+), 17 deletions(-)

diff --git a/src/gmalloc.c b/src/gmalloc.c
index 90a52a1..240408e 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -87,6 +87,20 @@ extern "C"
 
 #include <stddef.h>
 
+#ifdef emacs
+extern void emacs_abort (void);
+#endif
+
+/* Underlying allocation function; successive calls should
+   return contiguous pieces of memory.  */
+extern void *(*__morecore) (ptrdiff_t size);
+
+/* Default value of `__morecore'.  */
+extern void *__default_morecore (ptrdiff_t size);
+
+#ifdef HYBRID_MALLOC
+#define extern static
+#endif
 
 /* Allocate SIZE bytes of memory.  */
 extern void *malloc (size_t size) ATTRIBUTE_MALLOC_SIZE ((1));
@@ -99,8 +113,8 @@ extern void *calloc (size_t nmemb, size_t size) ATTRIBUTE_MALLOC_SIZE ((1,2));
 extern void free (void *ptr);
 
 /* Allocate SIZE bytes allocated to ALIGNMENT bytes.  */
-#ifdef MSDOS
 extern void *aligned_alloc (size_t, size_t);
+#ifdef MSDOS
 extern void *memalign (size_t, size_t);
 extern int posix_memalign (void **, size_t, size_t);
 #endif
@@ -110,10 +124,6 @@ extern int posix_memalign (void **, size_t, size_t);
 extern void malloc_enable_thread (void);
 #endif
 
-#ifdef emacs
-extern void emacs_abort (void);
-#endif
-
 /* The allocator divides the heap into blocks of fixed size; large
    requests receive one or more whole blocks, and small requests
    receive a fragment of a block.  Fragment sizes are powers of two,
@@ -249,13 +259,6 @@ extern int _malloc_thread_enabled_p;
    return the address of the beginning of the object.  */
 extern void *malloc_find_object_address (void *ptr);
 
-/* Underlying allocation function; successive calls should
-   return contiguous pieces of memory.  */
-extern void *(*__morecore) (ptrdiff_t size);
-
-/* Default value of `__morecore'.  */
-extern void *__default_morecore (ptrdiff_t size);
-
 /* If not NULL, this function is called after each time
    `__morecore' is called to increase the data size.  */
 extern void (*__after_morecore_hook) (void);
@@ -318,6 +321,8 @@ extern struct mstats mstats (void);
 /* Call WARNFUN with a warning message when memory usage is high.  */
 extern void memory_warnings (void *start, void (*warnfun) (const char *));
 
+#undef extern
+
 #ifdef	__cplusplus
 }
 #endif
@@ -346,6 +351,8 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 
 void *(*__morecore) (ptrdiff_t size) = __default_morecore;
 
+#ifndef HYBRID_MALLOC
+
 /* Debugging hook for `malloc'.  */
 void *(*__malloc_hook) (size_t size);
 
@@ -355,9 +362,6 @@ char *_heapbase;
 /* Block information table.  Allocated with align/__free (not malloc/free).  */
 malloc_info *_heapinfo;
 
-/* Number of info entries.  */
-static size_t heapsize;
-
 /* Search index in the info table.  */
 size_t _heapindex;
 
@@ -381,6 +385,15 @@ size_t __malloc_extra_blocks;
 void (*__malloc_initialize_hook) (void);
 void (*__after_morecore_hook) (void);
 
+#else
+
+static struct list _fraghead[BLOCKLOG];
+
+#endif /* HYBRID_MALLOC */
+
+/* Number of info entries.  */
+static size_t heapsize;
+
 #if defined GC_MALLOC_CHECK && defined GC_PROTECT_MALLOC_STATE
 
 /* Some code for hunting a bug writing into _heapinfo.
@@ -946,7 +959,7 @@ malloc (size_t size)
   return (hook != NULL ? *hook : _malloc_internal) (size);
 }
 \f
-#ifndef _LIBC
+#if !(defined (_LIBC) || defined (HYBRID_MALLOC))
 
 /* On some ANSI C systems, some libc functions call _malloc, _free
    and _realloc.  Make them use the GNU functions.  */
@@ -995,11 +1008,13 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
    or (US mail) as Mike Haertel c/o Free Software Foundation.  */
 
 
+#ifndef HYBRID_MALLOC
 /* Debugging hook for free.  */
 void (*__free_hook) (void *__ptr);
 
 /* List of blocks allocated by aligned_alloc.  */
 struct alignlist *_aligned_blocks = NULL;
+#endif
 
 /* Return memory to the heap.
    Like `_free_internal' but don't lock mutex.  */
@@ -1270,6 +1285,7 @@ free (void *ptr)
     _free_internal (ptr);
 }
 
+#ifndef HYBRID_MALLOC
 /* Define the `cfree' alias for `free'.  */
 #ifdef weak_alias
 weak_alias (free, cfree)
@@ -1280,6 +1296,7 @@ cfree (void *ptr)
   free (ptr);
 }
 #endif
+#endif
 /* Change the size of a block allocated by `malloc'.
    Copyright 1990, 1991, 1992, 1993, 1994, 1995 Free Software Foundation, Inc.
 		     Written May 1989 by Mike Haertel.
@@ -1304,8 +1321,10 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #define min(a, b) ((a) < (b) ? (a) : (b))
 #endif
 
+#ifndef HYBRID_MALLOC
 /* Debugging hook for realloc.  */
 void *(*__realloc_hook) (void *ptr, size_t size);
+#endif
 
 /* Resize the given region to the new size, returning a pointer
    to the (possibly moved) region.  This is optimized for speed;
@@ -1549,7 +1568,9 @@ General Public License for more details.
 You should have received a copy of the GNU General Public
 License along with this library.  If not, see <http://www.gnu.org/licenses/>.  */
 
+#ifndef HYBRID_MALLOC
 void *(*__memalign_hook) (size_t size, size_t alignment);
+#endif
 
 void *
 aligned_alloc (size_t alignment, size_t size)
@@ -1638,6 +1659,8 @@ aligned_alloc (size_t alignment, size_t size)
   return result;
 }
 
+/* Note that memalign and posix_memalign are not used in Emacs.  */
+#ifndef HYBRID_MALLOC
 /* An obsolete alias for aligned_alloc, for any old libraries that use
    this alias.  */
 
@@ -1649,7 +1672,6 @@ memalign (size_t alignment, size_t size)
 
 /* If HYBRID_MALLOC is defined, we may want to use the system
    posix_memalign below.  */
-#ifndef HYBRID_MALLOC
 int
 posix_memalign (void **memptr, size_t alignment, size_t size)
 {
@@ -1689,6 +1711,7 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
    The author may be reached (Email) at the address mike@ai.mit.edu,
    or (US mail) as Mike Haertel c/o Free Software Foundation.  */
 
+#ifndef HYBRID_MALLOC
 /* Allocate SIZE bytes on a page boundary.  */
 extern void *valloc (size_t);
 
@@ -1708,6 +1731,7 @@ valloc (size_t size)
 
   return aligned_alloc (pagesize, size);
 }
+#endif /* HYBRID_MALLOC */
 
 #undef malloc
 #undef realloc
-- 
2.6.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Compile separate gnulib.a for emacs with -Demacs --]
[-- Type: text/x-diff, Size: 2161 bytes --]

From 9610f313480b8e0da810c38aee0fbc07c88b0904 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Sun, 22 Feb 2015 04:15:34 +0100
Subject: [PATCH 3/5] Link temacs with gnulib objects compiled with -Demacs.

* lib/Makefile.am: Add incantation to install libegnu.a.
* src/Makefile.in ($(lib)/libgnu.a): Replace with libegnu.a
(temacs$(EXEEXT)): Use it.

This is done to support HYBRID_MALLOC, since some static variables
(e.g., last_environ in putenv.c) hold pointers to memory malloced
before dumping.
---
 lib/Makefile.am | 9 +++++++++
 src/Makefile.in | 6 +++---
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/lib/Makefile.am b/lib/Makefile.am
index cda9681..a1dd6a4 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -17,6 +17,15 @@ include gnulib.mk
 libgnu_a_SOURCES += openat-die.c save-cwd.c
 endif
 
+noinst_LIBRARIES += libegnu.a
+
+libegnu_a_SOURCES = $(libgnu_a_SOURCES)
+libegnu_a_LIBADD = $(patsubst %.o,e-%.o,$(libgnu_a_LIBADD))
+EXTRA_libegnu_a_SOURCES = $(EXTRA_libgnu_a_SOURCES)
+libegnu_a_SHORTNAME = e
+
+libegnu_a_CPPFLAGS = $(AM_CPPFLAGS) -Demacs
+
 .PHONY: bootstrap-clean
 
 bootstrap-clean: maintainer-clean
diff --git a/src/Makefile.in b/src/Makefile.in
index d7ad395..f1d780f 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -557,7 +557,7 @@ globals.h: gl-stamp; @true
 
 $(ALLOBJS): globals.h
 
-$(lib)/libgnu.a: $(config_h)
+$(lib)/libegnu.a: $(config_h)
 	$(MAKE) -C $(lib) all
 
 ## We have to create $(etc) here because init_cmdargs tests its
@@ -565,9 +565,9 @@ $(lib)/libgnu.a: $(config_h)
 ## This goes on to affect various things, and the emacs binary fails
 ## to start if Vinstallation_directory has the wrong value.
 temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) \
-	         $(lib)/libgnu.a $(EMACSRES) ${charsets} ${charscript}
+	         $(lib)/libegnu.a $(EMACSRES) ${charsets} ${charscript}
 	$(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
-	  -o temacs $(ALLOBJS) $(lib)/libgnu.a $(W32_RES_LINK) $(LIBES)
+	  -o temacs $(ALLOBJS) $(lib)/libegnu.a $(W32_RES_LINK) $(LIBES)
 	$(MKDIR_P) $(etc)
 ifneq ($(CANNOT_DUMP),yes)
 	$(PAXCTL_if_present) -r $@
-- 
2.6.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: The patch from musl, essentially --]
[-- Type: text/x-diff, Size: 4184 bytes --]

From f65fff5bb5f2f2eee3b258538f1abb70eeb911c6 Mon Sep 17 00:00:00 2001
From: Rich Felker <dalias@libc.org>
Date: Sun, 22 Feb 2015 04:25:03 +0100
Subject: [PATCH 4/5] Add musl patch to support HYBRID_MALLOC on elf systems.

* src/unexelf.c (unexec): Define bss_sbrk_did_unexec.
* src/gmalloc.c: Adjust for HYBRID_MALLOC in the non CYGWIN case.
(__default_morecore): Here, in particular.

* configure.ac: Define HYBRID_MALLOC when unexelf.o is used.
New variable SHEAP_OBJ.
* src/Makefile.in: Use it.

Except for build system fixes this is essentially the same as

http://git.alpinelinux.org/cgit/aports/tree/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6

The original patch contains one unrelated hunk, which will be
committed separately.
---
 configure.ac    | 12 ++++++++++--
 src/Makefile.in |  6 +++++-
 src/gmalloc.c   |  4 ++--
 src/unexelf.c   |  5 +++++
 4 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/configure.ac b/configure.ac
index 63e354e..56e2f12 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2117,7 +2117,13 @@ case "$opsys" in
   cygwin) hybrid_malloc=yes;;
 esac
 
+if test "${system_malloc}" != yes && test "${doug_lea_malloc}" != yes \
+   && test "${UNEXEC_OBJ}" = unexelf.o && test "${hybrid_malloc}" != no; then
+  hybrid_malloc=yes
+fi
+
 GMALLOC_OBJ=
+SHEAP_OBJ=
 if test "${system_malloc}" = "yes"; then
   AC_DEFINE([SYSTEM_MALLOC], 1,
     [Define to 1 to use the system memory allocator, even if it is not
@@ -2132,6 +2138,7 @@ elif test "$hybrid_malloc" = yes; then
   GNU_MALLOC=
   GNU_MALLOC_reason="only before dumping"
   GMALLOC_OBJ=gmalloc.o
+  SHEAP_OBJ=sheap.o
   VMLIMIT_OBJ=
 else
   test "$doug_lea_malloc" != "yes" && GMALLOC_OBJ=gmalloc.o
@@ -2152,9 +2159,10 @@ else
   fi
 fi
 AC_SUBST(GMALLOC_OBJ)
+AC_SUBST(SHEAP_OBJ)
 AC_SUBST(VMLIMIT_OBJ)
 
-if test "$doug_lea_malloc" = "yes" ; then
+if test "$doug_lea_malloc" = "yes" && test "$hybrid_malloc" != yes; then
   if test "$GNU_MALLOC" = yes ; then
     GNU_MALLOC_reason="
       (Using Doug Lea's new malloc from the GNU C Library.)"
@@ -4970,7 +4978,7 @@ fi
 AC_SUBST(RALLOC_OBJ)
 
 if test "$opsys" = "cygwin"; then
-  CYGWIN_OBJ="sheap.o cygw32.o"
+  CYGWIN_OBJ="cygw32.o"
   ## Cygwin differs because of its unexec().
   PRE_ALLOC_OBJ=
   POST_ALLOC_OBJ=lastfile.o
diff --git a/src/Makefile.in b/src/Makefile.in
index f1d780f..ece8858 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -243,7 +243,10 @@ XFIXES_CFLAGS = @XFIXES_CFLAGS@
 ## widget.o if USE_X_TOOLKIT, otherwise empty.
 WIDGET_OBJ=@WIDGET_OBJ@
 
-## sheap.o if CYGWIN, otherwise empty.
+## sheap.o if HYBRID_MALLOC, otherwise empty.
+SHEAP_OBJ=@SHEAP_OBJ@
+
+## cygw32.o if CYGWIN, otherwise empty.
 CYGWIN_OBJ=@CYGWIN_OBJ@
 
 ## fontset.o fringe.o image.o if we have any window system
@@ -384,6 +387,7 @@ base_obj = dispnew.o frame.o scroll.o xdisp.o menu.o $(XMENU_OBJ) window.o \
 	region-cache.o sound.o atimer.o \
 	doprnt.o intervals.o textprop.o composite.o xml.o $(NOTIFY_OBJ) \
 	profiler.o decompress.o \
+	$(SHEAP_OBJ) \
 	$(MSDOS_OBJ) $(MSDOS_X_OBJ) $(NS_OBJ) $(CYGWIN_OBJ) $(FONT_OBJ) \
 	$(W32_OBJ) $(WINDOW_SYSTEM_OBJ) $(XGSELOBJ)
 obj = $(base_obj) $(NS_OBJC_OBJ)
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 240408e..530b3b2 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -70,7 +70,7 @@ extern void emacs_abort (void);
 #define aligned_alloc galigned_alloc
 #define free gfree
 
-#ifdef CYGWIN
+#ifdef HYBRID_MALLOC
 extern void *bss_sbrk (ptrdiff_t size);
 extern int bss_sbrk_did_unexec;
 extern char bss_sbrk_buffer[];
@@ -1542,7 +1542,7 @@ void *
 __default_morecore (ptrdiff_t increment)
 {
   void *result;
-#if defined (CYGWIN)
+#ifdef HYBRID_MALLOC
   if (!DUMPED)
     {
       return bss_sbrk (increment);
diff --git a/src/unexelf.c b/src/unexelf.c
index c10c7f2..3d10aa0 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -227,6 +227,11 @@ unexec (const char *new_name, const char *old_name)
   off_t new_file_size;
   void *new_break;
 
+#ifdef HYBRID_MALLOC
+  extern int bss_sbrk_did_unexec;
+  bss_sbrk_did_unexec = 1;
+#endif
+
   /* Pointers to the base of the image of the two files.  */
   caddr_t old_base, new_base;
 
-- 
2.6.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: No special case for FreeBSD wrt mmap --]
[-- Type: text/x-diff, Size: 634 bytes --]

From d042d56201d1ab00fcc405976954d0d0585c70bd Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Sun, 22 Feb 2015 04:26:36 +0100
Subject: [PATCH 5/5] * configure.ac: Stop using mmap for buffers for FreeBSD.

---
 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 56e2f12..af8de47 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2186,7 +2186,7 @@ fi
 
 use_mmap_for_buffers=no
 case "$opsys" in
-  cygwin|mingw32|freebsd|irix6-5) use_mmap_for_buffers=yes ;;
+  cygwin|mingw32|irix6-5) use_mmap_for_buffers=yes ;;
 esac
 
 AC_FUNC_MMAP
-- 
2.6.3

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Those patches look reasonable, except that one is listed as being by Rich 
Felker, who has not signed copyright papers for GNU Emacs as far as I know. I'll 
CC: this to him to give him a heads-up. Rich, would you mind transferring 
copyright to that patch to the FSF? Here's a link:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Wed, Dec 16, 2015 at 12:28:48AM -0800, Paul Eggert wrote:
> Those patches look reasonable, except that one is listed as being by
> Rich Felker, who has not signed copyright papers for GNU Emacs as
> far as I know. I'll CC: this to him to give him a heads-up. Rich,
> would you mind transferring copyright to that patch to the FSF?
> Here's a link:
> 
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch

I have no objection, but I figured it was trivial enough not to need
any explicit assignment. Who do I need to contact about doing the
assignment papers?

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Eli Zaretskii]

> Date: Wed, 16 Dec 2015 12:15:42 -0500
> From: Rich Felker <dalias@aerifal.cx>
> Cc: Wolfgang Jenkner <wjenkner@inode.at>, 22086@debbugs.gnu.org
> 
> Who do I need to contact about doing the assignment papers?

Us.  I can send you the form with instructions (off-list) if you are
willing to do this.

Thanks.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

On Wed, Dec 16 2015, Paul Eggert wrote:

> Those patches look reasonable, except that one is listed as being by
> Rich Felker, who has not signed copyright papers for GNU Emacs as far
> as I know. I'll CC: this to him to give him a heads-up. Rich, would
> you mind transferring copyright to that patch to the FSF? Here's
> a link:
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch


Rich Felker has asked me in private mail (please, please CC your replies
to 22086@debbugs.gnu.org) to clarify the authorship of that patch.
I actually stated in the commit message

    Except for build system fixes this is essentially the same as

    http://git.alpinelinux.org/cgit/aports/tree/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6

So, I wrote the build system fixes, and, strictly speaking, only the two
lines in src/unexelf.c which declare and initialize bss_sbrk_did_unexec
are directly from the original "Alpine Linux" patch above.

I can split the patch accordingly and ascribe only the latter part to
Rich Felker if this is preferred.  The original patch still helped me to
quickly find where I had to change something; this is why a attributed
the whole patch to Rich Felker as it is just an expansion, as it were,
of his version.

Wolfgang

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Thu, 17 Dec 2015 14:16:41 +0100
> Cc: Rich Felker <dalias@aerifal.cx>, 22086@debbugs.gnu.org
> 
> I can split the patch accordingly and ascribe only the latter part to
> Rich Felker if this is preferred.

If this is not too much trouble, yes, please split it.  It is better
to have each part attributed to the person who wrote it, instead of
both of them as a whole attributed to both of you.

Thanks.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Thu, Dec 17, 2015 at 02:16:41PM +0100, Wolfgang Jenkner wrote:
> On Wed, Dec 16 2015, Paul Eggert wrote:
> 
> > Those patches look reasonable, except that one is listed as being by
> > Rich Felker, who has not signed copyright papers for GNU Emacs as far
> > as I know. I'll CC: this to him to give him a heads-up. Rich, would
> > you mind transferring copyright to that patch to the FSF? Here's
> > a link:
> >
> > http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch
> 
> 
> Rich Felker has asked me in private mail (please, please CC your replies
> to 22086@debbugs.gnu.org) to clarify the authorship of that patch.
> I actually stated in the commit message
> 
>     Except for build system fixes this is essentially the same as
> 
>     http://git.alpinelinux.org/cgit/aports/tree/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6
> 
> So, I wrote the build system fixes, and, strictly speaking, only the two
> lines in src/unexelf.c which declare and initialize bss_sbrk_did_unexec
> are directly from the original "Alpine Linux" patch above.
> 
> I can split the patch accordingly and ascribe only the latter part to
> Rich Felker if this is preferred.  The original patch still helped me to
> quickly find where I had to change something; this is why a attributed
> the whole patch to Rich Felker as it is just an expansion, as it were,
> of his version.

I'm not concerned with how it's split or attributed as long as the FSF
is fine with it. I just wanted to avoid misrepresenting parts that I
did not actually write when describing my changes for copyright
assignment. But this does solve my confusion about why I thought the
patch was trivial -- my original version was hardly anything but some
//'s and #if 0's.

Assuming you already have the proper assignment on file, does it
suffice for me to just mention that parts of the patch are your work?

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

On Thu, Dec 17 2015, Rich Felker wrote:

> I'm not concerned with how it's split or attributed as long as the FSF
> is fine with it. I just wanted to avoid misrepresenting parts that I
> did not actually write when describing my changes for copyright
> assignment.

Let's just split the patch.  If you intend to sign the assignment for
"past and future changes" the usual procedure is to wait until this is
on file and then to simply commit the changes under your name.  I don't
think that

    [Which files have you changed so far, and which new files have you written
    so far?]

means that you have to describe those changes beforehand (but I could be
wrong).

Wolfgang

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Fri, 18 Dec 2015 01:06:48 +0100
> Cc: Paul Eggert <eggert@cs.ucla.edu>, 22086@debbugs.gnu.org
> 
> I don't think that
> 
>     [Which files have you changed so far, and which new files have you written
>     so far?]
> 
> means that you have to describe those changes beforehand (but I could be
> wrong).

You need only to mention the file names where the changes were made,
that's all.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

On Fri, Dec 18 2015, Eli Zaretskii wrote:

>> From: Wolfgang Jenkner <wjenkner@inode.at>
>> Date: Fri, 18 Dec 2015 01:06:48 +0100
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, 22086@debbugs.gnu.org
>> 
>> I don't think that
>> 
>>     [Which files have you changed so far, and which new files have you written
>>     so far?]
>> 
>> means that you have to describe those changes beforehand (but I could be
>> wrong).
>
> You need only to mention the file names where the changes were made,
> that's all.

Yes, but do you have to mention file names for changes which are not yet
in emacs?

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Cc: eggert@cs.ucla.edu,  dalias@aerifal.cx,  22086@debbugs.gnu.org
> Date: Fri, 18 Dec 2015 14:15:07 +0100
> 
> > You need only to mention the file names where the changes were made,
> > that's all.
> 
> Yes, but do you have to mention file names for changes which are not yet
> in emacs?

No.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

While thinking over this patch I'd like to propose what should be a simpler 
approach. This new proposal is more radical, and so should not be applied to the 
emacs-25 branch, but it should make the port to musl etc. automatic.

The simpler approach is to remove gmalloc.c, and to use the system memory 
allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all platforms.

We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't work on 
Cygwin for some reason; and we can support the similar hybrid on Darwin, if it's 
still needed. But in neither approach should we override the system malloc; any 
Emacs-specific allocation function we define should be called (say) emalloc 
instead of malloc, so that it does not conflict with the system malloc. That 
way, we don't have to worry about name-space collisions, either at compile-time 
or at link-time.

If I'm wrong about gmalloc.c and it is still needed on some platforms for some 
reason, we can continue to use it, but it should define emalloc etc., and not 
attempt to override the C standard library.

Long ago as I recall, we really needed to override the C standard library on 
some platforms, due to the funny way in which undumped storage was made 
read-only. That need is obsolete, though, which should let us simplify things now.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Sun, Dec 20, 2015 at 02:33:22PM -0800, Paul Eggert wrote:
> While thinking over this patch I'd like to propose what should be a
> simpler approach. This new proposal is more radical, and so should
> not be applied to the emacs-25 branch, but it should make the port
> to musl etc. automatic.
> 
> The simpler approach is to remove gmalloc.c, and to use the system
> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
> all platforms.
> 
> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC
> wouldn't work on Cygwin for some reason; and we can support the
> similar hybrid on Darwin, if it's still needed. But in neither
> approach should we override the system malloc; any Emacs-specific
> allocation function we define should be called (say) emalloc instead
> of malloc, so that it does not conflict with the system malloc. That
> way, we don't have to worry about name-space collisions, either at
> compile-time or at link-time.
> 
> If I'm wrong about gmalloc.c and it is still needed on some
> platforms for some reason, we can continue to use it, but it should
> define emalloc etc., and not attempt to override the C standard
> library.
> 
> Long ago as I recall, we really needed to override the C standard
> library on some platforms, due to the funny way in which undumped
> storage was made read-only. That need is obsolete, though, which
> should let us simplify things now.

I don't object to this change if you can reliably ensure that nothing
in the pre-dump emacs will call malloc. When I looked, though, this
seemed to be very difficult; I had been looking (albeit casually) for
a clean solution to this problem for nearly a decade before hybrid
malloc showed up. Unless there's a quick solution I think switching to
hybrid malloc would be a first good step, but I like eliminating
gmalloc.c better.

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Rich Felker wrote:
> I don't object to this change if you can reliably ensure that nothing
> in the pre-dump emacs will call malloc.

We can't reliably ensure that, I'm afraid, as Emacs might call some library 
function that in turn calls malloc. It might be a 3rd-party library that Emacs 
has no control over.

Why can't pre-dump emacs call malloc? Is it a performance thing or a correctness 
thing? If a performance thing it shouldn't be much of a problem, as we can 
arrange for Emacs to never directly call malloc, and that should be the vast 
majority of the calls.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Sun, Dec 20, 2015 at 06:37:35PM -0800, Paul Eggert wrote:
> Rich Felker wrote:
> >I don't object to this change if you can reliably ensure that nothing
> >in the pre-dump emacs will call malloc.
> 
> We can't reliably ensure that, I'm afraid, as Emacs might call some
> library function that in turn calls malloc. It might be a 3rd-party
> library that Emacs has no control over.
> 
> Why can't pre-dump emacs call malloc? Is it a performance thing or a
> correctness thing? If a performance thing it shouldn't be much of a
> problem, as we can arrange for Emacs to never directly call malloc,
> and that should be the vast majority of the calls.

It's a correctness thing. The dumper can only dump the brk heap that's
contiguous with .data/.bss (and even that's not contiguous on modern
hardened kernels). This is why fake brk in static storage has to be
used by hybrid malloc. But even if the system malloc did guarantee
contiguity, it's not valid to keep using heap data structures from one
instance of the allocator with a new instance of the allocator, and
there are different failure modes for static and dynamic linking, but
both are bad.

In practice it _might_ work to varying degrees, but it's fundamentally
fragile and wrong.

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Ken Brown]

On 12/20/2015 5:33 PM, Paul Eggert wrote:
> While thinking over this patch I'd like to propose what should be a
> simpler approach. This new proposal is more radical, and so should not
> be applied to the emacs-25 branch, but it should make the port to musl
> etc. automatic.
>
> The simpler approach is to remove gmalloc.c, and to use the system
> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
> platforms.
>
> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
> work on Cygwin for some reason; and we can support the similar hybrid on
> Darwin, if it's still needed.

SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's malloc 
doesn't support malloc_set_state and malloc_get_state.  There may be 
other problems too.  (It's been a while since I tried it.)

Ken

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 20 Dec 2015 14:33:22 -0800
> Cc: Rich Felker <dalias@aerifal.cx>, 22086@debbugs.gnu.org
> 
> The simpler approach is to remove gmalloc.c, and to use the system memory 
> allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all platforms.
> 
> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't work on 
> Cygwin for some reason; and we can support the similar hybrid on Darwin, if it's 
> still needed. But in neither approach should we override the system malloc; any 
> Emacs-specific allocation function we define should be called (say) emalloc 
> instead of malloc, so that it does not conflict with the system malloc. That 
> way, we don't have to worry about name-space collisions, either at compile-time 
> or at link-time.
> 
> If I'm wrong about gmalloc.c and it is still needed on some platforms for some 
> reason, we can continue to use it, but it should define emalloc etc., and not 
> attempt to override the C standard library.

Wouldn't memory allocated with emallooc conflict with uses of malloc
in the startup code?

Other than that, I'm okay with leaving gmalloc.c's allocator under a
different name.  Removing it is something we should consider
separately.

> Long ago as I recall, we really needed to override the C standard library on 
> some platforms, due to the funny way in which undumped storage was made 
> read-only. That need is obsolete, though, which should let us simplify things now.

Why is it obsolete?

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
> On 12/20/2015 5:33 PM, Paul Eggert wrote:
> >While thinking over this patch I'd like to propose what should be a
> >simpler approach. This new proposal is more radical, and so should not
> >be applied to the emacs-25 branch, but it should make the port to musl
> >etc. automatic.
> >
> >The simpler approach is to remove gmalloc.c, and to use the system
> >memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
> >platforms.
> >
> >We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
> >work on Cygwin for some reason; and we can support the similar hybrid on
> >Darwin, if it's still needed.
> 
> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
> malloc doesn't support malloc_set_state and malloc_get_state.  There
> may be other problems too.  (It's been a while since I tried it.)

I don't see how this is possible; malloc_[gs]et_state do not exist on
other systems either. Presumably this is some hack needed for the
dumper, which wouldn't be needed if malloc weren't used pre-dumping.

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Rich Felker wrote:
> In practice it_might_  work to varying degrees, but it's fundamentally
> fragile and wrong.

Yes, of course. But I don't see how it's any more fundamentally fragile and 
wrong than what we're doing already, as we cannot prevent library functions from 
calling library allocators.

> The dumper can only dump the brk heap that's
> contiguous with .data/.bss (and even that's not contiguous on modern
> hardened kernels).

That should be fixed, then.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Eli Zaretskii wrote:
> Wouldn't memory allocated with emallooc conflict with uses of malloc
> in the startup code?

It shouldn't conflict, as long as memory allocated by the system malloc is freed 
by the system free, and memory allocated with emalloc is freed by efree. If 
efree is the hybrid implementation, it'd even be OK to combine the system malloc 
with efree, or to combine emalloc with the system free in some cases. Unless I'm 
misunderstanding the question?

>> >Long ago as I recall, we really needed to override the C standard library on
>> >some platforms, due to the funny way in which undumped storage was made
>> >read-only. That need is obsolete, though, which should let us simplify things now.
> Why is it obsolete?

Formerly Emacs made both static storage and storage allocated via malloc 
read-only in the dumped Emacs, for efficiency reasons. As I recall, this 
required replacing the system malloc with gmalloc. This approach had a lot of 
problems though, and we stopped making that storage read-only a while ago.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Ken Brown]

On 12/20/2015 11:06 PM, Rich Felker wrote:
> On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
>> On 12/20/2015 5:33 PM, Paul Eggert wrote:
>>> While thinking over this patch I'd like to propose what should be a
>>> simpler approach. This new proposal is more radical, and so should not
>>> be applied to the emacs-25 branch, but it should make the port to musl
>>> etc. automatic.
>>>
>>> The simpler approach is to remove gmalloc.c, and to use the system
>>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
>>> platforms.
>>>
>>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
>>> work on Cygwin for some reason; and we can support the similar hybrid on
>>> Darwin, if it's still needed.
>>
>> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
>> malloc doesn't support malloc_set_state and malloc_get_state.  There
>> may be other problems too.  (It's been a while since I tried it.)
>
> I don't see how this is possible; malloc_[gs]et_state do not exist on
> other systems either. Presumably this is some hack needed for the
> dumper, which wouldn't be needed if malloc weren't used pre-dumping.

You're right.  I wasn't thinking clearly.  But several years ago, before 
Cygwin started putting the heap in high memory, there were still issues 
that made it impossible to use the system malloc.  I've forgotten the 
details.

Ken

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

On Sun, Dec 20 2015, Paul Eggert wrote:

> The simpler approach is to remove gmalloc.c, and to use the system
> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
> all platforms.

The system malloc can't manage the "static heap" array implemented in
sheap.c (because usually there's no malloc_hook), so, presumably this
would be ditched as well.  So... I wonder what you are proposing here.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Eli Zaretskii]

> Cc: wjenkner@inode.at, dalias@aerifal.cx, 22086@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 21 Dec 2015 03:18:03 -0800
> 
> Eli Zaretskii wrote:
> > Wouldn't memory allocated with emallooc conflict with uses of malloc
> > in the startup code?
> 
> It shouldn't conflict, as long as memory allocated by the system malloc is freed 
> by the system free, and memory allocated with emalloc is freed by
> efree.

I don't see how this could be arranged.  E.g., the environ array might
be created by the startup code, and then the application could
manipulate it with setenv and unsetenv.  The former will use the libc
malloc, the latter emalloc/efree.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

On Mon, Dec 21 2015, Wolfgang Jenkner wrote:

> sheap.c (because usually there's no malloc_hook), so, presumably this
--------------------------------------^
I meant __morecore, sorry.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Wolfgang Jenkner wrote:
>> The simpler approach is to remove gmalloc.c, and to use the system
>> >memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
>> >all platforms.
> The system malloc can't manage the "static heap" array implemented in
> sheap.c (because usually there's no malloc_hook), so, presumably this
> would be ditched as well.  So... I wonder what you are proposing here.

If we don't need the static heap, then let's ditch it. If we do need it, then 
let's have our own allocator (named emalloc, say), which uses the static heap 
and/or the system malloc as needed. Regardless, we shouldn't be trying to 
redefine 'malloc', nor should we be supplying our own allocator merely because 
that was a good idea back in 1989.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Eli Zaretskii wrote:
>> It shouldn't conflict, as long as memory allocated by the system malloc is freed
>> >by the system free, and memory allocated with emalloc is freed by
>> >efree.
> I don't see how this could be arranged.  E.g., the environ array might
> be created by the startup code, and then the application could
> manipulate it with setenv and unsetenv.  The former will use the libc
> malloc, the latter emalloc/efree.

If by "startup code" you mean the C library, then it's not a problem, as 
portable C code already cannot free environ or environ[0] or environ[1] ... (and 
Emacs respects this, both when run as temacs and when run as dumped). If by 
"startup code" you mean something else, then I'm afraid I'm not following the 
example.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Wolfgang Jenkner]

On Mon, Dec 21 2015, Paul Eggert wrote:

> Wolfgang Jenkner wrote:
>>> The simpler approach is to remove gmalloc.c, and to use the system
>>> >memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
>>> >all platforms.
>> The system malloc can't manage the "static heap" array implemented in
>> sheap.c (because usually there's no malloc_hook), so, presumably this
>> would be ditched as well.  So... I wonder what you are proposing here.
>
> If we don't need the static heap, then let's ditch it.

Is it needed for what you have in mind?

> If we do need
> it, then let's have our own allocator (named emalloc, say), which uses
> the static heap and/or the system malloc as needed.

Well, that's exactly what hybrid malloc does.

> Regardless, we
> shouldn't be trying to redefine 'malloc', nor should we be supplying
> our own allocator merely because that was a good idea back in 1989.

Isn't this a minor problem since it concerns only those systems which
will still need the malloc from src/gmalloc.c after dumping (non-ELF
unix-like systems)?

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Mon, Dec 21, 2015 at 03:10:04AM -0800, Paul Eggert wrote:
> Rich Felker wrote:
> >In practice it_might_  work to varying degrees, but it's fundamentally
> >fragile and wrong.
> 
> Yes, of course. But I don't see how it's any more fundamentally
> fragile and wrong than what we're doing already, as we cannot
> prevent library functions from calling library allocators.

Unless the library code is static-linked to emacs, there's only an
issue if emacs actually saves a direct/indirect reference to the
allocated memory. If the library allocates it internally (e.g. in its
global ctors or init functions, with the pointer(s) saved to static
storage in the library) then all such references will be lost during
dumping and it doesn't matter whatsoever whether the memory is
properly dumped and available at runtime later.

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On 12/20/2015 08:06 PM, Rich Felker wrote:
> On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
>> On 12/20/2015 5:33 PM, Paul Eggert wrote:
>>> While thinking over this patch I'd like to propose what should be a
>>> simpler approach. This new proposal is more radical, and so should not
>>> be applied to the emacs-25 branch, but it should make the port to musl
>>> etc. automatic.
>>>
>>> The simpler approach is to remove gmalloc.c, and to use the system
>>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
>>> platforms.
>>>
>>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
>>> work on Cygwin for some reason; and we can support the similar hybrid on
>>> Darwin, if it's still needed.
>>
>> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
>> malloc doesn't support malloc_set_state and malloc_get_state.  There
>> may be other problems too.  (It's been a while since I tried it.)
> 
> I don't see how this is possible; malloc_[gs]et_state do not exist on
> other systems either. Presumably this is some hack needed for the
> dumper, which wouldn't be needed if malloc weren't used pre-dumping.

We really shouldn't be dumping the native heap at all, really.
Eventually, Emacs should be a position-independent executable (as should
every other program), and to unexec a PIE, we need to emit relocations,
and to emit relocations, we need to know where all the pointers are. We
can't do that if the internal heap structure is opaque to us.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Rich Felker]

On Mon, Dec 21, 2015 at 12:08:46PM -0800, Daniel Colascione wrote:
> On 12/20/2015 08:06 PM, Rich Felker wrote:
> > On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
> >> On 12/20/2015 5:33 PM, Paul Eggert wrote:
> >>> While thinking over this patch I'd like to propose what should be a
> >>> simpler approach. This new proposal is more radical, and so should not
> >>> be applied to the emacs-25 branch, but it should make the port to musl
> >>> etc. automatic.
> >>>
> >>> The simpler approach is to remove gmalloc.c, and to use the system
> >>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
> >>> platforms.
> >>>
> >>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
> >>> work on Cygwin for some reason; and we can support the similar hybrid on
> >>> Darwin, if it's still needed.
> >>
> >> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
> >> malloc doesn't support malloc_set_state and malloc_get_state.  There
> >> may be other problems too.  (It's been a while since I tried it.)
> > 
> > I don't see how this is possible; malloc_[gs]et_state do not exist on
> > other systems either. Presumably this is some hack needed for the
> > dumper, which wouldn't be needed if malloc weren't used pre-dumping.
> 
> We really shouldn't be dumping the native heap at all, really.
> Eventually, Emacs should be a position-independent executable (as should
> every other program), and to unexec a PIE, we need to emit relocations,
> and to emit relocations, we need to know where all the pointers are. We
> can't do that if the internal heap structure is opaque to us.

Actually you can, because the internal heap structure is not what you
need to dump. The "lisp heap" is what you need to dump, and it's
walkable in the same way you walk it now for garbage collection
purposes. It should be possible to adapt the GC code into a dumper
that dumps only the lisp data (with relocations in a form emacs can
internally process, i.e. not relying on writing a new executable
binary with ELF-level or other system-specific relocs) and no unwanted
additional state; then, even static linking should work correctly.

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Daniel Colascione]

On December 21, 2015 12:49:13 PM PST, Rich Felker <dalias@aerifal.cx> wrote:
>On Mon, Dec 21, 2015 at 12:08:46PM -0800, Daniel Colascione wrote:
>> On 12/20/2015 08:06 PM, Rich Felker wrote:
>> > On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
>> >> On 12/20/2015 5:33 PM, Paul Eggert wrote:
>> >>> While thinking over this patch I'd like to propose what should be
>a
>> >>> simpler approach. This new proposal is more radical, and so
>should not
>> >>> be applied to the emacs-25 branch, but it should make the port to
>musl
>> >>> etc. automatic.
>> >>>
>> >>> The simpler approach is to remove gmalloc.c, and to use the
>system
>> >>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined
>on all
>> >>> platforms.
>> >>>
>> >>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC
>wouldn't
>> >>> work on Cygwin for some reason; and we can support the similar
>hybrid on
>> >>> Darwin, if it's still needed.
>> >>
>> >> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
>> >> malloc doesn't support malloc_set_state and malloc_get_state. 
>There
>> >> may be other problems too.  (It's been a while since I tried it.)
>> > 
>> > I don't see how this is possible; malloc_[gs]et_state do not exist
>on
>> > other systems either. Presumably this is some hack needed for the
>> > dumper, which wouldn't be needed if malloc weren't used
>pre-dumping.
>> 
>> We really shouldn't be dumping the native heap at all, really.
>> Eventually, Emacs should be a position-independent executable (as
>should
>> every other program), and to unexec a PIE, we need to emit
>relocations,
>> and to emit relocations, we need to know where all the pointers are.
>We
>> can't do that if the internal heap structure is opaque to us.
>
>Actually you can, because the internal heap structure is not what you
>need to dump. The "lisp heap" is what you need to dump, and it's
>walkable in the same way you walk it now for garbage collection
>purposes. It should be possible to adapt the GC code into a dumper
>that dumps only the lisp data (with relocations in a form emacs can
>internally process, i.e. not relying on writing a new executable
>binary with ELF-level or other system-specific relocs) and no unwanted
>additional state; then, even static linking should work correctly.
>
>Rich

Sure. That's the XEmacs portable dumper approach, and it works all right. I'm just worried that we might have implicit dependencies on the C heap being preserved across unexec.

The same code we use to relocate when Emacs is a PIE would also help us do GC compaction. (We'd have to pin conservatively reachable roots.)

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Rich Felker wrote:
> Unless the library code is static-linked to emacs, there's only an
> issue if emacs actually saves a direct/indirect reference to the
> allocated memory.

It shouldn't be a problem, then. I doubt whether there are many such instances; 
if there are any, we can arrange for them to be copied before dumping or 
recreated after.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

[Paul Eggert]

Wolfgang Jenkner wrote:

>> If we don't need the static heap, then let's ditch it.
>
> Is it needed for what you have in mind?

I hope not. Perhaps it is needed for Cygwin, for reasons I don't understand. If 
it is needed then obviously we should keep it, and use the hybrid allocator. If 
not, not.

>> Regardless, we
>> shouldn't be trying to redefine 'malloc', nor should we be supplying
>> our own allocator merely because that was a good idea back in 1989.
>
> Isn't this a minor problem since it concerns only those systems which
> will still need the malloc from src/gmalloc.c after dumping (non-ELF
> unix-like systems)?

Yes, it's a minor problem, in that none of these systems actually need gmalloc.c 
any more.  If I understand things correctly, the main "problem" here (and it is 
a good problem to have) is that we need to carefully excise gmalloc.c out of 
Emacs, without breaking things.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

The recent emacs-devel thread "Removal of unexec support" has raised the 
priority of this bug, so I redid the patches to separate out Rich Felker's 
contribution, which is so small as to not require copyright papers, and fixed 
several problems I found with the resulting approach. I came up with the 
attached set of patches relative to commit 
ef760b899ad89f941f552ed2d3ac9e45156f3e3c. I would like to commit this patch set 
to the emacs-25 branch soon, and am sending this email to give you (particularly 
Eli) a heads-up about this.

These patches attempt to be more conservative than the other alternatives 
discussed in Bug#22086. They don't try to build a better dumper or remove 
gmalloc.c or anything like that. All they try to do, is to disentangle Emacs 
from glibc malloc internals, by renaming functions whose APIs are no longer 
compatible with glibc, and by using glibc's <malloc.h> rather than guessing what 
it will say, and that sort of thing. The goal is for the resulting Emacs to not 
only port to musl, but also to port to future glibc with less likelihood of trouble.

[-- Attachment #2: 0001-Internal-linkage-for-gmalloc-etc.-if-HYBRID_MALLOC.patch --]
[-- Type: text/x-diff, Size: 7150 bytes --]

From dd92b84893e2d59a76b088e9e6e36a959ecb36a4 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Sat, 30 Jan 2016 00:47:24 -0800
Subject: [PATCH 01/12] Internal linkage for gmalloc etc. if HYBRID_MALLOC

This avoids clashes with symbols if the after-dump malloc is
derived from Doug Lea's implementation (Bug#22086).

* src/gmalloc.c (emacs_abort, __morecore, __default_morecore):
Move declarations up.  For HYBRID_MALLOC, turn all `extern'
declarations below to `static' ones.
(aligned_alloc): Declare for !MSDOS as well.
(heapsize, _fraghead): Move resp. copy declaration downwards.
For HYBRID_MALLOC, conditionalize out the other definitions,
since the previous `static' declarations double as tentative
definitions, anyway.
(_malloc, _free, _realloc, __free_hook, _aligned_blocks)
(__realloc_hook, __memalign_hook): Conditionalize out.
(cfree, memalign, valloc): Ditto.
---
 src/gmalloc.c | 58 +++++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 41 insertions(+), 17 deletions(-)

diff --git a/src/gmalloc.c b/src/gmalloc.c
index 00b8364..72d65af 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -87,6 +87,20 @@ extern "C"
 
 #include <stddef.h>
 
+#ifdef emacs
+extern void emacs_abort (void);
+#endif
+
+/* Underlying allocation function; successive calls should
+   return contiguous pieces of memory.  */
+extern void *(*__morecore) (ptrdiff_t size);
+
+/* Default value of `__morecore'.  */
+extern void *__default_morecore (ptrdiff_t size);
+
+#ifdef HYBRID_MALLOC
+#define extern static
+#endif
 
 /* Allocate SIZE bytes of memory.  */
 extern void *malloc (size_t size) ATTRIBUTE_MALLOC_SIZE ((1));
@@ -99,8 +113,8 @@ extern void *calloc (size_t nmemb, size_t size) ATTRIBUTE_MALLOC_SIZE ((1,2));
 extern void free (void *ptr);
 
 /* Allocate SIZE bytes allocated to ALIGNMENT bytes.  */
-#ifdef MSDOS
 extern void *aligned_alloc (size_t, size_t);
+#ifdef MSDOS
 extern void *memalign (size_t, size_t);
 extern int posix_memalign (void **, size_t, size_t);
 #endif
@@ -110,10 +124,6 @@ extern int posix_memalign (void **, size_t, size_t);
 extern void malloc_enable_thread (void);
 #endif
 
-#ifdef emacs
-extern void emacs_abort (void);
-#endif
-
 /* The allocator divides the heap into blocks of fixed size; large
    requests receive one or more whole blocks, and small requests
    receive a fragment of a block.  Fragment sizes are powers of two,
@@ -249,13 +259,6 @@ extern int _malloc_thread_enabled_p;
    return the address of the beginning of the object.  */
 extern void *malloc_find_object_address (void *ptr);
 
-/* Underlying allocation function; successive calls should
-   return contiguous pieces of memory.  */
-extern void *(*__morecore) (ptrdiff_t size);
-
-/* Default value of `__morecore'.  */
-extern void *__default_morecore (ptrdiff_t size);
-
 /* If not NULL, this function is called after each time
    `__morecore' is called to increase the data size.  */
 extern void (*__after_morecore_hook) (void);
@@ -318,6 +321,8 @@ extern struct mstats mstats (void);
 /* Call WARNFUN with a warning message when memory usage is high.  */
 extern void memory_warnings (void *start, void (*warnfun) (const char *));
 
+#undef extern
+
 #ifdef	__cplusplus
 }
 #endif
@@ -346,6 +351,8 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 
 void *(*__morecore) (ptrdiff_t size) = __default_morecore;
 
+#ifndef HYBRID_MALLOC
+
 /* Debugging hook for `malloc'.  */
 void *(*__malloc_hook) (size_t size);
 
@@ -355,9 +362,6 @@ char *_heapbase;
 /* Block information table.  Allocated with align/__free (not malloc/free).  */
 malloc_info *_heapinfo;
 
-/* Number of info entries.  */
-static size_t heapsize;
-
 /* Search index in the info table.  */
 size_t _heapindex;
 
@@ -381,6 +385,15 @@ size_t __malloc_extra_blocks;
 void (*__malloc_initialize_hook) (void);
 void (*__after_morecore_hook) (void);
 
+#else
+
+static struct list _fraghead[BLOCKLOG];
+
+#endif /* HYBRID_MALLOC */
+
+/* Number of info entries.  */
+static size_t heapsize;
+
 #if defined GC_MALLOC_CHECK && defined GC_PROTECT_MALLOC_STATE
 
 /* Some code for hunting a bug writing into _heapinfo.
@@ -946,7 +959,7 @@ malloc (size_t size)
   return (hook != NULL ? *hook : _malloc_internal) (size);
 }
 \f
-#ifndef _LIBC
+#if !(defined (_LIBC) || defined (HYBRID_MALLOC))
 
 /* On some ANSI C systems, some libc functions call _malloc, _free
    and _realloc.  Make them use the GNU functions.  */
@@ -995,11 +1008,13 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
    or (US mail) as Mike Haertel c/o Free Software Foundation.  */
 
 
+#ifndef HYBRID_MALLOC
 /* Debugging hook for free.  */
 void (*__free_hook) (void *__ptr);
 
 /* List of blocks allocated by aligned_alloc.  */
 struct alignlist *_aligned_blocks = NULL;
+#endif
 
 /* Return memory to the heap.
    Like `_free_internal' but don't lock mutex.  */
@@ -1270,6 +1285,7 @@ free (void *ptr)
     _free_internal (ptr);
 }
 
+#ifndef HYBRID_MALLOC
 /* Define the `cfree' alias for `free'.  */
 #ifdef weak_alias
 weak_alias (free, cfree)
@@ -1280,6 +1296,7 @@ cfree (void *ptr)
   free (ptr);
 }
 #endif
+#endif
 /* Change the size of a block allocated by `malloc'.
    Copyright 1990, 1991, 1992, 1993, 1994, 1995 Free Software Foundation, Inc.
 		     Written May 1989 by Mike Haertel.
@@ -1304,8 +1321,10 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #define min(a, b) ((a) < (b) ? (a) : (b))
 #endif
 
+#ifndef HYBRID_MALLOC
 /* Debugging hook for realloc.  */
 void *(*__realloc_hook) (void *ptr, size_t size);
+#endif
 
 /* Resize the given region to the new size, returning a pointer
    to the (possibly moved) region.  This is optimized for speed;
@@ -1549,7 +1568,9 @@ General Public License for more details.
 You should have received a copy of the GNU General Public
 License along with this library.  If not, see <http://www.gnu.org/licenses/>.  */
 
+#ifndef HYBRID_MALLOC
 void *(*__memalign_hook) (size_t size, size_t alignment);
+#endif
 
 void *
 aligned_alloc (size_t alignment, size_t size)
@@ -1638,6 +1659,8 @@ aligned_alloc (size_t alignment, size_t size)
   return result;
 }
 
+/* Note that memalign and posix_memalign are not used in Emacs.  */
+#ifndef HYBRID_MALLOC
 /* An obsolete alias for aligned_alloc, for any old libraries that use
    this alias.  */
 
@@ -1649,7 +1672,6 @@ memalign (size_t alignment, size_t size)
 
 /* If HYBRID_MALLOC is defined, we may want to use the system
    posix_memalign below.  */
-#ifndef HYBRID_MALLOC
 int
 posix_memalign (void **memptr, size_t alignment, size_t size)
 {
@@ -1689,6 +1711,7 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
    The author may be reached (Email) at the address mike@ai.mit.edu,
    or (US mail) as Mike Haertel c/o Free Software Foundation.  */
 
+#ifndef HYBRID_MALLOC
 /* Allocate SIZE bytes on a page boundary.  */
 extern void *valloc (size_t);
 
@@ -1708,6 +1731,7 @@ valloc (size_t size)
 
   return aligned_alloc (pagesize, size);
 }
+#endif /* HYBRID_MALLOC */
 
 #undef malloc
 #undef realloc
-- 
2.5.0


[-- Attachment #3: 0002-Link-temacs-with-gnulib-compiled-with-Demacs.patch --]
[-- Type: text/x-diff, Size: 2165 bytes --]

From c9dadfaa6c91d1f48d2a3a0e43c56dfa76815727 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Sat, 30 Jan 2016 00:47:24 -0800
Subject: [PATCH 02/12] Link temacs with gnulib compiled with -Demacs

This is done to support HYBRID_MALLOC, since some static variables
(e.g., last_environ in putenv.c) hold pointers to memory malloced
before dumping (Bug#22086).
* lib/Makefile.am: Add incantation to install libegnu.a.
* src/Makefile.in ($(lib)/libgnu.a): Replace with libegnu.a
(temacs$(EXEEXT)): Use it.
---
 lib/Makefile.am | 9 +++++++++
 src/Makefile.in | 6 +++---
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/lib/Makefile.am b/lib/Makefile.am
index cda9681..a1dd6a4 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -17,6 +17,15 @@ include gnulib.mk
 libgnu_a_SOURCES += openat-die.c save-cwd.c
 endif
 
+noinst_LIBRARIES += libegnu.a
+
+libegnu_a_SOURCES = $(libgnu_a_SOURCES)
+libegnu_a_LIBADD = $(patsubst %.o,e-%.o,$(libgnu_a_LIBADD))
+EXTRA_libegnu_a_SOURCES = $(EXTRA_libgnu_a_SOURCES)
+libegnu_a_SHORTNAME = e
+
+libegnu_a_CPPFLAGS = $(AM_CPPFLAGS) -Demacs
+
 .PHONY: bootstrap-clean
 
 bootstrap-clean: maintainer-clean
diff --git a/src/Makefile.in b/src/Makefile.in
index fab10ae..fc96e61 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -576,7 +576,7 @@ globals.h: gl-stamp; @true
 
 $(ALLOBJS): globals.h
 
-$(lib)/libgnu.a: $(config_h)
+$(lib)/libegnu.a: $(config_h)
 	$(MAKE) -C $(lib) all
 
 ## We have to create $(etc) here because init_cmdargs tests its
@@ -584,9 +584,9 @@ $(lib)/libgnu.a: $(config_h)
 ## This goes on to affect various things, and the emacs binary fails
 ## to start if Vinstallation_directory has the wrong value.
 temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) \
-	         $(lib)/libgnu.a $(EMACSRES) ${charsets} ${charscript}
+	         $(lib)/libegnu.a $(EMACSRES) ${charsets} ${charscript}
 	$(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
-	  -o temacs $(ALLOBJS) $(lib)/libgnu.a $(W32_RES_LINK) $(LIBES)
+	  -o temacs $(ALLOBJS) $(lib)/libegnu.a $(W32_RES_LINK) $(LIBES)
 	$(MKDIR_P) $(etc)
 ifneq ($(CANNOT_DUMP),yes)
 	$(PAXCTL_if_present) -r $@
-- 
2.5.0


[-- Attachment #4: 0003-unexelf.c-hook-to-support-HYBRID_MALLOC-on-ELF.patch --]
[-- Type: text/x-diff, Size: 826 bytes --]

From 31f48ed3a6ca66ae0f13e6057270d4aed2372d3a Mon Sep 17 00:00:00 2001
From: Rich Felker <dalias@libc.org>
Date: Sat, 30 Jan 2016 00:47:24 -0800
Subject: [PATCH 03/12] unexelf.c hook to support HYBRID_MALLOC on ELF

* src/unexelf.c (unexec) [HYBRID_MALLOC]:
Define bss_sbrk_did_unexec (Bug#22086).
Copyright-paperwork-exempt: yes
---
 src/unexelf.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/unexelf.c b/src/unexelf.c
index e901994..32aa1b2 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -227,6 +227,11 @@ unexec (const char *new_name, const char *old_name)
   off_t new_file_size;
   void *new_break;
 
+#ifdef HYBRID_MALLOC
+  extern int bss_sbrk_did_unexec;
+  bss_sbrk_did_unexec = 1;
+#endif
+
   /* Pointers to the base of the image of the two files.  */
   caddr_t old_base, new_base;
 
-- 
2.5.0


[-- Attachment #5: 0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch --]
[-- Type: text/x-diff, Size: 3421 bytes --]

From 3049e05754993fb1e80a29ae48b2bf661e3ed773 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Sat, 30 Jan 2016 00:47:24 -0800
Subject: [PATCH 04/12] Add musl patch to support HYBRID_MALLOC on elf systems

* src/gmalloc.c: Adjust for HYBRID_MALLOC in the non CYGWIN case.
(__default_morecore): Here, in particular.
* configure.ac: Define HYBRID_MALLOC when unexelf.o is used.
New variable SHEAP_OBJ.
* src/Makefile.in: Use it.
(Bug#22086)
---
 configure.ac    | 12 ++++++++++--
 src/Makefile.in |  6 +++++-
 src/gmalloc.c   |  4 ++--
 3 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/configure.ac b/configure.ac
index fad556f..c01e238 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2121,7 +2121,13 @@ case "$opsys" in
   cygwin) hybrid_malloc=yes;;
 esac
 
+if test "${system_malloc}" != yes && test "${doug_lea_malloc}" != yes \
+   && test "${UNEXEC_OBJ}" = unexelf.o && test "${hybrid_malloc}" != no; then
+  hybrid_malloc=yes
+fi
+
 GMALLOC_OBJ=
+SHEAP_OBJ=
 if test "${system_malloc}" = "yes"; then
   AC_DEFINE([SYSTEM_MALLOC], 1,
     [Define to 1 to use the system memory allocator, even if it is not
@@ -2136,6 +2142,7 @@ elif test "$hybrid_malloc" = yes; then
   GNU_MALLOC=
   GNU_MALLOC_reason="only before dumping"
   GMALLOC_OBJ=gmalloc.o
+  SHEAP_OBJ=sheap.o
   VMLIMIT_OBJ=
 else
   test "$doug_lea_malloc" != "yes" && GMALLOC_OBJ=gmalloc.o
@@ -2156,9 +2163,10 @@ else
   fi
 fi
 AC_SUBST(GMALLOC_OBJ)
+AC_SUBST(SHEAP_OBJ)
 AC_SUBST(VMLIMIT_OBJ)
 
-if test "$doug_lea_malloc" = "yes" ; then
+if test "$doug_lea_malloc" = "yes" && test "$hybrid_malloc" != yes; then
   if test "$GNU_MALLOC" = yes ; then
     GNU_MALLOC_reason="
       (Using Doug Lea's new malloc from the GNU C Library.)"
@@ -5035,7 +5043,7 @@ fi
 AC_SUBST(RALLOC_OBJ)
 
 if test "$opsys" = "cygwin"; then
-  CYGWIN_OBJ="sheap.o cygw32.o"
+  CYGWIN_OBJ="cygw32.o"
   ## Cygwin differs because of its unexec().
   PRE_ALLOC_OBJ=
   POST_ALLOC_OBJ=lastfile.o
diff --git a/src/Makefile.in b/src/Makefile.in
index fc96e61..25dc3f8 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -254,7 +254,10 @@ XFIXES_CFLAGS = @XFIXES_CFLAGS@
 ## widget.o if USE_X_TOOLKIT, otherwise empty.
 WIDGET_OBJ=@WIDGET_OBJ@
 
-## sheap.o if CYGWIN, otherwise empty.
+## sheap.o if HYBRID_MALLOC, otherwise empty.
+SHEAP_OBJ=@SHEAP_OBJ@
+
+## cygw32.o if CYGWIN, otherwise empty.
 CYGWIN_OBJ=@CYGWIN_OBJ@
 
 ## fontset.o fringe.o image.o if we have any window system
@@ -397,6 +400,7 @@ base_obj = dispnew.o frame.o scroll.o xdisp.o menu.o $(XMENU_OBJ) window.o \
 	doprnt.o intervals.o textprop.o composite.o xml.o $(NOTIFY_OBJ) \
 	$(XWIDGETS_OBJ) \
 	profiler.o decompress.o \
+	$(SHEAP_OBJ) \
 	$(MSDOS_OBJ) $(MSDOS_X_OBJ) $(NS_OBJ) $(CYGWIN_OBJ) $(FONT_OBJ) \
 	$(W32_OBJ) $(WINDOW_SYSTEM_OBJ) $(XGSELOBJ)
 obj = $(base_obj) $(NS_OBJC_OBJ)
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 72d65af..30e0131 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -70,7 +70,7 @@ extern void emacs_abort (void);
 #define aligned_alloc galigned_alloc
 #define free gfree
 
-#ifdef CYGWIN
+#ifdef HYBRID_MALLOC
 extern void *bss_sbrk (ptrdiff_t size);
 extern int bss_sbrk_did_unexec;
 extern char bss_sbrk_buffer[];
@@ -1542,7 +1542,7 @@ void *
 __default_morecore (ptrdiff_t increment)
 {
   void *result;
-#if defined (CYGWIN)
+#ifdef HYBRID_MALLOC
   if (!DUMPED)
     {
       return bss_sbrk (increment);
-- 
2.5.0


[-- Attachment #6: 0005-Pacify-GCC-on-extern-decls.patch --]
[-- Type: text/x-diff, Size: 955 bytes --]

From ee520757a32999ac04030027af5693b4095984a7 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:25 -0800
Subject: [PATCH 05/12] Pacify GCC on extern decls

* src/unexelf.c (bss_sbrk_did_unexec): Move decl to top level
to pacify recent GCC (Bug#22086).
---
 src/unexelf.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/unexelf.c b/src/unexelf.c
index 32aa1b2..3dc0456 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -211,6 +211,10 @@ entry_address (void *section_h, ptrdiff_t idx, ptrdiff_t entsize)
 
 typedef unsigned char byte;
 
+#ifdef HYBRID_MALLOC
+extern int bss_sbrk_did_unexec;
+#endif
+
 /* ****************************************************************
  * unexec
  *
@@ -228,7 +232,6 @@ unexec (const char *new_name, const char *old_name)
   void *new_break;
 
 #ifdef HYBRID_MALLOC
-  extern int bss_sbrk_did_unexec;
   bss_sbrk_did_unexec = 1;
 #endif
 
-- 
2.5.0


[-- Attachment #7: 0006-Report-static-heap-usage-on-non-Cygwin-too.patch --]
[-- Type: text/x-diff, Size: 1815 bytes --]

From 3db9402fe355eb2ea60837148a0426bd41df09d1 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:25 -0800
Subject: [PATCH 06/12] Report static heap usage on non-Cygwin, too

* src/emacs.c (Fdump_emacs) [HYBRID_MALLOC]: Report sheap usage here ...
* src/unexcw.c (unexec): ... instead of here, since sheap can be used
on platforms other than Cygwin (Bug#22086).
---
 src/emacs.c  | 5 +++++
 src/unexcw.c | 4 ----
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/src/emacs.c b/src/emacs.c
index e3cfad0..40075b4 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -134,6 +134,7 @@ extern void unexec_init_emacs_zone (void);
 #endif
 
 extern void malloc_enable_thread (void);
+extern void report_sheap_usage (int);
 
 /* If true, Emacs should not attempt to use a window-specific code,
    but instead should use the virtual terminal under which it was started.  */
@@ -2070,6 +2071,10 @@ You must run Emacs in batch mode in order to dump it.  */)
   tem = Vpurify_flag;
   Vpurify_flag = Qnil;
 
+#ifdef HYBRID_MALLOC
+  report_sheap_usage (1);
+#endif
+
   fflush (stdout);
   /* Tell malloc where start of impure now is.  */
   /* Also arrange for warnings when nearly out of space.  */
diff --git a/src/unexcw.c b/src/unexcw.c
index febe939..e4aa356 100644
--- a/src/unexcw.c
+++ b/src/unexcw.c
@@ -30,8 +30,6 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #define DOTEXE ".exe"
 
-extern void report_sheap_usage (int);
-
 extern int bss_sbrk_did_unexec;
 
 /*
@@ -276,8 +274,6 @@ unexec (const char *outfile, const char *infile)
   int ret;
   int ret2;
 
-  report_sheap_usage (1);
-
   infile = add_exe_suffix_if_necessary (infile, infile_buffer);
   outfile = add_exe_suffix_if_necessary (outfile, outfile_buffer);
 
-- 
2.5.0


[-- Attachment #8: 0007-Pacify-enable-gcc-warnings-when-HYBRID_MALLOC.patch --]
[-- Type: text/x-diff, Size: 15985 bytes --]

From 62de6240b57bc148eccfe0c6a723a7dc2e67a526 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:25 -0800
Subject: [PATCH 07/12] Pacify --enable-gcc-warnings when HYBRID_MALLOC

* src/buffer.c (init_buffer):
* src/emacs.c (main):
* src/xsmfns.c (smc_save_yourself_CB, x_session_initialize):
Use emacs_get_current_dir_name, not get_current_dir_name.
* src/conf_post.h (aligned_alloc) [HYBRID_MALLOC && emacs]: New macro.
(HYBRID_GET_CURRENT_DIR_NAME, get_current_dir_name): Remove.
* src/emacs.c: Include "sheap.h".
(report_sheap_usage): Remove decl.
(Fdump_emacs) [HYBRID_MALLOC]: Report usage directly.
Don't assume ptrdiff_t can be printed as int.
* src/gmalloc.c [HYBRID_MALLOC]:
Include "sheap.h" rather than declaring its contents by hand.
(get_current_dir_name, gget_current_dir_name)
(hybrid_get_current_dir_name): Remove.
(emacs_abort): Remove duplicate decl.
(aligned_alloc): Undef, like malloc etc.
(ALLOCATED_BEFORE_DUMPING): Now a static function, not a macro.
Make it a bit more efficient.
(malloc_find_object_address): Remove unused decl.
(enum mcheck_status, mcheck, mprobe, mtrace, muntrace, struct mstats)
(mstats, memory_warnings): Declare only if GC_MCHECK.
* src/lisp.h (emacs_get_current_dir_name):
New decl, replacing get_current_dir_name.
* src/sheap.c: Include sheap.h first.
(STATIC_HEAP_SIZE): Remove; now in sheap.h.
(debug_sheap): Now static.
(bss_sbrk_buffer_end): Remove; no longer used.
(bss_sbrk_ptr): Now static and private.
(bss_sbrk_did_unexec): Now bool.
(BLOCKSIZE): Remove, to avoid GCC warning about its not being used.
(bss_sbrk): Don't treat request_size 0 as special, since the code
works without this being a special case.
Avoid overflow if request size exceeds INT_MAX.
(report_sheap_usage): Remove; now done in emacs.c.
* src/sheap.h: New file.
* src/sysdep.c (get_current_dir_name): Remove macro.
Include "sheap.h".
(emacs_get_current_dir_name): Rename function from
get_current_dir_name.  Handle HYBRID_MALLOC here;
this is simpler.
(Bug#22086)
---
 src/buffer.c    |  2 +-
 src/conf_post.h |  5 +---
 src/emacs.c     | 13 ++++++---
 src/gmalloc.c   | 51 +++++++++--------------------------
 src/lisp.h      |  4 +--
 src/sheap.c     | 83 ++++++++++++++++++++-------------------------------------
 src/sheap.h     | 31 +++++++++++++++++++++
 src/sysdep.c    | 29 ++++++++++----------
 src/xsmfns.c    |  4 +--
 9 files changed, 102 insertions(+), 120 deletions(-)
 create mode 100644 src/sheap.h

diff --git a/src/buffer.c b/src/buffer.c
index 74b6fb6..bd5425d 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -5277,7 +5277,7 @@ init_buffer (int initialized)
   if (NILP (BVAR (&buffer_defaults, enable_multibyte_characters)))
     Fset_buffer_multibyte (Qnil);
 
-  pwd = get_current_dir_name ();
+  pwd = emacs_get_current_dir_name ();
 
   if (!pwd)
     {
diff --git a/src/conf_post.h b/src/conf_post.h
index 5c332a0..9f4becd 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -100,11 +100,8 @@ typedef bool bool_bf;
 #define malloc hybrid_malloc
 #define realloc hybrid_realloc
 #define calloc hybrid_calloc
+#define aligned_alloc hybrid_aligned_alloc
 #define free hybrid_free
-#if defined HAVE_GET_CURRENT_DIR_NAME && !defined BROKEN_GET_CURRENT_DIR_NAME
-#define HYBRID_GET_CURRENT_DIR_NAME 1
-#define get_current_dir_name hybrid_get_current_dir_name
-#endif
 #endif
 #endif	/* HYBRID_MALLOC */
 
diff --git a/src/emacs.c b/src/emacs.c
index 40075b4..d31ba5a 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -79,6 +79,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "composite.h"
 #include "dispextern.h"
 #include "regex.h"
+#include "sheap.h"
 #include "syntax.h"
 #include "sysselect.h"
 #include "systime.h"
@@ -134,7 +135,6 @@ extern void unexec_init_emacs_zone (void);
 #endif
 
 extern void malloc_enable_thread (void);
-extern void report_sheap_usage (int);
 
 /* If true, Emacs should not attempt to use a window-specific code,
    but instead should use the virtual terminal under which it was started.  */
@@ -772,7 +772,7 @@ main (int argc, char **argv)
       filename_from_ansi (ch_to_dir, newdir);
       ch_to_dir = newdir;
 #endif
-      original_pwd = get_current_dir_name ();
+      original_pwd = emacs_get_current_dir_name ();
       if (chdir (ch_to_dir) != 0)
         {
           fprintf (stderr, "%s: Can't chdir to %s: %s\n",
@@ -2072,7 +2072,14 @@ You must run Emacs in batch mode in order to dump it.  */)
   Vpurify_flag = Qnil;
 
 #ifdef HYBRID_MALLOC
-  report_sheap_usage (1);
+  {
+    static char const fmt[] = "%d of %d static heap bytes used";
+    char buf[sizeof fmt + 2 * (INT_STRLEN_BOUND (int) - 2)];
+    int max_usage = max_bss_sbrk_ptr - bss_sbrk_buffer;
+    sprintf (buf, fmt, max_usage, STATIC_HEAP_SIZE);
+    /* Don't log messages, because at this point buffers cannot be created.  */
+    message1_nolog (buf);
+  }
 #endif
 
   fflush (stdout);
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 30e0131..4fd3246 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -28,11 +28,6 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #include <string.h>
 #include <limits.h>
 #include <stdint.h>
-
-#ifdef HYBRID_GET_CURRENT_DIR_NAME
-#undef get_current_dir_name
-#endif
-
 #include <unistd.h>
 
 #ifdef USE_PTHREAD
@@ -43,10 +38,6 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #include <w32heap.h>	/* for sbrk */
 #endif
 
-#ifdef emacs
-extern void emacs_abort (void);
-#endif
-
 /* If HYBRID_MALLOC is defined, then temacs will use malloc,
    realloc... as defined in this file (and renamed gmalloc,
    grealloc... via the macros that follow).  The dumped emacs,
@@ -63,6 +54,7 @@ extern void emacs_abort (void);
 #undef malloc
 #undef realloc
 #undef calloc
+#undef aligned_alloc
 #undef free
 #define malloc gmalloc
 #define realloc grealloc
@@ -71,13 +63,13 @@ extern void emacs_abort (void);
 #define free gfree
 
 #ifdef HYBRID_MALLOC
-extern void *bss_sbrk (ptrdiff_t size);
-extern int bss_sbrk_did_unexec;
-extern char bss_sbrk_buffer[];
-extern void *bss_sbrk_buffer_end;
-#define DUMPED bss_sbrk_did_unexec
-#define ALLOCATED_BEFORE_DUMPING(P) \
-  ((P) < bss_sbrk_buffer_end && (P) >= (void *) bss_sbrk_buffer)
+# include "sheap.h"
+# define DUMPED bss_sbrk_did_unexec
+static bool
+ALLOCATED_BEFORE_DUMPING (char *p)
+{
+  return bss_sbrk_buffer <= p && p < bss_sbrk_buffer + STATIC_HEAP_SIZE;
+}
 #endif
 
 #ifdef	__cplusplus
@@ -87,10 +79,6 @@ extern "C"
 
 #include <stddef.h>
 
-#ifdef emacs
-extern void emacs_abort (void);
-#endif
-
 /* Underlying allocation function; successive calls should
    return contiguous pieces of memory.  */
 extern void *(*__morecore) (ptrdiff_t size);
@@ -255,10 +243,6 @@ extern int _malloc_thread_enabled_p;
 #define UNLOCK_ALIGNED_BLOCKS()
 #endif
 
-/* Given an address in the middle of a malloc'd object,
-   return the address of the beginning of the object.  */
-extern void *malloc_find_object_address (void *ptr);
-
 /* If not NULL, this function is called after each time
    `__morecore' is called to increase the data size.  */
 extern void (*__after_morecore_hook) (void);
@@ -279,6 +263,8 @@ extern void *(*__malloc_hook) (size_t size);
 extern void *(*__realloc_hook) (void *ptr, size_t size);
 extern void *(*__memalign_hook) (size_t size, size_t alignment);
 
+#ifdef GC_MCHECK
+
 /* Return values for `mprobe': these are the kinds of inconsistencies that
    `mcheck' enables detection of.  */
 enum mcheck_status
@@ -321,6 +307,8 @@ extern struct mstats mstats (void);
 /* Call WARNFUN with a warning message when memory usage is high.  */
 extern void memory_warnings (void *start, void (*warnfun) (const char *));
 
+#endif
+
 #undef extern
 
 #ifdef	__cplusplus
@@ -1797,7 +1785,7 @@ hybrid_aligned_alloc (size_t alignment, size_t size)
 #endif
 }
 #endif
-  
+
 void *
 hybrid_realloc (void *ptr, size_t size)
 {
@@ -1825,19 +1813,6 @@ hybrid_realloc (void *ptr, size_t size)
   return result;
 }
 
-#ifdef HYBRID_GET_CURRENT_DIR_NAME
-/* Defined in sysdep.c.  */
-char *gget_current_dir_name (void);
-
-char *
-hybrid_get_current_dir_name (void)
-{
-  if (DUMPED)
-    return get_current_dir_name ();
-  return gget_current_dir_name ();
-}
-#endif
-
 #else	/* ! HYBRID_MALLOC */
 
 void *
diff --git a/src/lisp.h b/src/lisp.h
index 82cbca8..53f123d 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4254,9 +4254,7 @@ struct tty_display_info;
 struct terminal;
 
 /* Defined in sysdep.c.  */
-#ifndef HAVE_GET_CURRENT_DIR_NAME
-extern char *get_current_dir_name (void);
-#endif
+extern char *emacs_get_current_dir_name (void);
 extern void stuff_char (char c);
 extern void init_foreground_group (void);
 extern void sys_subshell (void);
diff --git a/src/sheap.c b/src/sheap.c
index 1451eca..fe905ca 100644
--- a/src/sheap.c
+++ b/src/sheap.c
@@ -19,87 +19,62 @@ You should have received a copy of the GNU General Public License
 along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include <config.h>
+
+#include "sheap.h"
+
 #include <stdio.h>
 #include "lisp.h"
 #include <unistd.h>
 #include <stdlib.h>		/* for exit */
 
-#ifdef ENABLE_CHECKING
-#define STATIC_HEAP_SIZE	(28 * 1024 * 1024)
-#else
-#define STATIC_HEAP_SIZE	(19 * 1024 * 1024)
-#endif
-
-int debug_sheap = 0;
-
-#define BLOCKSIZE 4096
+static int debug_sheap;
 
 char bss_sbrk_buffer[STATIC_HEAP_SIZE];
-/* The following is needed in gmalloc.c */
-void *bss_sbrk_buffer_end = bss_sbrk_buffer + STATIC_HEAP_SIZE;
-char *bss_sbrk_ptr;
 char *max_bss_sbrk_ptr;
-int bss_sbrk_did_unexec;
+bool bss_sbrk_did_unexec;
 
 void *
 bss_sbrk (ptrdiff_t request_size)
 {
+  static char *bss_sbrk_ptr;
+
   if (!bss_sbrk_ptr)
     {
       max_bss_sbrk_ptr = bss_sbrk_ptr = bss_sbrk_buffer;
 #ifdef CYGWIN
-      sbrk (BLOCKSIZE);		/* force space for fork to work */
+      /* Force space for fork to work.  */
+      sbrk (4096);
 #endif
     }
 
-  if (!(int) request_size)
-    {
-      return (bss_sbrk_ptr);
-    }
-  else if (bss_sbrk_ptr + (int) request_size < bss_sbrk_buffer)
+  int used = bss_sbrk_ptr - bss_sbrk_buffer;
+
+  if (request_size < -used)
     {
-      printf
-	("attempt to free too much: avail %d used %d failed request %d\n",
-	 STATIC_HEAP_SIZE, bss_sbrk_ptr - bss_sbrk_buffer,
-	 (int) request_size);
+      printf (("attempt to free too much: "
+	       "avail %d used %d failed request %"pD"d\n"),
+	      STATIC_HEAP_SIZE, used, request_size);
       exit (-1);
       return 0;
     }
-  else if (bss_sbrk_ptr + (int) request_size >
-	   bss_sbrk_buffer + STATIC_HEAP_SIZE)
+  else if (STATIC_HEAP_SIZE - used < request_size)
     {
-      printf ("static heap exhausted: avail %d used %d failed request %d\n",
-	      STATIC_HEAP_SIZE,
-	      bss_sbrk_ptr - bss_sbrk_buffer, (int) request_size);
+      printf ("static heap exhausted: avail %d used %d failed request %"pD"d\n",
+	      STATIC_HEAP_SIZE, used, request_size);
       exit (-1);
       return 0;
     }
-  else if ((int) request_size < 0)
-    {
-      bss_sbrk_ptr += (int) request_size;
-      if (debug_sheap)
-	printf ("freed size %d\n", request_size);
-      return bss_sbrk_ptr;
-    }
-  else
+
+  void *ret = bss_sbrk_ptr;
+  bss_sbrk_ptr += request_size;
+  if (max_bss_sbrk_ptr < bss_sbrk_ptr)
+    max_bss_sbrk_ptr = bss_sbrk_ptr;
+  if (debug_sheap)
     {
-      char *ret = bss_sbrk_ptr;
-      if (debug_sheap)
-	printf ("allocated 0x%08x size %d\n", ret, request_size);
-      bss_sbrk_ptr += (int) request_size;
-      if (bss_sbrk_ptr > max_bss_sbrk_ptr)
-	max_bss_sbrk_ptr = bss_sbrk_ptr;
-      return ret;
+      if (request_size < 0)
+	printf ("freed size %"pD"d\n", request_size);
+      else
+	printf ("allocated %p size %"pD"d\n", ret, request_size);
     }
-}
-
-void
-report_sheap_usage (int die_if_pure_storage_exceeded)
-{
-  char buf[200];
-  sprintf (buf, "Maximum static heap usage: %d of %d bytes",
-	   max_bss_sbrk_ptr - bss_sbrk_buffer, STATIC_HEAP_SIZE);
-  /* Don't log messages, cause at this point, we're not allowed to create
-     buffers.  */
-  message1_nolog (buf);
+  return ret;
 }
diff --git a/src/sheap.h b/src/sheap.h
new file mode 100644
index 0000000..4af3cf4
--- /dev/null
+++ b/src/sheap.h
@@ -0,0 +1,31 @@
+/* Static heap allocation for GNU Emacs.
+
+Copyright 2016 Free Software Foundation, Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include <stddef.h>
+
+#ifdef ENABLE_CHECKING
+# define STATIC_HEAP_SIZE (28 * 1024 * 1024)
+#else
+# define STATIC_HEAP_SIZE (19 * 1024 * 1024)
+#endif
+
+extern char bss_sbrk_buffer[STATIC_HEAP_SIZE];
+extern char *max_bss_sbrk_ptr;
+extern bool bss_sbrk_did_unexec;
+extern void *bss_sbrk (ptrdiff_t);
diff --git a/src/sysdep.c b/src/sysdep.c
index 3c3da0d..c224143 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -19,14 +19,6 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include <config.h>
 
-/* If HYBRID_GET_CURRENT_DIR_NAME is defined in conf_post.h, then we
-   need the following before including unistd.h, in order to pick up
-   the right prototype for gget_current_dir_name.  */
-#ifdef HYBRID_GET_CURRENT_DIR_NAME
-#undef get_current_dir_name
-#define get_current_dir_name gget_current_dir_name
-#endif
-
 #include <execinfo.h>
 #include "sysstdio.h"
 #ifdef HAVE_PWD_H
@@ -40,6 +32,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include <utimens.h>
 
 #include "lisp.h"
+#include "sheap.h"
 #include "sysselect.h"
 #include "blockinput.h"
 
@@ -137,14 +130,21 @@ static const int baud_convert[] =
     1800, 2400, 4800, 9600, 19200, 38400
   };
 
-#if !defined HAVE_GET_CURRENT_DIR_NAME || defined BROKEN_GET_CURRENT_DIR_NAME \
-  || (defined HYBRID_GET_CURRENT_DIR_NAME)
-/* Return the current working directory.  Returns NULL on errors.
-   Any other returned value must be freed with free. This is used
-   only when get_current_dir_name is not defined on the system.  */
+/* Return the current working directory.  The result should be freed
+   with 'free'.  Return NULL on errors.  */
 char *
-get_current_dir_name (void)
+emacs_get_current_dir_name (void)
 {
+# if HAVE_GET_CURRENT_DIR_NAME && !BROKEN_GET_CURRENT_DIR_NAME
+#  ifdef HYBRID_MALLOC
+  bool use_libc = bss_sbrk_did_unexec;
+#  else
+  bool use_libc = true;
+#  endif
+  if (use_libc)
+    return get_current_dir_name ();
+# endif
+
   char *buf;
   char *pwd = getenv ("PWD");
   struct stat dotstat, pwdstat;
@@ -192,7 +192,6 @@ get_current_dir_name (void)
     }
   return buf;
 }
-#endif
 
 \f
 /* Discard pending input on all input descriptors.  */
diff --git a/src/xsmfns.c b/src/xsmfns.c
index 7641449..df5c46b 100644
--- a/src/xsmfns.c
+++ b/src/xsmfns.c
@@ -204,7 +204,7 @@ smc_save_yourself_CB (SmcConn smcConn,
   props[props_idx]->vals[0].value = SDATA (user_login_name);
   ++props_idx;
 
-  char *cwd = get_current_dir_name ();
+  char *cwd = emacs_get_current_dir_name ();
   if (cwd)
     {
       props[props_idx] = &prop_ptr[props_idx];
@@ -401,7 +401,7 @@ x_session_initialize (struct x_display_info *dpyinfo)
   ptrdiff_t name_len = 0;
 
   /* libSM seems to crash if pwd is missing - see bug#18851.  */
-  if (! get_current_dir_name ())
+  if (! emacs_get_current_dir_name ())
     {
       fprintf (stderr, "Disabling session management due to pwd error: %s\n",
                emacs_strerror (errno));
-- 
2.5.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-src-alloc.c-Include-sheap.h.patch --]
[-- Type: text/x-diff; name="0008-src-alloc.c-Include-sheap.h.patch", Size: 4345 bytes --]

From 77b034d4eca1d66e678bcd3c9d31dfba503a616d Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:25 -0800
Subject: [PATCH 08/12] * src/alloc.c: Include "sheap.h".
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

(alloc_unexec_pre, alloc_unexec_post) [HYBRID_MALLOC]:
Set and clear bss_sbrk_did_unexec, on all platforms not just Cygwin.
* src/lisp.h (alloc_unexec_pre, alloc_unexec_post) [!DOUG_LEA_MALLOC]:
Declare unconditionally.
* src/unexcw.c, src/unexelf.c (bss_sbrk_did_unexec): Remove decl.
(unexec): Don’t set or clear bss_sbrk_did_unexec;
the caller now does this.
(Bug#22086)
---
 src/alloc.c   | 37 +++++++++++++++++++++++++------------
 src/lisp.h    |  5 -----
 src/unexcw.c  |  4 ----
 src/unexelf.c |  8 --------
 4 files changed, 25 insertions(+), 29 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index d379761..617148e 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -35,6 +35,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "dispextern.h"
 #include "intervals.h"
 #include "puresize.h"
+#include "sheap.h"
 #include "systime.h"
 #include "character.h"
 #include "buffer.h"
@@ -117,18 +118,6 @@ my_heap_start (void)
    inside glibc's malloc.  */
 static void *malloc_state_ptr;
 
-/* Get and free this pointer; useful around unexec.  */
-void
-alloc_unexec_pre (void)
-{
-  malloc_state_ptr = malloc_get_state ();
-}
-void
-alloc_unexec_post (void)
-{
-  free (malloc_state_ptr);
-}
-
 /* Restore the dumped malloc state.  Because malloc can be invoked
    even before main (e.g. by the dynamic linker), the dumped malloc
    state must be restored as early as possible using this special hook.  */
@@ -177,6 +166,30 @@ voidfuncptr __MALLOC_HOOK_VOLATILE __malloc_initialize_hook
 
 #endif
 
+/* Allocator-related actions to do just before and after unexec.  */
+
+void
+alloc_unexec_pre (void)
+{
+#ifdef DOUG_LEA_MALLOC
+  malloc_state_ptr = malloc_get_state ();
+#endif
+#ifdef HYBRID_MALLOC
+  bss_sbrk_did_unexec = true;
+#endif
+}
+
+void
+alloc_unexec_post (void)
+{
+#ifdef DOUG_LEA_MALLOC
+  free (malloc_state_ptr);
+#endif
+#ifdef HYBRID_MALLOC
+  bss_sbrk_did_unexec = false;
+#endif
+}
+
 /* Mark, unmark, query mark bit of a Lisp string.  S must be a pointer
    to a struct Lisp_String.  */
 
diff --git a/src/lisp.h b/src/lisp.h
index 53f123d..3c8e3dd 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3602,13 +3602,8 @@ extern void mark_object (Lisp_Object);
 #if defined REL_ALLOC && !defined SYSTEM_MALLOC && !defined HYBRID_MALLOC
 extern void refill_memory_reserve (void);
 #endif
-#ifdef DOUG_LEA_MALLOC
 extern void alloc_unexec_pre (void);
 extern void alloc_unexec_post (void);
-#else
-INLINE void alloc_unexec_pre (void) {}
-INLINE void alloc_unexec_post (void) {}
-#endif
 extern const char *pending_malloc_warning;
 extern Lisp_Object zero_vector;
 extern Lisp_Object *stack_base;
diff --git a/src/unexcw.c b/src/unexcw.c
index e4aa356..6ebd8c6 100644
--- a/src/unexcw.c
+++ b/src/unexcw.c
@@ -30,8 +30,6 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #define DOTEXE ".exe"
 
-extern int bss_sbrk_did_unexec;
-
 /*
 ** header for Windows executable files
 */
@@ -298,9 +296,7 @@ unexec (const char *outfile, const char *infile)
   ret = emacs_close (fd_in);
   assert (ret == 0);
 
-  bss_sbrk_did_unexec = 1;
   fixup_executable (fd_out);
-  bss_sbrk_did_unexec = 0;
 
   ret = emacs_close (fd_out);
   assert (ret == 0);
diff --git a/src/unexelf.c b/src/unexelf.c
index 3dc0456..e901994 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -211,10 +211,6 @@ entry_address (void *section_h, ptrdiff_t idx, ptrdiff_t entsize)
 
 typedef unsigned char byte;
 
-#ifdef HYBRID_MALLOC
-extern int bss_sbrk_did_unexec;
-#endif
-
 /* ****************************************************************
  * unexec
  *
@@ -231,10 +227,6 @@ unexec (const char *new_name, const char *old_name)
   off_t new_file_size;
   void *new_break;
 
-#ifdef HYBRID_MALLOC
-  bss_sbrk_did_unexec = 1;
-#endif
-
   /* Pointers to the base of the image of the two files.  */
   caddr_t old_base, new_base;
 
-- 
2.5.0


[-- Attachment #10: 0009-Include-malloc.h-when-advisable.patch --]
[-- Type: text/x-diff, Size: 18375 bytes --]

From dc4f8bc0fefa61b00a9d9df662a591d162a50010 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:25 -0800
Subject: [PATCH 09/12] Include <malloc.h> when advisable

This should help insulate us better from future glibc changes.
It is good hygiene to include .h files for APIs that Emacs uses.
Fix type clashes between Emacs and GNU <malloc.h> (Bug#22086).
* configure.ac: Check for malloc.h.
* src/alloc.c: Include <malloc.h> depending on HAVE_MALLOC_H,
not on DOUG_LEA_MALLOC.
* src/emacs.c, src/gmalloc.c (malloc_enable_thread):
Remove decl (now in lisp.h).
* src/gmalloc.c: Include stddef.h earlier, for ptrdiff_t.
[emacs]: Include lisp.h.
[HAVE_MALLOC_H]: Include <malloc.h>.
(__MALLOC_HOOK_VOLATILE): New macro, if not already defined.
(__after_morecore_hook, __malloc_initialize_hook, __morecore)
(__default_morecore):
[!HAVE_MALLOC_H]: New decls near non-inclusion of <malloc.h>.
(calloc): Make it clear that the macro should not be used.
Remove unused decl.
(malloc_info): New macro, to avoid clash with glibc <malloc.h>.
(__morecore, __default_morecore, __after_morecore_hook)
(__malloc_extra_blocks, __malloc_initialize_hook, __free_hook)
(__malloc_hook, __realloc_hook, __memalign_hook, memory_warnings):
Remove later decls.
(gmalloc_hook, gfree_hook, grealloc_hook):
Rename from __malloc_hook, __free_hook, __realloc_hook to
avoid type collision with glibc <malloc.h>.  All uses changed.
(gmalloc_hook):
(__malloc_extra_blocks) [DOUG_LEA_MALLOC||HYBRID_MALLOC||SYSTEM_MALLOC]:
Now static.
(gmalloc_hook, __malloc_extra_blocks): Define even if [!HYBRID_MALLOC].
(__malloc_initialize_hook, __after_morecore_hook):
Declare with types compatible with glibc.
(__memalign_hook, hybrid_calloc) [HYBRID_MALLOC]:
Remove.  All uses removed.
* src/lisp.h (__malloc_extra_blocks, malloc_enable_thread): New decls.
* src/ralloc.c, src/vm-limit.c:
Simplify includes and include <malloc.h> if available.
---
 configure.ac    |   1 +
 src/alloc.c     |   6 ++-
 src/conf_post.h |   1 -
 src/emacs.c     |   2 -
 src/gmalloc.c   | 156 ++++++++++++++++++++++++--------------------------------
 src/lisp.h      |   6 +++
 src/ralloc.c    |  30 +++--------
 src/vm-limit.c  |  11 +++-
 8 files changed, 95 insertions(+), 118 deletions(-)

diff --git a/configure.ac b/configure.ac
index c01e238..fe3a990 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1589,6 +1589,7 @@ fi
 
 dnl checks for header files
 AC_CHECK_HEADERS_ONCE(
+  malloc.h
   sys/systeminfo.h
   sys/sysinfo.h
   coff.h pty.h
diff --git a/src/alloc.c b/src/alloc.c
index 617148e..039b728 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -59,6 +59,10 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "dosfns.h"		/* For dos_memory_info.  */
 #endif
 
+#ifdef HAVE_MALLOC_H
+# include <malloc.h>
+#endif
+
 #if (defined ENABLE_CHECKING			\
      && defined HAVE_VALGRIND_VALGRIND_H	\
      && !defined USE_VALGRIND)
@@ -107,8 +111,6 @@ my_heap_start (void)
 
 #ifdef DOUG_LEA_MALLOC
 
-#include <malloc.h>
-
 /* Specify maximum number of areas to mmap.  It would be nice to use a
    value that explicitly means "no limit".  */
 
diff --git a/src/conf_post.h b/src/conf_post.h
index 9f4becd..c5eec5a 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -99,7 +99,6 @@ typedef bool bool_bf;
 #ifdef emacs
 #define malloc hybrid_malloc
 #define realloc hybrid_realloc
-#define calloc hybrid_calloc
 #define aligned_alloc hybrid_aligned_alloc
 #define free hybrid_free
 #endif
diff --git a/src/emacs.c b/src/emacs.c
index d31ba5a..73c857d 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -134,8 +134,6 @@ bool might_dump;
 extern void unexec_init_emacs_zone (void);
 #endif
 
-extern void malloc_enable_thread (void);
-
 /* If true, Emacs should not attempt to use a window-specific code,
    but instead should use the virtual terminal under which it was started.  */
 bool inhibit_window_system;
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 4fd3246..4feff83 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -25,6 +25,7 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #define USE_PTHREAD
 #endif
 
+#include <stddef.h>
 #include <string.h>
 #include <limits.h>
 #include <stdint.h>
@@ -38,6 +39,26 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #include <w32heap.h>	/* for sbrk */
 #endif
 
+#ifdef emacs
+# include "lisp.h"
+#endif
+
+#ifdef HAVE_MALLOC_H
+# if 4 < __GNUC__ + (2 <= __GNUC_MINOR__)
+#  pragma GCC diagnostic ignored "-Wdeprecated-declarations"
+# endif
+# include <malloc.h>
+#endif
+#ifndef __MALLOC_HOOK_VOLATILE
+# define __MALLOC_HOOK_VOLATILE volatile
+#endif
+#ifndef HAVE_MALLOC_H
+extern void (*__MALLOC_HOOK_VOLATILE __after_morecore_hook) (void);
+extern void (*__MALLOC_HOOK_VOLATILE __malloc_initialize_hook) (void);
+extern void *(*__morecore) (ptrdiff_t);
+extern void *__default_morecore (ptrdiff_t);
+#endif
+
 /* If HYBRID_MALLOC is defined, then temacs will use malloc,
    realloc... as defined in this file (and renamed gmalloc,
    grealloc... via the macros that follow).  The dumped emacs,
@@ -58,9 +79,10 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #undef free
 #define malloc gmalloc
 #define realloc grealloc
-#define calloc gcalloc
+#define calloc do_not_call_me /* Emacs never calls calloc.  */
 #define aligned_alloc galigned_alloc
 #define free gfree
+#define malloc_info gmalloc_info
 
 #ifdef HYBRID_MALLOC
 # include "sheap.h"
@@ -77,15 +99,6 @@ extern "C"
 {
 #endif
 
-#include <stddef.h>
-
-/* Underlying allocation function; successive calls should
-   return contiguous pieces of memory.  */
-extern void *(*__morecore) (ptrdiff_t size);
-
-/* Default value of `__morecore'.  */
-extern void *__default_morecore (ptrdiff_t size);
-
 #ifdef HYBRID_MALLOC
 #define extern static
 #endif
@@ -95,9 +108,7 @@ extern void *malloc (size_t size) ATTRIBUTE_MALLOC_SIZE ((1));
 /* Re-allocate the previously allocated block
    in ptr, making the new block SIZE bytes long.  */
 extern void *realloc (void *ptr, size_t size) ATTRIBUTE_ALLOC_SIZE ((2));
-/* Allocate NMEMB elements of SIZE bytes each, all initialized to 0.  */
-extern void *calloc (size_t nmemb, size_t size) ATTRIBUTE_MALLOC_SIZE ((1,2));
-/* Free a block allocated by `malloc', `realloc' or `calloc'.  */
+/* Free a block.  */
 extern void free (void *ptr);
 
 /* Allocate SIZE bytes allocated to ALIGNMENT bytes.  */
@@ -107,11 +118,6 @@ extern void *memalign (size_t, size_t);
 extern int posix_memalign (void **, size_t, size_t);
 #endif
 
-#ifdef USE_PTHREAD
-/* Set up mutexes and make malloc etc. thread-safe.  */
-extern void malloc_enable_thread (void);
-#endif
-
 /* The allocator divides the heap into blocks of fixed size; large
    requests receive one or more whole blocks, and small requests
    receive a fragment of a block.  Fragment sizes are powers of two,
@@ -243,26 +249,11 @@ extern int _malloc_thread_enabled_p;
 #define UNLOCK_ALIGNED_BLOCKS()
 #endif
 
-/* If not NULL, this function is called after each time
-   `__morecore' is called to increase the data size.  */
-extern void (*__after_morecore_hook) (void);
-
-/* Number of extra blocks to get each time we ask for more core.
-   This reduces the frequency of calling `(*__morecore)'.  */
-extern size_t __malloc_extra_blocks;
-
 /* Nonzero if `malloc' has been called and done its initialization.  */
 extern int __malloc_initialized;
 /* Function called to initialize malloc data structures.  */
 extern int __malloc_initialize (void);
 
-/* Hooks for debugging versions.  */
-extern void (*__malloc_initialize_hook) (void);
-extern void (*__free_hook) (void *ptr);
-extern void *(*__malloc_hook) (size_t size);
-extern void *(*__realloc_hook) (void *ptr, size_t size);
-extern void *(*__memalign_hook) (size_t size, size_t alignment);
-
 #ifdef GC_MCHECK
 
 /* Return values for `mprobe': these are the kinds of inconsistencies that
@@ -304,9 +295,6 @@ struct mstats
 /* Pick up the current statistics. */
 extern struct mstats mstats (void);
 
-/* Call WARNFUN with a warning message when memory usage is high.  */
-extern void memory_warnings (void *start, void (*warnfun) (const char *));
-
 #endif
 
 #undef extern
@@ -337,13 +325,11 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 
 #include <errno.h>
 
-void *(*__morecore) (ptrdiff_t size) = __default_morecore;
+/* Debugging hook for 'malloc'.  */
+static void *(*__MALLOC_HOOK_VOLATILE gmalloc_hook) (size_t);
 
 #ifndef HYBRID_MALLOC
 
-/* Debugging hook for `malloc'.  */
-void *(*__malloc_hook) (size_t size);
-
 /* Pointer to the base of the first block.  */
 char *_heapbase;
 
@@ -368,10 +354,9 @@ size_t _bytes_free;
 /* Are you experienced?  */
 int __malloc_initialized;
 
-size_t __malloc_extra_blocks;
-
-void (*__malloc_initialize_hook) (void);
-void (*__after_morecore_hook) (void);
+void (*__MALLOC_HOOK_VOLATILE __malloc_initialize_hook) (void);
+void (*__MALLOC_HOOK_VOLATILE __after_morecore_hook) (void);
+void *(*__morecore) (ptrdiff_t);
 
 #else
 
@@ -379,6 +364,13 @@ static struct list _fraghead[BLOCKLOG];
 
 #endif /* HYBRID_MALLOC */
 
+/* Number of extra blocks to get each time we ask for more core.
+   This reduces the frequency of calling `(*__morecore)'.  */
+#if defined DOUG_LEA_MALLOC || defined HYBRID_MALLOC || defined SYSTEM_MALLOC
+static
+#endif
+size_t __malloc_extra_blocks;
+
 /* Number of info entries.  */
 static size_t heapsize;
 
@@ -935,15 +927,15 @@ malloc (size_t size)
   if (!__malloc_initialized && !__malloc_initialize ())
     return NULL;
 
-  /* Copy the value of __malloc_hook to an automatic variable in case
-     __malloc_hook is modified in another thread between its
+  /* Copy the value of gmalloc_hook to an automatic variable in case
+     gmalloc_hook is modified in another thread between its
      NULL-check and the use.
 
      Note: Strictly speaking, this is not a right solution.  We should
      use mutexes to access non-read-only variables that are shared
      among multiple threads.  We just leave it for compatibility with
-     glibc malloc (i.e., assignments to __malloc_hook) for now.  */
-  hook = __malloc_hook;
+     glibc malloc (i.e., assignments to gmalloc_hook) for now.  */
+  hook = gmalloc_hook;
   return (hook != NULL ? *hook : _malloc_internal) (size);
 }
 \f
@@ -995,10 +987,10 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
    The author may be reached (Email) at the address mike@ai.mit.edu,
    or (US mail) as Mike Haertel c/o Free Software Foundation.  */
 
+/* Debugging hook for free.  */
+static void (*__MALLOC_HOOK_VOLATILE gfree_hook) (void *);
 
 #ifndef HYBRID_MALLOC
-/* Debugging hook for free.  */
-void (*__free_hook) (void *__ptr);
 
 /* List of blocks allocated by aligned_alloc.  */
 struct alignlist *_aligned_blocks = NULL;
@@ -1251,7 +1243,7 @@ _free_internal_nolock (void *ptr)
 }
 
 /* Return memory to the heap.
-   Like `free' but don't call a __free_hook if there is one.  */
+   Like 'free' but don't call a hook if there is one.  */
 void
 _free_internal (void *ptr)
 {
@@ -1265,7 +1257,7 @@ _free_internal (void *ptr)
 void
 free (void *ptr)
 {
-  void (*hook) (void *) = __free_hook;
+  void (*hook) (void *) = gfree_hook;
 
   if (hook != NULL)
     (*hook) (ptr);
@@ -1309,10 +1301,8 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 #define min(a, b) ((a) < (b) ? (a) : (b))
 #endif
 
-#ifndef HYBRID_MALLOC
 /* Debugging hook for realloc.  */
-void *(*__realloc_hook) (void *ptr, size_t size);
-#endif
+static void *(*grealloc_hook) (void *, size_t);
 
 /* Resize the given region to the new size, returning a pointer
    to the (possibly moved) region.  This is optimized for speed;
@@ -1456,7 +1446,7 @@ realloc (void *ptr, size_t size)
   if (!__malloc_initialized && !__malloc_initialize ())
     return NULL;
 
-  hook = __realloc_hook;
+  hook = grealloc_hook;
   return (hook != NULL ? *hook : _realloc_internal) (ptr, size);
 }
 /* Copyright (C) 1991, 1992, 1994 Free Software Foundation, Inc.
@@ -1479,6 +1469,7 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 
 /* Allocate an array of NMEMB elements each SIZE bytes long.
    The entire array is initialized to zeros.  */
+#ifndef calloc
 void *
 calloc (size_t nmemb, size_t size)
 {
@@ -1496,6 +1487,7 @@ calloc (size_t nmemb, size_t size)
     return memset (result, 0, bytes);
   return result;
 }
+#endif
 /* Copyright (C) 1991, 1992, 1993, 1994, 1995 Free Software Foundation, Inc.
 This file is part of the GNU C Library.
 
@@ -1541,6 +1533,9 @@ __default_morecore (ptrdiff_t increment)
     return NULL;
   return result;
 }
+
+void *(*__morecore) (ptrdiff_t) = __default_morecore;
+
 /* Copyright (C) 1991, 92, 93, 94, 95, 96 Free Software Foundation, Inc.
 
 This library is free software; you can redistribute it and/or
@@ -1556,19 +1551,11 @@ General Public License for more details.
 You should have received a copy of the GNU General Public
 License along with this library.  If not, see <http://www.gnu.org/licenses/>.  */
 
-#ifndef HYBRID_MALLOC
-void *(*__memalign_hook) (size_t size, size_t alignment);
-#endif
-
 void *
 aligned_alloc (size_t alignment, size_t size)
 {
   void *result;
   size_t adj, lastadj;
-  void *(*hook) (size_t, size_t) = __memalign_hook;
-
-  if (hook)
-    return (*hook) (alignment, size);
 
   /* Allocate a block with enough extra space to pad the block with up to
      (ALIGNMENT - 1) bytes if necessary.  */
@@ -1731,7 +1718,6 @@ valloc (size_t size)
 /* Declare system malloc and friends.  */
 extern void *malloc (size_t size);
 extern void *realloc (void *ptr, size_t size);
-extern void *calloc (size_t nmemb, size_t size);
 extern void free (void *ptr);
 #ifdef HAVE_ALIGNED_ALLOC
 extern void *aligned_alloc (size_t alignment, size_t size);
@@ -1750,14 +1736,6 @@ hybrid_malloc (size_t size)
   return gmalloc (size);
 }
 
-void *
-hybrid_calloc (size_t nmemb, size_t size)
-{
-  if (DUMPED)
-    return calloc (nmemb, size);
-  return gcalloc (nmemb, size);
-}
-
 void
 hybrid_free (void *ptr)
 {
@@ -1947,9 +1925,9 @@ freehook (void *ptr)
   else
     hdr = NULL;
 
-  __free_hook = old_free_hook;
+  gfree_hook = old_free_hook;
   free (hdr);
-  __free_hook = freehook;
+  gfree_hook = freehook;
 }
 
 static void *
@@ -1957,9 +1935,9 @@ mallochook (size_t size)
 {
   struct hdr *hdr;
 
-  __malloc_hook = old_malloc_hook;
+  gmalloc_hook = old_malloc_hook;
   hdr = malloc (sizeof *hdr + size + 1);
-  __malloc_hook = mallochook;
+  gmalloc_hook = mallochook;
   if (hdr == NULL)
     return NULL;
 
@@ -1985,13 +1963,13 @@ reallochook (void *ptr, size_t size)
 	memset ((char *) ptr + size, FREEFLOOD, osize - size);
     }
 
-  __free_hook = old_free_hook;
-  __malloc_hook = old_malloc_hook;
-  __realloc_hook = old_realloc_hook;
+  gfree_hook = old_free_hook;
+  gmalloc_hook = old_malloc_hook;
+  grealloc_hook = old_realloc_hook;
   hdr = realloc (hdr, sizeof *hdr + size + 1);
-  __free_hook = freehook;
-  __malloc_hook = mallochook;
-  __realloc_hook = reallochook;
+  gfree_hook = freehook;
+  gmalloc_hook = mallochook;
+  grealloc_hook = reallochook;
   if (hdr == NULL)
     return NULL;
 
@@ -2048,12 +2026,12 @@ mcheck (void (*func) (enum mcheck_status))
   /* These hooks may not be safely inserted if malloc is already in use.  */
   if (!__malloc_initialized && !mcheck_used)
     {
-      old_free_hook = __free_hook;
-      __free_hook = freehook;
-      old_malloc_hook = __malloc_hook;
-      __malloc_hook = mallochook;
-      old_realloc_hook = __realloc_hook;
-      __realloc_hook = reallochook;
+      old_free_hook = gfree_hook;
+      gfree_hook = freehook;
+      old_malloc_hook = gmalloc_hook;
+      gmalloc_hook = mallochook;
+      old_realloc_hook = grealloc_hook;
+      grealloc_hook = reallochook;
       mcheck_used = 1;
     }
 
diff --git a/src/lisp.h b/src/lisp.h
index 3c8e3dd..4f4ec2c 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3769,6 +3769,12 @@ extern void check_cons_list (void);
 INLINE void (check_cons_list) (void) { lisp_h_check_cons_list (); }
 #endif
 
+#if !defined DOUG_LEA_MALLOC && !defined HYBRID_MALLOC && !defined SYSTEM_MALLOC
+/* Defined in gmalloc.c.  */
+extern size_t __malloc_extra_blocks;
+#endif
+extern void malloc_enable_thread (void);
+
 #ifdef REL_ALLOC
 /* Defined in ralloc.c.  */
 extern void *r_alloc (void **, size_t) ATTRIBUTE_ALLOC_SIZE ((2));
diff --git a/src/ralloc.c b/src/ralloc.c
index 12d2fa9..d1a9e01 100644
--- a/src/ralloc.c
+++ b/src/ralloc.c
@@ -22,31 +22,15 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
    rather than all of them.  This means allowing for a possible
    hole between the first bloc and the end of malloc storage.  */
 
-#ifdef emacs
-
 #include <config.h>
 
-#include "lisp.h"		/* Needed for VALBITS.  */
-#include "blockinput.h"
-
-#include <unistd.h>
-
-#ifdef DOUG_LEA_MALLOC
-#define M_TOP_PAD           -2
-extern int mallopt (int, int);
-#else /* not DOUG_LEA_MALLOC */
-#if !defined SYSTEM_MALLOC && !defined HYBRID_MALLOC
-extern size_t __malloc_extra_blocks;
-#endif /* not SYSTEM_MALLOC and not HYBRID_MALLOC */
-#endif /* not DOUG_LEA_MALLOC */
-
-#else /* not emacs */
-
 #include <stddef.h>
-#include <malloc.h>
-
-#endif	/* not emacs */
 
+#ifdef emacs
+# include "lisp.h"
+# include "blockinput.h"
+# include <unistd.h>
+#endif
 
 #include "getpagesize.h"
 
@@ -95,7 +79,9 @@ static int extra_bytes;
 /* The hook `malloc' uses for the function which gets more space
    from the system.  */
 
-#if !defined SYSTEM_MALLOC && !defined HYBRID_MALLOC
+#ifdef HAVE_MALLOC_H
+# include <malloc.h>
+#else
 extern void *(*__morecore) (ptrdiff_t);
 #endif
 
diff --git a/src/vm-limit.c b/src/vm-limit.c
index 0c6dbdd..42f0470 100644
--- a/src/vm-limit.c
+++ b/src/vm-limit.c
@@ -51,9 +51,16 @@ char data_start[1] = { 1 };
 # endif
 #endif
 
-/* From gmalloc.c.  */
-extern void (* __after_morecore_hook) (void);
+#ifdef HAVE_MALLOC_H
+# include <malloc.h>
+#endif
+#ifndef __MALLOC_HOOK_VOLATILE
+# define __MALLOC_HOOK_VOLATILE volatile
+#endif
+#ifndef HAVE_MALLOC_H
 extern void *(*__morecore) (ptrdiff_t);
+extern void (*__MALLOC_HOOK_VOLATILE __after_morecore_hook) (void);
+#endif
 
 /* From ralloc.c.  */
 #ifdef REL_ALLOC
-- 
2.5.0


[-- Attachment #11: 0010-Build-lib-e-.o-only-on-platforms-that-need-it.patch --]
[-- Type: text/x-diff, Size: 7963 bytes --]

From 925eb6dc43fe4a7fd3ad90caf5dd29101c7577a8 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:26 -0800
Subject: [PATCH 10/12] Build lib/e-*.o only on platforms that need it

* configure.ac (hybrid malloc): Simplify configuration.
(SHEAP_OBJ): Remove; no longer needed.
(HYBRID_MALLOC): New var. Subst it.
(HYBRID_MALLOC_LIB): New Automake conditional.
* lib/Makefile.am (noinst_LIBRARIES): Add libegnu.a only if
HYBRID_MALLOC_LIB.
(libegnu_a_CPPFLAGS): Omit AM_CPPFLAGS; not needed.
(MOSTLYCLEANFILES): Add libegnu.a.
* src/Makefile.in (SHEAP_OBJ): Remove.
(HYBRID_MALLOC): New macro.
(base_obj): Use it to conditionally add sheap.o.
(LIBEGNU_ARCHIVE): New macro.
($(LIBEGNU_ARCHIVE)): New rule, replacing $(lib)/libegnu.a.
All uses of the latter replaced by the former.
* src/alloc.c (USE_ALIGNED_ALLOC): Simplify configuration.
Correct misspelling ALIGNED_ALLOC to HAVE_ALIGNED_ALLOC.
* src/gmalloc.c: Update comment.
* src/lisp.h (aligned_alloc)
[!DOUG_LEA_MALLOC && !HYBRID_MALLOC && !SYSTEM_MALLOC]:
New decl.
(Bug#22086)
---
 configure.ac    |  9 +++++----
 lib/Makefile.am | 23 +++++++++++++++++++++--
 src/Makefile.in | 12 +++++++-----
 src/alloc.c     | 13 ++++---------
 src/gmalloc.c   |  8 +-------
 src/lisp.h      |  1 +
 6 files changed, 39 insertions(+), 27 deletions(-)

diff --git a/configure.ac b/configure.ac
index fe3a990..3d70893 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2123,12 +2123,12 @@ case "$opsys" in
 esac
 
 if test "${system_malloc}" != yes && test "${doug_lea_malloc}" != yes \
-   && test "${UNEXEC_OBJ}" = unexelf.o && test "${hybrid_malloc}" != no; then
+   && test "${UNEXEC_OBJ}" = unexelf.o; then
   hybrid_malloc=yes
 fi
 
 GMALLOC_OBJ=
-SHEAP_OBJ=
+HYBRID_MALLOC=
 if test "${system_malloc}" = "yes"; then
   AC_DEFINE([SYSTEM_MALLOC], 1,
     [Define to 1 to use the system memory allocator, even if it is not
@@ -2140,10 +2140,10 @@ if test "${system_malloc}" = "yes"; then
 elif test "$hybrid_malloc" = yes; then
   AC_DEFINE(HYBRID_MALLOC, 1,
     [Define to use gmalloc before dumping and the system malloc after.])
+  HYBRID_MALLOC=1
   GNU_MALLOC=
   GNU_MALLOC_reason="only before dumping"
   GMALLOC_OBJ=gmalloc.o
-  SHEAP_OBJ=sheap.o
   VMLIMIT_OBJ=
 else
   test "$doug_lea_malloc" != "yes" && GMALLOC_OBJ=gmalloc.o
@@ -2163,8 +2163,9 @@ else
        of the main data segment.])
   fi
 fi
+AC_SUBST([HYBRID_MALLOC])
+AM_CONDITIONAL([HYBRID_MALLOC_LIB], [test -n "$HYBRID_MALLOC"])
 AC_SUBST(GMALLOC_OBJ)
-AC_SUBST(SHEAP_OBJ)
 AC_SUBST(VMLIMIT_OBJ)
 
 if test "$doug_lea_malloc" = "yes" && test "$hybrid_malloc" != yes; then
diff --git a/lib/Makefile.am b/lib/Makefile.am
index a1dd6a4..74bab4e 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -1,3 +1,20 @@
+# Copyright 2016 Free Software Foundation, Inc.
+
+# This file is part of GNU Emacs.
+
+# GNU Emacs is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# GNU Emacs is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
+
 BUILT_SOURCES =
 CLEANFILES =
 EXTRA_DIST =
@@ -17,14 +34,16 @@ include gnulib.mk
 libgnu_a_SOURCES += openat-die.c save-cwd.c
 endif
 
+if HYBRID_MALLOC_LIB
 noinst_LIBRARIES += libegnu.a
+endif
 
 libegnu_a_SOURCES = $(libgnu_a_SOURCES)
 libegnu_a_LIBADD = $(patsubst %.o,e-%.o,$(libgnu_a_LIBADD))
 EXTRA_libegnu_a_SOURCES = $(EXTRA_libgnu_a_SOURCES)
 libegnu_a_SHORTNAME = e
-
-libegnu_a_CPPFLAGS = $(AM_CPPFLAGS) -Demacs
+libegnu_a_CPPFLAGS = -Demacs
+MOSTLYCLEANFILES += libegnu.a
 
 .PHONY: bootstrap-clean
 
diff --git a/src/Makefile.in b/src/Makefile.in
index 25dc3f8..cc9a6e9 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -254,8 +254,7 @@ XFIXES_CFLAGS = @XFIXES_CFLAGS@
 ## widget.o if USE_X_TOOLKIT, otherwise empty.
 WIDGET_OBJ=@WIDGET_OBJ@
 
-## sheap.o if HYBRID_MALLOC, otherwise empty.
-SHEAP_OBJ=@SHEAP_OBJ@
+HYBRID_MALLOC = @HYBRID_MALLOC@
 
 ## cygw32.o if CYGWIN, otherwise empty.
 CYGWIN_OBJ=@CYGWIN_OBJ@
@@ -400,6 +399,7 @@ base_obj = dispnew.o frame.o scroll.o xdisp.o menu.o $(XMENU_OBJ) window.o \
 	doprnt.o intervals.o textprop.o composite.o xml.o $(NOTIFY_OBJ) \
 	$(XWIDGETS_OBJ) \
 	profiler.o decompress.o \
+	$(if $(HYBRID_MALLOC),sheap.o) \
 	$(SHEAP_OBJ) \
 	$(MSDOS_OBJ) $(MSDOS_X_OBJ) $(NS_OBJ) $(CYGWIN_OBJ) $(FONT_OBJ) \
 	$(W32_OBJ) $(WINDOW_SYSTEM_OBJ) $(XGSELOBJ)
@@ -580,7 +580,9 @@ globals.h: gl-stamp; @true
 
 $(ALLOBJS): globals.h
 
-$(lib)/libegnu.a: $(config_h)
+LIBEGNU_ARCHIVE = $(lib)/lib$(if $(HYBRID_MALLOC),e)gnu.a
+
+$(LIBEGNU_ARCHIVE): $(config_h)
 	$(MAKE) -C $(lib) all
 
 ## We have to create $(etc) here because init_cmdargs tests its
@@ -588,9 +590,9 @@ $(lib)/libegnu.a: $(config_h)
 ## This goes on to affect various things, and the emacs binary fails
 ## to start if Vinstallation_directory has the wrong value.
 temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) \
-	         $(lib)/libegnu.a $(EMACSRES) ${charsets} ${charscript}
+	         $(LIBEGNU_ARCHIVE) $(EMACSRES) ${charsets} ${charscript}
 	$(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
-	  -o temacs $(ALLOBJS) $(lib)/libegnu.a $(W32_RES_LINK) $(LIBES)
+	  -o temacs $(ALLOBJS) $(LIBEGNU_ARCHIVE) $(W32_RES_LINK) $(LIBES)
 	$(MKDIR_P) $(etc)
 ifneq ($(CANNOT_DUMP),yes)
 	$(PAXCTL_if_present) -r $@
diff --git a/src/alloc.c b/src/alloc.c
index 039b728..b1d3f2e 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -1123,17 +1123,12 @@ lisp_free (void *block)
    clang 3.3 anyway.  */
 
 #if ! ADDRESS_SANITIZER
-# if !defined SYSTEM_MALLOC && !defined DOUG_LEA_MALLOC && !defined HYBRID_MALLOC
-#  define USE_ALIGNED_ALLOC 1
-/* Defined in gmalloc.c.  */
-void *aligned_alloc (size_t, size_t);
-# elif defined HYBRID_MALLOC
-#  if defined ALIGNED_ALLOC || defined HAVE_POSIX_MEMALIGN
+# if defined HYBRID_MALLOC
+#  if defined HAVE_ALIGNED_ALLOC || defined HAVE_POSIX_MEMALIGN
 #   define USE_ALIGNED_ALLOC 1
-#   define aligned_alloc hybrid_aligned_alloc
-/* Defined in gmalloc.c.  */
-void *aligned_alloc (size_t, size_t);
 #  endif
+# elif !defined SYSTEM_MALLOC && !defined DOUG_LEA_MALLOC
+#  define USE_ALIGNED_ALLOC 1
 # elif defined HAVE_ALIGNED_ALLOC
 #  define USE_ALIGNED_ALLOC 1
 # elif defined HAVE_POSIX_MEMALIGN
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 4feff83..ca86276 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -65,13 +65,7 @@ extern void *__default_morecore (ptrdiff_t);
    however, will use the system malloc, realloc....  In other source
    files, malloc, realloc... are renamed hybrid_malloc,
    hybrid_realloc... via macros in conf_post.h.  hybrid_malloc and
-   friends are wrapper functions defined later in this file.
-   aligned_alloc is defined as a macro only in alloc.c.
-
-   As of this writing (August 2014), Cygwin is the only platform on
-   which HYBRID_MACRO is defined.  Any other platform that wants to
-   define it will have to define the macros DUMPED and
-   ALLOCATED_BEFORE_DUMPING, defined below for Cygwin.  */
+   friends are wrapper functions defined later in this file.  */
 #undef malloc
 #undef realloc
 #undef calloc
diff --git a/src/lisp.h b/src/lisp.h
index 4f4ec2c..cafcfde 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3772,6 +3772,7 @@ INLINE void (check_cons_list) (void) { lisp_h_check_cons_list (); }
 #if !defined DOUG_LEA_MALLOC && !defined HYBRID_MALLOC && !defined SYSTEM_MALLOC
 /* Defined in gmalloc.c.  */
 extern size_t __malloc_extra_blocks;
+extern void *aligned_alloc (size_t, size_t);
 #endif
 extern void malloc_enable_thread (void);
 
-- 
2.5.0


[-- Attachment #12: 0011-Fix-extern-symbols-defined-and-not-used.patch --]
[-- Type: text/x-diff, Size: 4331 bytes --]

From 8834194fddbeb093f6ffda952a13316c7a6faaa3 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:26 -0800
Subject: [PATCH 11/12] Fix extern symbols defined and not used

* src/alloc.c: Always include <signal.h>.
(malloc_warning) [!SIGDANGER && (SYSTEM_MALLOC || HYBRID_MALLOC)]:
Do not define; unused.
* src/emacs.c, src/lisp.h (might_dump) [!DOUG_LEA_MALLOC]: Now static.
* src/gmalloc.c (gdefault_morecore): Rename from __default_morecore,
to avoid collision with glibc.  Now static.  All uses changed.
* src/lastfile.c (my_edata): Define only if
((!defined SYSTEM_MALLOC && !defined HYBRID_MALLOC && !defined
WINDOWSNT) \ || defined CYGWIN || defined DARWIN_OS).
(Bug#22086)
---
 src/alloc.c    | 8 ++++----
 src/emacs.c    | 3 +++
 src/gmalloc.c  | 7 +++----
 src/lastfile.c | 3 +++
 src/lisp.h     | 2 ++
 5 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index b1d3f2e..57ef4c5 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -22,10 +22,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include <stdio.h>
 #include <limits.h>		/* For CHAR_BIT.  */
-
-#ifdef ENABLE_CHECKING
-#include <signal.h>		/* For SIGABRT.  */
-#endif
+#include <signal.h>		/* For SIGABRT, SIGDANGER.  */
 
 #ifdef HAVE_PTHREAD
 #include <pthread.h>
@@ -565,6 +562,8 @@ static struct Lisp_Finalizer doomed_finalizers;
 				Malloc
  ************************************************************************/
 
+#if defined SIGDANGER || (!defined SYSTEM_MALLOC && !defined HYBRID_MALLOC)
+
 /* Function malloc calls this if it finds we are near exhausting storage.  */
 
 void
@@ -573,6 +572,7 @@ malloc_warning (const char *str)
   pending_malloc_warning = str;
 }
 
+#endif
 
 /* Display an already-pending malloc warning.  */
 
diff --git a/src/emacs.c b/src/emacs.c
index 73c857d..c65c76c 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -128,6 +128,9 @@ Lisp_Object Vlibrary_cache;
 bool initialized;
 
 /* Set to true if this instance of Emacs might dump.  */
+#ifndef DOUG_LEA_MALLOC
+static
+#endif
 bool might_dump;
 
 #ifdef DARWIN_OS
diff --git a/src/gmalloc.c b/src/gmalloc.c
index ca86276..282216a 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -56,7 +56,6 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 extern void (*__MALLOC_HOOK_VOLATILE __after_morecore_hook) (void);
 extern void (*__MALLOC_HOOK_VOLATILE __malloc_initialize_hook) (void);
 extern void *(*__morecore) (ptrdiff_t);
-extern void *__default_morecore (ptrdiff_t);
 #endif
 
 /* If HYBRID_MALLOC is defined, then temacs will use malloc,
@@ -1512,8 +1511,8 @@ extern void *__sbrk (ptrdiff_t increment);
 /* Allocate INCREMENT more bytes of data space,
    and return the start of data space, or NULL on errors.
    If INCREMENT is negative, shrink data space.  */
-void *
-__default_morecore (ptrdiff_t increment)
+static void *
+gdefault_morecore (ptrdiff_t increment)
 {
   void *result;
 #ifdef HYBRID_MALLOC
@@ -1528,7 +1527,7 @@ __default_morecore (ptrdiff_t increment)
   return result;
 }
 
-void *(*__morecore) (ptrdiff_t) = __default_morecore;
+void *(*__morecore) (ptrdiff_t) = gdefault_morecore;
 
 /* Copyright (C) 1991, 92, 93, 94, 95, 96 Free Software Foundation, Inc.
 
diff --git a/src/lastfile.c b/src/lastfile.c
index 2d0bcc7..9c73fb4 100644
--- a/src/lastfile.c
+++ b/src/lastfile.c
@@ -38,7 +38,10 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include "lisp.h"
 
+#if ((!defined SYSTEM_MALLOC && !defined HYBRID_MALLOC && !defined WINDOWSNT) \
+     || defined CYGWIN || defined DARWIN_OS)
 char my_edata[] = "End of Emacs initialized data";
+#endif
 
 /* Help unexec locate the end of the .bss area used by Emacs (which
    isn't always a separate section in NT executables).  */
diff --git a/src/lisp.h b/src/lisp.h
index cafcfde..ef86c4f 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -619,7 +619,9 @@ extern _Noreturn Lisp_Object wrong_type_argument (Lisp_Object, Lisp_Object);
 extern _Noreturn void wrong_choice (Lisp_Object, Lisp_Object);
 
 /* Defined in emacs.c.  */
+#ifdef DOUG_LEA_MALLOC
 extern bool might_dump;
+#endif
 /* True means Emacs has already been initialized.
    Used during startup to detect startup of dumped Emacs.  */
 extern bool initialized;
-- 
2.5.0


[-- Attachment #13: 0012-Shrink-static-heap-a-bit.patch --]
[-- Type: text/x-diff, Size: 1260 bytes --]

From ab3300b50479d5f8a5ca5c461f53e8e7eb4443f8 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 30 Jan 2016 00:47:26 -0800
Subject: [PATCH 12/12] Shrink static heap a bit

* src/sheap.h: Include lisp.h, for Lisp_Object.
(STATIC_HEAP_SIZE): Now an enum constant, not a macro.
Make it 2 MiB * sizeof (Lisp_Object), which is a bit more
conservative than the old value.
(Bug#22086)
---
 src/sheap.h | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/sheap.h b/src/sheap.h
index 4af3cf4..db059d2 100644
--- a/src/sheap.h
+++ b/src/sheap.h
@@ -18,12 +18,12 @@ You should have received a copy of the GNU General Public License
 along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include <stddef.h>
+#include "lisp.h"
 
-#ifdef ENABLE_CHECKING
-# define STATIC_HEAP_SIZE (28 * 1024 * 1024)
-#else
-# define STATIC_HEAP_SIZE (19 * 1024 * 1024)
-#endif
+/* Size of the static heap.  Guess a value that is probably too large,
+   by up to a factor of two or so.  Typically the unused part is not
+   paged in and so does not cost much.  */
+enum { STATIC_HEAP_SIZE = sizeof (Lisp_Object) << 21 };
 
 extern char bss_sbrk_buffer[STATIC_HEAP_SIZE];
 extern char *max_bss_sbrk_ptr;
-- 
2.5.0

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Eli Zaretskii]

> Cc: 22086@debbugs.gnu.org, Rich Felker <dalias@aerifal.cx>,
>  Daniel Colascione <dancol@dancol.org>, Ken Brown <kbrown@cornell.edu>,
>  Eli Zaretskii <eliz@gnu.org>
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 30 Jan 2016 01:17:21 -0800
> 
> The recent emacs-devel thread "Removal of unexec support" has raised the 
> priority of this bug, so I redid the patches to separate out Rich Felker's 
> contribution, which is so small as to not require copyright papers, and fixed 
> several problems I found with the resulting approach. I came up with the 
> attached set of patches relative to commit 
> ef760b899ad89f941f552ed2d3ac9e45156f3e3c. I would like to commit this patch set 
> to the emacs-25 branch soon, and am sending this email to give you (particularly 
> Eli) a heads-up about this.

I'm sorry, I can't afford testing this large patchset at this time,
while preparations to the pretest are under way.  I also don't think
such pervasive changes should be done on the release branch.  I think
when the time comes you should commit this to the master branch, not
to emacs-25.  In the meantime, a public feature branch will allow
people to try it and see if it breaks anything.

Thanks.

> These patches attempt to be more conservative than the other alternatives 
> discussed in Bug#22086. They don't try to build a better dumper or remove 
> gmalloc.c or anything like that. All they try to do, is to disentangle Emacs 
> from glibc malloc internals, by renaming functions whose APIs are no longer 
> compatible with glibc, and by using glibc's <malloc.h> rather than guessing what 
> it will say, and that sort of thing. The goal is for the resulting Emacs to not 
> only port to musl, but also to port to future glibc with less likelihood of trouble.

As the issue with glibc is only relevant to GNU/Linux systems, I
wonder if a solution that is limited to some file(s) specific to that
target could be possible.  Even then I'd hesitate to do this on the
release branch, since making Emacs less stable even on that single
platform will most probably delay the v25.1 release too much -- we
cannot possibly release Emacs 25 that is not stable on GNU/Linux.  But
at least we won't need to review and debug the results of this on
platforms that don't need these changes at all.

The glibc guys said the change won't happen too soon, so I see no
reason to delay Emacs 25 on behalf of this issue.

Thanks again for doing this work.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Paul Eggert]

Eli Zaretskii wrote:
> I think
> when the time comes you should commit this to the master branch, not
> to emacs-25.

I think it's ready enough for the master branch, so I committed it there and 
will mark this bug as done.

I had to do a merge first, which was pretty painful, but that should be a 
different thread.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Rich Felker]

On Sat, Jan 30, 2016 at 01:17:21AM -0800, Paul Eggert wrote:
> The recent emacs-devel thread "Removal of unexec support" has raised
> the priority of this bug, so I redid the patches to separate out
> Rich Felker's contribution, which is so small as to not require
> copyright papers, and fixed several problems I found with the
> resulting approach. I came up with the attached set of patches
> relative to commit ef760b899ad89f941f552ed2d3ac9e45156f3e3c. I would
> like to commit this patch set to the emacs-25 branch soon, and am
> sending this email to give you (particularly Eli) a heads-up about
> this.

Hey! I'm very sorry I dropped the ball on assignment papers. I've been
busy with a lot of other stuff and forgot that was still pending. Let
me know if you'd still prefer to have them, but I agree totally that
my part of the patch was trivial and it was my opinion from the
beginning that I had no copyright claim to it.

Rich

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Wolfgang Jenkner]

On Sat, Jan 30 2016, Paul Eggert wrote:

> I think it's ready enough for the master branch, so I committed it
> there and will mark this bug as done.

Thanks, I think patch 5 of my OP (for turning off direct use of mmap for
buffer allocations on FreeBSD) should also be applied.

When I do this, the build is correctly configured for hybrid malloc on
such a system, but linking temacs now fails with

  CCLD     temacs
gmalloc.o: In function `align':
/usr/opt/src/emacs-paul-test/src/gmalloc.c:440: undefined reference to `__after_morecore_hook'
/usr/opt/src/emacs-paul-test/src/gmalloc.c:441: undefined reference to `__after_morecore_hook'
gmalloc.o: In function `malloc_initialize_1':
/usr/opt/src/emacs-paul-test/src/gmalloc.c:553: undefined reference to `__malloc_initialize_hook'
/usr/opt/src/emacs-paul-test/src/gmalloc.c:554: undefined reference to `__malloc_initialize_hook'
collect2: error: ld returned 1 exit status
Makefile:594: recipe for target 'temacs' failed

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

Wolfgang Jenkner wrote:

> Thanks, I think patch 5 of my OP (for turning off direct use of mmap for
> buffer allocations on FreeBSD) should also be applied.

Sorry, I missed that one. I don't recall why it's needed but I'll take your word 
for it. Done.

> /usr/opt/src/emacs-paul-test/src/gmalloc.c:440: undefined reference to `__after_morecore_hook'
> ...
> /usr/opt/src/emacs-paul-test/src/gmalloc.c:553: undefined reference to `__malloc_initialize_hook'

I installed the attached patch to try to fix these; please give it a try.


[-- Attachment #2: 0001-Port-new-hybrid-malloc-to-FreeBSD.patch --]
[-- Type: text/x-diff, Size: 1559 bytes --]

From 1005257b104d2dac80479eac2c98246ad29147ec Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 31 Jan 2016 09:50:07 -0800
Subject: [PATCH] Port new hybrid malloc to FreeBSD

Problem reported by Wolfgang Jenkner in: http://bugs.gnu.org/22086#118
* src/gmalloc.c (__malloc_initialize_hook, __after_morecore_hook)
(__morecore) [HYBRID_MALLOC]: Define in this case too.
---
 src/gmalloc.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/src/gmalloc.c b/src/gmalloc.c
index 282216a..0b76aee 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -321,6 +321,13 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 /* Debugging hook for 'malloc'.  */
 static void *(*__MALLOC_HOOK_VOLATILE gmalloc_hook) (size_t);
 
+/* Replacements for traditional glibc malloc hooks, for platforms that
+   do not already have these hooks.  Platforms with these hooks all
+   used relaxed ref/def, so it is OK to define them here too.  */
+void (*__MALLOC_HOOK_VOLATILE __malloc_initialize_hook) (void);
+void (*__MALLOC_HOOK_VOLATILE __after_morecore_hook) (void);
+void *(*__morecore) (ptrdiff_t);
+
 #ifndef HYBRID_MALLOC
 
 /* Pointer to the base of the first block.  */
@@ -347,10 +354,6 @@ size_t _bytes_free;
 /* Are you experienced?  */
 int __malloc_initialized;
 
-void (*__MALLOC_HOOK_VOLATILE __malloc_initialize_hook) (void);
-void (*__MALLOC_HOOK_VOLATILE __after_morecore_hook) (void);
-void *(*__morecore) (ptrdiff_t);
-
 #else
 
 static struct list _fraghead[BLOCKLOG];
-- 
2.5.0

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Wolfgang Jenkner]

On Sun, Jan 31 2016, Paul Eggert wrote:

> Wolfgang Jenkner wrote:
>
>> Thanks, I think patch 5 of my OP (for turning off direct use of mmap for
>> buffer allocations on FreeBSD) should also be applied.
>
> Sorry, I missed that one. I don't recall why it's needed but I'll take
> your word for it. Done.

Thanks.  Actually, I haven't bothered to check if the value of
use_mmap_for_buffers has an effect when hybrid malloc is enabled, but
it's a clean-up at least and the configure summary would be misleading.

> I installed the attached patch to try to fix these; please give it a try.

Thanks, this almost worked, I had just to double the static heap size.

In the ensuing successful build, I got the following message after
dumping:

18788992 of 33554432 static heap bytes used

However, this is with my standard configuration, which doesn't include
everything:

ssystem-configuration-features is a variable defined in ‘C source code’.
Its value is
"XPM JPEG TIFF GIF PNG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB X11 MODULES"

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Paul Eggert]

On 02/01/2016 07:15 AM, Wolfgang Jenkner wrote:
> I had just to double the static heap size.

OK, I installed a patch doing that into Emacs master.

> In the ensuing successful build, I got the following message after
> dumping:
>
> 18788992 of 33554432 static heap bytes used
>
> However, this is with my standard configuration, which doesn't include
> everything:
>
> ssystem-configuration-features is a variable defined in ‘C source code’.
> Its value is
> "XPM JPEG TIFF GIF PNG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB X11 MODULES"

I wonder why you're using way more static heap bytes than I am? I get 
this on Fedora 23 x86-64 when configured via './configure 
--enable-gcc-warnings --with-modules emacs_cv_var_doug_lea_malloc=no':

11892256 of 33554432 static heap bytes used
92482 pure bytes used

10765856 of 33554432 static heap bytes used
2781901 pure bytes used

The first is when building bootstrap-emacs, the second is when building 
emacs. On my build, system-configuration-features is "XPM JPEG TIFF GIF 
PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS NOTIFY ACL 
LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB 
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES", which is a superset of your features.

Is the FreeBSD runtime calling malloc a lot on its own behalf? It might 
be interesting to find out why.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Wolfgang Jenkner]

On Mon, Feb 01 2016, Paul Eggert wrote:

> On 02/01/2016 07:15 AM, Wolfgang Jenkner wrote:

>> 18788992 of 33554432 static heap bytes used

> I wonder why you're using way more static heap bytes than I am? I get
> this on Fedora 23 x86-64 when configured via
> './configure --enable-gcc-warnings --with-modules
> emacs_cv_var_doug_lea_malloc=no':
>
> 11892256 of 33554432 static heap bytes used
> 92482 pure bytes used
>
> 10765856 of 33554432 static heap bytes used
> 2781901 pure bytes used
>
> The first is when building bootstrap-emacs, the second is when
> building emacs.

Ah, indeed, the static heap size is much smaller in the final dump:

Pure-hashed: 24083 strings, 3801 vectors, 38521 conses, 3706 bytecodes, 102 others
Dumping under the name emacs
9663104 of 33554432 static heap bytes used
2777644 pure bytes used
Adding name emacs-25.1.50.1
: paxctl -zex emacs

Also, I get an only slightly larger number when dumping bootstrap-emacs
with the pre-compiled elisp files.  Perhaps, you got the first number
with the *.elc files still being around from a previous test?

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Mon, 01 Feb 2016 19:34:20 +0100
> Cc: 22086@debbugs.gnu.org
> 
> On Mon, Feb 01 2016, Paul Eggert wrote:
> 
> > On 02/01/2016 07:15 AM, Wolfgang Jenkner wrote:
> 
> >> 18788992 of 33554432 static heap bytes used
> 
> > I wonder why you're using way more static heap bytes than I am? I get
> > this on Fedora 23 x86-64 when configured via
> > './configure --enable-gcc-warnings --with-modules
> > emacs_cv_var_doug_lea_malloc=no':
> >
> > 11892256 of 33554432 static heap bytes used
> > 92482 pure bytes used
> >
> > 10765856 of 33554432 static heap bytes used
> > 2781901 pure bytes used
> >
> > The first is when building bootstrap-emacs, the second is when
> > building emacs.
> 
> Ah, indeed, the static heap size is much smaller in the final dump:
> 
> Pure-hashed: 24083 strings, 3801 vectors, 38521 conses, 3706 bytecodes, 102 others
> Dumping under the name emacs
> 9663104 of 33554432 static heap bytes used
> 2777644 pure bytes used
> Adding name emacs-25.1.50.1
> : paxctl -zex emacs
> 
> Also, I get an only slightly larger number when dumping bootstrap-emacs
> with the pre-compiled elisp files.  Perhaps, you got the first number
> with the *.elc files still being around from a previous test?

You guys are aware that the amount of memory used when dumping depends
on such factors as the length of the absolute file name of the
directory where you build Emacs, right?

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Paul Eggert]

On 02/01/2016 11:38 AM, Eli Zaretskii wrote:
>> From: Wolfgang Jenkner <wjenkner@inode.at>
>> Date: Mon, 01 Feb 2016 19:34:20 +0100
>>
>>
>> Also, I get an only slightly larger number when dumping bootstrap-emacs
>> with the pre-compiled elisp files.  Perhaps, you got the first number
>> with the *.elc files still being around from a previous test?

Yes, that's probably it. Though I'm still surprised by the discrepancy. 
The biggest .el file is not that big. Is Emacs reading all the .el files 
into memory, into a single Emacs instance that buffers all the .el files 
at once?

> You guys are aware that the amount of memory used when dumping depends
> on such factors as the length of the absolute file name of the
> directory where you build Emacs, right?

Oh yes. Even doing the exact same dump twice can yield different 
results, presumably due to ASLR.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Wolfgang Jenkner]

[-- Attachment #1: Type: text/plain, Size: 215 bytes --]

On Sat, Jan 30 2016, Paul Eggert wrote:

> I think it's ready enough for the master branch, so I committed it
> there and will mark this bug as done.

I think, (g)calloc and hybrid_calloc are still needed, though?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Restore calloc and friends. --]
[-- Type: text/x-diff, Size: 3068 bytes --]

From cead27eae68b5d8d7db36b1a2965c541c4a7eb54 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner@inode.at>
Date: Mon, 8 Feb 2016 17:16:15 +0100
Subject: [PATCH] Restore the calloc family.

* src/gmalloc.c (calloc, gcalloc, hybrid_calloc): Restore definitions.
They were lost in a4817d8 but calloc is still (marginally) used in
code statically liked with emacs, so hybrid_calloc is needed.
Also, in the non-hybrid case, we can't get rid of calloc anyway as
other libraries liked with emacs may need it.
* src/conf_post.h: Restore redefinition of calloc to hybrid_calloc.
---
 src/conf_post.h |  1 +
 src/gmalloc.c   | 15 ++++++++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/src/conf_post.h b/src/conf_post.h
index c5eec5a..2788abf 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -100,6 +100,7 @@ typedef bool bool_bf;
 #define malloc hybrid_malloc
 #define realloc hybrid_realloc
 #define aligned_alloc hybrid_aligned_alloc
+#define calloc hybrid_calloc
 #define free hybrid_free
 #endif
 #endif	/* HYBRID_MALLOC */
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 0b76aee..dd18293 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -72,7 +72,7 @@ extern void *(*__morecore) (ptrdiff_t);
 #undef free
 #define malloc gmalloc
 #define realloc grealloc
-#define calloc do_not_call_me /* Emacs never calls calloc.  */
+#define calloc gcalloc
 #define aligned_alloc galigned_alloc
 #define free gfree
 #define malloc_info gmalloc_info
@@ -101,6 +101,8 @@ extern void *malloc (size_t size) ATTRIBUTE_MALLOC_SIZE ((1));
 /* Re-allocate the previously allocated block
    in ptr, making the new block SIZE bytes long.  */
 extern void *realloc (void *ptr, size_t size) ATTRIBUTE_ALLOC_SIZE ((2));
+/* Allocate NMEMB elements of SIZE bytes each, all initialized to 0.  */
+extern void *calloc (size_t nmemb, size_t size) ATTRIBUTE_MALLOC_SIZE ((1,2));
 /* Free a block.  */
 extern void free (void *ptr);
 
@@ -1465,7 +1467,6 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 
 /* Allocate an array of NMEMB elements each SIZE bytes long.
    The entire array is initialized to zeros.  */
-#ifndef calloc
 void *
 calloc (size_t nmemb, size_t size)
 {
@@ -1483,7 +1484,6 @@ calloc (size_t nmemb, size_t size)
     return memset (result, 0, bytes);
   return result;
 }
-#endif
 /* Copyright (C) 1991, 1992, 1993, 1994, 1995 Free Software Foundation, Inc.
 This file is part of the GNU C Library.
 
@@ -1714,6 +1714,7 @@ valloc (size_t size)
 /* Declare system malloc and friends.  */
 extern void *malloc (size_t size);
 extern void *realloc (void *ptr, size_t size);
+extern void *calloc (size_t nmemb, size_t size);
 extern void free (void *ptr);
 #ifdef HAVE_ALIGNED_ALLOC
 extern void *aligned_alloc (size_t alignment, size_t size);
@@ -1732,6 +1733,14 @@ hybrid_malloc (size_t size)
   return gmalloc (size);
 }
 
+void *
+hybrid_calloc (size_t nmemb, size_t size)
+{
+  if (DUMPED)
+    return calloc (nmemb, size);
+  return gcalloc (nmemb, size);
+}
+
 void
 hybrid_free (void *ptr)
 {
-- 
2.7.0

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Paul Eggert]

> I think, (g)calloc and hybrid_calloc are still needed, though?

I suppose you're right, so I installed your patch in master. Though this 
stuff is all pretty iffy. Why does Emacs redefine calloc but not 
aligned_alloc or posix_memalign, for example? Is it because we know no 
library uses these newer allocators?

Anyway, I suppose it's better to play it safe and continue to redefine 
calloc, the way Emacs has done for decades, until we have a better way 
to do dumping and restoring and memory allocation for such.

bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo

[Wolfgang Jenkner]

On Tue, Feb 09 2016, Paul Eggert wrote:

>> I think, (g)calloc and hybrid_calloc are still needed, though?
>
> I suppose you're right, so I installed your patch in master.

Thanks, I could have pushed it myself, I just posted it here to avoid
interfering with your plans concerning this stuff.

> this stuff is all pretty iffy. Why does Emacs redefine calloc but not
> aligned_alloc or posix_memalign, for example?

Here on FreeBSD 10, in the non-hybrid case both aligned_alloc and
posix_memalign are redefined; in the hybrid case there is only
hybrid_aligned_alloc:

/opt/src/emacs-calloc-non-hybrid;!2;: nm -g src/emacs | grep align
00000000005e4210 R QCalign_to
0000000000b83df8 D _aligned_blocks
0000000000b83df0 D _aligned_blocks_mutex
00000000005c92b0 T aligned_alloc
00000000005c8e70 T galigned_alloc
00000000005c90e0 T memalign
00000000005c90f0 T posix_memalign
/opt/src/emacs-calloc-non-hybrid;!3;: cd ../emacs-calloc-hybrid/
/opt/src/emacs-calloc-hybrid;!4;: nm -g src/emacs | grep align
00000000005e3380 R QCalign_to
                 U aligned_alloc@@FBSD_1.3
00000000005c71a0 T hybrid_aligned_alloc
/opt/src/emacs-calloc-hybrid;!5;: 

> Is it because we know no library uses these newer allocators?

Just for the record, as you know this stuff way better than I do:

IIUC, we need hybrid_ versions for all allocation functions that are
actually used in code statically linked with emacs (at least those which
somewhere refer to memory allocated before dumping, but I don't think
there's much point in making a distinction here).

On the other hand, in the non-hybrid case, we have to override all of
them (if they exist in libc), so that references in shared libraries
linked with emacs to, say, posix_memalign don't resolve to the libc
version.

> Anyway, I suppose it's better to play it safe and continue to redefine
> calloc, the way Emacs has done for decades, until we have a better way
> to do dumping and restoring and memory allocation for such.

In the hybrid case we just do #define calloc hybrid_calloc etc. in the
emacs sources, so that's not a problem.  On the other hand, in the
non-hybrid case, what has been done for decades (i.e. overriding the
libc implementation of calloc etc.) has turned out to be a problem after
all (bug#22085).