hybrid_malloc on CANNOT_DUMP?

hybrid_malloc on CANNOT_DUMP?

[Daniel Colascione]

Emacs on GNU/Linux and X11 crashes on startup in a CANNOT_DUMP 
configuration due to some nonsense inside gmalloc. Why wouldn't we want 
to force hybrid_malloc=off in a CANNOT_DUMP configuration?

Re: hybrid_malloc on CANNOT_DUMP?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Sun, 20 Nov 2016 09:16:33 -0800
> 
> Emacs on GNU/Linux and X11 crashes on startup in a CANNOT_DUMP 
> configuration due to some nonsense inside gmalloc.

Can you show the details?  A CANNOT_DUMP configuration doesn't unexec,
so I don't understand how can something bad happen inside gmalloc in
that case.

> Why wouldn't we want to force hybrid_malloc=off in a CANNOT_DUMP
> configuration?

On what branch?  I believe we already do that on master.

Re: hybrid_malloc on CANNOT_DUMP?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

Daniel Colascione wrote:
> Emacs on GNU/Linux and X11 crashes on startup in a CANNOT_DUMP configuration due
> to some nonsense inside gmalloc. Why wouldn't we want to force hybrid_malloc=off
> in a CANNOT_DUMP configuration?

Yes, that makes sense. More generally, the CANNOT_DUMP code has been suffering 
from bitrot for some time. I just now made a pass through it to fix some gotchas 
and installed the attached into Emacs master. It's still a mess, but at least I 
can now do 'configure CANNOT_DUMP=yes emacs_cv_var_doug_lea_malloc=no' and run 
'make check', and most tests pass. Hope this helps.

Can we hope that you're working on supporting a fast CANNOT_DUMP implementation? 
That'd be nice....

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-CANNOT_DUMP-work-better-on-GNU-Linux.patch --]
[-- Type: text/x-diff; name="0001-Make-CANNOT_DUMP-work-better-on-GNU-Linux.patch", Size: 5818 bytes --]

From 8c95deec6b3d7ff5f0b295c3948a1e0fce7ba726 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 20 Nov 2016 16:57:17 -0800
Subject: [PATCH] Make CANNOT_DUMP work better on GNU/Linux

Clean up some of the bitrot affecting the CANNOT_DUMP code.  This
lets the build succeed again, and fixes the testing framework so
that most test cases now pass.  About twenty test cases still
fail, though, and we still have Bug#24974.
* configure.ac (CANNOT_DUMP): Now empty if CANNOT_DUMP.
(SYSTEM_MALLOC): Now true if CANNOT_DUMP.  There should no longer
be any point to messing with a private memory allocator unless
Emacs is dumping.
* src/alloc.c (alloc_unexec_pre, alloc_unexec_post, check_pure_size):
* src/image.c (reset_image_types):
* src/lastfile.c (my_endbss, _my_endbss, my_endbss_static):
Do not define if CANNOT_DUMP.
* src/emacs.c (might_dump) [CANNOT_DUMP]: Now always false and local.
(daemon_pipe) [!WINDOWSNT]: Now static.
* test/Makefile.in (mostlyclean): Remove *.tmp files.
(make-test-deps.mk): Elide CANNOT_DUMP chatter.
---
 configure.ac     |  2 ++
 src/alloc.c      | 22 ++++++++++++++--------
 src/emacs.c      | 10 +++++++---
 src/image.c      |  3 +++
 src/lastfile.c   |  4 ++++
 test/Makefile.in |  7 +++++--
 6 files changed, 35 insertions(+), 13 deletions(-)

diff --git a/configure.ac b/configure.ac
index c9759e1..2d116de 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1366,6 +1366,7 @@ AC_DEFUN
    UNEXEC_OBJ=unexsol.o
    ;;
 esac
+test "$CANNOT_DUMP" = "yes" && UNEXEC_OBJ=
 
 LD_SWITCH_SYSTEM=
 case "$opsys" in
@@ -2154,6 +2155,7 @@ AC_DEFUN
 hybrid_malloc=
 system_malloc=yes
 
+test "$CANNOT_DUMP" = yes ||
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
   darwin | mingw32 | nacl | sol2-10) ;;
diff --git a/src/alloc.c b/src/alloc.c
index 90c6f94..175dcab 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -173,31 +173,34 @@ voidfuncptr __MALLOC_HOOK_VOLATILE __malloc_initialize_hook EXTERNALLY_VISIBLE
 
 #endif
 
+#if defined DOUG_LEA_MALLOC || !defined CANNOT_DUMP
+
 /* Allocator-related actions to do just before and after unexec.  */
 
 void
 alloc_unexec_pre (void)
 {
-#ifdef DOUG_LEA_MALLOC
+# ifdef DOUG_LEA_MALLOC
   malloc_state_ptr = malloc_get_state ();
   if (!malloc_state_ptr)
     fatal ("malloc_get_state: %s", strerror (errno));
-#endif
-#ifdef HYBRID_MALLOC
+# endif
+# ifdef HYBRID_MALLOC
   bss_sbrk_did_unexec = true;
-#endif
+# endif
 }
 
 void
 alloc_unexec_post (void)
 {
-#ifdef DOUG_LEA_MALLOC
+# ifdef DOUG_LEA_MALLOC
   free (malloc_state_ptr);
-#endif
-#ifdef HYBRID_MALLOC
+# endif
+# ifdef HYBRID_MALLOC
   bss_sbrk_did_unexec = false;
-#endif
+# endif
 }
+#endif
 
 /* Mark, unmark, query mark bit of a Lisp string.  S must be a pointer
    to a struct Lisp_String.  */
@@ -5216,6 +5219,8 @@ pure_alloc (size_t size, int type)
 }
 
 
+#ifndef CANNOT_DUMP
+
 /* Print a warning if PURESIZE is too small.  */
 
 void
@@ -5226,6 +5231,7 @@ check_pure_size (void)
 	      " bytes needed)"),
 	     pure_bytes_used + pure_bytes_used_before_overflow);
 }
+#endif
 
 
 /* Find the byte sequence {DATA[0], ..., DATA[NBYTES-1], '\0'} from
diff --git a/src/emacs.c b/src/emacs.c
index efd4fa3..ac9b649 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -130,11 +130,15 @@ Lisp_Object Vlibrary_cache;
    on subsequent starts.  */
 bool initialized;
 
+#ifdef CANNOT_DUMP
+enum { might_dump = false };
+#else
 /* Set to true if this instance of Emacs might dump.  */
-#ifndef DOUG_LEA_MALLOC
+# ifndef DOUG_LEA_MALLOC
 static
-#endif
+# endif
 bool might_dump;
+#endif
 
 #ifdef DARWIN_OS
 extern void unexec_init_emacs_zone (void);
@@ -196,7 +200,7 @@ int daemon_type;
 #ifndef WINDOWSNT
 /* Pipe used to send exit notification to the background daemon parent at
    startup.  On Windows, we use a kernel event instead.  */
-int daemon_pipe[2];
+static int daemon_pipe[2];
 #else
 HANDLE w32_daemon_event;
 #endif
diff --git a/src/image.c b/src/image.c
index d82fedb..5614f39 100644
--- a/src/image.c
+++ b/src/image.c
@@ -9776,6 +9776,8 @@ lookup_image_type (Lisp_Object type)
   return NULL;
 }
 
+#if !defined CANNOT_DUMP && defined HAVE_WINDOW_SYSTEM
+
 /* Reset image_types before dumping.
    Called from Fdump_emacs.  */
 
@@ -9789,6 +9791,7 @@ reset_image_types (void)
       image_types = next;
     }
 }
+#endif
 
 void
 syms_of_image (void)
diff --git a/src/lastfile.c b/src/lastfile.c
index 9d70b00..27602bd 100644
--- a/src/lastfile.c
+++ b/src/lastfile.c
@@ -43,6 +43,8 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 char my_edata[] = "End of Emacs initialized data";
 #endif
 
+#ifndef CANNOT_DUMP
+
 /* Help unexec locate the end of the .bss area used by Emacs (which
    isn't always a separate section in NT executables).  */
 char my_endbss[1];
@@ -52,3 +54,5 @@ char my_endbss[1];
    of the bss area used by Emacs.  */
 static char _my_endbss[1];
 char * my_endbss_static = _my_endbss;
+
+#endif
diff --git a/test/Makefile.in b/test/Makefile.in
index 33e625f..f2f2763 100644
--- a/test/Makefile.in
+++ b/test/Makefile.in
@@ -190,6 +190,7 @@ .PHONY:
 
 mostlyclean:
 	-@for f in ${LOGFILES}; do test ! -f $$f || mv $$f $$f~; done
+	rm -f *.tmp
 
 clean:
 	-rm -f ${LOGFILES} ${LOGSAVEFILES}
@@ -206,5 +207,7 @@ maintainer-clean:
 make-test-deps.mk: $(ELFILES) make-test-deps.emacs-lisp
 	$(EMACS) --batch -l $(srcdir)/make-test-deps.emacs-lisp \
 	--eval "(make-test-deps \"$(srcdir)\")" \
-	2> $@
-# Makefile ends here.
+	2> $@.tmp
+	# Hack to elide any CANNOT_DUMP=yes chatter.
+	sed '/\.log: /!d' $@.tmp >$@
+	rm -f $@.tmp
-- 
2.7.4

Re: hybrid_malloc on CANNOT_DUMP?

[Ken Raeburn]

> On Nov 20, 2016, at 20:03, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
> Daniel Colascione wrote:
>> Emacs on GNU/Linux and X11 crashes on startup in a CANNOT_DUMP configuration due
>> to some nonsense inside gmalloc. Why wouldn't we want to force hybrid_malloc=off
>> in a CANNOT_DUMP configuration?
> 
> Yes, that makes sense. More generally, the CANNOT_DUMP code has been suffering from bitrot for some time. I just now made a pass through it to fix some gotchas and installed the attached into Emacs master. It's still a mess, but at least I can now do 'configure CANNOT_DUMP=yes emacs_cv_var_doug_lea_malloc=no' and run 'make check', and most tests pass. Hope this helps.

Sounds like it broke a little more since I tried it a few weeks ago. :-(
Back then, it didn’t break until it got to trying to byte-compile Lisp files.

Even with the configure options you specify here, I can’t get it to link on a GNU/Linux configuration:

emacs.o: In function `main':
/home/ken/dev/emacs/emacs/lx2/src/../../src/emacs.c:725: undefined reference to `my_endbss_static'
/home/ken/dev/emacs/emacs/lx2/src/../../src/emacs.c:725: undefined reference to `my_endbss'
/home/ken/dev/emacs/emacs/lx2/src/../../src/emacs.c:725: undefined reference to `my_endbss'
/home/ken/dev/emacs/emacs/lx2/src/../../src/emacs.c:725: undefined reference to `my_endbss_static'
collect2: error: ld returned 1 exit status
Makefile:611: recipe for target 'temacs’ failed
$

my_endbss is no longer defined if CANNOT_DUMP is true, but emacs.c references it if GNU_LINUX is defined.

Ken

Re: hybrid_malloc on CANNOT_DUMP?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

Ken Raeburn wrote:
> my_endbss is no longer defined if CANNOT_DUMP is true, but emacs.c references it if GNU_LINUX is defined.

Thanks, I suspect I didn't see that because my GCC optimized away all uses of 
my_endbss etc. before the linker could see them. I installed the attached to try 
to port to less-industrious compilers.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-undefined-refs-on-some-GNU-Linux-hosts.patch --]
[-- Type: text/x-diff; name="0001-Fix-undefined-refs-on-some-GNU-Linux-hosts.patch", Size: 1792 bytes --]

From 33b69e2d76e514b0cce39b1e516fb8ffe2cfd497 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 20 Nov 2016 20:55:35 -0800
Subject: [PATCH] Fix undefined refs on some GNU/Linux hosts

Problem reported by Ken Raeburn in:
http://lists.gnu.org/archive/html/emacs-devel/2016-11/msg00463.html
* src/emacs.c (heap_bss_diff) [CANNOT_DUMP]: Remove, as this is
not needed in the CANNOT_UNDUMP case.  All uses removed.  This
removes unwanted references to my_endbss and my_endbss_static,
which are not optimized away on some platforms.
---
 src/emacs.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/emacs.c b/src/emacs.c
index ac9b649..48df533 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -161,7 +161,7 @@ bool display_arg;
    Tells GC how to save a copy of the stack.  */
 char *stack_bottom;
 
-#ifdef GNU_LINUX
+#if defined GNU_LINUX && !defined CANNOT_DUMP
 /* The gap between BSS end and heap start as far as we can tell.  */
 static uprintmax_t heap_bss_diff;
 #endif
@@ -716,14 +716,14 @@ main (int argc, char **argv)
 
 #ifndef CANNOT_DUMP
   might_dump = !initialized;
-#endif
 
-#ifdef GNU_LINUX
+# ifdef GNU_LINUX
   if (!initialized)
     {
       char *heap_start = my_heap_start ();
       heap_bss_diff = heap_start - max (my_endbss, my_endbss_static);
     }
+# endif
 #endif
 
 #if defined WINDOWSNT || defined HAVE_NTGUI
@@ -2126,7 +2126,7 @@ You must run Emacs in batch mode in order to dump it.  */)
   if (!might_dump)
     error ("Emacs can be dumped only once");
 
-#ifdef GNU_LINUX
+#if defined GNU_LINUX && !defined CANNOT_DUMP
 
   /* Warn if the gap between BSS end and heap start is larger than this.  */
 # define MAX_HEAP_BSS_DIFF (1024*1024)
-- 
2.7.4

Re: hybrid_malloc on CANNOT_DUMP?

[Ken Raeburn]

> On Nov 20, 2016, at 23:58, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
> Ken Raeburn wrote:
>> my_endbss is no longer defined if CANNOT_DUMP is true, but emacs.c references it if GNU_LINUX is defined.
> 
> Thanks, I suspect I didn't see that because my GCC optimized away all uses of my_endbss etc. before the linker could see them. I installed the attached to try to port to less-industrious compilers.

That works for me, thanks.

I’ll try to get back to that load-path issue later this week.

Ken

Re: hybrid_malloc on CANNOT_DUMP?

[Robert Pluim]

Paul Eggert <eggert@cs.ucla.edu> writes:

> diff --git a/src/emacs.c b/src/emacs.c
> index efd4fa3..ac9b649 100644
> --- a/src/emacs.c
> +++ b/src/emacs.c
> @@ -130,11 +130,15 @@ Lisp_Object Vlibrary_cache;
>     on subsequent starts.  */
>  bool initialized;
>  
> +#ifdef CANNOT_DUMP
> +enum { might_dump = false };
> +#else
>  /* Set to true if this instance of Emacs might dump.  */
> -#ifndef DOUG_LEA_MALLOC
> +# ifndef DOUG_LEA_MALLOC
>  static
> -#endif
> +# endif
>  bool might_dump;
> +#endif


Hi Paul,

I guess you're using a more forgiving gcc than I am:

  GEN      globals.h
  CC       emacs.o
emacs.c:134:8: error: ‘might_dump’ redeclared as different kind of symbol
 enum { might_dump = false };
        ^
In file included from emacs.c:36:0:
lisp.h:608:13: note: previous declaration of ‘might_dump’ was here
 extern bool might_dump;


$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 

Re: hybrid_malloc on CANNOT_DUMP?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

On 11/21/2016 03:41 AM, Robert Pluim wrote:
> I guess you're using a more forgiving gcc than I am:

Thanks for reporting that. It looks like I was building with different 
config-time flags. Sorry; there are many combinations of flags here. I 
installed the attached patch to work around that problem, so let's try 
again.


[-- Attachment #2: 0001-Fix-another-CANNOT_DUMP-problem.patch --]
[-- Type: application/x-patch, Size: 1420 bytes --]

Re: hybrid_malloc on CANNOT_DUMP?

[Robert Pluim]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 11/21/2016 03:41 AM, Robert Pluim wrote:
>> I guess you're using a more forgiving gcc than I am:
>
> Thanks for reporting that. It looks like I was building with different
> config-time flags. Sorry; there are many combinations of flags here. I
> installed the attached patch to work around that problem, so let's try
> again.

OK, it now compiles for me (and then fails to find loadup.el, but I
guess that's expected for the moment)

Regards

Robert

Re: hybrid_malloc on CANNOT_DUMP?

[Daniel Colascione]

On Mon, Nov 21 2016, Robert Pluim wrote:
> Paul Eggert <eggert@cs.ucla.edu> writes:
>
>> On 11/21/2016 03:41 AM, Robert Pluim wrote:
>>> I guess you're using a more forgiving gcc than I am:
>>
>> Thanks for reporting that. It looks like I was building with different
>> config-time flags. Sorry; there are many combinations of flags here. I
>> installed the attached patch to work around that problem, so let's try
>> again.
>
> OK, it now compiles for me (and then fails to find loadup.el, but I
> guess that's expected for the moment)

Yeah. I don't know what's going on there. I worked around it by just
setting the Emacs load path via the environment, but I'd expect
CANNOT_DUMP emacs to act just like regular emacs in this respect.