Re: 22.0.99 emacs dumper (?) problem

[Chip Coldwell <coldwell@redhat.com>]

On Mon, 21 May 2007, Richard Stallman wrote:

> I cannot reconcile the things that you've said about the problem.
> 
> If this is the full extent of the problem
> 
>     This appears to be due to a change in the internal format of freed
>     blocks introduced between glibc-2.5.90-21 and glibc-2.5.90-22.  As a
>     result, Emacs binaries built using older versions of glibc may crash
>     when run using newer versions of glibc.
> 
> then it ought to work just to relink temacs with the newer libc
> and then dump Emacs again.
> 
> (Mr Coldwell, what happens if you do that?)

Yes, that works; the resulting dumped emacs binary does not seg-fault
on startup.

> However, if this change is needed also
> 
> +	MALLOC_MMAP_MAX_=0 LC_ALL=C $(RUN_TEMACS) -nl -batch -l loadup dump
> 
> to make Emacs work with the newer libc, it implies there is a worse
> problem.  It implies that Emacs fails to work correctly with the newer
> libc, and never mind what the older libc did.
> 
> Which is it?

My current belief is that it is the former -- changes in the internal
malloc data structures meant that the dumping temacs and dumped emacs
had to have the same glibc version.  I currently believe the
MALLOC_MMAP_MAX_=0 is a red herring; I've build several binaries
without it that work fine.

dump_emacs contains this line

  malloc_state_ptr = malloc_get_state();

just before the call to unexec.  Then malloc_initialize_hook (bound to
the weak symbol __malloc_initialize_hook) does

  if (initialized)
    {
    [ ... ]
      malloc_set_state (malloc_state_ptr);
#ifndef XMALLOC_OVERRUN_CHECK
      free (malloc_state_ptr);
#endif
    }

The "initialized" variable is zero in the dumping emacs and nonzero in
the dumped emacs.

So malloc_get_state returns a pointer to an opaque data structure in
the .bss segment (I believe all calls to malloc in temacs are guarded
such that malloc always uses sbrk not mmap).  The .bss gets dumped
into a new .data segment in the emacs binary with the opaque data
structure.  The dumped emacs binary then starts up and passes the
pointer to the opaque data structure in the call to malloc_set_state.
Since the temacs binary and the emacs binary could be linked to
different versions of glibc, there can be no incompatible changes to
this opaque data structure between glibc versions.  IOW, the data
structure is not quite as opaque as the glibc maintainers believed.

> Meanwhile, is this workaround really ok?

We are currently building without it; I believe this workaround isn't
even necessary.

> Can Mr Coldwell (or anyone) tell us which?

Once the glibc situation settles down, I will verify that I can dump
emacs on glibc-2.5.90-21 and run the resulting binary when it links to
glibc-2.6-whatever.  Then we can put this to rest.

> If it only prevents use of mmap when running temacs, then it is ok as
> a workaround, but I would like to generate it automatically thru the
> makefile mechanism rather than ask users to patch it by hand.

I think that the reason the workaround worked has something to do with
the specifics of the malloc saved state opaque structure.  If mmap was
off during dumping, then that saved state structure from previous
glibc versions was compatible with the new glibc version, just by
luck.

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426