dumping bug explained?

dumping bug explained?

[Richard Stallman]

    ** coldwell@redhat.com, May 18: 22.0.99 emacs dumper (?) problem
    This seems to be caused by a binary incompatible change for the malloc
    implementation in glibc. The problem would occur when emacs is built
    on one system and used on another. See 
    https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60
    for a possible complete explanation.

Are people sure that this is the cause of the problem?
If so, the only fix it needs is an entry in PROBLEMS.

Re: dumping bug explained?

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>     ** coldwell@redhat.com, May 18: 22.0.99 emacs dumper (?) problem
>     This seems to be caused by a binary incompatible change for the malloc
>     implementation in glibc. The problem would occur when emacs is built
>     on one system and used on another. See 
>     https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60
>     for a possible complete explanation.
>
> Are people sure that this is the cause of the problem?

There is already a patch available, see
<http://sourceware.org/ml/libc-hacker/2007-05/msg00015.html>.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: dumping bug explained?

[Chong Yidong]

Andreas Schwab <schwab@suse.de> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     ** coldwell@redhat.com, May 18: 22.0.99 emacs dumper (?) problem
>>     This seems to be caused by a binary incompatible change for the malloc
>>     implementation in glibc. The problem would occur when emacs is built
>>     on one system and used on another. See 
>>     https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60
>>     for a possible complete explanation.
>>
>> Are people sure that this is the cause of the problem?
>
> There is already a patch available, see
> <http://sourceware.org/ml/libc-hacker/2007-05/msg00015.html>.

To be precise (since RMS doesn't read URLs), this is a patch to glibc,
not Emacs.  However, there will be released versions of glibc out
there with this problem, so we should still document it in PROBLEMS.

Re: dumping bug explained?

[Chip Coldwell]

On Mon, 21 May 2007, Chong Yidong wrote:

> Andreas Schwab <schwab@suse.de> writes:
> 
> > Richard Stallman <rms@gnu.org> writes:
> >
> >>     ** coldwell@redhat.com, May 18: 22.0.99 emacs dumper (?) problem
> >>     This seems to be caused by a binary incompatible change for the malloc
> >>     implementation in glibc. The problem would occur when emacs is built
> >>     on one system and used on another. See 
> >>     https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60
> >>     for a possible complete explanation.
> >>
> >> Are people sure that this is the cause of the problem?
> >
> > There is already a patch available, see
> > <http://sourceware.org/ml/libc-hacker/2007-05/msg00015.html>.
> 
> To be precise (since RMS doesn't read URLs), this is a patch to glibc,
> not Emacs.  However, there will be released versions of glibc out
> there with this problem, so we should still document it in PROBLEMS.

Jakub Jelinek and Ulrich Drepper are working on this in glibc.  Jakub
explained it as follows: (text from
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60)

There are multiple bugs on the glibc side (unfortunately malloc_set_state the
way it is used by emacs makes many internal details of the malloc implementation
part of the ABI):
1) the
2007-04-30  Ulrich Drepper  <drepper@redhat.com>
            Jakub Jelinek  <jakub@redhat.com>

        [BZ #4349]
        * malloc/malloc.c: Keep separate list for first blocks on the bin
        lists with a given size.  This helps skipping over list elements
        we know won't fit in two places.
        Inspired by a patch by Tomash Brechko <tomash.brechko@gmail.com>.
   change affects the ABI, as if emacs is dumped with glibc before this patch
   and run with glibc after this patch, the fd_nextsize/bk_nextsize pointers
   are left uninitialized, but the new glibc relies on them having correct
   values.  This is solvable by recomputing these pointers in malloc_set_state.
2) I made an accidental commit that enabled MALLOC_DEBUG (in the
2007-05-07  Ulrich Drepper  <drepper@redhat.com>
            Jakub Jelinek  <jakub@redhat.com>

        * malloc/arena.c (heap_info): Add mprotect_size field, adjust pad.
        (new_heap): Initialize mprotect_size.
        (grow_heap): When growing, only mprotect from mprotect_size till
        new_size if mprotect_size is smaller.  When shrinking, use PROT_NONE
        MMAP for __libc_enable_secure only, otherwise use MADV_DONTNEED.
check-in) and
2007-05-13  Ulrich Drepper  <drepper@redhat.com>

        * malloc/malloc.c [MALLOC_DEBUG]: Keep track of current maximum
        number of mmaps.  n_mmaps_max is the target.
        * malloc/hooks.c: Likewise.
        * malloc/arena.c: Likewise.
change added a field to struct malloc_save_state without bumping version number.
That together means another ABI change for malloc_save_state.  We certainly
have to revert the MALLOC_DEBUG setting (that's expensive) and something
has to be done about the 05-13 change (state changing shouldn't depend on
MALLOC_DEBUG).

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

Re: dumping bug explained?

[Chong Yidong]

Chip Coldwell <coldwell@redhat.com> writes:

>> To be precise (since RMS doesn't read URLs), this is a patch to glibc,
>> not Emacs.  However, there will be released versions of glibc out
>> there with this problem, so we should still document it in PROBLEMS.
>
> Jakub Jelinek and Ulrich Drepper are working on this in glibc.  Jakub
> explained it as follows: (text from
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60)

Do you know if Emacs will still need to be patched after these changes
to glibc are done?

Re: dumping bug explained?

[Chip Coldwell]

On Mon, 21 May 2007, Chong Yidong wrote:

> Chip Coldwell <coldwell@redhat.com> writes:
> 
> >> To be precise (since RMS doesn't read URLs), this is a patch to glibc,
> >> not Emacs.  However, there will be released versions of glibc out
> >> there with this problem, so we should still document it in PROBLEMS.
> >
> > Jakub Jelinek and Ulrich Drepper are working on this in glibc.  Jakub
> > explained it as follows: (text from
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=239344#c60)
> 
> Do you know if Emacs will still need to be patched after these changes
> to glibc are done?

My understanding is that Emacs will not need to be patched after glibc
is updated.

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

Re: dumping bug explained?

[Ulrich Drepper]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chong Yidong wrote:
> Do you know if Emacs will still need to be patched after these changes
> to glibc are done?

Not all the symptoms described in the bug can be explained by the
changes.  It might very well be that changes are needed.

In general, the use of malloc in emacs is horrific.  We are now going to
work around some of problems created by the changes when it comes to
assumptions emacs makes, but there is no guarantee that we can keep this up.

- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFGUct42ijCOnn/RHQRAnuXAKChmJ9SwrUbeQeTmXNwwUB1RNI4ggCgxyNJ
zyTm12a0Glz8SK3Fs5/0qQU=
=DUhM
-----END PGP SIGNATURE-----

Re: dumping bug explained?

[Chip Coldwell]

On Mon, 21 May 2007, Ulrich Drepper wrote:

> Chong Yidong wrote:
> > Do you know if Emacs will still need to be patched after these changes
> > to glibc are done?
> 
> Not all the symptoms described in the bug can be explained by the
> changes.  It might very well be that changes are needed.

I don't know if this is the behavior you are thinking about, but IIUC
this message

*** glibc detected *** emacs: malloc(): memory corruption: 0x0000000002904d10 ***

was generated by an assert failing in malloc that Jakub Jelinek
described in his libc-hacker posting containing his patch to solve
this problem thus "removes a bogus assert mp_.n_mmaps <=
mp_.n_mmaps_max."  This would be the assert that would succeed if you
dumped emacs with MALLOC_MMAP_MAX_=0, right?

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

Re: dumping bug explained?

[Richard Stallman]

    In general, the use of malloc in emacs is horrific.

Which aspects do you find particularly troublesome?
We could try to look for ways to change them.