unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
@ 2011-05-27 21:50 Jim Meyering
  2011-05-27 21:57 ` Jim Meyering
  2011-05-29  5:44 ` Paul Eggert
  0 siblings, 2 replies; 10+ messages in thread
From: Jim Meyering @ 2011-05-27 21:50 UTC (permalink / raw)
  To: Emacs development discussions

Now, bootstrapping emacs on Fedora 15 x86_64
(with env settings as below) fails with a segfault.

If you develop emacs, or even if you just build from sources regularly,
on a glibc-based system, do us all a favor and build with these envvar
settings:

    export MALLOC_PERTURB_=$((RANDOM % 255 + 1))
    export MALLOC_CHECK_=3

Why?  Because that helps you expose malloc-related problems far earlier.
I've found numerous bugs that way.  The trouble is that you have to
be ready to debug, or at least ready to retry with

    env MALLOC_PERTURB_=0 MALLOC_CHECK_=0 make ...

to see if things work better without protection.

Until recently, I've been building like this to work around
a problem that I traced back to a commit in emacs several years ago:

    make bootstrap RUN_TEMACS='MALLOC_CHECK_=0 ./temacs'

However, today, even with that, make bootstrap is failing with a
segfault (in various places) that goes away when I turn off malloc
robustness checking via both variables for the whole process
and not just one of them when running temacs:

    MALLOC_PERTURB_=0 MALLOC_CHECK_=0 make bootstrap

That suggests a very recent change (within the last 24 hrs)
makes emacs act to differently when MALLOC_PERTURB_=N causes glibc
to scribble nonzero (N) bytes into each just freed buffer.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-27 21:50 please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars Jim Meyering
@ 2011-05-27 21:57 ` Jim Meyering
  2011-05-28  0:13   ` Thien-Thi Nguyen
  2011-05-28 20:19   ` Chong Yidong
  2011-05-29  5:44 ` Paul Eggert
  1 sibling, 2 replies; 10+ messages in thread
From: Jim Meyering @ 2011-05-27 21:57 UTC (permalink / raw)
  To: Emacs development discussions

Jim Meyering wrote:
> Now, bootstrapping emacs on Fedora 15 x86_64
> (with env settings as below) fails with a segfault.
>
> If you develop emacs, or even if you just build from sources regularly,
> on a glibc-based system, do us all a favor and build with these envvar
> settings:
>
>     export MALLOC_PERTURB_=$((RANDOM % 255 + 1))
>     export MALLOC_CHECK_=3
>
> Why?  Because that helps you expose malloc-related problems far earlier.
> I've found numerous bugs that way.  The trouble is that you have to
> be ready to debug, or at least ready to retry with
>
>     env MALLOC_PERTURB_=0 MALLOC_CHECK_=0 make ...
>
> to see if things work better without protection.
>
> Until recently, I've been building like this to work around
> a problem that I traced back to a commit in emacs several years ago:
>
>     make bootstrap RUN_TEMACS='MALLOC_CHECK_=0 ./temacs'
>
> However, today, even with that, make bootstrap is failing with a
> segfault (in various places) that goes away when I turn off malloc
> robustness checking via both variables for the whole process
> and not just one of them when running temacs:
>
>     MALLOC_PERTURB_=0 MALLOC_CHECK_=0 make bootstrap
>
> That suggests a very recent change (within the last 24 hrs)
> makes emacs act to differently when MALLOC_PERTURB_=N causes glibc
> to scribble nonzero (N) bytes into each just freed buffer.

Most of that argument is valid, but I have to confess this time
a compiler problem appears to be at fault.  To get "make" to succeed,
I have to build like this:

    env MALLOC_CHECK_=0 make CFLAGS=-O0

Use CFLAGS=-O or MALLOC_CHECK_=4 (or anything else in 1..255)
and the build segfaults when using the just-built temacs
to compile .el files.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-27 21:57 ` Jim Meyering
@ 2011-05-28  0:13   ` Thien-Thi Nguyen
  2011-05-28 18:52     ` Jim Meyering
  2011-05-28 20:19   ` Chong Yidong
  1 sibling, 1 reply; 10+ messages in thread
From: Thien-Thi Nguyen @ 2011-05-28  0:13 UTC (permalink / raw)
  To: Jim Meyering; +Cc: Emacs development discussions

() Jim Meyering <jim@meyering.net>
() Fri, 27 May 2011 23:57:40 +0200

   this time a compiler problem appears

Please describe specifics of this compiler (name, version, etc).



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-28  0:13   ` Thien-Thi Nguyen
@ 2011-05-28 18:52     ` Jim Meyering
  2011-05-30 10:30       ` Jim Meyering
  0 siblings, 1 reply; 10+ messages in thread
From: Jim Meyering @ 2011-05-28 18:52 UTC (permalink / raw)
  To: Thien-Thi Nguyen; +Cc: Emacs development discussions

Thien-Thi Nguyen wrote:
> () Jim Meyering <jim@meyering.net>
> () Fri, 27 May 2011 23:57:40 +0200
>
>    this time a compiler problem appears
>
> Please describe specifics of this compiler (name, version, etc).

It was the gcc from Fedora 15.

    $ rpm -q gcc
    gcc-4.6.0-7.fc15.x86_64

Using gcc built from the very latest (yesterday),

    gcc version 4.7.0 20110528 (experimental) (GCC)

everything works fine, and yes!, even with these envvar settings:

    MALLOC_PERTURB_=117
    MALLOC_CHECK_=3

So at least for a little while, I'll be using that.

Bottom line: using the very latest gcc-4.7.x, all is well.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-27 21:57 ` Jim Meyering
  2011-05-28  0:13   ` Thien-Thi Nguyen
@ 2011-05-28 20:19   ` Chong Yidong
  2011-05-28 20:33     ` Jim Meyering
  1 sibling, 1 reply; 10+ messages in thread
From: Chong Yidong @ 2011-05-28 20:19 UTC (permalink / raw)
  To: Jim Meyering; +Cc: Emacs development discussions

Jim Meyering <jim@meyering.net> writes:

>> If you develop emacs, or even if you just build from sources regularly,
>> on a glibc-based system, do us all a favor and build with these envvar
>> settings:
>>
>>     export MALLOC_PERTURB_=$((RANDOM % 255 + 1))
>>     export MALLOC_CHECK_=3
>>
>> Why?  Because that helps you expose malloc-related problems far earlier.
>> I've found numerous bugs that way.
>
> Most of that argument is valid, but I have to confess this time
> a compiler problem appears to be at fault.
> [...]
>     gcc version 4.7.0 20110528 (experimental) (GCC)
>
> everything works fine, and yes!, even with these envvar settings:
>
>     MALLOC_PERTURB_=117
>     MALLOC_CHECK_=3
>
> So at least for a little while, I'll be using that.

Maybe we should add these environment variables to the Hydra build.  But
what to use for MALLOC_PERTURB---117, or the value in your previous
message?



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-28 20:19   ` Chong Yidong
@ 2011-05-28 20:33     ` Jim Meyering
  0 siblings, 0 replies; 10+ messages in thread
From: Jim Meyering @ 2011-05-28 20:33 UTC (permalink / raw)
  To: Chong Yidong; +Cc: Emacs development discussions

Chong Yidong wrote:

> Jim Meyering <jim@meyering.net> writes:
>
>>> If you develop emacs, or even if you just build from sources regularly,
>>> on a glibc-based system, do us all a favor and build with these envvar
>>> settings:
>>>
>>>     export MALLOC_PERTURB_=$((RANDOM % 255 + 1))
>>>     export MALLOC_CHECK_=3
>>>
>>> Why?  Because that helps you expose malloc-related problems far earlier.
>>> I've found numerous bugs that way.
>>
>> Most of that argument is valid, but I have to confess this time
>> a compiler problem appears to be at fault.
>> [...]
>>     gcc version 4.7.0 20110528 (experimental) (GCC)
>>
>> everything works fine, and yes!, even with these envvar settings:
>>
>>     MALLOC_PERTURB_=117
>>     MALLOC_CHECK_=3
>>
>> So at least for a little while, I'll be using that.
>
> Maybe we should add these environment variables to the Hydra build.

Good idea.

> But
> what to use for MALLOC_PERTURB---117, or the value in your previous
> message?

You can use any value between 1..255 for MALLOC_PERTURB_
That chooses the byte that glibc will use to memset all freed buffers.
I set it like this in a shell start-up file:

    export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))

Using a random value is slightly better than using a fixed one
in case your fixed value is someday just the right/wrong value
to mask a problem.  At least with a random value, if you rerun
the test in a different shell, the odds are good you won't use
the unfortunate setting again.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-27 21:50 please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars Jim Meyering
  2011-05-27 21:57 ` Jim Meyering
@ 2011-05-29  5:44 ` Paul Eggert
  2011-05-29  7:32   ` Jim Meyering
  1 sibling, 1 reply; 10+ messages in thread
From: Paul Eggert @ 2011-05-29  5:44 UTC (permalink / raw)
  To: Jim Meyering; +Cc: Emacs development discussions

On 05/27/11 14:50, Jim Meyering wrote:
>     export MALLOC_PERTURB_=$((RANDOM % 255 + 1))
>     export MALLOC_CHECK_=3
> 

I tried that, on Fedora 14 x86-64, and the Emacs trunk build failed
as follows:

  Compiling language/thai-word.el

  In toplevel form:
  language/thai-word.el:10738:5:Error: Memory exhausted--use C-x s then exit and restart Emacs
  make[2]: *** [language/thai-word.elc] Error 1
  make[2]: Leaving directory `/home/eggert/src/gnu/emacs/int-hash/lisp'

As near as I can discover, Emacs was fine, but the malloc debugging
caused it to use so much more memory that Emacs ran out of memory
trying to compile thai-word.el.  This is on a host with
8 GiB of RAM.

Perhaps there is a real Emacs bug in there somewhere, but I spent
a reasonable amount of time looking for it unsuccessfully.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-29  5:44 ` Paul Eggert
@ 2011-05-29  7:32   ` Jim Meyering
  0 siblings, 0 replies; 10+ messages in thread
From: Jim Meyering @ 2011-05-29  7:32 UTC (permalink / raw)
  To: Paul Eggert; +Cc: Emacs development discussions

Paul Eggert wrote:

> On 05/27/11 14:50, Jim Meyering wrote:
>>     export MALLOC_PERTURB_=$((RANDOM % 255 + 1))
>>     export MALLOC_CHECK_=3
>>
>
> I tried that, on Fedora 14 x86-64, and the Emacs trunk build failed
> as follows:
>
>   Compiling language/thai-word.el
>
>   In toplevel form:
>   language/thai-word.el:10738:5:Error: Memory exhausted--use C-x s
> then exit and restart Emacs
>   make[2]: *** [language/thai-word.elc] Error 1
>   make[2]: Leaving directory `/home/eggert/src/gnu/emacs/int-hash/lisp'
>
> As near as I can discover, Emacs was fine, but the malloc debugging
> caused it to use so much more memory that Emacs ran out of memory
> trying to compile thai-word.el.  This is on a host with
> 8 GiB of RAM.
>
> Perhaps there is a real Emacs bug in there somewhere, but I spent
> a reasonable amount of time looking for it unsuccessfully.

Same here.  I even bisected back through several years of commits to
find the one that *appeared* to introduce this problem:
    http://thread.gmane.org/gmane.emacs.devel/137942
But all that was assuming a reliable compiler.

With those envvar settings, I've been bootstrapping emacs for
months, but only with this kludge:

     make bootstrap RUN_TEMACS='MALLOC_CHECK_=0 ./temacs'

Yesterday, I found that I can bootstrap with no kludge at all as long
as I use the latest gcc-4.7.0.  I.e., with that compiler, a plain
"make bootstrap" now succeeds in spite of my aggressive envvar settings.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-28 18:52     ` Jim Meyering
@ 2011-05-30 10:30       ` Jim Meyering
  2011-05-30 17:49         ` Andreas Schwab
  0 siblings, 1 reply; 10+ messages in thread
From: Jim Meyering @ 2011-05-30 10:30 UTC (permalink / raw)
  To: Emacs development discussions

Jim Meyering wrote:
> Thien-Thi Nguyen wrote:
>> () Jim Meyering <jim@meyering.net>
>> () Fri, 27 May 2011 23:57:40 +0200
>>
>>    this time a compiler problem appears
>>
>> Please describe specifics of this compiler (name, version, etc).
>
> It was the gcc from Fedora 15.
>
>     $ rpm -q gcc
>     gcc-4.6.0-7.fc15.x86_64
>
> Using gcc built from the very latest (yesterday),
>
>     gcc version 4.7.0 20110528 (experimental) (GCC)
>
> everything works fine, and yes!, even with these envvar settings:
>
>     MALLOC_PERTURB_=117
>     MALLOC_CHECK_=3
>
> So at least for a little while, I'll be using that.
>
> Bottom line: using the very latest gcc-4.7.x, all is well.

Here's an update.

Maybe that was just luck, because bootstrap failed this morning with
the aggressive MALLOC_* settings, even when using the latest version of gcc.

However, when I turned off MALLOC_CHECK_, it succeeded:

  MALLOC_CHECK_=0 MALLOC_PERTURB_=91 make bootstrap



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars
  2011-05-30 10:30       ` Jim Meyering
@ 2011-05-30 17:49         ` Andreas Schwab
  0 siblings, 0 replies; 10+ messages in thread
From: Andreas Schwab @ 2011-05-30 17:49 UTC (permalink / raw)
  To: Jim Meyering; +Cc: Emacs development discussions

MALLOC_CHECK_ has never worked on the dumped emacs.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."



^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2011-05-30 17:49 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-05-27 21:50 please set both MALLOC_PERTURB_ and MALLOC_CHECK_ envvars Jim Meyering
2011-05-27 21:57 ` Jim Meyering
2011-05-28  0:13   ` Thien-Thi Nguyen
2011-05-28 18:52     ` Jim Meyering
2011-05-30 10:30       ` Jim Meyering
2011-05-30 17:49         ` Andreas Schwab
2011-05-28 20:19   ` Chong Yidong
2011-05-28 20:33     ` Jim Meyering
2011-05-29  5:44 ` Paul Eggert
2011-05-29  7:32   ` Jim Meyering

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).