* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
@ 2011-08-09 20:09 Ken Brown
2011-08-10 0:24 ` Richard Stallman
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Ken Brown @ 2011-08-09 20:09 UTC (permalink / raw
To: 9273
The code in src/gmalloc.c makes assumptions about how a system maintains
its memory that are not necessarily valid. In particular, they will not
be valid on Cygwin starting with version 1.7.10 (which will almost
certainly be released before emacs 24.1). The problem is that malloc
initialization is done by temacs, and the results are dumped into emacs.
This includes the setting __malloc_initialized = 1, so no malloc
initialization is done when emacs is run. But the dumped value of
_heapbase, while appropriate for temacs, may not point to the beginning
of the runtime heap for emacs. This causes all code that uses the BLOCK
and ADDRESS macros to be invalid.
Here's what happens on Cygwin. temacs (on Cygwin) uses a static buffer
as its heap and a function bss_sbrk that simulates sbrk. (See
src/sheap.c.) The data in this buffer, including malloc information,
are then dumped into emacs.exe as initialized data. But when the dumped
emacs is run, it uses Cygwin's sbrk, which allocates memory on a heap
that won't (as of Cygwin 1.7.10) be contiguous with the static heap.
The saved value of _heapbase, which points into the static heap, is
never changed, but it will mess up later calculations as soon as sbrk is
called for the first time.
All of this is described in detail on the Cygwin mailing list in the
thread starting at
http://cygwin.com/ml/cygwin/2011-08/msg00153.html
See especially
http://cygwin.com/ml/cygwin/2011-08/msg00193.html
which contains a gdb session illustrating the problem. The context for
that session is that, as a result of the problem I'm reporting,
morecore_nolock went into an infinite loop. I attached gdb to that
looping process.
Maybe the solution is for emacs to do malloc initialization, including
the assignment of _heapbase, every time it starts, at least on systems
that use gmalloc.c. I made one naive attempt to do this, but it didn't
work (and it was Cygwin specific). Namely, I made unexec (for Cygwin)
set _malloc_initialized = 0 before dumping. The resulting emacs aborted
as soon as it was started. I haven't figured out what went wrong, but
I'm not sure that's the right answer anyway.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-09 20:09 bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime Ken Brown
@ 2011-08-10 0:24 ` Richard Stallman
2011-08-10 15:56 ` Eli Zaretskii
2011-08-12 23:51 ` grischka
2 siblings, 0 replies; 22+ messages in thread
From: Richard Stallman @ 2011-08-10 0:24 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
If Cygwin does something very strange, there are limits to how far
we would want to change Emacs to cope with that.
--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
Use free telephony http://directory.fsf.org/category/tel/
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-09 20:09 bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime Ken Brown
2011-08-10 0:24 ` Richard Stallman
@ 2011-08-10 15:56 ` Eli Zaretskii
2011-08-10 17:52 ` Ken Brown
2011-08-11 21:45 ` Ken Brown
2011-08-12 23:51 ` grischka
2 siblings, 2 replies; 22+ messages in thread
From: Eli Zaretskii @ 2011-08-10 15:56 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
> Date: Tue, 09 Aug 2011 16:09:48 -0400
> From: Ken Brown <kbrown@cornell.edu>
>
> The code in src/gmalloc.c makes assumptions about how a system maintains
> its memory that are not necessarily valid. In particular, they will not
> be valid on Cygwin starting with version 1.7.10 (which will almost
> certainly be released before emacs 24.1). The problem is that malloc
> initialization is done by temacs, and the results are dumped into emacs.
> This includes the setting __malloc_initialized = 1, so no malloc
> initialization is done when emacs is run. But the dumped value of
> _heapbase, while appropriate for temacs, may not point to the beginning
> of the runtime heap for emacs. This causes all code that uses the BLOCK
> and ADDRESS macros to be invalid.
If Cygwin developers cannot or won't add to the Cygwin memory
allocation enough features and knobs to cater to the special needs of
Emacs dumping, then your only hope is to make Cygwin-specific changes
to Emacs.
You will see that 2 other ports that need to live with Windows memory
allocation either have such knobs and features (DJGPP, used to build
the DOS port; see the beginning of src/msdos.c) or use their own
emulation of sbrk that upholds the contract expected by gmalloc.c and
ralloc.c (w32heap.c, for the native Windows build).
> But when the dumped emacs is run, it uses Cygwin's sbrk, which
> allocates memory on a heap that won't (as of Cygwin 1.7.10) be
> contiguous with the static heap. The saved value of _heapbase,
> which points into the static heap, is never changed, but it will
> mess up later calculations as soon as sbrk is called for the first
> time.
Are you sure this is all that's at work here? AFAIR, gmalloc does
have code to cope with non-contiguous memory regions returned by sbrk.
> All of this is described in detail on the Cygwin mailing list in the
> thread starting at
>
> http://cygwin.com/ml/cygwin/2011-08/msg00153.html
>
> See especially
>
> http://cygwin.com/ml/cygwin/2011-08/msg00193.html
I have read all the discussion there, but I'm sorry to say that I
cannot figure out what you are talking about: there's too much
Cygwin-isms in that thread that I couldn't penetrate.
> Maybe the solution is for emacs to do malloc initialization, including
> the assignment of _heapbase, every time it starts, at least on systems
> that use gmalloc.c.
Most supported systems don't need that. The native Windows build
indeed does, see w32heap.c. Perhaps you could reuse some or even most
of it for Cygwin. (What is so special about the Cygwin sbrk that is
worth sticking to it?)
> I made one naive attempt to do this, but it didn't work (and it was
> Cygwin specific). Namely, I made unexec (for Cygwin) set
> _malloc_initialized = 0 before dumping. The resulting emacs aborted
> as soon as it was started. I haven't figured out what went wrong,
> but I'm not sure that's the right answer anyway.
One more evidence that something else is at work here. I would
suggest to walk through the session that reinitializes the heap after
unexec and see what goes wrong there.
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-10 15:56 ` Eli Zaretskii
@ 2011-08-10 17:52 ` Ken Brown
2011-08-10 18:10 ` Eli Zaretskii
2011-08-11 21:45 ` Ken Brown
1 sibling, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-10 17:52 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/10/2011 11:56 AM, Eli Zaretskii wrote:
>> Date: Tue, 09 Aug 2011 16:09:48 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>
>> But when the dumped emacs is run, it uses Cygwin's sbrk, which
>> allocates memory on a heap that won't (as of Cygwin 1.7.10) be
>> contiguous with the static heap. The saved value of _heapbase,
>> which points into the static heap, is never changed, but it will
>> mess up later calculations as soon as sbrk is called for the first
>> time.
>
> Are you sure this is all that's at work here? AFAIR, gmalloc does
> have code to cope with non-contiguous memory regions returned by sbrk.
The issue isn't that sbrk returns non-contiguous regions. The issue is
that two different of sbrk are used. One is used when temacs is
running, and a different one is used when the dumped emacs.exe is
running. This is controlled by the Cygwin-specific code in gmalloc.c,
and it has to do with the way unexec works in the Cygwin build of emacs.
>> All of this is described in detail on the Cygwin mailing list in the
>> thread starting at
>>
>> http://cygwin.com/ml/cygwin/2011-08/msg00153.html
>>
>> See especially
>>
>> http://cygwin.com/ml/cygwin/2011-08/msg00193.html
>
> I have read all the discussion there, but I'm sorry to say that I
> cannot figure out what you are talking about: there's too much
> Cygwin-isms in that thread that I couldn't penetrate.
Thanks for trying.
>> Maybe the solution is for emacs to do malloc initialization, including
>> the assignment of _heapbase, every time it starts, at least on systems
>> that use gmalloc.c.
>
> Most supported systems don't need that. The native Windows build
> indeed does, see w32heap.c. Perhaps you could reuse some or even most
> of it for Cygwin. (What is so special about the Cygwin sbrk that is
> worth sticking to it?)
Thanks for the suggestions.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-10 17:52 ` Ken Brown
@ 2011-08-10 18:10 ` Eli Zaretskii
2011-08-10 18:49 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Eli Zaretskii @ 2011-08-10 18:10 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
> Date: Wed, 10 Aug 2011 13:52:39 -0400
> From: Ken Brown <kbrown@cornell.edu>
> CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>
>
> On 8/10/2011 11:56 AM, Eli Zaretskii wrote:
> >> Date: Tue, 09 Aug 2011 16:09:48 -0400
> >> From: Ken Brown<kbrown@cornell.edu>
> >
> >> But when the dumped emacs is run, it uses Cygwin's sbrk, which
> >> allocates memory on a heap that won't (as of Cygwin 1.7.10) be
> >> contiguous with the static heap. The saved value of _heapbase,
> >> which points into the static heap, is never changed, but it will
> >> mess up later calculations as soon as sbrk is called for the first
> >> time.
> >
> > Are you sure this is all that's at work here? AFAIR, gmalloc does
> > have code to cope with non-contiguous memory regions returned by sbrk.
>
> The issue isn't that sbrk returns non-contiguous regions. The issue is
> that two different of sbrk are used. One is used when temacs is
> running, and a different one is used when the dumped emacs.exe is
> running.
I still don't see the problem: the memory sbrk'ed before dumping is
frozen in the dumped Emacs, so I don't see how that could matter.
Perhaps I'm missing something.
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-10 18:10 ` Eli Zaretskii
@ 2011-08-10 18:49 ` Ken Brown
0 siblings, 0 replies; 22+ messages in thread
From: Ken Brown @ 2011-08-10 18:49 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/10/2011 2:10 PM, Eli Zaretskii wrote:
>> Date: Wed, 10 Aug 2011 13:52:39 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>
>> On 8/10/2011 11:56 AM, Eli Zaretskii wrote:
>>>> Date: Tue, 09 Aug 2011 16:09:48 -0400
>>>> From: Ken Brown<kbrown@cornell.edu>
>>>
>>>> But when the dumped emacs is run, it uses Cygwin's sbrk, which
>>>> allocates memory on a heap that won't (as of Cygwin 1.7.10) be
>>>> contiguous with the static heap. The saved value of _heapbase,
>>>> which points into the static heap, is never changed, but it will
>>>> mess up later calculations as soon as sbrk is called for the first
>>>> time.
>>>
>>> Are you sure this is all that's at work here? AFAIR, gmalloc does
>>> have code to cope with non-contiguous memory regions returned by sbrk.
>>
>> The issue isn't that sbrk returns non-contiguous regions. The issue is
>> that two different of sbrk are used. One is used when temacs is
>> running, and a different one is used when the dumped emacs.exe is
>> running.
>
> I still don't see the problem: the memory sbrk'ed before dumping is
> frozen in the dumped Emacs, so I don't see how that could matter.
> Perhaps I'm missing something.
The memory sbrk'ed before dumping is in the static heap, which is
somewhere in relatively low memory. All the variables that malloc uses
for keeping track of this involve these low addresses. But when the
dumped emacs is run, Cygwin's sbrk is called, and it returns addresses
starting at wherever Cygwin decides to put the heap (which will be
either 0x20000000 or 0x80000000 in Cygwin 1.7.10, depending on whether
or not large address awareness is enabled for emacs.exe and is supported
by the underlying Windows system).
The calculations done in gmalloc.c are based on the assumption that the
heap starts in the same place in the dumped executable as it did before
dumping. See especially the BLOCK and ADDRESS macros, which use the
_heapbase variable. But _heapbase was set before dumping, and it points
somewhere in the static heap; this is now much lower than the beginning
of the runtime heap.
The specific problem that led me to notice this was that under some
circumstances emacs went into an infinite loop when executing the
following (from morecore_nolock in gmalloc.c):
newsize = heapsize;
do
newsize *= 2;
while ((__malloc_size_t) BLOCK ((char *) result + size) > newsize);
Here `result' is very large, and BLOCK returns a large number because
it's using a small _heapbase. So the test is always true, newsize
becomes 0 because of overflow, and the loop never terminates.
Aside from the infinite loop, however, BLOCK and ADDRESS simply yield
results that don't make sense when the heap starts in high memory but
_heapbase points to low memory.
Surprisingly, I haven't yet run into any problems when Cygwin's heap
starts at 0x20000000. (It was 0x80000000 in the situation above.) I
don't know if there's a good reason for this or if it's just luck. I
think what happens is that malloc behaves as if it's allowed to allocate
memory ranging all the way from the static heap to 0x20000000 and
beyond. If for some reason it really is legal for malloc to use the
memory between the static heap and 0x20000000, then I guess there's no
harm done as long as the large addresses don't lead to overflow.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-10 15:56 ` Eli Zaretskii
2011-08-10 17:52 ` Ken Brown
@ 2011-08-11 21:45 ` Ken Brown
2011-08-12 6:54 ` Eli Zaretskii
1 sibling, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-11 21:45 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/10/2011 11:56 AM, Eli Zaretskii wrote:
>> Date: Tue, 09 Aug 2011 16:09:48 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>> I made one naive attempt to do this, but it didn't work (and it was
>> Cygwin specific). Namely, I made unexec (for Cygwin) set
>> _malloc_initialized = 0 before dumping. The resulting emacs aborted
>> as soon as it was started. I haven't figured out what went wrong,
>> but I'm not sure that's the right answer anyway.
The problem was that realloc got called on memory that had been
allocated prior to dumping, and the malloc information that was used
then had disappeared. I think there's an obvious solution to this. At
the time of reinitialization, we save the previous malloc state. Then
if realloc is called on a pointer to something in the static heap, we
temporarily restore the old state and let realloc proceed as it did in
temacs prior to dumping.
Unless I've (again) missed something obvious, it shouldn't be too hard
to do this. I'm about to go on vacation, but I should have a patch
ready within a couple weeks.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-11 21:45 ` Ken Brown
@ 2011-08-12 6:54 ` Eli Zaretskii
2011-08-12 10:10 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Eli Zaretskii @ 2011-08-12 6:54 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
> Date: Thu, 11 Aug 2011 17:45:41 -0400
> From: Ken Brown <kbrown@cornell.edu>
> CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>
>
> The problem was that realloc got called on memory that had been
> allocated prior to dumping, and the malloc information that was used
> then had disappeared.
Can you show the code which called realloc on that memory? I'm
surprised that Emacs does that, but perhaps I'm missing something.
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-12 6:54 ` Eli Zaretskii
@ 2011-08-12 10:10 ` Ken Brown
2011-08-12 11:33 ` Eli Zaretskii
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-12 10:10 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/12/2011 2:54 AM, Eli Zaretskii wrote:
>> Date: Thu, 11 Aug 2011 17:45:41 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>
>> The problem was that realloc got called on memory that had been
>> allocated prior to dumping, and the malloc information that was used
>> then had disappeared.
>
> Can you show the code which called realloc on that memory? I'm
> surprised that Emacs does that, but perhaps I'm missing something.
Here's the code that I stumbled across (as a result of a SEGV). I
haven't checked to see if there are other examples. From terminal.c:
/* Deletes the bootstrap terminal device.
Called through delete_terminal_hook. */
static void
delete_initial_terminal (struct terminal *terminal)
{
if (terminal != initial_terminal)
abort ();
delete_terminal (terminal);
initial_terminal = NULL;
}
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-12 10:10 ` Ken Brown
@ 2011-08-12 11:33 ` Eli Zaretskii
2011-08-12 12:18 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Eli Zaretskii @ 2011-08-12 11:33 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
> Date: Fri, 12 Aug 2011 06:10:35 -0400
> From: Ken Brown <kbrown@cornell.edu>
> CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>
>
> On 8/12/2011 2:54 AM, Eli Zaretskii wrote:
> >> Date: Thu, 11 Aug 2011 17:45:41 -0400
> >> From: Ken Brown<kbrown@cornell.edu>
> >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
> >>
> >> The problem was that realloc got called on memory that had been
> >> allocated prior to dumping, and the malloc information that was used
> >> then had disappeared.
> >
> > Can you show the code which called realloc on that memory? I'm
> > surprised that Emacs does that, but perhaps I'm missing something.
>
> Here's the code that I stumbled across (as a result of a SEGV). I
> haven't checked to see if there are other examples. From terminal.c:
>
> /* Deletes the bootstrap terminal device.
> Called through delete_terminal_hook. */
>
> static void
> delete_initial_terminal (struct terminal *terminal)
> {
> if (terminal != initial_terminal)
> abort ();
>
> delete_terminal (terminal);
> initial_terminal = NULL;
> }
delete_terminal doesn't call realloc, it just calls xfree.
Do the problems with the Cygwin build go away if the call to
delete_terminal is commented out? That is, does the infloop still
happen or not?
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-12 11:33 ` Eli Zaretskii
@ 2011-08-12 12:18 ` Ken Brown
2011-08-12 20:24 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-12 12:18 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/12/2011 7:33 AM, Eli Zaretskii wrote:
>> Date: Fri, 12 Aug 2011 06:10:35 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>
>> On 8/12/2011 2:54 AM, Eli Zaretskii wrote:
>>>> Date: Thu, 11 Aug 2011 17:45:41 -0400
>>>> From: Ken Brown<kbrown@cornell.edu>
>>>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>>>
>>>> The problem was that realloc got called on memory that had been
>>>> allocated prior to dumping, and the malloc information that was used
>>>> then had disappeared.
>>>
>>> Can you show the code which called realloc on that memory? I'm
>>> surprised that Emacs does that, but perhaps I'm missing something.
>>
>> Here's the code that I stumbled across (as a result of a SEGV). I
>> haven't checked to see if there are other examples. From terminal.c:
>>
>> /* Deletes the bootstrap terminal device.
>> Called through delete_terminal_hook. */
>>
>> static void
>> delete_initial_terminal (struct terminal *terminal)
>> {
>> if (terminal != initial_terminal)
>> abort ();
>>
>> delete_terminal (terminal);
>> initial_terminal = NULL;
>> }
>
> delete_terminal doesn't call realloc, it just calls xfree.
Maybe I mis-remembered where the call to realloc is. I'll reproduce it
later and let you know. (I don't have time at the moment.) But I
assure you that I did a backtrace showing that realloc was called on
something related to terminals.
> Do the problems with the Cygwin build go away if the call to
> delete_terminal is commented out?
No. At the very least, I have to force reinitialization of malloc.
Otherwise the BLOCK macro yields wrong results that lead to infinite
looping or crashing. After reinitialization, I have to be able to
handle calls to free() on memory allocated prior to dumping. Probably
it's OK to just ignore such calls. If I can also take care of calls to
realloc too, everything will be OK.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-12 12:18 ` Ken Brown
@ 2011-08-12 20:24 ` Ken Brown
2011-08-13 8:05 ` Eli Zaretskii
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-12 20:24 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/12/2011 8:18 AM, Ken Brown wrote:
> On 8/12/2011 7:33 AM, Eli Zaretskii wrote:
>>> Date: Fri, 12 Aug 2011 06:10:35 -0400
>>> From: Ken Brown<kbrown@cornell.edu>
>>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>>
>>> On 8/12/2011 2:54 AM, Eli Zaretskii wrote:
>>>>> Date: Thu, 11 Aug 2011 17:45:41 -0400
>>>>> From: Ken Brown<kbrown@cornell.edu>
>>>>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>>>>
>>>>> The problem was that realloc got called on memory that had been
>>>>> allocated prior to dumping, and the malloc information that was used
>>>>> then had disappeared.
>>>>
>>>> Can you show the code which called realloc on that memory? I'm
>>>> surprised that Emacs does that, but perhaps I'm missing something.
>>>
>>> Here's the code that I stumbled across (as a result of a SEGV). I
>>> haven't checked to see if there are other examples. From terminal.c:
>>>
>>> /* Deletes the bootstrap terminal device.
>>> Called through delete_terminal_hook. */
>>>
>>> static void
>>> delete_initial_terminal (struct terminal *terminal)
>>> {
>>> if (terminal != initial_terminal)
>>> abort ();
>>>
>>> delete_terminal (terminal);
>>> initial_terminal = NULL;
>>> }
>>
>> delete_terminal doesn't call realloc, it just calls xfree.
>
> Maybe I mis-remembered where the call to realloc is. I'll reproduce it
> later and let you know. (I don't have time at the moment.) But I
> assure you that I did a backtrace showing that realloc was called on
> something related to terminals.
>
>> Do the problems with the Cygwin build go away if the call to
>> delete_terminal is commented out?
>
> No. At the very least, I have to force reinitialization of malloc.
> Otherwise the BLOCK macro yields wrong results that lead to infinite
> looping or crashing. After reinitialization, I have to be able to
> handle calls to free() on memory allocated prior to dumping. Probably
> it's OK to just ignore such calls. If I can also take care of calls to
> realloc too, everything will be OK.
OK, here's a backtrace showing realloc being called on memory in the static heap (at 0x897040). This is after applying the patch appended at the end of this message. (I think it's self-explanatory, but I'll be glad to explain further.)
(gdb) r -Q
Starting program: /home/kbrown/src/emacs/test/src/emacs.exe -Q
[New Thread 4756.0x1144]
warning: cYgFFFFFFFF 611857C0
[New Thread 4756.0xd80]
warning: cYgstd 28ccf5 d 3
Program received signal SIGSEGV, Segmentation fault.
0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28)
at gmalloc.c:1394
1394 type = _heapinfo[block].busy.type;
(gdb) p block
$1 = 4294838425
(gdb) bt
#0 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28)
at gmalloc.c:1394
#1 0x00636bd7 in _realloc_internal (ptr=0x897040, size=28) at gmalloc.c:1499
#2 0x00636c42 in realloc (ptr=0x897040, size=28) at gmalloc.c:1516
#3 0x00596856 in xrealloc (block=0x897040, size=28) at alloc.c:711
#4 0x00589648 in regex_compile (pattern=0xa7ec60 "site-lisp", size=9,
syntax=3408388, bufp=0x846258) at regex.c:3684
#5 0x0059556d in re_compile_pattern (pattern=0xa7ec60 "site-lisp", length=9,
bufp=0x846258) at regex.c:6361
#6 0x005768d0 in compile_pattern_1 (cp=0x846248, pattern=9810241,
translate=8930309, posix=0) at search.c:150
#7 0x00576b32 in compile_pattern (pattern=9810241, regp=0x8475d8,
translate=8930309, posix=0, multibyte=0) at search.c:245
#8 0x005771b8 in string_match_1 (regexp=9810241, string=9810337,
start=8968218, posix=0) at search.c:401
#9 0x005773ab in Fstring_match (regexp=9810241, string=9810337, start=8968218)
at search.c:451
#10 0x005e4f91 in init_lread () at lread.c:4111
#11 0x0052866c in main (argc=2, argv=0x2001cc00) at emacs.c:1467
(gdb) p _heapbase
$3 = 0x20000000 ""
(gdb) p block
$1 = 4294838425
The SEGV comes from the ridiculous value of block, which was calculated by the BLOCK macro.
=== modified file 'src/gmalloc.c'
--- src/gmalloc.c 2011-08-04 17:04:39 +0000
+++ src/gmalloc.c 2011-08-12 19:47:21 +0000
@@ -584,6 +584,12 @@
mcheck (NULL);
#endif
+#ifdef CYGWIN
+ if (bss_sbrk_did_unexec)
+ /* we're reinitializing the dumped emacs. */
+ memset (_fraghead, 0, BLOCKLOG * sizeof (struct list));
+#endif
+
if (__malloc_initialize_hook)
(*__malloc_initialize_hook) ();
@@ -1054,6 +1060,12 @@
if (ptr == NULL)
return;
+#ifdef CYGWIN
+ if (ptr < _heapbase)
+ /* we're being asked to free something in the static heap */
+ return;
+#endif
+
PROTECT_MALLOC_STATE (0);
LOCK_ALIGNED_BLOCKS ();
=== modified file 'src/unexcw.c'
--- src/unexcw.c 2011-03-17 20:18:59 +0000
+++ src/unexcw.c 2011-08-12 15:37:47 +0000
@@ -33,6 +33,8 @@
extern int bss_sbrk_did_unexec;
+extern int __malloc_initialized;
+
/* emacs symbols that indicate where bss and data end for emacs internals */
extern char my_endbss[];
extern char my_edata[];
@@ -210,9 +212,12 @@
lseek (fd, (long) (exe_header->section_header[i].s_scnptr),
SEEK_SET);
assert (ret != -1);
+ /* force the dumped emacs to reinitialize malloc */
+ __malloc_initialized = 0;
ret =
write (fd, (char *) start_address,
my_endbss - (char *) start_address);
+ __malloc_initialized = 1;
assert (ret == (my_endbss - (char *) start_address));
if (debug_unexcw)
printf (" .bss, mem start 0x%08x mem length %d\n",
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-09 20:09 bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime Ken Brown
2011-08-10 0:24 ` Richard Stallman
2011-08-10 15:56 ` Eli Zaretskii
@ 2011-08-12 23:51 ` grischka
2 siblings, 0 replies; 22+ messages in thread
From: grischka @ 2011-08-12 23:51 UTC (permalink / raw
To: kbrown; +Cc: 9273
> No. At the very least, I have to force reinitialization of malloc.
> Otherwise the BLOCK macro yields wrong results that lead to infinite
> looping or crashing.
I don't think the results from the BLOCK macro are wrong. I think
your analysis of the problem is wrong. ;)
--- grischka
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-12 20:24 ` Ken Brown
@ 2011-08-13 8:05 ` Eli Zaretskii
2011-08-13 13:48 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Eli Zaretskii @ 2011-08-13 8:05 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
> Date: Fri, 12 Aug 2011 16:24:20 -0400
> From: Ken Brown <kbrown@cornell.edu>
> CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28)
> at gmalloc.c:1394
> 1394 type = _heapinfo[block].busy.type;
> (gdb) p block
> $1 = 4294838425
I'm confused: since you patched unexecw.c to set __malloc_initialized
to zero, the dumped Emacs should have called malloc_initialize_1,
which should have allocated a new copy of _heapinfo, that was supposed
to be consistent with the current heap. Why isn't that working? why
`block' still gets a value that is relative to the "old" _heapinfo?
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 8:05 ` Eli Zaretskii
@ 2011-08-13 13:48 ` Ken Brown
2011-08-13 14:41 ` Eli Zaretskii
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-13 13:48 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/13/2011 4:05 AM, Eli Zaretskii wrote:
>> Date: Fri, 12 Aug 2011 16:24:20 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28)
>> at gmalloc.c:1394
>> 1394 type = _heapinfo[block].busy.type;
>> (gdb) p block
>> $1 = 4294838425
>
> I'm confused: since you patched unexecw.c to set __malloc_initialized
> to zero, the dumped Emacs should have called malloc_initialize_1,
> which should have allocated a new copy of _heapinfo, that was supposed
> to be consistent with the current heap. Why isn't that working? why
> `block' still gets a value that is relative to the "old" _heapinfo?
_heapinfo is indeed consistent with the current heap. But the pointer
that was passed to realloc points into the old heap. So applying BLOCK
to that pointer yields an absurd result. I can easily catch such cases
by testing for ptr < _heapbase, as in my patch to _free_internal_nolock,
but I have to figure out the best way to handle them once I've caught
them. I have work in progress that tries to keep track of both heaps,
but I haven't got it working yet.
An alternative would be to have realloc return NULL (or some other
special value) in these cases, but then I would have to find all
possible callers of realloc (with pointers to the old heap) and make
sure they know how to deal with that return value. I'm guessing my
first approach is safer and easier to implement.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 13:48 ` Ken Brown
@ 2011-08-13 14:41 ` Eli Zaretskii
2011-08-13 14:53 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Eli Zaretskii @ 2011-08-13 14:41 UTC (permalink / raw
To: Ken Brown; +Cc: 9273
> Date: Sat, 13 Aug 2011 09:48:52 -0400
> From: Ken Brown <kbrown@cornell.edu>
> CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>
>
> _heapinfo is indeed consistent with the current heap. But the pointer
> that was passed to realloc points into the old heap. So applying BLOCK
> to that pointer yields an absurd result. I can easily catch such cases
> by testing for ptr < _heapbase, as in my patch to _free_internal_nolock,
> but I have to figure out the best way to handle them once I've caught
> them.
malloc a buffer, then copy the contents of the old one to the new
one. You will have to know the size of the old block, which means you
will have to access the old copy of _heapinfo.
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 14:41 ` Eli Zaretskii
@ 2011-08-13 14:53 ` Ken Brown
2011-08-13 15:07 ` Stefan Monnier
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-13 14:53 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 9273@debbugs.gnu.org
On 8/13/2011 10:41 AM, Eli Zaretskii wrote:
>> Date: Sat, 13 Aug 2011 09:48:52 -0400
>> From: Ken Brown<kbrown@cornell.edu>
>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org>
>>
>> _heapinfo is indeed consistent with the current heap. But the pointer
>> that was passed to realloc points into the old heap. So applying BLOCK
>> to that pointer yields an absurd result. I can easily catch such cases
>> by testing for ptr< _heapbase, as in my patch to _free_internal_nolock,
>> but I have to figure out the best way to handle them once I've caught
>> them.
>
> malloc a buffer, then copy the contents of the old one to the new
> one. You will have to know the size of the old block, which means you
> will have to access the old copy of _heapinfo.
Thanks! That's much easier than what I was trying to do.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 14:53 ` Ken Brown
@ 2011-08-13 15:07 ` Stefan Monnier
2011-08-13 15:33 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Monnier @ 2011-08-13 15:07 UTC (permalink / raw
To: Ken Brown; +Cc: 9273@debbugs.gnu.org
>>> _heapinfo is indeed consistent with the current heap. But the pointer
>>> that was passed to realloc points into the old heap. So applying BLOCK
>>> to that pointer yields an absurd result. I can easily catch such cases
>>> by testing for ptr< _heapbase, as in my patch to _free_internal_nolock,
>>> but I have to figure out the best way to handle them once I've caught
>>> them.
>> malloc a buffer, then copy the contents of the old one to the new
>> one. You will have to know the size of the old block, which means you
>> will have to access the old copy of _heapinfo.
> Thanks! That's much easier than what I was trying to do.
Wouldn't it be easier to try and convince malloc to keep using the old
_heapinfo?
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 15:07 ` Stefan Monnier
@ 2011-08-13 15:33 ` Ken Brown
2011-08-13 19:19 ` Stefan Monnier
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-13 15:33 UTC (permalink / raw
To: Stefan Monnier; +Cc: 9273@debbugs.gnu.org
On 8/13/2011 11:07 AM, Stefan Monnier wrote:
>>>> _heapinfo is indeed consistent with the current heap. But the pointer
>>>> that was passed to realloc points into the old heap. So applying BLOCK
>>>> to that pointer yields an absurd result. I can easily catch such cases
>>>> by testing for ptr< _heapbase, as in my patch to _free_internal_nolock,
>>>> but I have to figure out the best way to handle them once I've caught
>>>> them.
>>> malloc a buffer, then copy the contents of the old one to the new
>>> one. You will have to know the size of the old block, which means you
>>> will have to access the old copy of _heapinfo.
>> Thanks! That's much easier than what I was trying to do.
>
> Wouldn't it be easier to try and convince malloc to keep using the old
> _heapinfo?
Probably. That would keep me from having to get involved in the details
of how _heapinfo works, and it would probably be less error prone. So I
would do something like the following:
1. Temporarily restore the pre-dump malloc state.
2. Call xrealloc, getting storage of the new size in the old heap.
3. Copy the contents into a temporary buffer, and then free the recently
allocated storage in the old heap.
4. Restore the malloc state, call malloc, and then copy the contents of
the temporary buffer into the just-acquired storage in the new heap.
Is that what you had in mind?
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 15:33 ` Ken Brown
@ 2011-08-13 19:19 ` Stefan Monnier
2011-08-14 3:13 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Monnier @ 2011-08-13 19:19 UTC (permalink / raw
To: Ken Brown; +Cc: 9273@debbugs.gnu.org
> Probably. That would keep me from having to get involved in the details of
> how _heapinfo works, and it would probably be less error prone. So I would
> do something like the following:
> 1. Temporarily restore the pre-dump malloc state.
> 2. Call xrealloc, getting storage of the new size in the old heap.
> 3. Copy the contents into a temporary buffer, and then free the recently
> allocated storage in the old heap.
> 4. Restore the malloc state, call malloc, and then copy the contents of the
> temporary buffer into the just-acquired storage in the new heap.
> Is that what you had in mind?
No, I meant, during initialization of Emacs, make sure we get the old
_heapinfo state, so that malloc/realloc/... just works.
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-13 19:19 ` Stefan Monnier
@ 2011-08-14 3:13 ` Ken Brown
2011-08-16 13:30 ` Ken Brown
0 siblings, 1 reply; 22+ messages in thread
From: Ken Brown @ 2011-08-14 3:13 UTC (permalink / raw
To: Stefan Monnier; +Cc: 9273@debbugs.gnu.org
On 8/13/2011 3:19 PM, Stefan Monnier wrote:
>> Probably. That would keep me from having to get involved in the details of
>> how _heapinfo works, and it would probably be less error prone. So I would
>> do something like the following:
>
>> 1. Temporarily restore the pre-dump malloc state.
>> 2. Call xrealloc, getting storage of the new size in the old heap.
>> 3. Copy the contents into a temporary buffer, and then free the recently
>> allocated storage in the old heap.
>> 4. Restore the malloc state, call malloc, and then copy the contents of the
>> temporary buffer into the just-acquired storage in the new heap.
>
>> Is that what you had in mind?
>
> No, I meant, during initialization of Emacs, make sure we get the old
> _heapinfo state, so that malloc/realloc/... just works.
No, that wouldn't be good because the old _heapinfo state uses the static heap, which is very small. It's big enough for temacs, but not for a usable emacs. In any case, Eli's suggestion turned out to be very easy to implement. My revised patch is appended below, and it seems to solve all the problems I encountered. I still want to test it further, but I think it's OK. Eli, thanks for your help.
Ken
=== modified file 'src/gmalloc.c'
--- src/gmalloc.c 2011-08-04 17:04:39 +0000
+++ src/gmalloc.c 2011-08-13 22:32:45 +0000
@@ -351,11 +351,19 @@
#endif
#include <errno.h>
-/* How to really get more memory. */
-#if defined(CYGWIN)
+/* On Cygwin there are two heaps. temacs uses the static heap
+ (defined in sheap.c and managed with bss_sbrk), and the dumped
+ emacs uses the Cygwin heap (managed with sbrk). When emacs starts
+ on Cygwin, it reinitializes malloc, and we save the old info for
+ use by free and realloc if they're called with a pointer into the
+ static heap. */
+#ifdef CYGWIN
extern __ptr_t bss_sbrk PP ((ptrdiff_t __size));
extern int bss_sbrk_did_unexec;
+char *bss_sbrk_heapbase;
+malloc_info *bss_sbrk_heapinfo;
#endif
+
__ptr_t (*__morecore) PP ((__malloc_ptrdiff_t __size)) = __default_morecore;
/* Debugging hook for `malloc'. */
@@ -584,6 +592,16 @@
mcheck (NULL);
#endif
+#ifdef CYGWIN
+ if (bss_sbrk_did_unexec)
+ /* we're reinitializing the dumped emacs */
+ {
+ bss_sbrk_heapbase = _heapbase;
+ bss_sbrk_heapinfo = _heapinfo;
+ memset (_fraghead, 0, BLOCKLOG * sizeof (struct list));
+ }
+#endif
+
if (__malloc_initialize_hook)
(*__malloc_initialize_hook) ();
@@ -1054,6 +1072,12 @@
if (ptr == NULL)
return;
+#ifdef CYGWIN
+ if (ptr < _heapbase)
+ /* We're being asked to free something in the static heap. */
+ return;
+#endif
+
PROTECT_MALLOC_STATE (0);
LOCK_ALIGNED_BLOCKS ();
@@ -1346,9 +1370,33 @@
#include <malloc.h>
#endif
-
#define min(A, B) ((A) < (B) ? (A) : (B))
+/* On Cygwin the dumped emacs may try to realloc storage allocated in
+ the static heap. We just malloc space in the new heap and copy the
+ data. */
+#ifdef CYGWIN
+__ptr_t
+special_realloc (ptr, size)
+ __ptr_t ptr;
+ __malloc_size_t size;
+{
+ __ptr_t result;
+ int type;
+ __malloc_size_t block, oldsize;
+
+ block = ((char *) ptr - bss_sbrk_heapbase) / BLOCKSIZE + 1;
+ type = bss_sbrk_heapinfo[block].busy.type;
+ oldsize =
+ type == 0 ? bss_sbrk_heapinfo[block].busy.info.size * BLOCKSIZE
+ : (__malloc_size_t) 1 << type;
+ result = _malloc_internal_nolock (size);
+ if (result != NULL)
+ memcpy (result, ptr, min (oldsize, size));
+ return result;
+}
+#endif
+
/* Debugging hook for realloc. */
__ptr_t (*__realloc_hook) PP ((__ptr_t __ptr, __malloc_size_t __size));
@@ -1375,6 +1423,12 @@
else if (ptr == NULL)
return _malloc_internal_nolock (size);
+#ifdef CYGWIN
+ if (ptr < _heapbase)
+ /* ptr points into the static heap */
+ return special_realloc (ptr, size);
+#endif
+
block = BLOCK (ptr);
PROTECT_MALLOC_STATE (0);
=== modified file 'src/unexcw.c'
--- src/unexcw.c 2011-03-17 20:18:59 +0000
+++ src/unexcw.c 2011-08-12 20:10:03 +0000
@@ -33,6 +33,8 @@
extern int bss_sbrk_did_unexec;
+extern int __malloc_initialized;
+
/* emacs symbols that indicate where bss and data end for emacs internals */
extern char my_endbss[];
extern char my_edata[];
@@ -210,9 +212,12 @@
lseek (fd, (long) (exe_header->section_header[i].s_scnptr),
SEEK_SET);
assert (ret != -1);
+ /* force the dumped emacs to reinitialize malloc */
+ __malloc_initialized = 0;
ret =
write (fd, (char *) start_address,
my_endbss - (char *) start_address);
+ __malloc_initialized = 1;
assert (ret == (my_endbss - (char *) start_address));
if (debug_unexcw)
printf (" .bss, mem start 0x%08x mem length %d\n",
^ permalink raw reply [flat|nested] 22+ messages in thread
* bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime
2011-08-14 3:13 ` Ken Brown
@ 2011-08-16 13:30 ` Ken Brown
0 siblings, 0 replies; 22+ messages in thread
From: Ken Brown @ 2011-08-16 13:30 UTC (permalink / raw
To: Stefan Monnier, Eli Zaretskii; +Cc: 9273-done
I've committed the changes and am closing the bug.
Ken
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2011-08-16 13:30 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-08-09 20:09 bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime Ken Brown
2011-08-10 0:24 ` Richard Stallman
2011-08-10 15:56 ` Eli Zaretskii
2011-08-10 17:52 ` Ken Brown
2011-08-10 18:10 ` Eli Zaretskii
2011-08-10 18:49 ` Ken Brown
2011-08-11 21:45 ` Ken Brown
2011-08-12 6:54 ` Eli Zaretskii
2011-08-12 10:10 ` Ken Brown
2011-08-12 11:33 ` Eli Zaretskii
2011-08-12 12:18 ` Ken Brown
2011-08-12 20:24 ` Ken Brown
2011-08-13 8:05 ` Eli Zaretskii
2011-08-13 13:48 ` Ken Brown
2011-08-13 14:41 ` Eli Zaretskii
2011-08-13 14:53 ` Ken Brown
2011-08-13 15:07 ` Stefan Monnier
2011-08-13 15:33 ` Ken Brown
2011-08-13 19:19 ` Stefan Monnier
2011-08-14 3:13 ` Ken Brown
2011-08-16 13:30 ` Ken Brown
2011-08-12 23:51 ` grischka
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).