bug#11519: "Wrong type argument: characterp" building custom-deps while boostrapping

[Eli Zaretskii <eliz@gnu.org>]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Richard Stallman <rms@gnu.org>,  handa@gnu.org,  schwab@linux-m68k.org,  lekktu@gmail.com,  11519@debbugs.gnu.org
> Date: Wed, 23 May 2012 10:16:17 -0400
> 
> > Which other places use C pointers to buffer text and call functions
> > that can allocate memory?
> 
> IIUC any place that uses STRING_CHAR_AND_LENGTH on buffer text is
> vulnerable to the problem.

That's not true.  As long as you access buffer text through character
position, you are safe.  The only situation where you are vulnerable
is if you store a C pointer to buffer text, e.g., like this:

   char *text = BEGV_ADDR;
or
   char *text = BYTE_POS_ADDR (current_buffer->pt);

then invoke some function that can allocate or reallocate memory, and
_then_ access buffer text through that pointer.

If you find such a code anywhere, then that's a bug similar to this
one.

> If you really want to install your workaround on the emacs-24 branch, go
> for it but let's try to find a real fix for the trunk.

What kind of real fix are you looking for?  I agree with Handa-san:
being able to suppress relocation in select places is a good feature.
Why shouldn't it be the fix in this case, and what better fix can we
invent when we use an essentially externally maintained code (AFAIR,
regex will at some point be re-sync'ed with gnulib) that cannot be
expected to change its code radically so as not to access buffer text
through C pointers?

> >> >> I wonder: why do we use REL_ALLOC?
> >> > AFAIK, we do that only on platforms that don't support mmap for
> >> > allocating buffer text.
> >> So, IIUC the only reason to use it is so that we can more often return
> >> memory to the OS even for the non-mmap case?  Is that because returning
> >> memory can only be done via sbrk style memory management?
> > I don't think this is only about _returning_ memory.  It is first and
> > foremost about not _asking_ for more memory when we can come up with
> > it by reshuffling buffer text.
> 
> So you're saying it's use for fragmentation reasons?

Yes.

> But on other platforms where we use mmap, we do suffer from this
> fragmentation, and yet it doesn't seem to be a real source of problem.

I don't know enough about mmap to answer that.  I vaguely recollect
that mmap avoids such fragmentation as an inherent feature, but I may
be wrong.

> That's why I think the only real reason is because memory can only be
> returned via sbrk-style memory management (i.e. only free memory at the
> end of the heap can be returned).  Is that right?

Yes, AFAIK.

> I guess my question turns into "why do we use gmalloc.c instead of
> a malloc library that uses mmap (or some other mechanism that lets it
> return large free chunks to the OS)"?

Use of gmalloc is a different issue.  We were talking about ralloc.c.
You could use one, but not the other.

> AFAIK, Windows is pretty much the only system where we use gmalloc.c and
> ralloc.c nowadays.

My reading of configure is that we use it on more than just Windows
(and MS-DOS).  Basically, any platform that uses gmalloc.c (which is
the default, turned off only for GNU/Linux and Darwin) also uses
ralloc.c.

> Does anyone remember why we don't use the system malloc under
> Windows (and Cygwin)?

I find it hard to believe that going through system malloc on
MS-Windows will let us use buffers as large as 1.5 GB (on a 32-bit
machine).  To achieve this today, we reserve a 2GB contiguous chunk of
address space at startup, and then commit and uncommit parts of it as
needed (see w32heap.c).  ralloc.c has an important part in this
arrangement.