bug#36431: Crash in marker.c:337

[Eli Zaretskii <eliz@gnu.org>]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: wl@gnu.org,  36431@debbugs.gnu.org
> Date: Tue, 02 Jul 2019 13:51:30 -0400
> 
> - we insert the new bytes at the beginning of the gap, in order to have
>   room to grow if there are more bytes than expected, and also in case
>   there are fewer bytes than expected (in which case we'd otherwise
>   have to move the bytes we just read so they properly end at the end
>   of the gap).

Also, you will see in insert-file-contents that it supports quitting
while reading a huge file, and also the REPLACE argument, where we
detect the same contents at beginning and end of the file and the
buffer.

> - decode_coding_gap wants the new input bytes to be at the end of the
>   gap, so that we can put the decoded chars at the beginning of the gap
>   and as one grows the other shrinks, so we don't need space for "IN +
>   OUT" bytes but only for "OUT" bytes.  Is that right (I'm trying to
>   find some comment or other evidence that this is the case, but
>   haven't found it yet).

That's right.  The comment you are looking for (well, at least part of
it) is in the commentary before decode_coding, where it explains the
semantics of CODING->src_pos.  You will see at the beginning of
decode_coding_gap how it sets things up according to that hairy
protocol.

> IOW, it should be possible to optimize the common case by reading the
> new bytes into the end of the gap to avoid moving everything in the
> common case (if the number of bytes read is different from originally
> expected, we'll have to do extra work, but for the common case where we
> know the file size upfront and it doesn't change while we read it, this
> will save us some work).
> 
> But the effort is probably not worth the trouble: a memmove of a few
> gigabytes costs relatively little compared to the cost of actually
> decoding those same gigabytes.

Right.  Also, there are the other subtle issues with quitting, the
REPLACE argument, special files, etc.