bug#56793: Infinite loop in pure_alloc when dumping strings longer than 10K bytes

bug#56793: Infinite loop in pure_alloc when dumping strings longer than 10K bytes

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2584 bytes --]

I apologize for not being able to include significant details of the build,
as this is happening on a sandboxed system in a proprietary context.
I've been attempting to dump emacs built from the 28.1 tarball with a large
number of core libraries preloaded.  I have observed runaway allocation
when attempting to dump with native-compilation enabled and with
native-compilation disabled.
Using gdb I was able to identify the culprit as an infinite "goto again"
loop in pure_alloc:

static void *
pure_alloc (size_t size, int type)
{
void *result;
again:
if (type >= 0)
{
/* Allocate space for a Lisp object from the beginning of the free
space with taking account of alignment. */
result = pointer_align (purebeg + pure_bytes_used_lisp, LISP_ALIGNMENT);
pure_bytes_used_lisp = ((char *)result - (char *)purebeg) + size;
}
else
{


/* Allocate space for a non-Lisp object from the end of the free
space. */
ptrdiff_t unaligned_non_lisp = pure_bytes_used_non_lisp + size;
char *unaligned = purebeg + pure_size - unaligned_non_lisp;
int decr = (intptr_t) unaligned & (-1 - type);
pure_bytes_used_non_lisp = unaligned_non_lisp + decr;
result = unaligned - decr;
}
pure_bytes_used = pure_bytes_used_lisp + pure_bytes_used_non_lisp;
if (pure_bytes_used <= pure_size)
return result;
/* Don't allocate a large amount here,
because it might get mmap'd and then its address
might not be usable. */
int small_amount = 10000;
eassert (size <= small_amount - LISP_ALIGNMENT);
purebeg = xzalloc (small_amount);
pure_size = small_amount;
pure_bytes_used_before_overflow += pure_bytes_used - size;
pure_bytes_used = 0;
pure_bytes_used_lisp = pure_bytes_used_non_lisp = 0;
/* Can't GC if pure storage overflowed because we can't determine
if something is a pure object or not. */
garbage_collection_inhibited++;
goto again;
}

For example, the issue was triggered while attempting to dump ibuffer.el  -
a docstring has 10,655 characters, and the "eassert" is turned off with the
usual CFLAGS, so it just never stops attempting to allocate.
Since unexec is disabled, I don't even understand why avoiding mmap'ed
memory is important.  Pure space is just going to be copied into the cold
storage area of the pdmp file anyway, no?
In any case, changing the small amount to 20000 made the issue go away for
the purpose of testing, but I'm not sure why there isn't some comparison of
the size of the object to the small amount in the logic - either
determining the minimum to allocate in one chunk, or attempting to obtain a
contiguous chunk by successive xzallocs until the required size is obtained.

Lynn

[-- Attachment #2: Type: text/html, Size: 3019 bytes --]

bug#56793: Infinite loop in pure_alloc when dumping strings longer than 10K bytes

[Pip Cet]

On Wed, Jul 27, 2022 at 1:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> I apologize for not being able to include significant details of the build, as this is happening on a sandboxed system in a proprietary context.
> I've been attempting to dump emacs built from the 28.1 tarball with a large number of core libraries preloaded.  I have observed runaway allocation when attempting to dump with native-compilation enabled and with native-compilation disabled.

I believe this is a duplicate of #46916, which was closed as WONTFIX.
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46916) There's a patch
there, IIRC.

Pip

bug#56793: Infinite loop in pure_alloc when dumping strings longer than 10K bytes

[Lynn Winebarger]

On Wed, Jul 27, 2022 at 10:00 AM Pip Cet <pipcet@gmail.com> wrote:
>
> On Wed, Jul 27, 2022 at 1:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> > I apologize for not being able to include significant details of the build, as this is happening on a sandboxed system in a proprietary context.
> > I've been attempting to dump emacs built from the 28.1 tarball with a large number of core libraries preloaded.  I have observed runaway allocation when attempting to dump with native-compilation enabled and with native-compilation disabled.
>
> I believe this is a duplicate of #46916, which was closed as WONTFIX.
> (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46916) There's a patch
> there, IIRC.
>
True, but the triggering issue doesn't really seem to be pure space
exhaustion per se.  Nominal pure space exhaustion doesn't even seem to
prevent producing a portable dump file - unless the object is over 10K
bytes in size.

I understand the long-term solution, but in the meantime we should
still be able to dump emacs with additional files without runaway
allocation.

Lynn

bug#56793: Infinite loop in pure_alloc when dumping strings longer than 10K bytes

[Lynn Winebarger]

On Wed, Jul 27, 2022 at 10:21 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> On Wed, Jul 27, 2022 at 10:00 AM Pip Cet <pipcet@gmail.com> wrote:
> >
> > On Wed, Jul 27, 2022 at 1:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> > > I apologize for not being able to include significant details of the build, as this is happening on a sandboxed system in a proprietary context.
> > > I've been attempting to dump emacs built from the 28.1 tarball with a large number of core libraries preloaded.  I have observed runaway allocation when attempting to dump with native-compilation enabled and with native-compilation disabled.
> >
> > I believe this is a duplicate of #46916, which was closed as WONTFIX.
> > (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46916) There's a patch
> > there, IIRC.
> >
> True, but the triggering issue doesn't really seem to be pure space
> exhaustion per se.  Nominal pure space exhaustion doesn't even seem to
> prevent producing a portable dump file - unless the object is over 10K
> bytes in size.
>
> I understand the long-term solution, but in the meantime we should
> still be able to dump emacs with additional files without runaway
> allocation.

Also, in that archived bug correspondence, it wasn't clear that Eli
noticed that the user is no longer notified that pure space has been
exhausted to know that they need to increase it.  The manual indicates
that there will be an error stating that pure space has been exhausted
- I didn't realize it was until I tracked down this bug.  I was
thinking I would get some indication of how much additional pure space
would be required.
All I know for sure is that setting SITELOAD_PURESIZE_EXTRA to 10^6
did not prevent the issue.