Re: MPS: out-of-memory

[Eli Zaretskii <eliz@gnu.org>]

[I've changed the Subject, since this ceased to be about weak hash
tables long ago.]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Helmut Eller <eller.helmut@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   pipcet@protonmail.com,  emacs-devel@gnu.org
> Date: Mon, 08 Jul 2024 11:24:25 +0200
> 
> Andrea Corallo <acorallo@gnu.org> writes:
> 
> >>   ./src/emacs -Q -batch -l \
> >>   ~/.emacs.d/elpa/elisp-benchmarks-1.14/elisp-benchmarks.el --eval \
> >>   '(elisp-benchmarks-run "pidigits" nil 1)'
> >>
> >> finishes and requires 41.82 seconds compared to the 12.26 of the non-MPS
> >> version.  And it uses something like 2.8 GB compared to 350 MB of the
> >> non-MPS version (estimated by looking at the RES column in top).
> >
> > Such a difference in memory footprint is because MPS can't GC at the
> > speed the main thread is allocating or for some other reason?
> 
> Yes, that's it basically. The client is allocating at a high rate,
> and not much is dying until the end of that phase. Both together mean
> that MPS is scanning like wild to make some room, without having a
> chance to find something it can discard.

Btw, the current GC in Emacs has a feature for when Emacs runs out of
memory: it sets some small amount of memory aside, and uses it when
"memory-full" condition happens, because some memory is needed to show
the memory-full error message, and then shut down in an orderly
fashion:

  void
  memory_full (size_t nbytes)
  {
    if (!initialized)
      fatal ("memory exhausted");

    /* Do not go into hysterics merely because a large request failed.  */
    bool enough_free_memory = false;
    if (SPARE_MEMORY < nbytes)
      {
	void *p;

	MALLOC_BLOCK_INPUT;
	p = malloc (SPARE_MEMORY);
	if (p)
	  {
	    free (p);
	    enough_free_memory = true;
	  }
	MALLOC_UNBLOCK_INPUT;
      }

    if (! enough_free_memory)
      {
	Vmemory_full = Qt;
	consing_until_gc = min (consing_until_gc, memory_full_cons_threshold);

	/* The first time we get here, free the spare memory.  */
	for (int i = 0; i < ARRAYELTS (spare_memory); i++)
	  if (spare_memory[i])
	    {
	      if (i == 0)
		free (spare_memory[i]);
	      else if (i >= 1 && i <= 4)
		lisp_align_free (spare_memory[i]);
	      else
		lisp_free (spare_memory[i]);
	      spare_memory[i] = 0;
	    }
      }

    /* This used to call error, but if we've run out of memory, we could
       get infinite recursion trying to build the string.  */
    xsignal (Qnil, Vmemory_signal_data);
  }

See also refill_memory_reserve.

Do we perhaps need something like that with MPS?