Re: MPS: dangling markers

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 01 Jul 2024 17:14:11 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: Ihor Radchenko <yantar92@posteo.net>, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org, eller.helmut@gmail.com
> 
> On Monday, July 1st, 2024 at 04:22, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> > Pip Cet pipcet@protonmail.com writes:
> > 
> > > On Sunday, June 30th, 2024 at 19:22, Gerd Möllmann gerd.moellmann@gmail.com wrote:
> > > 
> > > > Thanks! What do youo think about making a patch containing only your
> > > > weak hash tables, and leaving the BUF_MARKERS alone for now?
> > > 
> > > I think that's the best way forward. Patch attached.
> > 
> > Could you please send me something from git format-patch? That way I'd
> > have commit message and your authorship would also be clear. Or even
> > better, if you have the rights could you please commit to the branch?
> 
> I'll do that. Please let me know what I got wrong.
> 
> > > > That way
> > > > igc could support the existing uses of weak hash tables (I remember one
> > > > in the CLOS department somehwere), and they would be somewhat tested.
> > > > Don't remember if we have unit tests for them.
> > > 
> > > It seems MPS isn't very eager about splatting weak references during
> > > ordinary automatic GC, FWIW. What I'm observing with
> > > 
> > > (while t
> > > (dotimes (i 10000)
> > > (puthash (cons 1 2) (cons 3 4) table))
> > > (message "%S" (hash-table-count table))
> > > (sit-for 0.1))
> > > 
> > > is that the hash table starts out at 0, grows quickly, resets to
> > > count=0 once, then keeps growing and never splats any references after
> > > that. It's quite possible this is a bug in my code, of course.
> > 
> > Yes, it's not eagerly splatting. Don't know. Which reminds me that I
> > wanted to look if the AWL pool maybe has some paramter that one could
> > set, or something else influences that, like the mortality rate of the
> > generation chain. Or something completely different.
> 
> I debugged this a little, and it turns out that when we alternate between two weak hash tables, splatting works fine. It seems that if MPS receives a SIGSEGV in a segment belonging to a weak hash table, it scans it in "exact" mode, not "weak" mode, in order to continue execution as soon as possible. That's how I read this comment in mps/trace.c:
> 
>  * If the trace band is EXACT then we scan EXACT. This might prevent
>  * finalisation messages and may preserve objects pointed to only by weak
>  * references but tough luck -- the mutator wants to look.
> 
> So I don't think this will be a problem in practice...

The 32-bit build of the branch is now broken: dumping dies with

  lisp.h:1241: Emacs fatal error: assertion failed: !FIXNUM_OVERFLOW_P (n)

Here's the backtrace:

  lisp.h:1241: Emacs fatal error: assertion failed: !FIXNUM_OVERFLOW_P (n)

  Thread 1 hit Breakpoint 1, terminate_due_to_signal (sig=sig@entry=22,
      backtrace_limit=backtrace_limit@entry=2147483647) at emacs.c:443
  443     {
  (gdb) bt
  #0  terminate_due_to_signal (sig=sig@entry=22,
      backtrace_limit=backtrace_limit@entry=2147483647) at emacs.c:443
  #1  0x009ca26d in die (
      msg=msg@entry=0xf6ee2d <i_fwd+1057> "!FIXNUM_OVERFLOW_P (n)",
      file=file@entry=0xf6edec <i_fwd+992> "lisp.h", line=line@entry=1241)
      at alloc.c:8356
  #2  0x009ff199 in make_fixnum (n=<optimized out>) at lisp.h:1241
  #3  0x00a0fdb8 in make_fixnum (n=<optimized out>) at fns.c:5620
  #4  maybe_resize_weak_hash_table (h=<optimized out>, h=<optimized out>)
      at fns.c:5598
  #5  weak_hash_put (h=<optimized out>, h@entry=0xb45c1b8, key=<optimized out>,
      key@entry=XIL(0xb46065b), value=<optimized out>,
      value@entry=XIL(0xa4088b8), hash=<optimized out>, hash@entry=3269884494)
      at fns.c:5665
  #6  0x00a0fed2 in Fputhash (key=XIL(0xb46065b), value=XIL(0xa4088b8),
      table=XIL(0xb45c1bd)) at fns.c:6453
  #7  0x00a4b55d in exec_byte_code (fun=XIL(0xf447a5), args_template=514,
      args_template@entry=0, nargs=3, nargs@entry=0, args=0x1a956144,
      args@entry=0x0) at lisp.h:759
  #8  0x00a4be43 in Fbyte_code (bytestr=<optimized out>, vector=XIL(0xb46027d),
      maxdepth=make_fixnum(6)) at bytecode.c:330
  #9  0x009fa6e8 in eval_sub (form=form@entry=XIL(0xb45feab)) at eval.c:2629
  #10 0x00a360c8 in readevalloop (readcharfun=readcharfun@entry=XIL(0x48a8),
      infile0=infile0@entry=0x749f048,
      sourcename=sourcename@entry=XIL(0xb448914),
      printflag=printflag@entry=false, unibyte=unibyte@entry=XIL(0),
      readfun=readfun@entry=XIL(0), start=start@entry=XIL(0),
      end=<optimized out>, end@entry=XIL(0)) at lread.c:2541
  #11 0x00a36b1f in Fload (file=<optimized out>, noerror=XIL(0),
      nomessage=XIL(0), nosuffix=XIL(0), must_suffix=<optimized out>)
      at lisp.h:1194
  #12 0x009fa69b in eval_sub (form=form@entry=XIL(0xb4486ab)) at eval.c:2637
  #13 0x00a360c8 in readevalloop (readcharfun=readcharfun@entry=XIL(0x48a8),
      infile0=infile0@entry=0x749f638,
      sourcename=sourcename@entry=XIL(0xa84807c),
      printflag=printflag@entry=false, unibyte=unibyte@entry=XIL(0),
      readfun=readfun@entry=XIL(0), start=start@entry=XIL(0),
      end=<optimized out>, end@entry=XIL(0)) at lread.c:2541
  #14 0x00a36b1f in Fload (file=<optimized out>, noerror=XIL(0),
      nomessage=XIL(0), nosuffix=XIL(0), must_suffix=<optimized out>)
      at lisp.h:1194
  #15 0x009fa69b in eval_sub (form=form@entry=XIL(0xa847d43)) at eval.c:2637
  #16 0x009fc7be in Feval (form=XIL(0xa847d43), lexical=lexical@entry=XIL(0x18))
      at eval.c:2482
  #17 0x0095593e in top_level_2 () at lisp.h:1194
  #18 0x009f4bc2 in internal_condition_case (
      bfun=bfun@entry=0x9558e0 <top_level_2>, handlers=handlers@entry=XIL(0x48),
      hfun=hfun@entry=0x95f47e <cmd_error>) at eval.c:1629
  #19 0x00956063 in top_level_1 (ignore=XIL(0)) at lisp.h:1194
  #20 0x009f4adc in internal_catch (tag=tag@entry=XIL(0x93d8),
      func=func@entry=0x95603a <top_level_1>, arg=arg@entry=XIL(0))
      at eval.c:1308
  #21 0x009556ff in command_loop () at lisp.h:1194
  #22 0x0095f039 in recursive_edit_1 () at keyboard.c:765
  #23 0x0095f329 in Frecursive_edit () at keyboard.c:848
  #24 0x00b9f109 in main (argc=<optimized out>, argv=<optimized out>)
      at emacs.c:2651

This code:

  static void
  maybe_resize_weak_hash_table (struct Lisp_Weak_Hash_Table *h)
  {
    if (XFIXNUM (h->strong->next_free) < 0)
      {
	ptrdiff_t old_size = WEAK_HASH_TABLE_SIZE (h);
	ptrdiff_t min_size = 6;
	ptrdiff_t base_size = min (max (old_size, min_size), PTRDIFF_MAX / 2);
	/* Grow aggressively at small sizes, then just double.  */
	ptrdiff_t new_size =
	  old_size == 0
	  ? min_size
	  : (base_size <= 64 ? base_size * 4 : base_size * 2);

is unsafe, since AFAIU it could produce new_size = PTRDIFF_MAX, and
that cannot fit in a fixnum, not even on a 64-bit system (although in
a 32-bit build this is much easier to reach).  So this loop:

      for (ptrdiff_t i = 0; i < new_size - 1; i++)
	strong->next[i].lisp_object = make_fixnum (i + 1);

will then cause a fixnum overflow, which happens here.

However, using MOST_POSITIVE_FIXNUM instead of PTRDIFF_MAX doesn't
help, so something else is at work here.