Re: MPS: weak hash tables

[Pip Cet <pipcet@protonmail.com>]

On Wednesday, July 3rd, 2024 at 07:04, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> Pip Cet pipcet@protonmail.com writes:
> 
> > > I can understand that Ravenbrook documents these restrictions, for
> > > future (payed) developments and so on, but, you know... implementing
> > > them gets pretty ugly pretty quickly. (And I wonder if the emulation
> > > brings enough to warrant the effort.)
> > 
> > Frankly, I wonder how they'd feel about a patch to make emulation
> > optional on all architectures, so the restrictions would be optional
> > as well. I'm playing with a qemu IA32 environment, but haven't been
> > able to trigger the emulation code yet (even on IA32, only some very
> > simple instructions are emulated).
> 
> Right, they say themselves that only simple instructions are handled.
> 
> Hm, one could try. Maybe you could submit an issue for that on their
> github project? The only "problem" is see is that they are not very
> responsive. Not that they aren't very friendly and so on, it just takes
> them some time to react, and sometimes they don't.

I believe we should leave our options open as far as modifying MPS goes: my understanding is we could, if we really wanted to (but IANAL), but I'd strongly prefer to discuss that only if and when scratch/igc is merged. Everything else is premature.

> (When I sent a mail to info@ravenbrook.com some months ago to make them
> aware that something in Emacs is going on wrt MPS and to thank them for
> MPS (couldn't find another address), someone told me that Ravenbrook
> isn't that large anymore, or something like that. My guess is that they
> are getting old, too. I think MPS had its first release, commercial,
> in 1997.)
> 
> > I've run into another issue: finalization. MPS's take on that is
> > rather unusual, in that an object can be "finalized" while weak
> > references to it still exist (and destruction can be vetoed by the
> > finalization code creating a new strong reference to it, IIUC). The
> > upshot of this is that this code:
> > 
> > (setq bignum (1+ most-positive-fixnum))
> > (setq table (make-hash-table :test 'eq :weakness 'key))
> > (puthash bignum t table)
> > table
> > (setq bignum nil)
> > (setq values nil)
> > (igc--collect)
> > table
> > 
> > produces a table with a nonsensical/random bignum as key, because the
> > memory has been freed and reused for something else.
> > 
> > I have a patch here which "splats" finalized pvecs so they become
> > PVEC_FREE, ignores such objects during iteration, and gets rid of the
> > "count" element, instead counting the elements for (hash-table-count
> > weak-table). I'll install it after some more testing, unless a better
> > solution occurs to someone.
> 
> I have a vague memory that the docs say somewhere that a finalizer can
> decline finalization by creating a strong reference, but I can't find it
> anymore, as usual. Could we use that somehow, maybe?

https://memory-pool-system.readthedocs.io/en/latest/topic/finalization.html#topic-finalization:

The client program may choose to keep the finalized block alive by keeping a strong reference to the finalized object after discarding the finalization message.

This process is known as resurrection and in some finalization systems requires special handling, but in the MPS this just is just the usual result of the rule that strong references keep objects alive.


We could use that for key-or-value hash tables, but those aren't really my priority.

I don't see a way to use it for the more usual weak hash tables, but maybe I'm missing something.

> > > It would be nice if the ugliness could be encapsulated so that one
> > > doesn't have to see it all the time, as far as that it possible in C
> > > :-). Or conditionalized, maybe, because with Helmut's idea (which I find
> > > the right one), we're using and additional word for weak references.
> > 
> > How hard would it be to "just" add struct igc_headers to the remaining
> > non-headered objects? I don't really want to reopen the "get rid of
> > pure space" discussion again, but that's probably the hard part?
> 
> I'd say it's not hard to add the headers to pure space. (Let me briefly
> sprinkle in that pure space should die :-)). There is a function
> pure_alloc that is used to allocate objects in pure space, AFAIK.

git merge scratch/no-purespace :-)

> For lispsym, main_threads, and subrs (DEFUN etc.), one would have to
> create static structs with an igc_header member. That's probably also
> not really difficult but could result in a lot of work.
> 
> And of course igc_header would have to be made visible everywhere, and
> so on. A lot of work...

You're right. I think Helmut's idea is great, actually, and I don't mind the extra cost for now. Large weak hash tables are expensive, and memory is the least of that problem.

Pip