Re: MPS: weak hash tables

[Pip Cet <pipcet@protonmail.com>]

On Wednesday, July 3rd, 2024 at 06:11, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> Helmut Eller eller.helmut@gmail.com writes:
> > On Tue, Jul 02 2024, Pip Cet wrote:
> > 
> > > Also, there's the whole caution thing about weak objects containing
> > > only unaligned words or words pointing directly to a base object,
> > > which is only relevant on Unix/i386, IIRC. (MPS emulates instructions
> > > to simulate fine-grained barriers, which is a really cool idea; I'd
> > > still like an option to turn it off though...). That would mean we
> > > have to replace Lisp_Objects and use the ptr member of our union (and
> > > that's the reason I'm using fixnums rather than plain integers for the
> > > hash).
> > 
> > Why do you think that the restriction only applies to 32-bit systems?
> > My interpretation of
> > 
> > Section 7.4. Caution
> > ...
> > “Aligned pointer” means a word whose numeric value (that is, its value
> > when treated as an unsigned integer) is a multiple of the size of a
> > pointer. If you’re using a 64-bit architecture, that means that an
> > aligned pointer is a multiple of 8 and its bottom three bits are zero.
> > ...
> > 
> > is that it applies to 64-bit machines as well.
> 
> 
> OTOH, when I see this in a bit broader context, namely
> 
> 7.3
> ...
> 
> Emulation of accesses to protected objects happens when all of the
> following are true:
> 
> The object is a weak object allocated in an AWL pool.
> 
> The MPS is running on Linux/IA-32 or Windows/IA-32. Extending this
> list to new (reasonable) operating systems should be tolerable (for
> example, macOS/IA-32). Extending this to new processor architectures
> requires more work.
> 
> The processor instruction that is accessing the object is of a
> suitable simple form. The MPS doesn’t contain an emulator for all
> possible instructions that might access memory, so currently it only
> recognizes and emulates a simple MOV from memory to a register or
> vice-versa.
> 
> Contact us if you need emulation of access to weak references for new
> operating systems, processor architectures, or memory access
> instructions.
> 
> 7.4. Caution
> 
> Because of the instruction emulation described in Protection faults
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> above, AWL places the following restriction on the format of objects
> allocated in it:
> 
> Each slot in an object must either be a valid word-aligned reference,
> or else the bottom bits of the word must be non-zero so that it does
> not look like an aligned pointer.
> 
> “Aligned pointer” means...
> 
> I'd bet that these restrictions don't matter when emulation is not done,
> which is the case for 64 bit processors, not IA32 etc. And indeed, on my
> machine with arm64 splatting in the marker vectors works just fine.
> 
> I can understand that Ravenbrook documents these restrictions, for
> future (payed) developments and so on, but, you know... implementing
> them gets pretty ugly pretty quickly. (And I wonder if the emulation
> brings enough to warrant the effort.)

Frankly, I wonder how they'd feel about a patch to make emulation optional on all architectures, so the restrictions would be optional as well. I'm playing with a qemu IA32 environment, but haven't been able to trigger the emulation code yet (even on IA32, only some very simple instructions are emulated).

I've run into another issue: finalization. MPS's take on that is rather unusual, in that an object can be "finalized" while weak references to it still exist (and destruction can be vetoed by the finalization code creating a new strong reference to it, IIUC). The upshot of this is that this code:

(setq bignum (1+ most-positive-fixnum))
(setq table (make-hash-table :test 'eq :weakness 'key))
(puthash bignum t table)
table
(setq bignum nil)
(setq values nil)
(igc--collect)
table

produces a table with a nonsensical/random bignum as key, because the memory has been freed and reused for something else.

I have a patch here which "splats" finalized pvecs so they become PVEC_FREE, ignores such objects during iteration, and gets rid of the "count" element, instead counting the elements for (hash-table-count weak-table). I'll install it after some more testing, unless a better solution occurs to someone.

> It would be nice if the ugliness could be encapsulated so that one
> doesn't have to see it all the time, as far as that it possible in C
> :-). Or conditionalized, maybe, because with Helmut's idea (which I find
> the right one), we're using and additional word for weak references.

How hard would it be to "just" add struct igc_headers to the remaining non-headered objects? I don't really want to reopen the "get rid of pure space" discussion again, but that's probably the hard part?

Pip