Re: Dumper problems and a possible solutions

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> > Thanks for the feedback. Can you elaborate on how/why the hash
>> > changes, and where it's stored that would need to be updated?
>> When placing an object in a hash-table, the hashing function often just
>> uses the address as "the hash value".  So any hash-table that uses such
>> a hash-function will need to be rehashed after relocation.
> I see. Is this hashing all at the C level, or is it happening in lisp
> code?

It's in C.

> Can the lisp code even see the address value for lisp objects?

It usually doesn't see it, but the `sxhash' function does return values
which can depend on the address of objects (not for cons cells or
arrays, but for objects such as processes, markers, buffer, overlays,
...).
I think it happens rarely enough that we can hope to be OK on this front.

> If it's purely at the C level I doubt it would be hard to re-do the
> hashes but I obviously haven't read the relevant code.

Indeed, we just need to rehash all the hash-tables we find while
traversing the heap.

>> Yes, the GC already knows how to find the references that are inside
>> Lisp objects, but there can also be references coming from global
>> variables (for sure) or non-Lisp data-structures or maybe from the stack
>> (not sure about those last two).
> How does the GC avoid freeing objects that have these kinds of
> references?

It knows about some of those pointers (via `staticpro' for global
variables and via conservative stack scanning for the stack).

> BTW, at the point of dumping, my impression is that there should not
> be relevant references from the stack;

That'd be my hope as well.

>> We could support relocation at mmap-time to solve this.
> Yes, but that's conceptually just as difficult as dumping to a C
> array: you have to patch up all the addresses and the hash values will
> change.

Agreed.  Relocation is the big issue and pretty much any technique we
may like to use will need to address the problem.
I wonder how smalltalk machines deal with it.

> I agree completely. The current situation makes it nearly impossible
> to port emacs to a system that's not making strong guarantees about
> its implementation internals, and (at least from my understanding
> reading list archives) it's imposing ugly constraints on existing
> implementations (glibc) not to change internals in ways that would
> break emacs' dumper. I would really like to see fixing this issue
> treated as a priority in the future direction of emacs.

It's been a latent problem for the last 20 years or so, but it rarely
bites, so it's not of terribly high priority in general, especially
since new systems don't show up very often.
But it's important enough that we might be willing to pay some price
(e.g. the relocation code will likely either require significant
changes to the GC code, or it will duplicate significant chunks of the
GC code).


        Stefan