Re: Dumper problems and a possible solutions

[Rich Felker <dalias@libc.org>]

On Tue, Jun 24, 2014 at 04:24:47PM -0400, Stefan Monnier wrote:
> > Thanks for the feedback. Can you elaborate on how/why the hash
> > changes, and where it's stored that would need to be updated?
> 
> When placing an object in a hash-table, the hashing function often just
> uses the address as "the hash value".  So any hash-table that uses such
> a hash-function will need to be rehashed after relocation.

I see. Is this hashing all at the C level, or is it happening in lisp
code? Can the lisp code even see the address value for lisp objects?
If it's purely at the C level I doubt it would be hard to re-do the
hashes but I obviously haven't read the relevant code.

> > As far as the relocation, my impression is that it would just need to
> > be able to identify pointers in lisp objects (this is already possible
> > since the GC needs to do it, right?), and rewrite them to
> > (essentially) "static_lisp_heap + offset_of_pointed_to_object" when
> > writing the dump out as a C array.
> 
> Yes, the GC already knows how to find the references that are inside
> Lisp objects, but there can also be references coming from global
> variables (for sure) or non-Lisp data-structures or maybe from the stack
> (not sure about those last two).

How does the GC avoid freeing objects that have these kinds of
references?

BTW, at the point of dumping, my impression is that there should not
be relevant references from the stack; the whole call frame at the
time of dumping is thrown away and replaced by the new invocation of
main. Conceptually, the dumper can be thought of as a longjmp back to
the start of main.

> > This is the xemacs "portable dumper" approach, and I believe it's
> > inferrior because it depends on being able to map back at the same
> > location.
> 
> We could support relocation at mmap-time to solve this.

Yes, but that's conceptually just as difficult as dumping to a C
array: you have to patch up all the addresses and the hash values will
change.

> Neither solution is clearly superior to the other, it just depends on
> you priority.  For me, either way would be an improvement over what we
> have (tho, again, it might still depend on the constraints imposed by
> the need to perform relocation).

I agree completely. The current situation makes it nearly impossible
to port emacs to a system that's not making strong guarantees about
its implementation internals, and (at least from my understanding
reading list archives) it's imposing ugly constraints on existing
implementations (glibc) not to change internals in ways that would
break emacs' dumper. I would really like to see fixing this issue
treated as a priority in the future direction of emacs.

Rich