thinking out loud: wip-rtl, ELF, pages, and mmap

[Andy Wingo <wingo@pobox.com>]

Hi,

I've been working on wip-rtl recently.  The goal is to implement good
debugging.  I'll give a bit of background and then get to my problem.

In master, ".go" files are written in the ELF format.  ELF is nice
because it embodies common wisdom on how to structure object files, and
this wisdom applies to Guile fairly directly.  To simplify, ELF files
are cheap to load and useful to introspect.  The former is achieved with
"segments", which basically correspond to mmap'd blocks of memory.  The
latter is achieved by "sections", which describe parts of the file.  The
table of segments is usually written at the beginning of the file, to
make loading easier, and the table of sections is usually at the end, as
it's not usually needed at runtime.  There are usually fewer segments
than sections.  You can have segments in the file that are marked as not
being loaded into memory at runtime.  Usually this is the case for
debugging information.

OK, so that's ELF.  The conventional debugging format to use with ELF is
DWARF, and it's pretty well thought out.  In Guile we'll probably use
DWARF, along with some more basic metadata in .symtab sections.

I should mention that in master, the ELF files are simple wrappers over
2.0-style objcode.  The wip-rtl branch takes more advantage of ELF --
for example, to allocate some constants in read-only shareable memory,
and to statically allocate any constants that need initialization or
relocation at runtime.  ELF also has advantages when we start to do
native compilation: native code can go in another section, for example.

                            *   *   *

OK, so that's the thing.  I recently added support for writing .symtab
sections, and have been looking on how to load that up at runtime, for
example when disassembling functions.  To be complete, there are a few
other common operations that would require loading debug information:

  * Procedure names.
  * Line/column information, for example in backtraces.
  * Arity information and argument names.
  * Local variable names and live ranges (the ,locals REPL command).
  * Generic procedure metadata.

Anyway!  How do you avoid loading this information at runtime?

The original solution I had in mind was to put them in ELF segments that
don't get loaded.  Then at runtime you would somehow map from an IP to
an ELF object, and at that point you would lazily load the unloaded ELF
sections.

But that has a few disadvantages.  One is that it's difficult to ensure
that the lazily-loaded object is the same as the one that you originally
loaded.  We don't keep .go file descriptors open currently, and
debugging would be a bad reason to do so.

Another more serious is that this is a lot of work, actually.  There's a
constant overhead of the data about what is loaded and how to load what
isn't, and the cross-references from the debug info to the loaded info
is tricky.

Then I realized: why am I doing all of this if the kernel has a virtual
memory system already that does all this for me?

So I have a new plan, I think.  I'll change the linker to always emit
sections and segments that correspond exactly in their on-disk layout
and in their in-memory layout.  (In ELF terms: segments are contiguous,
with p_memsz == p_filesz.)  I'll put commonly needed things at the
beginning, and debugging info and the section table at the end.  Then
I'll just map the whole thing with PROT_READ, and set PROT_WRITE on
those page-aligned segments that need it.  (Obviously in the future,
PROT_EXEC as well.)

Then I'll just record a list of ELF objects that have been loaded.
Simple bisection will map IP -> ELF, and from there we have the section
table in memory (lazily paged in by the virtual memory system) and can
find the symtab and other debug info.

So that's the plan.  It's a significant change, and I wondered if folks
had some experience or reactions.

Note that we have a read()-based fallback if mmap is not available.
This strategy also makes the read-based fallback easier.

Thoughts?

Andy
-- 
http://wingolog.org/