Guile 2.2 TODO

Guile 2.2 TODO

[Andy Wingo]

Hi!

So, the RTL VM and the CPS compiler landed.  I think the consensus is
that Guile 2.2 is basically what's in master, all compiled to RTL.
Anything else we manage to get in is great but that's the fundamental
bit.

To that end, we should start to think about what is necessary to finish
the job, now that we're close.  I'm going to try to leave off things
that aren't necessary but would be nice to have, such as particular
optimization passes.  Those can always be added later.

* Test cases

I'm going to go ahead and mention this one first -- though it's more
part of "the way to get there" and not of "where we're going", writing
tests will be really useful to determine which parts of the compiler are
working and which are not.  This is the easiest, most productive way for
someone interested in this work to contribute.  Add a test to
test-suite/tests/rtl-compilation.test and commit the patch!  If it
fails, even better.

* Source information.

The plan here would be to use DWARF.  We don't use DWARF yet, so there
are a few parts of this.

  - Add a DWARF parser; see the wip-dwarf branch for that.  

  - Emit debugging info entries (DIEs) for each function in a
    compilation unit, into the .debug_info section.  Initially they
    would be minimal, just having the function's address and maybe its
    name.

  - Add machinery to (system vm debug) to get a DIE for a procedure, if
    it is present.

  - Add a macro-instruction to the assembler that records source
    location info.  Have the assembler collect this info.

  - Have the assembler write out source location info into a .debug_line
    section, and link the DIE to that info.

  - Add an appropriate interface to (system vm debug) to parse out line
    number information.  Perhaps model this interface on what is already
    in (system vm program).

  - Shim the existing (system vm program) interfaces to call the new
    (system vm debug) interfaces for RTL programs.

* Local variable information.

This is similar.  Assuming source information is done, you have to add
DW_TAG_variable entries for each variable to their function's DIE, and
link those entries to location lists in the .debug_loc section.  A
location list specifies the bytecode ranges for which a variable is
live.  Then you have to parse it out as well; the existing interfaces in
(system vm program) can be a guide.

This involves some compiler hacking, because we don't explicitly compute
liveness ranges.  I expect we'd have to compute an order in which to
emit labelled expressions (continuations), and use that order in the
slot allocator, the bytecode emitter, and of course when emitting
live/dead labels.  Maybe not, though; maybe it would be sufficient to
just associate live-set used in the slot allocator with each
continuation.  Dunno.

Ideally this gets linked up to the disassembler too.

* Replace bytecode trampolines with RTL trampolines.

foreign.c, control.c, gsubr.c, and continuations.c all have inline
bytecode trampolines to implement calls to "foreign" things.  These need
to be replaced with RTL trampolines.  This can be done now, I think, and
doesn't depend on anything else, given that the two VMs interoperate.

* Prompt, abort, and call/cc.

These bits have stub implementations in the VM and aren't yet
implemented in the compiler.  I think Mark Weaver is on this though, so
hopefully this situation shouldn't last too long.

* Compiling all of Guile to RTL.

As I mentioned, guild compile -t rtl foo.scm will compile to RTL, and
that interoperates fine with the old VM (modulo tail calls).  We can try
this by hand, file by file, checking that things work.  Then we would
switch the order of the compilers in (language tree-il spec) to make CPS
preferred over GLIL.

* Rip out GLIL, assembly, bytecode, objcode, the old VM, etc..

This will be a final glorious set of commits :)

* Update documentation

The manual will need updating.  I'm OK with putting this off to the end.


I think that's pretty much it.  Obviously it would be nice to get a lot
of other things into 2.2 but this is the necessary bit.  I think we
should shoot for a 2.1.0 release around 1 December; dunno.

WDYT?  If no one else is up to it, I'll get on source location info in a
couple weeks.

Andy
-- 
http://wingolog.org/

Re: Guile 2.2 TODO

[Peter TB Brett]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

Hi Andy,

All of this stuff sounds great, and I can't wait for it to be available!
Can I please request an addition to your to-do list: a document that
explains what changes to libguile-using programs will be required?
Getting people from 1.8 to 2.0 is hard enough, and soon Guile will be
*two* major releases ahead of many of my users... I still need to
support people using 1.8.x.

Thanks,

Peter

-- 
Dr Peter Brett <peter@peter-b.co.uk>

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Guile 2.2 TODO

[Andy Wingo]

On Sun 01 Sep 2013 21:07, Peter TB Brett <peter@peter-b.co.uk> writes:

> Can I please request an addition to your to-do list: a document that
> explains what changes to libguile-using programs will be required?

You mean like a history of user-visible changes? :)

  http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=NEWS;h=59133019d63a18fae6156c580826176d7cb15493;hb=HEAD

Cheers,

Andy
-- 
http://wingolog.org/

Re: Guile 2.2 TODO

[Ludovic Courtès]

Hi,

Thanks for the road map!

Andy Wingo <wingo@pobox.com> skribis:

>   - Add machinery to (system vm debug) to get a DIE for a procedure, if
>     it is present.

It would be great if it would support .gnu_debuglink too (info "(gdb)
Separate Debug Files").

Another thing I noticed is that in 2.0 we have location info for code,
but not for (global) variable definitions in general (which prevents
M-. from doing anything useful for non-procedure top-level variables.)

How would that work with DWARF?

> I think that's pretty much it.  Obviously it would be nice to get a lot
> of other things into 2.2 but this is the necessary bit.  I think we
> should shoot for a 2.1.0 release around 1 December; dunno.

Sounds like a good plan.

Personally I’m still very much on the Guix (dark?) side of things, but
I’d like to get on the 2.1 train sometime in the coming months.

Ludo’.

Re: Guile 2.2 TODO

[Andy Wingo]

On Sat 14 Sep 2013 15:48, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>>   - Add machinery to (system vm debug) to get a DIE for a procedure, if
>>     it is present.
>
> It would be great if it would support .gnu_debuglink too (info "(gdb)
> Separate Debug Files").

The guile-dltools stuff does support this, so it should be possible.
Initially I think it makes sense to include it in the .go directly
though, and add things incrementally.

> Another thing I noticed is that in 2.0 we have location info for code,
> but not for (global) variable definitions in general (which prevents
> M-. from doing anything useful for non-procedure top-level variables.)
>
> How would that work with DWARF?

It could work!  Currently M-. just fails if the thing isn't a
procedure.  But this info could go in the DIE.  What would it be
associated with, however?  The scoped name or the variable object?
Anyway we can make this happen.

> Personally I’m still very much on the Guix (dark?) side of things, but
> I’d like to get on the 2.1 train sometime in the coming months.

Cool.  Happy hacking :)

Andy
-- 
http://wingolog.org/

Re: Guile 2.2 TODO

[Andy Wingo]

Hi,

On Sun 01 Sep 2013 13:03, Andy Wingo <wingo@pobox.com> writes:

> * Source information.
>
> The plan here would be to use DWARF.  We don't use DWARF yet, so there
> are a few parts of this.
>
>   - Add a DWARF parser; see the wip-dwarf branch for that.  
>
>   - Emit debugging info entries (DIEs) for each function in a
>     compilation unit, into the .debug_info section.  Initially they
>     would be minimal, just having the function's address and maybe its
>     name.
>
>   - Add machinery to (system vm debug) to get a DIE for a procedure, if
>     it is present.
>
>   - Add a macro-instruction to the assembler that records source
>     location info.  Have the assembler collect this info.
>
>   - Have the assembler write out source location info into a .debug_line
>     section, and link the DIE to that info.
>
>   - Add an appropriate interface to (system vm debug) to parse out line
>     number information.  Perhaps model this interface on what is already
>     in (system vm program).
>
>   - Shim the existing (system vm program) interfaces to call the new
>     (system vm debug) interfaces for RTL programs.

Done!  Woo hoo :)

> * Local variable information.

Still to do, will put off until later.

> * Replace bytecode trampolines with RTL trampolines.
> * Prompt, abort, and call/cc.
> * Compiling all of Guile to RTL.

These are next.

Andy
-- 
http://wingolog.org/

Re: Guile 2.2 TODO

[Andy Wingo]

On Sun 01 Sep 2013 13:03, Andy Wingo <wingo@pobox.com> writes:

> * Replace bytecode trampolines with RTL trampolines.
>
> foreign.c, control.c, gsubr.c, and continuations.c all have inline
> bytecode trampolines to implement calls to "foreign" things.  These need
> to be replaced with RTL trampolines.  This can be done now, I think, and
> doesn't depend on anything else, given that the two VMs interoperate.

I've now done this for gsubr.c.  The initial results look great from RTL
code, but as most things are stack VM code the temporary result is a
slowdown.  Hopefully we can switch over to RTL within a few weeks
though.

> * Prompt, abort, and call/cc.

I did prompt and abort, sorta -- it actually ends up recursing through
the stack VM to abort and so continuations are not rewindable.  That
will get fixed soon but perhaps not urgently, given that compiling the
compiler doesn't depend on rewinding partial continuations.

> * Compiling all of Guile to RTL.

eval.scm compiles fine and appears to be around 50% faster, currently.
However it can't be loaded at boot-time due to some module bootstrapping
issues.  I need to fix that before we can bootstrap Guile off of an
RTL-compiled eval.  Boot-9 is tricky for similar reasons.  The rest of
things should work though; I'll be going through them one by one looking
for miscompilations.

> * Rip out GLIL, assembly, bytecode, objcode, the old VM, etc..
>
> This will be a final glorious set of commits :)

I'm getting very excited about this :)

> I think that's pretty much it.  Obviously it would be nice to get a lot
> of other things into 2.2 but this is the necessary bit.  I think we
> should shoot for a 2.1.0 release around 1 December; dunno.

I think we'll be ready on that date, code-wise, but documentation makes
a 2.1.0 release much less likely -- documenting things takes time.

Regards,

Andy
-- 
http://wingolog.org/

Re: Guile 2.2 TODO

[Ludovic Courtès]

Andy Wingo <wingo@pobox.com> skribis:

> On Sun 01 Sep 2013 13:03, Andy Wingo <wingo@pobox.com> writes:

[...]

>> I think that's pretty much it.  Obviously it would be nice to get a lot
>> of other things into 2.2 but this is the necessary bit.  I think we
>> should shoot for a 2.1.0 release around 1 December; dunno.
>
> I think we'll be ready on that date, code-wise, but documentation makes
> a 2.1.0 release much less likely -- documenting things takes time.

That’s because the doc of the VM in 2.0 put the bar very high.  :-)

FWIW, I think it’s fine for an odd release series to have incomplete
documentation (especially about the internals) if that can be fixed
before the stable series comes out.

Thank you for the impressive work!

Ludo’.