vm branch now uses vm repl by default

vm branch now uses vm repl by default

[Andy Wingo]

Hi,

I've enabled the VM repl by default on the vm branch. Here's a brief
annotated tour:

    $ ./pre-inst-guile
    Guile Scheme interpreter 0.5 on Guile 1.9.0
    Copyright (C) 2001-2008 Free Software Foundation, Inc.

    Enter `,help' for help.

A very pleasant introduction, no? And totally configurable with a nice
programming interface.

    scheme@(guile-user)> 'foo
    $1 = foo

A normal repl. The `scheme' indicates the current language, and
(guile-user) is the current module. The $1 = foo is from the history
module. If you don't have this in your ~/.guile, you really really want
it:

    (use-modules (ice-9 readline) (ice-9 history))
    (activate-readline)

Anyway, moving on:

    scheme@(guile-user)> (lambda () (pk a #:bar))
    $2 = #<program b755ecf8>

Entering in expressions actually compiles and executes them. In this
case we compiled and loaded a thunk. Compiled procedures are "programs",
and print as such.

    scheme@(guile-user)> (define a '(a . pair))
    scheme@(guile-user)> ($2)

    ;;; ((a . pair) #:bar)
    $3 = #:bar

Procedures resolve toplevel bindings lazily, as in the interpreter, so
you get letrec semantics in the repl.

    scheme@(guile-user)> ,x $2

There is a wealth of meta-commands at the repl, commands that start with
`,'. This command, `,x', is an abbreviation for `,disassemble'. Its
output is this:

    Disassembly of #<program b755ecf8>:

    nargs = 0  nrest = 0  nlocs = 0  nexts = 0

The program has no arguments, no rest arguments, no local variables, and
no external (lexically-bound) variables.

    Bytecode:

       0    (late-variable-ref 0)
       2    (late-variable-ref 1)
       4    (object-ref 2)                  ;; #:bar
       6    (tail-call 2)

    Objects:

       0    #<variable b80057f0 value: #<program b8005858>>
       1    #<variable b7569af0 value: (a . pair)>
       2    #:bar

Late-variable-ref looks at a cell in the object vector. If it is a
symbol, it is resolved relative to the module that was current when the
program was made. The object cell is then replaced with the resulting
resolved variable. Here we see that objects 0 and 1 were already
resolved. Object 2 is just the constant, #:bar.

All of the ref instructions push their values on the stack. Call
instructions pop off arguments, if any, then call the program on the top
of the stack. In this case it is a tail call.

    Sources:

       8    #(1 11 #f)

Some instructions are annotated with source information. In this case,
when the instruction pointer is at 8 (right after the tail-call -- one
byte for tail-call and one for the number of arguments, 2), the original
source was at line 1 and column 11 in an unnamed port (stdin in this
case).

    scheme@(guile-user)> ,option interp #t
    scheme@(guile-user)> ,option
    trace	#f
    interp	#t

Here we tell the repl that, given the option, we prefer to interpret
rather than compile. Of course, if the current language doesn't support
compilation, we always interpret.

    scheme@(guile-user)> (lambda () (pk a #:bar))
    $4 = #<procedure #f ()>

An interpreted procedure, like in olden times.

Happy hacking!

Andy
-- 
http://wingolog.org/

Re: vm branch now uses vm repl by default

[Neil Jerram]

Hi Andy,

This is looking like fun!  But I'm not fully understanding...

2008/9/9 Andy Wingo <wingo@pobox.com>:

>    scheme@(guile-user)> (lambda () (pk a #:bar))
>    $2 = #<program b755ecf8>

[...]

>    scheme@(guile-user)> ,x $2
>
> There is a wealth of meta-commands at the repl, commands that start with
> `,'. This command, `,x', is an abbreviation for `,disassemble'. Its
> output is this:
>
>    Disassembly of #<program b755ecf8>:
>
>    nargs = 0  nrest = 0  nlocs = 0  nexts = 0
>
> The program has no arguments, no rest arguments, no local variables, and
> no external (lexically-bound) variables.
>
>    Bytecode:
>
>       0    (late-variable-ref 0)
>       2    (late-variable-ref 1)
>       4    (object-ref 2)                  ;; #:bar
>       6    (tail-call 2)
>
>    Objects:
>
>       0    #<variable b80057f0 value: #<program b8005858>>
>       1    #<variable b7569af0 value: (a . pair)>
>       2    #:bar

Where in the bytecode is `pk', or some kind of reference to it?

Regards,
          Neil

Re: vm branch now uses vm repl by default

[Andy Wingo]

Hi,

On Tue 09 Sep 2008 10:27, "Neil Jerram" <neiljerram@googlemail.com> writes:

> 2008/9/9 Andy Wingo <wingo@pobox.com>:
>
>>    scheme@(guile-user)> (lambda () (pk a #:bar))
>>    $2 = #<program b755ecf8>
>>    scheme@(guile-user)> ,x $2
>>    Disassembly of #<program b755ecf8>:
>>
>>    nargs = 0  nrest = 0  nlocs = 0  nexts = 0
>>
>>    Bytecode:
>>
>>       0    (late-variable-ref 0)
>>       2    (late-variable-ref 1)
>>       4    (object-ref 2)                  ;; #:bar
>>       6    (tail-call 2)
>>
>>    Objects:
>>
>>       0    #<variable b80057f0 value: #<program b8005858>>
>>       1    #<variable b7569af0 value: (a . pair)>
>>       2    #:bar
>
> Where in the bytecode is `pk', or some kind of reference to it?

It's object 0:

    scheme@(guile-user)> ,x (lambda () (pk a #:bar))
    Disassembly of #<program b7fedfc8>:

    nargs = 0  nrest = 0  nlocs = 0  nexts = 0

    Bytecode:

       0    (late-variable-ref 0)
       2    (late-variable-ref 1)
       4    (object-ref 2)                  ;; #:bar
       6    (tail-call 2)

    Objects:

       0    pk
       1    a
       2    #:bar

    Sources:

       8    #(0 14 #f)

The objects are stored in a vector. When late-variable-ref or -set is
called and the top of the stack is a symbol, it gets looked up and the
resulting variable replaced into where the symbol was. See:

   http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=libguile/vm-i-system.c;h=54689116942bc5d5f942ebcc6500e1e9b610e4c9;hb=refs/heads/vm#l262

Andy
-- 
http://wingolog.org/

Re: vm branch now uses vm repl by default

[Neil Jerram]

2008/9/9 Andy Wingo <wingo@pobox.com>:
> Hi,
>
> On Tue 09 Sep 2008 10:27, "Neil Jerram" <neiljerram@googlemail.com> writes:
>>
>> Where in the bytecode is `pk', or some kind of reference to it?
>
> It's object 0:

Thanks, I see now.  (I guess I misread the #<program> in object 0 as
being the same as the #<program> being disassembled.... or something.)

      Neil

Re: vm branch now uses vm repl by default

[Ludovic Courtès]

Hey!

Andy Wingo <wingo@pobox.com> writes:

> I've enabled the VM repl by default on the vm branch. Here's a brief
> annotated tour:
>
>     $ ./pre-inst-guile
>     Guile Scheme interpreter 0.5 on Guile 1.9.0
>     Copyright (C) 2001-2008 Free Software Foundation, Inc.
>
>     Enter `,help' for help.

Cool!

> Late-variable-ref looks at a cell in the object vector. If it is a
> symbol, it is resolved relative to the module that was current when the
> program was made. The object cell is then replaced with the resulting
> resolved variable. Here we see that objects 0 and 1 were already
> resolved. Object 2 is just the constant, #:bar.

Aaah, nice!  So, previously, there was `variable-ref':

 -- Instruction: variable-ref
     Dereference the variable object which is on top of the stack and
     replace it by the value of the variable it represents.

Now, there's also a vector associated with each closure to store
references to global variables, right?  Looks better!

(Hint: the doc is outdated.  :-))

>     scheme@(guile-user)> ,option interp #t
>     scheme@(guile-user)> ,option
>     trace	#f
>     interp	#t
>
> Here we tell the repl that, given the option, we prefer to interpret
> rather than compile. Of course, if the current language doesn't support
> compilation, we always interpret.

That means good old `CEVAL ()' is used, right?

When that is the case, one can still use `{eval,debug}-options', right?

It'd be nice if we could find a way to "do something" with the
`current-reader' fluid at compilation time, like detecting top-level
`(fluid-set! current-reader ...)' statements and use that to switch the
compiler's reader (hacky...).

Thanks for the good news!

Ludo'.

Re: vm branch now uses vm repl by default

[Andy Wingo]

On Tue 09 Sep 2008 10:41, ludo@gnu.org (Ludovic Courtès) writes:

> So, previously, there was `variable-ref':

There still is. It is used when a variable is bound immediately, when it
is pushed on the stack by a link-now instruction.

    scheme@(guile-user)> ,c (define x a)
    Disassembly of #<objcode b7feb6d0>:

    nlocs = 0  nexts = 0

       0    (load-symbol "a")               ;; a
       3    (link-now)
       4    (variable-ref)
       5    (define "x")
       8    (variable-set)
       9    (void)
      10    (return)

> Now, there's also a vector associated with each closure to store
> references to global variables, right?  Looks better!

That's always been the case IIRC, only before it used to push and pop a
bit more -- instead of

    (late-variable-ref 0) 

it would be

    (object-ref 0)
    (variable-ref)

where the object was instantiated by a load-symbol / link-now pair. But
that's not how Guile's toplevel lookups occur, so I added the new
behavior, which has the added benefit of not pushing and popping so
much.

> (Hint: the doc is outdated.  :-))

It's still correct, just not complete :-) I'll get on it at some point,
probably folding into Guile's docs somehow.

>>     scheme@(guile-user)> ,option interp #t
>
> That means good old `CEVAL ()' is used, right?

Yep, whatever (language-evaluator (repl-language repl)) does -- which
for `scheme' is `eval'.

    http://git.savannah.gnu.org/gitweb/?p=guile.git;a=commitdiff;h=02ed0d3df2607c5d78fbc38cbb82a65df1bc7080

> When that is the case, one can still use `{eval,debug}-options', right?

Yep

> It'd be nice if we could find a way to "do something" with the
> `current-reader' fluid at compilation time, like detecting top-level
> `(fluid-set! current-reader ...)' statements and use that to switch the
> compiler's reader (hacky...).

Perhaps, there is already a repl-reader fluid for readline's benefit.
Note also that languages have readers as well, so that e.g. elisp can
read differently from scheme.

Andy
-- 
http://wingolog.org/

Re: vm branch now uses vm repl by default

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> On Tue 09 Sep 2008 10:41, ludo@gnu.org (Ludovic Courtès) writes:
>
>> So, previously, there was `variable-ref':
>
> There still is. It is used when a variable is bound immediately, when it
> is pushed on the stack by a link-now instruction.

OK, I see.

>> Now, there's also a vector associated with each closure to store
>> references to global variables, right?  Looks better!
>
> That's always been the case IIRC, only before it used to push and pop a
> bit more -- instead of
>
>     (late-variable-ref 0) 
>
> it would be
>
>     (object-ref 0)
>     (variable-ref)

Oh, right.

>> It'd be nice if we could find a way to "do something" with the
>> `current-reader' fluid at compilation time, like detecting top-level
>> `(fluid-set! current-reader ...)' statements and use that to switch the
>> compiler's reader (hacky...).
>
> Perhaps, there is already a repl-reader fluid for readline's benefit.
> Note also that languages have readers as well, so that e.g. elisp can
> read differently from scheme.

Right, but `current-reader' is a dynamic thing, which complicates the
situation.

A use case is the following:

  (define-module (foo))

  (fluid-set! current-reader %my-favorite-reader)

  ;; use non-standard syntax extensions from now on

I use it this way in Skribilo, but I may well be the only user, who
knows.  ;-)

Anyway, if we are to handle this at all, we're probably going to have to
pattern-match this in `translate.scm' and switch readers when we
encounter it.

Thanks,
Ludo'.

Re: vm branch now uses vm repl by default

[Andy Wingo]

Howdy,

On Tue 09 Sep 2008 23:01, ludo@gnu.org (Ludovic Courtès) writes:

>> On Tue 09 Sep 2008 10:41, ludo@gnu.org (Ludovic Courtès) writes:
>
>>> It'd be nice if we could find a way to "do something" with the
>>> `current-reader' fluid at compilation time, like detecting top-level
>>> `(fluid-set! current-reader ...)' statements and use that to switch the
>>> compiler's reader (hacky...).
>
> A use case is the following:
>
>   (define-module (foo))
>
>   (fluid-set! current-reader %my-favorite-reader)
>
>   ;; use non-standard syntax extensions from now on

This doesn't map very well to the current semantics of "guile-vm" -- I
was going to say "the compiler", but the compiler's fine with whatever,
it's more "the expected model of compilation" or something.

Currently when compiling an entire file, the entire file is read in and
compiled to bytecode as if it were all wrapped in a `begin'. The whole
file is compiled at once, because it was all read in at once. But that
means we have no semantic meaning to extract from the code at read-time.
What to do?

I would suggest going down a common-lispy route, and introducing read
macros. This is pretty firmly extension land anyway. Or, you could write
a custom compiler for skribilo -- probably four hours' work or so.

> Anyway, if we are to handle this at all, we're probably going to have to
> pattern-match this in `translate.scm' and switch readers when we
> encounter it.

Ew. You are a bad person ;)

Cheers,

Andy
-- 
http://wingolog.org/

Re: vm branch now uses vm repl by default

[Neil Jerram]

2008/9/9 Andy Wingo <wingo@pobox.com>:
>
>    Sources:
>
>       8    #(1 11 #f)
>
> Some instructions are annotated with source information.

I guess source information is of interest for debugging, and I think
you've observed previously elsewhere that the VM doesn't yet have much
debugging support - by which I presume you mean something like the
traps that the evaluator has.

So I was just wondering if we actually _need_ any debugging support in
the VM.  If we can assume that the VM will always behave equivalently
to the evaluator (except faster), then whenever a piece of code needs
step-by-step debugging, the developer can drop back to using the
evaluator in order to do that.

Does that make sense?

(Then another question is whether we can assume that the VM behaves
equivalently to the evaluator.  I wonder if there is some test
methodology for partly proving this, by automatically running the
evaluator in parallel with the VM, at least for code that doesn't have
side effects?

I'm sorry, this email has ended up a bit vague....)

      Neil

Re: vm branch now uses vm repl by default

[Andy Wingo]

Hi,

On Wed 10 Sep 2008 00:43, "Neil Jerram" <neiljerram@googlemail.com> writes:

> I guess source information is of interest for debugging

Of course :)

> you've observed previously elsewhere that the VM doesn't yet have much
> debugging support - by which I presume you mean something like the
> traps that the evaluator has.

Oh it has some. Here are the hooks:

    vm-next-hook vm-apply-hook vm-boot-hook vm-return-hook
    vm-break-hook vm-exit-hook vm-halt-hook vm-enter-hook

They sound sufficient, but perhaps more could be added. Unless I
misunderstand what the traps do (which is likely).

> So I was just wondering if we actually _need_ any debugging support in
> the VM.

Certainly we do for debugging the VM itself. But besides that, if the VM
will be the way that people will want to run their programs, they need
good backtraces at the very least. Which implies at least source
information.

I think the only thing that is missing at this point is procedure names
(which I haven't spent enough time figuring out how to push them down
the compilation pipeline, but the support is there), and spaghetti
stacks, which are also needed to support call/cc.

> Then another question is whether we can assume that the VM behaves
> equivalently to the evaluator.

I think that this is a precondition for merging vm to master.

> code that doesn't have side effects?

If you believe that, I've got a type system in Scotland I'd like to sell
you ;-)

Peace,

Andy
-- 
http://wingolog.org/

Re: vm branch now uses vm repl by default

[Neil Jerram]

2008/9/10 Andy Wingo <wingo@pobox.com>:
>
> Oh it has some. Here are the hooks:
>
>    vm-next-hook vm-apply-hook vm-boot-hook vm-return-hook
>    vm-break-hook vm-exit-hook vm-halt-hook vm-enter-hook
>
> They sound sufficient, but perhaps more could be added. Unless I
> misunderstand what the traps do (which is likely).

No, I agree that those sound sufficient.

> Certainly we do for debugging the VM itself. But besides that, if the VM
> will be the way that people will want to run their programs, they need
> good backtraces at the very least.

Yes, of course, silly me.  (I was thinking too much about breakpoints
and single-stepping, forgot about the basic error case.)

Regards,
        Neil