case-lambda integration

case-lambda integration

[Andy Wingo]

Hello,

I am climbing out of a case-lambda hole, and am wondering about
integration. If you missed the first part of this, check:

  http://article.gmane.org/gmane.lisp.guile.devel/9436

The deal is that we need case-lambda, and then some kind of
typecase-lambda to get efficient dispatch of the default generic
function protocol into the VM.

OK then, my wip-case-lambda branch is shaping up, with support for
case-lambda at the VM, debugging, assembly, GLIL, and tree-il levels. I
had to adapt psyntax too to make the new tree-il structures.

In order to expose the case-lambda functionality to Scheme though, we
need a way to make lambdas with more than one case; so I think I will
export case-lambda as an identifier by default.

That is probably uncontroversial.

But what might be is that while I was at it, I made allowances for
efficient optional and keyword argument dispatch -- so we can have
keyword arguments without consing, and arguments which are positionally
optional yet can have keywords. This is so lambda* and define* can be
implemented nicely.

There are hooks in <lambda-case> for these, but again the question is
how to expose to Scheme. As I see it there are two options:

   1. Implement lambda* (and define*) in psyntax

   -or-

   2. Make the stock lambda accept #:optional, #:keyword, et al
      arguments (also a hack to psyntax)

I'm kindof leaning towards (2) actually. But I could go either way. Both
ways expose lambda* capability in the default environment -- because
psyntax boots before modules have booted, and these things need access
to psyntax internals.

Perhaps we should do (1) in the interest of interoperability; but you
still have to know you want to code portable Scheme in order to code
portable Scheme. Dunno.

Anyways, input appreciated.

Ciao,

Andy
-- 
http://wingolog.org/

Re: case-lambda integration

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> In order to expose the case-lambda functionality to Scheme though, we
> need a way to make lambdas with more than one case; so I think I will
> export case-lambda as an identifier by default.
>
> That is probably uncontroversial.

It’s fine with me, so certainly uncontroversial.  ;-)

> But what might be is that while I was at it, I made allowances for
> efficient optional and keyword argument dispatch -- so we can have
> keyword arguments without consing, and arguments which are positionally
> optional yet can have keywords. This is so lambda* and define* can be
> implemented nicely.

I threw a glance at the branch (in particular “flesh out glil support
for optional and keyword arguments”), but it’s unclear to me how it
works.  The callee reserves space on the stack for its parameters, but
how are keyword parameters associated with a reserved slot on the stack?

> There are hooks in <lambda-case> for these, but again the question is
> how to expose to Scheme. As I see it there are two options:
>
>    1. Implement lambda* (and define*) in psyntax
>
>    -or-
>
>    2. Make the stock lambda accept #:optional, #:keyword, et al
>       arguments (also a hack to psyntax)
>
> I'm kindof leaning towards (2) actually. But I could go either way. Both
> ways expose lambda* capability in the default environment -- because
> psyntax boots before modules have booted, and these things need access
> to psyntax internals.

As you rightfully guessed I lean towards (1), but then again I don’t
understand how this affects the implementation.  Would option (1) be
somehow harder to implement, or would it tend to duplicate code or
something like that?

> Perhaps we should do (1) in the interest of interoperability; but you
> still have to know you want to code portable Scheme in order to code
> portable Scheme. Dunno.

Yes, right.

OTOH, how difficult would it be to accommodate different keyword
argument APIs, such as ‘(ice-9 optargs)’, SRFI-89, and DSSSL, which have
subtle differences, once we have built-in support for one flavor?

Thanks,
Ludo’.

Re: case-lambda integration

[Andy Wingo]

Good evening Ludovic :)

On Tue 13 Oct 2009 22:35, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> I made allowances for efficient optional and keyword argument
>> dispatch -- so we can have keyword arguments without consing, and
>> arguments which are positionally optional yet can have keywords. This
>> is so lambda* and define* can be implemented nicely.
>
> I threw a glance at the branch (in particular “flesh out glil support
> for optional and keyword arguments”), but it’s unclear to me how it
> works.  The callee reserves space on the stack for its parameters, but
> how are keyword parameters associated with a reserved slot on the
> stack?

All args are passed on the stack. A procedure may have required,
optional, keyword, and rest arguments. The N required arguments are
bound to the first N local variable slots. The M optional arguments are
bound to the next M local variable slots. If an optional variable is not
positionally present, SCM_UNDEFINED is placed in the slot. If there are
keywords as well, then things get complicated.

I wanted to allow positional arguments to also have keywords. I had to
make a decision what to do if you have 2 required args, 1 optional, and
1 keyword -- if the 3rd argument is a keyword, does it go in the
optional slot or does it start the keyword processing? I chose the
latter. So once you run out of optionals or see a keyword, the remaining
arguments are shuffled up on the stack, to be placed above the empty
slots for keyword arguments -- /if any/. Then the keyword arguments are
traversed, looking at the keyword alist ((#:KW . N) ...), to determinine
the local slot that a given keyword corresponds to. If no slot is found,
that may be an error or it may be ignored -- depending on the
allow-other-keywords? option.

If a rest arg is also given, it will hold all keyword args as well, and
the rest must be a valid keyword arg list.

It's quite complicated, and very much "feature on top of feature". You
could emulate this with rest arguments, as (ice-9 optargs) does, but
with callee-parsed arguments we can make keyword arg procedures pay for
it, but without the cost of allocation on each procedure call.

So that's the deal. The tree-il interface will require initializers for
all args, so after all the args are bound, if any given arg is
uninitialized, it will be set to the value of some user-given
expression. This is useful for Scheme, where you never have
uninitialized lexicals except in erroroneous use of letrec, and for e.g.
javascript, where you want optionals to have certain values.

The system is very flexible, but should you need more flexibility, you
can have that too -- by using the rest arg interface. It costs, in terms
of allocation, but there's only so much we can do.

>> There are hooks in <lambda-case> for these, but again the question is
>> how to expose to Scheme. As I see it there are two options:
>>
>>    1. Implement lambda* (and define*) in psyntax
>>
>>    -or-
>>
>>    2. Make the stock lambda accept #:optional, #:keyword, et al
>>       arguments (also a hack to psyntax)
>>
>> I'm kindof leaning towards (2) actually. But I could go either way. Both
>> ways expose lambda* capability in the default environment -- because
>> psyntax boots before modules have booted, and these things need access
>> to psyntax internals.
>
> As you rightfully guessed I lean towards (1), but then again I don’t
> understand how this affects the implementation.  Would option (1) be
> somehow harder to implement, or would it tend to duplicate code or
> something like that?

So it's either introduce new identifiers (lambda* and define*) in the
default environment, or add functionality to the existing ones. I don't
care very much either way, TBH, though I lean to enhancing lambda.

> OTOH, how difficult would it be to accommodate different keyword
> argument APIs, such as ‘(ice-9 optargs)’, SRFI-89, and DSSSL, which have
> subtle differences, once we have built-in support for one flavor?

The Scheme interface will be the same as optargs, I think -- possibly
with some extensions. I haven't looked at srfi-89. I think we can do
DSSSL as it's a subset of optargs afaik.

What do you think?

Andy
-- 
http://wingolog.org/

Re: case-lambda integration

[Ludovic Courtès]

Hey,

Andy Wingo <wingo@pobox.com> writes:

> All args are passed on the stack. A procedure may have required,
> optional, keyword, and rest arguments. The N required arguments are
> bound to the first N local variable slots. The M optional arguments are
> bound to the next M local variable slots. If an optional variable is not
> positionally present, SCM_UNDEFINED is placed in the slot. If there are
> keywords as well, then things get complicated.

Understood.

> I wanted to allow positional arguments to also have keywords. I had to
> make a decision what to do if you have 2 required args, 1 optional, and
> 1 keyword -- if the 3rd argument is a keyword, does it go in the
> optional slot or does it start the keyword processing? I chose the
> latter. So once you run out of optionals or see a keyword, the remaining
> arguments are shuffled up on the stack, to be placed above the empty
> slots for keyword arguments -- /if any/. Then the keyword arguments are
> traversed, looking at the keyword alist ((#:KW . N) ...), to determinine
> the local slot that a given keyword corresponds to. If no slot is found,
> that may be an error or it may be ignored -- depending on the
> allow-other-keywords? option.

So the keyword alist is new meta-data stored alongside the procedure,
right?

> If a rest arg is also given, it will hold all keyword args as well, and
> the rest must be a valid keyword arg list.
>
> It's quite complicated, and very much "feature on top of feature". You
> could emulate this with rest arguments, as (ice-9 optargs) does, but
> with callee-parsed arguments we can make keyword arg procedures pay for
> it, but without the cost of allocation on each procedure call.

... which sounds like a big win to me.  That means one would be less
hesitant in using keyword arguments, which is good IMO.

>> As you rightfully guessed I lean towards (1), but then again I don’t
>> understand how this affects the implementation.  Would option (1) be
>> somehow harder to implement, or would it tend to duplicate code or
>> something like that?
>
> So it's either introduce new identifiers (lambda* and define*) in the
> default environment, or add functionality to the existing ones. I don't
> care very much either way, TBH, though I lean to enhancing lambda.

I’m OK with the introduction of ‘lambda*’ and ‘define*’ in the name space.

>> OTOH, how difficult would it be to accommodate different keyword
>> argument APIs, such as ‘(ice-9 optargs)’, SRFI-89, and DSSSL, which have
>> subtle differences, once we have built-in support for one flavor?
>
> The Scheme interface will be the same as optargs, I think -- possibly
> with some extensions.

So (ice-9 optargs) could very much be deprecated?  There may be subtle
corner cases needing attention, though.

Thanks,
Ludo’.

Re: case-lambda integration

[Andy Wingo]

Hi Ludovic,

On Thu 22 Oct 2009 00:00, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> So once you run out of optionals or see a keyword, the remaining
>> arguments are shuffled up on the stack, to be placed above the empty
>> slots for keyword arguments -- /if any/. Then the keyword arguments are
>> traversed, looking at the keyword alist ((#:KW . N) ...), to determinine
>> the local slot that a given keyword corresponds to. If no slot is found,
>> that may be an error or it may be ignored -- depending on the
>> allow-other-keywords? option.
>
> So the keyword alist is new meta-data stored alongside the procedure,
> right?

Yes. It is stored in the procedure's object table.

The metadata itself does change on this branch. Before the metadata was
a thunk returning (BINDINGS SOURCES . PROPERTIES); now the thunk returns
(BINDINGS SOURCES ARITIES . PROPERTIES). Arities is an
extents-delimited description of the procedure's arities, for debugging
and printing purposes.

>> with callee-parsed arguments we can make keyword arg procedures pay for
>> it, but without the cost of allocation on each procedure call.
>
> ... which sounds like a big win to me.  That means one would be less
> hesitant in using keyword arguments, which is good IMO.

Yes, I think so too! Keyword args are a big win for maintainability of a
library, IMO -- and this makes kwargs first-class citizens of Guile at
the same time as making them faster.

>> So it's either introduce new identifiers (lambda* and define*) in the
>> default environment, or add functionality to the existing ones.
>
> I’m OK with the introduction of ‘lambda*’ and ‘define*’ in the name
> space.

OK. There will also be case-lambda and case-lambda*.

One more thing in lambda* -- I have added a #:predicate option, so that
this particular lambda case only matches if evaluating the predicate in
the lexical context of the arguments returns a true value.

This should allow:

  (typecase-lambda
    (((a <foo>) (b <bar>))
     (specific-frob a b))
    ((a b)
     (general-frob a b)))

  => (case-lambda*
       ((a b #:predicate (and (eq? (class-of a) <foo>)
                              (eq? (class-of b) <bar>)))
        (specific-frob a b))
       ((a b)
        (general-frob a b)))

Thus it allows effective-method implementation in Scheme and not
using evaluator #@dispatch hacks :-)))

>>> OTOH, how difficult would it be to accommodate different keyword
>>> argument APIs, such as ‘(ice-9 optargs)’, SRFI-89, and DSSSL, which have
>>> subtle differences, once we have built-in support for one flavor?
>>
>> The Scheme interface will be the same as optargs, I think -- possibly
>> with some extensions.
>
> So (ice-9 optargs) could very much be deprecated?  There may be subtle
> corner cases needing attention, though.

Sure that would be possible. For 1.8 -> 2.0 reasons we can just make
optargs re-export lambda* from the base environment, and deprecate
optargs in 2.2.

Andy
-- 
http://wingolog.org/

Re: case-lambda integration

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@pobox.com> writes:

> On Thu 22 Oct 2009 00:00, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> So the keyword alist is new meta-data stored alongside the procedure,
>> right?
>
> Yes. It is stored in the procedure's object table.
>
> The metadata itself does change on this branch. Before the metadata was
> a thunk returning (BINDINGS SOURCES . PROPERTIES); now the thunk returns
> (BINDINGS SOURCES ARITIES . PROPERTIES). Arities is an
> extents-delimited description of the procedure's arities, for debugging
> and printing purposes.

OK, thanks for explaining.

> One more thing in lambda* -- I have added a #:predicate option, so that
> this particular lambda case only matches if evaluating the predicate in
> the lexical context of the arguments returns a true value.
>
> This should allow:
>
>   (typecase-lambda
>     (((a <foo>) (b <bar>))
>      (specific-frob a b))
>     ((a b)
>      (general-frob a b)))
>
>   => (case-lambda*
>        ((a b #:predicate (and (eq? (class-of a) <foo>)
>                               (eq? (class-of b) <bar>)))
>         (specific-frob a b))
>        ((a b)
>         (general-frob a b)))
>
> Thus it allows effective-method implementation in Scheme and not
> using evaluator #@dispatch hacks :-)))

Sounds cool, and indeed refined compared to the evaluator hacks.

I’m looking forward to seeing all this land in ‘master’!

>> So (ice-9 optargs) could very much be deprecated?  There may be subtle
>> corner cases needing attention, though.
>
> Sure that would be possible. For 1.8 -> 2.0 reasons we can just make
> optargs re-export lambda* from the base environment, and deprecate
> optargs in 2.2.

Yeah, excellent.

Ludo’.