Elisp flet construct

Elisp flet construct

[Daniel Kraft]

Hi all,

in the %nil thread a suggestion was brought up to support the `flet' 
construct (and `flet*' as well, if we choose to do so at all, I favour) 
in Guile's upcoming elisp implementation that behaves just like a let 
for function-slot bindings, enabling dynamic scoping for them.

It is no "official" elisp construct, but according to what I heard 
there, can be useful at some times (I guess the use is mainly to locally 
alter bindings of standard functions for some code executed without a 
risk of permanently messing things up).  So I don't know how you regard 
addition of "extensions"...?

 From an implementation point of view, I see both an argument against 
and in favour of this extension:

* As implemented at the moment, it would be fairly easy to add 
flet/flet* to the compiler.

* If we do not implement flet, we can implement the function-slots 
without indirection via fluids but rather use the Guile symbol bindings 
directly.  This is for sure a simplification especially performance 
wise, but I can't say how much it really affects.  Most bindings in use 
are, I guess, variables, so we save the fluid-references only in a 
fraction of cases.

Any comments welcome, so we can decide upon this.

Thanks,
Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri

Re: Elisp flet construct

[Daniel Kraft]

Daniel Kraft wrote:
> in the %nil thread a suggestion was brought up to support the `flet' 
> construct (and `flet*' as well, if we choose to do so at all, I favour) 
> in Guile's upcoming elisp implementation that behaves just like a let 
> for function-slot bindings, enabling dynamic scoping for them.
> 
> It is no "official" elisp construct, but according to what I heard 
> there, can be useful at some times (I guess the use is mainly to locally 
> alter bindings of standard functions for some code executed without a 
> risk of permanently messing things up).  So I don't know how you regard 
> addition of "extensions"...?

Just a little addition to the subject of extensions:  I'd very much like 
to add lexical-let and lexical-let* as another set of extensions, 
because this gives the possibility to use "fast" lexical variables 
without the dynamic-scoping-fluid-pain.

Currently, I did implement some control constructs that could be done as 
macros still in the compiler directly (like prog1 or dolist) because 
there I can make use of lexical helper variables; lexical-let would 
allow using this feature directly from elisp (and implementing these 
constructs equivalently as macros).

So, what do you think about this extension?

Yours,
Daniel

Re: Elisp flet construct

[Ken Raeburn]

On Jul 21, 2009, at 09:10, Daniel Kraft wrote:
> Just a little addition to the subject of extensions:  I'd very much  
> like to add lexical-let and lexical-let* as another set of  
> extensions, because this gives the possibility to use "fast" lexical  
> variables without the dynamic-scoping-fluid-pain.
>
> Currently, I did implement some control constructs that could be  
> done as macros still in the compiler directly (like prog1 or dolist)  
> because there I can make use of lexical helper variables; lexical- 
> let would allow using this feature directly from elisp (and  
> implementing these constructs equivalently as macros).
>
> So, what do you think about this extension?

Is this modeled on the lexical-let work being done for Emacs (probably  
to be merged for Emacs 24)?  If so, I wouldn't call it an extension so  
much as being compatible in advance. :-)  And if not, you should  
probably take a look at that work and make sure yours is compatible.

It definitely sounds like a good idea to me.

Ken

Re: Elisp flet construct

[Daniel Kraft]

Hi Ken,

Ken Raeburn wrote:
> On Jul 21, 2009, at 09:10, Daniel Kraft wrote:
>> Just a little addition to the subject of extensions:  I'd very much 
>> like to add lexical-let and lexical-let* as another set of extensions, 
>> because this gives the possibility to use "fast" lexical variables 
>> without the dynamic-scoping-fluid-pain.
>>
>> Currently, I did implement some control constructs that could be done 
>> as macros still in the compiler directly (like prog1 or dolist) 
>> because there I can make use of lexical helper variables; lexical-let 
>> would allow using this feature directly from elisp (and implementing 
>> these constructs equivalently as macros).
>>
>> So, what do you think about this extension?
> 
> Is this modeled on the lexical-let work being done for Emacs (probably 
> to be merged for Emacs 24)?  If so, I wouldn't call it an extension so 
> much as being compatible in advance. :-)  And if not, you should 
> probably take a look at that work and make sure yours is compatible.

hm...  I just found this page which is not decidedly about a planned 
addition to emacs 24, but I think it is quite reasonable that this 
addition will be the extension/feature described here:

http://www.delorie.com/gnu/docs/emacs/cl_21.html

This seems also (as far as I'm no expert on the details) to be just like 
let behaves in Scheme and I plan to implement lexical-let just using 
Guile's real let handling (not the dynamic one based on fluids crafted 
for elisp) -- I hope this is "compatible".

If this is not quite correct, please point me to another source and/or 
let me know the differences between Scheme's let to keep in mind!

Thanks, Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri

Re: Elisp flet construct

[Ken Raeburn]

On Jul 21, 2009, at 15:32, Daniel Kraft wrote:
> hm...  I just found this page which is not decidedly about a planned  
> addition to emacs 24, but I think it is quite reasonable that this  
> addition will be the extension/feature described here:

I was thinking about this:

http://lists.gnu.org/archive/html/emacs-devel/2009-06/msg00260.html

It's not set in stone, but since the author is a current Emacs  
maintainer, I think it's likely.
The "lexbind" branch has been in the Emacs CVS code for some time now,  
but Emacs 23 has been in feature freeze for a while too.

> http://www.delorie.com/gnu/docs/emacs/cl_21.html

It looks like that's about the common-lisp compatibility package Emacs  
includes.  I don't know how similar that lexical-let is to what Miles  
Bader has been doing on the lexbind branch.  Nor can I tell you how  
they'll compare to Scheme; I've never used either of the Emacs versions.

Ken

Re: Elisp flet construct

[Andy Wingo]

Hi Daniel,

On Mon 20 Jul 2009 19:53, Daniel Kraft <d@domob.eu> writes:

> in the %nil thread a suggestion was brought up to support the `flet'
> construct (and `flet*' as well, if we choose to do so at all, I favour)
> in Guile's upcoming elisp implementation that behaves just like a let
> for function-slot bindings, enabling dynamic scoping for them.
>
> It is no "official" elisp construct, but according to what I heard
> there, can be useful at some times (I guess the use is mainly to locally
> alter bindings of standard functions for some code executed without a
> risk of permanently messing things up).  So I don't know how you regard
> addition of "extensions"...?

Personally? I think what's out there would be the priority, but
extensions are fun too :)

> * If we do not implement flet, we can implement the function-slots
> without indirection via fluids but rather use the Guile symbol bindings
> directly.  This is for sure a simplification especially performance
> wise, but I can't say how much it really affects.  Most bindings in use
> are, I guess, variables, so we save the fluid-references only in a
> fraction of cases.

From a Guile perspective, I would like to avoid the use of
symbol-function. It seems that the module and variable hacks we
discussed before would be sufficiently fast. I could be wrong of course.

Andy
-- 
http://wingolog.org/

Re: Elisp flet construct

[Andy Wingo]

On Tue 21 Jul 2009 15:10, Daniel Kraft <d@domob.eu> writes:

> Just a little addition to the subject of extensions:  I'd very much like
> to add lexical-let and lexical-let* as another set of extensions,
> because this gives the possibility to use "fast" lexical variables
> without the dynamic-scoping-fluid-pain.

Yes, yes. I totally agree. This allows stack allocation of the variables
as well, which can be a significant speed win (because of not making so
much garbage).

> Currently, I did implement some control constructs that could be done as
> macros still in the compiler directly (like prog1 or dolist) because
> there I can make use of lexical helper variables; lexical-let would
> allow using this feature directly from elisp (and implementing these
> constructs equivalently as macros).
>
> So, what do you think about this extension?

I think it sounds great!

Andy
-- 
http://wingolog.org/

Re: Elisp flet construct

[Ken Raeburn]

On Jul 23, 2009, at 16:46, Andy Wingo wrote:
> On Tue 21 Jul 2009 15:10, Daniel Kraft <d@domob.eu> writes:
> Just a little addition to the subject of extensions:  I'd very much  
> like
>> to add lexical-let and lexical-let* as another set of extensions,
>> because this gives the possibility to use "fast" lexical variables
>> without the dynamic-scoping-fluid-pain.
>
> Yes, yes. I totally agree. This allows stack allocation of the  
> variables
> as well, which can be a significant speed win (because of not making  
> so
> much garbage).

In some cases, but not all.  Consider the Lisp version of an example I  
remember from my Scheme intro class:

   (defun make-counter ()
     (lexical-let ((count 0))
       (lambda ()
          (setq count (+ 1 count))
          count)))

In the Scheme version, and in Emacs with the cl package, a separate  
binding of "count" exists for each generated counter "object", so you  
can create and use several counters independently.  If we want this to  
work in Guile-Elisp too, I don't think "count" can live on the stack  
in this case, since it has to survive past the return from make- 
counter.  The dynamic-scoping-fluid-pain is replaced by lambda- 
environment-tracking-pain. :)

Without lambda or defun forms inside the lexical-let, yes, I think  
stack slots can be used.  (Well, I'm kind of fuzzy on the whole call/ 
cc thing, but I assume that's already addressed elsewhere....)

Ken

Re: Elisp flet construct

[Andy Wingo]

On Thu 23 Jul 2009 23:53, Ken Raeburn <raeburn@raeburn.org> writes:

> On Jul 23, 2009, at 16:46, Andy Wingo wrote:
>> On Tue 21 Jul 2009 15:10, Daniel Kraft <d@domob.eu> writes:
>> Just a little addition to the subject of extensions:  I'd very much
>> like
>>> to add lexical-let and lexical-let* as another set of extensions,
>>> because this gives the possibility to use "fast" lexical variables
>>> without the dynamic-scoping-fluid-pain.
>>
>> Yes, yes. I totally agree. This allows stack allocation of the
>> variables
>> as well, which can be a significant speed win (because of not making
>> so
>> much garbage).
>
> In some cases, but not all.  Consider the Lisp version of an example I
> remember from my Scheme intro class:
>
>   (defun make-counter ()
>     (lexical-let ((count 0))
>       (lambda ()
>          (setq count (+ 1 count))
>          count)))

Of course :) But with lexical scoping you can statically determine when
variables are assigned. The tree-il->glil compiler already does this.

> Without lambda or defun forms inside the lexical-let, yes, I think stack
> slots can be used.  (Well, I'm kind of fuzzy on the whole call/ cc
> thing, but I assume that's already addressed elsewhere....)

If the continuation may be captured, you have to heap-allocate mutated
vars. So we just heap-allocate all mutated vars.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Elisp flet construct

[Daniel Kraft]

Andy Wingo wrote:
>> in the %nil thread a suggestion was brought up to support the `flet'
>> construct (and `flet*' as well, if we choose to do so at all, I favour)
>> in Guile's upcoming elisp implementation that behaves just like a let
>> for function-slot bindings, enabling dynamic scoping for them.
>>
>> It is no "official" elisp construct, but according to what I heard
>> there, can be useful at some times (I guess the use is mainly to locally
>> alter bindings of standard functions for some code executed without a
>> risk of permanently messing things up).  So I don't know how you regard
>> addition of "extensions"...?
> 
> Personally? I think what's out there would be the priority, but
> extensions are fun too :)

Ok, so then I'll implement flet :)  It should be nearly trivial, because 
I can just seperate the existing let/let* code out and use it for 
functions, also.

>> * If we do not implement flet, we can implement the function-slots
>> without indirection via fluids but rather use the Guile symbol bindings
>> directly.  This is for sure a simplification especially performance
>> wise, but I can't say how much it really affects.  Most bindings in use
>> are, I guess, variables, so we save the fluid-references only in a
>> fraction of cases.
> 
>>From a Guile perspective, I would like to avoid the use of
> symbol-function. It seems that the module and variable hacks we
> discussed before would be sufficiently fast. I could be wrong of course.

Hm... I'm not sure if we understand each other correctly; what do you 
mean with symbol-function here?  My point was that at the moment, we 
store in the respective module under the variable's name a fluid, and 
then use this fluid's value when calling the function:

(define foobar (make-fluid))
(fluid-set! foobar (lambda (a b) (+ a b)))
...
((fluid-ref foobar) 1 2)

If we don't want to have flet and dynamic bindings for functions as a 
consequence, we can get rid of the fluids and do just (as we would in 
Scheme):

(define foobar (lambda (a b) (+ a b)))
...
(foobar 1 2)

Which would be simpler (at runtime at least) and maybe faster, although 
I don't think the difference is significant.  But let's just add flet 
and keep using fluids ;)

Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri

Re: Elisp flet construct

[Daniel Kraft]

Andy Wingo wrote:
> On Tue 21 Jul 2009 15:10, Daniel Kraft <d@domob.eu> writes:
> 
>> Just a little addition to the subject of extensions:  I'd very much like
>> to add lexical-let and lexical-let* as another set of extensions,
>> because this gives the possibility to use "fast" lexical variables
>> without the dynamic-scoping-fluid-pain.
> 
> Yes, yes. I totally agree. This allows stack allocation of the variables
> as well, which can be a significant speed win (because of not making so
> much garbage).

Ok, cool!  And yes, if you believe there might be some code depending on 
the exact semantics of the cl implementation, I will implement our 
lexical-let with the same semantics.

I think we can't just treat inner let's like lexical-let's, because they 
might also bind variables for which no lexical binding is established. 
And those need still get their dynamic binding (tests confirmed); but 
that is of course not much of a problem, either.

And BTW, inner lambda expressions still bind their arguments 
dynamically; this special semantics is just with let, and might be some 
problem with its implementation and not really designed as such?  Well, 
anyways, just do it :)

>> Currently, I did implement some control constructs that could be done as
>> macros still in the compiler directly (like prog1 or dolist) because
>> there I can make use of lexical helper variables; lexical-let would
>> allow using this feature directly from elisp (and implementing these
>> constructs equivalently as macros).
>>
>> So, what do you think about this extension?
> 
> I think it sounds great!

Actually, there's one more advantage of implementing those control 
structures directly from the compiler over macros I did not think about, 
namely not just the dynamic binding but also that I don't do the check 
for void value when I know it can't be void.  I'm getting suspicious 
that the main performance hit with elisp variables is not the fluids but 
rather this check, and here lexical-let does not help, unfortunatly.  We 
would need another extension that disables this check for some 
variables, but I'm not sure if we should keep on adding just extensions 
like crazy.

But more on this later, and maybe I can supply real timings soon, too, 
so we really know where the most time is spent (fluids vs. void).

Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri