Closures - do you understand them well?

Closures - do you understand them well?

[Michael Heerdegen]

Hi,

Maybe you will have fun with this new exercise:

What is the return value of the following expression, and what's your
reasoning?  Answer without asking Emacs:

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(let ((i 0)
      (funs '()))
  (while (< (setq i (1+ i))
            4)
    (push (lambda () i)
          funs))
  (apply #'+ (mapcar #'funcall funs)))
#+end_src

Please no "you better write this as ... and then it returns ..."
answers: I mean this code and what it returns.


Have fun!

Michael.

RE: [External] : Closures - do you understand them well?

[Drew Adams]

Just a mention for others, that '() == ().
No need to quote t or nil (including its
alter ego ()).

Re: [External] : Closures - do you understand them well?

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Just a mention for others, that '() == ().
> No need to quote t or nil (including its
> alter ego ()).

You are lucky that your remark doesn't fulfill the "you better write
this as ... and then it returns ..." answer property because it doesn't
change the return value ;-)

But yes, like explicitly specifying a nil binding (X nil) instead of (X)
this is bout readability and personal preferences.  (X '()) is my
hint to the reader that X will be used as a list type accumulator in the
BODY.  I prefer that over (X ()), (X nil) and (X).  It's a stylistic
habit.

Why?  Because I want to make clear that the variable is initialized with
an empty list, compared to what the evaluation of an empty list aka nil
returns (which accidentally happens to be the same again in Lisp).

Thanks for mentioning this point Drew,

Michael.

RE: [External] : Closures - do you understand them well?

[Drew Adams]

> But yes, like explicitly specifying a nil binding (X nil) instead of (X)
> this is bout readability and personal preferences.  (X '()) is my
> hint to the reader that X will be used as a list type accumulator in the
> BODY.  I prefer that over (X ()), (X nil) and (X).  It's a stylistic
> habit.
> 
> Why?  Because I want to make clear that the variable is initialized with
> an empty list, compared to what the evaluation of an empty list aka nil
> returns (which accidentally happens to be the same again in Lisp).

I have _exactly_ the same concern as you:
human-reader readability (and in most cases
that's me).

I use (let ((xs    ())
            (foop  nil)
            (bar   nil)
            (yy))

to tell me that:

1. xs and foop are bound initially.
   yy gets its meaningful value only from
   the `let' body - its initial nil value
   is (should be) meaningless/ignored.

2. xs has a list value (at least initially).
   IOW, I use () for the same reason you use
   '().

3. foop and bar have non-(non-nil) values
   (at least initially).

4. foop is used as a Boolean (true/false).

   bar is likely not used just as a Boolean.
   (bar might have several possible values,
   where nil might mean some base case,
   which could, but need not, mean false.

Wrt #4: I use the suffix code convention
"p"/"-p"  for variable names, as well as for
function names.  I find this _very_ helpful.
Throughout the body, I'm reminded that foop
is Boolean.
___

You say, "which accidentally happens to be
the same again in Lisp".  To me it's not an
accident; it's by design.  It's arguable,
and it's been argued, but to me this design
is (1) clever/elegant and (2) _very_ handy.
Is #1 important?  Maybe not, but to me it's
lovely.

RE: [External] : Closures - do you understand them well?

[Drew Adams]

Typo - forgot to mention bar here:

> I use (let ((xs    ())
>             (foop  nil)
>             (bar   nil)
>             (yy))
> to tell me that:
> 
> 1. xs and foop are bound initially.
                ^^^^
               and bar
>    yy gets its meaningful value only from
>    the `let' body - its initial nil value
>    is (should be) meaningless/ignored.

Re: [External] : Closures - do you understand them well?

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> You say, "which accidentally happens to be
> the same again in Lisp".  To me it's not an
> accident; it's by design.

Sure it's by design, and it's convenient.

But for readability purposes that doesn't matter that much.

When the reader encounters "()" I expect it to construct an empty list.
That's what I want, so I quote it to get exactly that when that
expression is evaluated.

If we do not limit ourselves to Emacs Lisp, the (not trivial) question
would else be: what is the return value when evaluating an empty list?  Not
trivial because there can be Lisps where an empty list and a boolean
"false" are different (AFAIR such Lisps exist).

So the result of evaluating an empty list could be:

  - an empty list/ the same empty list (self-evaluating)
  - undefined
  - a Boolean value false
  - ...maybe something else?

Because I don't want to tangent this question I prefer to quote the
empty list.

What are your reasons to prefer to evaluate it and use the result?

Michael.

Re: Closures - do you understand them well?

[Tassilo Horn]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> What is the return value of the following expression, and what's your
> reasoning?  Answer without asking Emacs:

Obviously 6.

> #+begin_src emacs-lisp
> ;; -*- lexical-binding: t -*-
>
> (let ((i 0)
>       (funs '()))
>   (while (< (setq i (1+ i))
>             4)
>     (push (lambda () i)
>           funs))
>   (apply #'+ (mapcar #'funcall funs)))
> #+end_src

Shocking!!!  Common Lisp agrees:

--8<---------------cut here---------------start------------->8---
(let ((funs '()))
  (loop for i from 1 to 3
     do (push (lambda () i) funs))
  (apply #'+ (mapcar #'funcall funs)))
;=> 12
--8<---------------cut here---------------end--------------->8---

So I guess the reason is that the `i' is not an integer but a place
where the integers 1, 2, and 3?

Well, at least rust gets this right:

--8<---------------cut here---------------start------------->8---
fn main() {
    let mut funs = vec![];
    let mut i: i32 = 1;
    while i < 4 {
        funs.push(move || i);
        i += 1;
    }
    let result: i32 = funs.iter().map(|f| f()).sum();
    println!("Result = {}", result); // 6, that's right!
}
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

RE: [External] : Closures - do you understand them well?

[Drew Adams]

> When the reader encounters "()" I expect it to construct an empty list.
> That's what I want, so I quote it to get exactly that when that
> expression is evaluated.
> 
> If we do not limit ourselves to Emacs Lisp, the (not trivial) question
> would else be: what is the return value when evaluating an empty list?
> Not trivial because there can be Lisps where an empty list and a boolean
> "false" are different (AFAIR such Lisps exist).

Scheme.

As CLTL2 says at the outset:

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node9.html#SECTION00522000000000000000

 In Common Lisp, as in most Lisp dialects, the symbol
                 ^^^^^^^^^^^^^^^^^^^^^^^^
 nil is used to represent both the empty list and the
 ``false'' value for Boolean tests.  An empty list may,
 of course, also be written (); this normally denotes
 the same object as nil.... These two notations may be
 used interchangeably as far as the Lisp system is
 concerned.  However, as a matter of style, this book
 uses the notation () when it is desirable to emphasize
 the use of an empty list, and uses the notation nil
 when it is desirable to emphasize the use of the
 Boolean ``false''.  The notation 'nil (note the
 explicit quotation mark) is used to emphasize the use
 of a symbol.  For example:

 (defun three () 3)      ;Emphasize empty parameter list 

 (append '() '()) => ()  ;Emphasize use of empty lists 

 (not nil) => t          ;Emphasize use as Boolean ``false'' 

 (get 'nil 'color)       ;Emphasize use as a symbol

 Any data object other than nil is construed to be Boolean
 ``not false'', that is, ``true''.  The symbol t is
 conventionally used to mean ``true'' when no other value
 is more appropriate.  When a function is said to ``return
 false'' or to ``be false'' in some circumstance, this
 means that it returns nil.  However, when a function is
 said to ``return true'' or to ``be true'' in some
 circumstance, this means that it returns some value other
 than nil, but not necessarily t.

(Note that the second example does what you do.
(Of course at this point in the book there's
been no mention of t and nil/() being
self-evaluating.)

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node57.html#SECTION00911000000000000000

> Because I don't want to tangent this question I prefer
> to quote the empty list.  What are your reasons to
> prefer to evaluate it and use the result?

No reason not to do that, i.e., no reason to quote it.  

Why not emphasize and take advantage of the fact that 
(always) '() = ()?  No more reason to quote it than
there is (in Elisp, unlike in Common Lisp) to quote a
lambda expression (not speaking about #' "quoting" here).

Re: Closures - do you understand them well?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hi,
>
> Maybe you will have fun with this new exercise:
>
> What is the return value of the following expression, and what's your
> reasoning?  Answer without asking Emacs:

I lost! I really thought that with lexical-binding non-nil it would do
what I expected. I guess this is why I try not to use closures.

RE: [External] : Closures - do you understand them well?

[Drew Adams]

I probably should have also cited these Elisp
messages about nil:

https://www.gnu.org/software/emacs/manual/html_node/eintr/nil-explained.html

https://www.gnu.org/software/emacs/manual/html_node/elisp/nil-and-t.html

Re: Closures - do you understand them well?

[Michael Heerdegen]

Tassilo Horn <tsdh@gnu.org> writes:

> Shocking!!!

The example is written in a way to suggest a wrong result.  But don't we
all rely on the behavior demonstrated from time to time?  The
implementation of `letrec' for example:

(macroexpand-1 '(letrec ((f (lambda () f)))))
==> (let (f) (setq f (lambda nil f)))

We expect that the value of F in the closure bound to F references the
closure itself, not the initial value.

>  Common Lisp agrees:
>
> (let ((funs '()))
>   (loop for i from 1 to 3
>      do (push (lambda () i) funs))
>   (apply #'+ (mapcar #'funcall funs)))
> ;=> 12

I guess this depends a bit on the implementation of `loop'.  But I think
the result is expected, yes.  Don't you remember what you learned about
shared environments that different places in the code have access to
(including the ability to modify)?

> So I guess the reason is that the `i' is not an integer but a place
> where the integers 1, 2, and 3?

It's a free variable in all the lambdas, and it refers to the binding in
the same environment.  When the `apply' call is made the binding in that
environment is the one from the moment when it was last changed in the
loop, and that is `4`.  That's all.

Michael.

Re: Closures - do you understand them well?

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> (let ((funs '()))
>>   (loop for i from 1 to 3
>>      do (push (lambda () i) funs))
>>   (apply #'+ (mapcar #'funcall funs)))
>> ;=> 12
>
> I guess this depends a bit on the implementation of `loop'.

I haven't checked but for the similar code using `dolist` or `dotimes`
Common Lisp indeed leaves it unspecified ig the same `i` binding is used
through the loop or if a different binding is used at each iteration.

In ELisp, both `dotimes` and `dolist` create a new binding for `i` at
each iteration of the loop.


        Stefan

Re: Closures - do you understand them well?

[Emanuel Berg]

Eric Abrahamsen wrote:

> I lost! I really thought that with lexical-binding non-nil
> it would do what I expected. I guess this is why I try not
> to use closures.

Oh, use them! They are very useful :)

I understand the two use cases I've found (share access to
variables between functions, persistent values in variables
between function calls) and it works well for those.

Example with both those:

  https://dataswamp.org/~incal/emacs-init/w3m/w3m-survivor.el

But I'd like to understand this riddle as well ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> In ELisp, both `dotimes` and `dolist` create a new binding for `i` at
> each iteration of the loop.

This is probably what most people expect from a loop.  `cl-loop'
doesn't.

I guess it's not good to rely on either behavior of built-in looping
constructs.

Michael.

Re: Closures - do you understand them well?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I lost! I really thought that with lexical-binding non-nil it would do
> what I expected. I guess this is why I try not to use closures.

No no, sharpen your mental model.

The exercise originates from a mistake I made myself a while ago.
Sometimes your brain establishes shortcuts that lead to wrong
assumptions. It's human to fall into such traps.

Michael.

Re: Closures - do you understand them well?

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> In ELisp, both `dotimes` and `dolist` create a new binding for `i` at
>> each iteration of the loop.
> This is probably what most people expect from a loop.

I find this behavior cleaner, indeed (which is why I changed `dotimes`
and `dolist` to provide that behavior :-).

> `cl-loop' doesn't.

Indeed, when I looked at it, I decided it was too much trouble figuring
out how to change `cl-loop` to provide that behavior :-(
IMO `cl-loop` is too complex for its own good.


        Stefan

RE: [External] : Re: Closures - do you understand them well?

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

> IMO `cl-loop` is too complex for its own good.

Maybe ... "Don't Loop, Iterate"?

https://iterate.common-lisp.dev/doc/index.html

(Dunno what binding it uses in this context.)

___

https://iterate.common-lisp.dev/doc/Don_0027t-Loop-Iterate.html#Don_0027t-Loop-Iterate

https://dspace.mit.edu/handle/1721.1/41498?show=full

[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 13481 bytes --]

Re: [External] : Re: Closures - do you understand them well?

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> IMO `cl-loop` is too complex for its own good.
> Maybe ... "Don't Loop, Iterate"?
> https://iterate.common-lisp.dev/doc/index.html

IMO it's still "too complex for its own good".

> (Dunno what binding it uses in this context.)

`iterate` explicitly defines those vars to use a single binding over the
whole iteration (I suspect the same holds for `loop` as well, actually).
IOW it's very fundamentally "imperative", where iteration variables are
defined to be "uninitialized" at the beginning at are updated by
side-effect during the course of the iteration.
[ I tend to cringe at the idea of uninitialized variables.  ]


        Stefan

Re: Closures - do you understand them well?

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> The exercise originates from a mistake I made myself a while ago.
> Sometimes your brain establishes shortcuts that lead to wrong
> assumptions. It's human to fall into such traps.

But I see that exercises that are just traps can be demotivating.

I'm not sure how to write more motivating exercises, however, how would
they look like?  Maybe like, you think it returns 3 but instead it
returns...the perfect Christmas present you were looking for?

#+begin_src emacs-lisp
  (if (you-think-this-is-non-nil-but-isnt)
      3
    "You should buy XYZ.")
#+end_src

Have to think about it...

Michael.

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>> In ELisp, both `dotimes` and `dolist` create a new binding
>> for `i` at each iteration of the loop.
>
> This is probably what most people expect from a loop.
> `cl-loop' doesn't.
>
> I guess it's not good to rely on either behavior of built-in
> looping constructs.

OK, so is it the closures? Or the loops?

Or ... the lamdas?

(setq x 111)

(setq f (lambda () x))

(funcall f)

(setq x 222)

(funcall f)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> OK, so is it the closures? Or the loops?
>
> Or ... the lamdas?
>
> (setq x 111)
>
> (setq f (lambda () x))
>
> (funcall f)
>
> (setq x 222)
>
> (funcall f)

Maybe I'm too tired, but I don't understand the question.  What's "it"
in the first question?

Michael.

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 565 bytes --]

On Fri, Dec 09, 2022 at 05:01:23AM +0100, Michael Heerdegen wrote:
> Emanuel Berg <incal@dataswamp.org> writes:
> 
> > OK, so is it the closures? Or the loops?
> >
> > Or ... the lamdas?
> >
> > (setq x 111)
> >
> > (setq f (lambda () x))
> >
> > (funcall f)
> >
> > (setq x 222)
> >
> > (funcall f)
> 
> Maybe I'm too tired, but I don't understand the question.  What's "it"
> in the first question?

My gut reaction was such that the answer to Emanuel's question
would have been "it's the bindings".

But I may be wrong.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Thu, Dec 08, 2022 at 05:00:14PM -0500, Stefan Monnier via Users list for the GNU Emacs text editor wrote:
> >> In ELisp, both `dotimes` and `dolist` create a new binding for `i` at
> >> each iteration of the loop.
> > This is probably what most people expect from a loop.
> 
> I find this behavior cleaner, indeed (which is why I changed `dotimes`
> and `dolist` to provide that behavior :-).

This is funny. I did come up with the right answer. Truth be
said, Michael's "tone" was set up in a way to raise awareness
("now, look carefully"), so this has surely played a role.

That said, this is actually the behaviour I expect (and like),
so reading your note taught me something: expectations here
seem to vary, and mine isn't the "only", much less the "right"
one. And I might meet loop constructs with local bindings :)

So I learnt something from the riddle, after all [1].

> > `cl-loop' doesn't.
> 
> Indeed, when I looked at it, I decided it was too much trouble figuring
> out how to change `cl-loop` to provide that behavior :-(
> IMO `cl-loop` is too complex for its own good.

Uh-huh. I never wanted to learn a whole language to just do
looping :-)

Thanks for the riddle and all the follow-up!

[1] The most general rule seems still to be: "all generalisations
   suck"

-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

RE: [External] : Re: Closures - do you understand them well?

[Tomas Hlavaty]

On Thu 08 Dec 2022 at 22:25, Drew Adams <drew.adams@oracle.com> wrote:
> Maybe ... "Don't Loop, Iterate"?
>
> https://iterate.common-lisp.dev/doc/index.html

iterate is very bad idea
i wish people would not depend on it
and used the simplest form to do the job instead

Re: Closures - do you understand them well?

[Emanuel Berg]

tomas wrote:

>>> OK, so is it the closures? Or the loops?
>>>
>>> Or ... the lamdas?
>>>
>>> (setq x 111)
>>>
>>> (setq f (lambda () x))
>>>
>>> (funcall f)
>>>
>>> (setq x 222)
>>>
>>> (funcall f)
>> 
>> Maybe I'm too tired, but I don't understand the question.
>> What's "it" in the first question?
>
> My gut reaction was such that the answer to Emanuel's
> question would have been "it's the bindings".

This is why you should never quote lambdas because if you do,
you get the intuitive result:

(let ((i 0)
      (funs '()))
  (while (< (setq i (1+ i))
            4)
    (push `(lambda () ,i)
          funs))
  (apply #'+ (mapcar #'funcall funs))) ; 6

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> The exercise originates from a mistake I made myself a while ago.
>> Sometimes your brain establishes shortcuts that lead to wrong
>> assumptions. It's human to fall into such traps.
>
> But I see that exercises that are just traps can be demotivating.
>
> I'm not sure how to write more motivating exercises, however, how would
> they look like?  Maybe like, you think it returns 3 but instead it
> returns...the perfect Christmas present you were looking for?
>
> #+begin_src emacs-lisp
>   (if (you-think-this-is-non-nil-but-isnt)
>       3
>     "You should buy XYZ.")
> #+end_src
>
> Have to think about it...

It's true I don't enjoy "exercises" in general -- a nice Christmas
present would be a motivation.

I would love to know closures well enough to put them into use without
having to spend extra time thinking and testing. But the fact is I *do*
need to spend extra time on it, and I rarely hit upon a problem that's
begging to be resolved with a closure, so...

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> This is why you should never quote lambdas because if you do,
> you get the intuitive result:
>
> (let ((i 0)
>       (funs '()))
>   (while (< (setq i (1+ i))
>             4)
>     (push `(lambda () ,i)
>           funs))
>   (apply #'+ (mapcar #'funcall funs))) ; 6

I think the intended goal was to see that shortcuts can cause trouble,
not to try to find a better shortcut...

Michael.

Re: Closures - do you understand them well?

[Emanuel Berg]

Eric Abrahamsen wrote:

> I would love to know closures well enough to put them into
> use without having to spend extra time thinking and testing.
> But the fact is I *do* need to spend extra time on it

Eval this:

  (lambda () x)

Same thing, no let-closure. Aren't lambdas like
that everywhere?

let-closures are not hard to understand, put a bunch of
`defun' in a `let' and use the variables like they were
globals, only they are not ...

> and I rarely hit upon a problem that's begging to be
> resolved with a closure, so...

Grep your source for global variables. Those are the ones you
can get rid of using let-closures, and without changing the
code using them. (Possible exception, for practical reasons:
Emacs options.)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I would love to know closures well enough to put them into use without
> having to spend extra time thinking and testing. But the fact is I *do*
> need to spend extra time on it, and I rarely hit upon a problem that's
> begging to be resolved with a closure, so...

Or maybe you don't hear them begging?

What were your resources for learning?


Michael.

Re: Closures - do you understand them well?

[Michael Heerdegen]

<tomas@tuxteam.de> writes:

> So I learnt something from the riddle, after all [1].

Cool that there was something to learn for some people.

The second part of the exercise is now of course (this goes to
everyone): how to "fix" the code so that what some people expected is
the end result?...:

 Write an expression that returns a list of 100 functions accepting zero
 arguments.  `funcall'ing the Nth function must return N.  Use a `while'
 loop or whatever you like but please not `dolist' or any tool that
 already handles this "problem" specifically.


> [1] The most general rule seems still to be: "all generalisations
>    suck"

I think this is the first instance of Russell's paradox I have ever met
in real life.

Michael.

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

On Fri, Dec 09, 2022 at 08:40:50PM +0100, Michael Heerdegen wrote:
> <tomas@tuxteam.de> writes:
> 
> > So I learnt something from the riddle, after all [1].
> 
> Cool that there was something to learn for some people.

It was enjoyable, yes :)

> The second part of the exercise is now of course (this goes to
> everyone): how to "fix" the code so that what some people expected is
> the end result?...:
> 
>  Write an expression that returns a list of 100 functions accepting zero
>  arguments.  `funcall'ing the Nth function must return N.  Use a `while'
>  loop or whatever you like but please not `dolist' or any tool that
>  already handles this "problem" specifically.

Hm. I imagine in the middle of my code "fred's function factory", like
so:

  (defun fred-s-function-factory (n)
    ...)

(i won't tell more now ;)

> > [1] The most general rule seems still to be: "all generalisations
> >    suck"
> 
> I think this is the first instance of Russell's paradox I have ever met
> in real life.

Then I would like you to meet my barber...

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Closures - do you understand them well?

[Tassilo Horn]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> everyone): how to "fix" the code so that what some people expected is
> the end result?...:
>
>  Write an expression that returns a list of 100 functions accepting zero
>  arguments.  `funcall'ing the Nth function must return N.  Use a `while'
>  loop or whatever you like but please not `dolist' or any tool that
>  already handles this "problem" specifically.

This time I'm prepared and have three versions making a new binding.
Take this, Michael! :-)

--8<---------------cut here---------------start------------->8---
(let ((i 0)
      (m (lambda (i)
           (lambda () i)))
      funs)
  (while (< i 100)
    (push (funcall m i) funs)
    (cl-incf i))
  (mapcar #'funcall funs))

(let ((i 0)
      funs)
  (while (< i 100)
    (let ((j i))
      (push (lambda () j) funs))
    (cl-incf i))
  (mapcar #'funcall funs))

(let ((i 0)
      funs)
  (while (< i 100)
    (push `(lambda () ,i) funs)
    (cl-incf i))
  (mapcar #'funcall funs))
--8<---------------cut here---------------end--------------->8---

Oh, I forgot to `nreverse` funs so my Nth function returns 99-N instead.
Good enough...

Bye,
Tassilo

Re: Closures - do you understand them well?

[Michael Heerdegen]

Tassilo Horn <tsdh@gnu.org> writes:

> This time I'm prepared and have three versions making a new binding.
> Take this, Michael! :-)

Wow - ok, let's see...

> (let ((i 0)
>       (m (lambda (i)
>            (lambda () i)))
>       funs)
>   (while (< i 100)
>     (push (funcall m i) funs)
>     (cl-incf i))
>   (mapcar #'funcall funs))

Yes, I think one can do this (the argument variable i shadows the
variable of the same in the `let', but it also exists to conserve the
value from the outside i so one can argue that it's ok to use the same
name - it's semantically not incorrect anyway).  Nice solution.

> (let ((i 0)
>       funs)
>   (while (< i 100)
>     (let ((j i))
>       (push (lambda () j) funs))
>     (cl-incf i))
>   (mapcar #'funcall funs))

This is the trick that is used in the implementation of `dolist': create
a binding that lives only inside the evaluation of the body of the loop
in one iteration step.

> (let ((i 0)
>       funs)
>   (while (< i 100)
>     (push `(lambda () ,i) funs)
>     (cl-incf i))
>   (mapcar #'funcall funs))

Hmm...hmm.  No, we don't want to quote lambdas.  When Stefan sees that
he might install a patch that will make the above error (he really could
do this!).  Better only do that privately.

> Oh, I forgot to `nreverse` funs so my Nth function returns 99-N instead.
> Good enough...

Ok, it still counts (but maybe backwards).


Regards,

Michael.

Re: Closures - do you understand them well?

[Emanuel Berg]

Tassilo Horn wrote:

> This time I'm prepared and have three versions making a new
> binding. Take this, Michael! :-)
>
> (let ((i 0)
>       (m (lambda (i)
>            (lambda () i)))
>       funs)
>   (while (< i 100)
>     (push (funcall m i) funs)
>     (cl-incf i))
>   (mapcar #'funcall funs))

Same.

> (let ((i 0)
>       funs)
>   (while (< i 100)
>     (let ((j i))
>       (push (lambda () j) funs))
>     (cl-incf i))
>   (mapcar #'funcall funs))

DNC ...

> (let ((i 0)
>       funs)
>   (while (< i 100)
>     (push `(lambda () ,i) funs)
>     (cl-incf i))
>   (mapcar #'funcall funs))

Yes, but here there is no "new binding" exactly, rather the
value is hard-coded onto the lambda ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>> (let ((i 0)
>>       (m (lambda (i)
>>            (lambda () i)))
>>       funs)
>>   (while (< i 100)
>>     (push (funcall m i) funs)
>>     (cl-incf i))
>>   (mapcar #'funcall funs))
>
> Yes, I think one can do this (the argument variable
> i shadows the variable of the same in the `let', but it also
> exists to conserve the value from the outside i so one can
> argue that it's ok to use the same name - it's semantically
> not incorrect anyway). Nice solution.

Okay so now it works for me as well, maybe some encoding in
messed up evaluation ...

>> (let ((i 0)
>>       funs)
>>   (while (< i 100)
>>     (let ((j i))
>>       (push (lambda () j) funs))
>>     (cl-incf i))
>>   (mapcar #'funcall funs))
>
> This is the trick that is used in the implementation of
> `dolist': create a binding that lives only inside the
> evaluation of the body of the loop in one iteration step.

Same, works.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

>  Write an expression that returns a list of 100 functions accepting zero
>  arguments.  `funcall'ing the Nth function must return N.  Use a `while'
>  loop or whatever you like but please not `dolist' or any tool that
>  already handles this "problem" specifically.

Would this be a valid solution?

#+begin_src emacs-lisp
(let ((i 0) (funs))
  (while (<= (cl-incf i) 100)
    (push (apply-partially #'identity i) funs))
  (nreverse funs))
#+end_src

Michael.

Re: Closures - do you understand them well?

[Emanuel Berg]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

> Indeed, when I looked at it, I decided it was too much
> trouble figuring out how to change `cl-loop` to provide that
> behavior :-( IMO `cl-loop` is too complex for its own good.

All of Lisp is, and the reason is it's so simple to begin with ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Closures - do you understand them well?

[Emanuel Berg]

Drew Adams wrote:

>> IMO `cl-loop` is too complex for its own good.
>
> Maybe ... "Don't Loop, Iterate"?

What does that saying say?

Traversal of data structures?

If so, `cl-loop' can do that ... in many ways :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Closures - do you understand them well?

[Emanuel Berg]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

> `iterate` explicitly defines those vars to use a single
> binding over the whole iteration (I suspect the same holds
> for `loop` as well, actually). IOW it's very fundamentally
> "imperative", where iteration variables are defined to be
> "uninitialized" at the beginning at are updated by
> side-effect during the course of the iteration. [ I tend to
> cringe at the idea of uninitialized variables. ]

Iteration meaning a for loop is imperative, a traversal of
a data structure is pretty functional IMO ... set, aggregate,
map, higher-order ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Closures - do you understand them well?

[Emanuel Berg]

Tomas Hlavaty wrote:

>> Maybe ... "Don't Loop, Iterate"? [...]
>
> iterate is very bad idea
> i wish people would not depend on it
> and used the simplest form to do the job instead

Donatello King of Salsa, what are you talking about?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Tassilo Horn wrote:

> (let ((i 0)
>       (m (lambda (i)
>            (lambda () i)))
>       funs)
>   (while (< i 100)
>     (push (funcall m i) funs)
>     (cl-incf i))
>   (mapcar #'funcall funs))
>
> (let ((i 0)
>       funs)
>   (while (< i 100)
>     (let ((j i))
>       (push (lambda () j) funs))
>     (cl-incf i))
>   (mapcar #'funcall funs))

OK, so `lexical-binding' required, that's why they don't work
in the article buffer but suddenly work when I yank them into
an Elisp buffer ...

(setq-default lexical-binding t)

anyone?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Closures - do you understand them well?

[Emanuel Berg]

Drew Adams wrote:

>> What are your reasons to prefer to evaluate it and use
>> the result?
>
> No reason not to do that, i.e., no reason to quote it.
>
> Why not emphasize and take advantage of the fact that
> (always) '() = ()? No more reason to quote it than there is
> (in Elisp, unlike in Common Lisp) to quote a lambda
> expression (not speaking about #' "quoting" here).

I think we need a special syntax to denote list quoting ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>> OK, so is it the closures? Or the loops?
>>
>> Or ... the lamdas?
>>
>> (setq x 111)
>>
>> (setq f (lambda () x))
>>
>> (funcall f)
>>
>> (setq x 222)
>>
>> (funcall f)
>
> Maybe I'm too tired, but I don't understand the question.  What's "it"
> in the first question?

I don't know, and I studied it at the university even ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Tassilo Horn]

Michael Heerdegen <michael_heerdegen@web.de> writes:

>>  Write an expression that returns a list of 100 functions accepting
>>  zero arguments.  `funcall'ing the Nth function must return N.  Use a
>>  `while' loop or whatever you like but please not `dolist' or any
>>  tool that already handles this "problem" specifically.
>
> Would this be a valid solution?
>
> #+begin_src emacs-lisp
> (let ((i 0) (funs))
>   (while (<= (cl-incf i) 100)
>     (push (apply-partially #'identity i) funs))
>   (nreverse funs))
> #+end_src

Apparently it works although I don't see why.  Where's the difference to
this inlined version which returns (101 ... 101)?

--8<---------------cut here---------------start------------->8---
(let ((i 0) (funs))
  (while (<= (cl-incf i) 100)
    (push (lambda (&rest args2)
            (apply #'identity (append (list i) args2)))
          funs))
  (mapcar #'funcall (nreverse funs)))
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

Re: Closures - do you understand them well?

[Emanuel Berg]

Tassilo Horn wrote:
 
> Apparently it works although I don't see why. Where's the
> difference to this inlined version which returns (101 ...
> 101)?
>
> (let ((i 0) (funs))
>   (while (<= (cl-incf i) 100)
>     (push (lambda (&rest args2)
>             (apply #'identity (append (list i) args2)))
>           funs))
>   (mapcar #'funcall (nreverse funs)))

I doesn't work for me and I also don't see why ...

(101 ... 101)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

On Fri, Dec 09, 2022 at 10:23:21PM +0100, Emanuel Berg wrote:
> Tassilo Horn wrote:

[...]

> > (let ((i 0)
> >       funs)
> >   (while (< i 100)
> >     (push `(lambda () ,i) funs)
> >     (cl-incf i))
> >   (mapcar #'funcall funs))
> 
> Yes, but here there is no "new binding" exactly, rather the
> value is hard-coded onto the lambda ...

Look closely. There is one. Just at macro expansion (aka compile)
time. Welcome to Lisp's multiple personality :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Mutation - do you understand it really? (was: Closures - do you understand them well?)

[Stefan Monnier via Users list for the GNU Emacs text editor]

Tassilo Horn [2022-12-10 10:34:24] wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:
>>>  Write an expression that returns a list of 100 functions accepting
>>>  zero arguments.  `funcall'ing the Nth function must return N.  Use a
>>>  `while' loop or whatever you like but please not `dolist' or any
>>>  tool that already handles this "problem" specifically.
>>
>> Would this be a valid solution?
>>
>> #+begin_src emacs-lisp
>> (let ((i 0) (funs))
>>   (while (<= (cl-incf i) 100)
>>     (push (apply-partially #'identity i) funs))
>>   (nreverse funs))
>> #+end_src
>
> Apparently it works although I don't see why.

BTW, for those people here who find this all confusing.
I'll point out that the culprit is not "closures" but `setq` (here
hidden within `cl-incf`).

Immutable variables are just a lot better behaved (and easier to work
with for the compiler, which is why GCC/LLVM/... use an SSA conversion
to replace those nasty mutable variables with immutable ones).


        Stefan

RE: [External] : Re: Closures - do you understand them well?

[Drew Adams]

> > "Don't Loop, Iterate"
> 
> What does that saying say?

Just a pun:

1. "loop" and "iterate" are synonyms.
2. Common Lisp `loop' and the writer's macro `iterate'
   are different (but similar).

The author of macro `iterate' is saying, with that
title, that macro `iterate' is preferable to `loop'.
The point of his paper is to say how/why he thinks
that's the case.

RE: [External] : Re: Closures - do you understand them well?

[Drew Adams]

> > Indeed, when I looked at it, I decided it was too much
> > trouble figuring out how to change `cl-loop` to provide that
> > behavior :-( IMO `cl-loop` is too complex for its own good.
> 
> All of Lisp is, and the reason is it's so simple to begin with ...

Most likely Stefan meant that the code implementing
`cl-loop' is complex - even too complex for its own
good (it's a bother to maintain/improve/correct).

But it's also the case that for a _user_ the syntax
of `(cl-)loop' is complex - certainly much more
complex than the usual Lisp syntax.  You essentially
have to learn another language - `loop' - to use it.

That's one of the arguments in favor of using macro
`iterate' instead: its syntax is more "lispy".

Users can of course learn the language of `loop',
just as they can learn the language of Unix `find',
and just as they can learn the language (patterns)
of regular expressions, or those of `pcase'.

And for the simplest, most common `loop' cases they
need not learn/memorize everything in the language.

Nevertheless, it remains true that a second/separate
language is introduced.  That can present advantages
(code can often look like simple English commands),
but it also presents disadvantages.

It's largely a question of taste/style, and
"Des goûts et des couleurs, on ne discute pas".

https://dictionary.reverso.net/french-english/des+go%C3%BBts+et+des+couleurs+on+ne+discute+pas

On the other hand, Occam might have something to
say here about multiplying things unnecessarily.

https://en.wikipedia.org/wiki/Occam%27s_razor

(The rule behind Occam's taste & color preferences,
perhaps, but a good rule of thumb nevertheless.
The taste question comes in when deciding just
which multiplying is "unnecessary".)

Iteration macros (was: [External] : Re: Closures - do you understand them well?)

[Stefan Monnier via Users list for the GNU Emacs text editor]

> But it's also the case that for a _user_ the syntax
> of `(cl-)loop' is complex - certainly much more
> complex than the usual Lisp syntax.  You essentially
> have to learn another language - `loop' - to use it.

That's what I meant, actually, yes.

I'm not opposed to introducing specialized sublanguages (after all, I've
done that myself for `pcase`, `setf`, and `bindat`, and I'm not opposed
to things like `rx` or `peg`), but I think what annoys me in `cl-loop`
is that you cannot understand each subelement independently because the
effect of each element often depends on the presence/absence of other
elements or the place where it appears, ...: it's not just a separate
language but that language is not modular (and hence in my view is a bad
language design).

Shiver's [Anatomy of
a Loop](https://www.ccs.neu.edu/home/shivers/papers/loop.pdf) is much
better in this respect, but I'm still not a big fan because for example
(bind <bindings>) changes the environment of *subsequent* expressions,
which I find ugly (admittedly, he does that in Scheme where `define`
already suffers from the same problem).


        Stefan "who votes for `named-let` :-)"

Re: Mutation - do you understand it really?

[Michael Heerdegen]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Immutable variables are just a lot better behaved (and easier to work
> with for the compiler, which is why GCC/LLVM/... use an SSA conversion
> to replace those nasty mutable variables with immutable ones).

The original question used `setq'.  I presented it to a friend that
knows lambda calculus and Haskell very well, but Lisp not at all.  When
he asked "what does this setq do" and I gave an answer he was shocked
that these bindings are mutual in Lisp and his reaction was like
"WTF?...Ok, then...".

But mutable variable bindings are a legitimate aspect of semantics of
closures in Lisp - I hope I didn't scare anybody.  When there is an
intention then it is to hint at pitfalls so that people know them before
they cause trouble.

Michael.

RE: Iteration macros (was: [External] : Re: Closures - do you understand them well?)

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

> > But it's also the case that for a _user_ the syntax
> > of `(cl-)loop' is complex - certainly much more
> > complex than the usual Lisp syntax.  You essentially
> > have to learn another language - `loop' - to use it.
> 
> That's what I meant, actually, yes.
> 
> I'm not opposed to introducing specialized sublanguages (after all, I've
> done that myself for `pcase`, `setf`, and `bindat`, and I'm not opposed
> to things like `rx` or `peg`), but I think what annoys me in `cl-loop`
> is that you cannot understand each subelement independently because the
> effect of each element often depends on the presence/absence of other
> elements or the place where it appears, ...: it's not just a separate
> language but that language is not modular (and hence in my view is a bad
> language design).

100% agreement.

> Shiver's [Anatomy of a Loop](
> https://www.ccs.neu.edu/home/shivers/papers/loop.pdf) is much
> better in this respect, but I'm still not a big fan because for example
> (bind <bindings>) changes the environment of *subsequent* expressions,
> which I find ugly (admittedly, he does that in Scheme where `define`
> already suffers from the same problem).

Thanks for that; wasn't familiar with it.

[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 14573 bytes --]

Re: Closures - do you understand them well?

[Michael Heerdegen]

Tassilo Horn <tsdh@gnu.org> writes:

> > Would this be a valid solution?
> >
> > #+begin_src emacs-lisp
> > (let ((i 0) (funs))
> >   (while (<= (cl-incf i) 100)
> >     (push (apply-partially #'identity i) funs))
> >   (nreverse funs))
> > #+end_src
>
> Apparently it works although I don't see why.

I do think the above is a valid solution.

> Where's the difference to
> this inlined version which returns (101 ... 101)?
>
> (let ((i 0) (funs))
>   (while (<= (cl-incf i) 100)
>     (push (lambda (&rest args2)
>             (apply #'identity (append (list i) args2)))
>           funs))
>   (mapcar #'funcall (nreverse funs)))

Well - `apply-partially' is a function and I use a simple function call,
so it is precisely not just something inlined: The important point is
that only the _value_ of the variable i passed, not the variable or
a reference to it:

#+begin_src emacs-lisp
(let ((i 27))
  (apply-partially #'identity i))
==>
  (closure
      ((args . (27))
       (fun . identity))
      (&rest args2)
    (apply fun (append args args2)))
#+end_src


Michael.

Re: Closures - do you understand them well?

[Tassilo Horn]

Michael Heerdegen <michael_heerdegen@web.de> writes:

>> Where's the difference to
>> this inlined version which returns (101 ... 101)?
>>
>> (let ((i 0) (funs))
>>   (while (<= (cl-incf i) 100)
>>     (push (lambda (&rest args2)
>>             (apply #'identity (append (list i) args2)))
>>           funs))
>>   (mapcar #'funcall (nreverse funs)))
>
> Well - `apply-partially' is a function and I use a simple function call,
> so it is precisely not just something inlined: The important point is
> that only the _value_ of the variable i passed, not the variable or
> a reference to it:
>
> #+begin_src emacs-lisp
> (let ((i 27))
>   (apply-partially #'identity i))
> ==>
>   (closure
>       ((args . (27))
>        (fun . identity))
>       (&rest args2)
>     (apply fun (append args args2)))
> #+end_src

Thanks, that explains it.

Bye,
Tassilo

Re: Closures - do you understand them well?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I would love to know closures well enough to put them into use without
>> having to spend extra time thinking and testing. But the fact is I *do*
>> need to spend extra time on it, and I rarely hit upon a problem that's
>> begging to be resolved with a closure, so...
>
> Or maybe you don't hear them begging?

I'm sure if I had a better intuitive grasp, they would present
themselves as the right solution more often.

> What were your resources for learning?

Nothing, I guess -- like all my programming knowledge it started with
trying to solve a specific problem, and ended with more in-depth
reading. But I've always found multiple layers of indirection confusing
(recursion/TCO and nested macros included), and any time anything starts
looking like references to references to... I try "fix it" by doing
something else.

Re: Closures - do you understand them well?

[Michael Heerdegen]

<tomas@tuxteam.de> writes:

> > > (let ((i 0)
> > >       funs)
> > >   (while (< i 100)
> > >     (push `(lambda () ,i) funs)
> > >     (cl-incf i))
> > >   (mapcar #'funcall funs))
> >
> > Yes, but here there is no "new binding" exactly, rather the
> > value is hard-coded onto the lambda ...
>
> Look closely. There is one. Just at macro expansion (aka compile)
> time.

?!?

The expansion of (push `(lambda () ,i) funs) is

#+begin_src emacs-lisp
(setq funs
      (cons
       (list 'lambda nil i)
       funs))
#+end_src

No binding is created here, neither at compile time (where the symbol i
is not touched), nor at run-time (where the value of `i` is just passed
as an argument to `list').

Stefan would say that this accumulates a list of function expressions in
FUNS, not a list of functions, strictly speaking.  Works only because
the Lisp interpreter is made to be nice to everyone.

Michael.

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]

On Mon, Dec 12, 2022 at 02:18:04AM +0100, Michael Heerdegen wrote:
> <tomas@tuxteam.de> writes:
> 
> > > > (let ((i 0)
> > > >       funs)
> > > >   (while (< i 100)
> > > >     (push `(lambda () ,i) funs)
> > > >     (cl-incf i))
> > > >   (mapcar #'funcall funs))
> > >
> > > Yes, but here there is no "new binding" exactly, rather the
> > > value is hard-coded onto the lambda ...
> >
> > Look closely. There is one. Just at macro expansion (aka compile)
> > time.
> 
> ?!?
> 
> The expansion of (push `(lambda () ,i) funs) is
                                     ^^^

To me, that one (well, if you squint, the others too)
counts as a compile time binding, of sorts. Or well,
pre-compile or something.

It was the "poor human's closure" in pre-lexical times,
after all.

> #+begin_src emacs-lisp
> (setq funs
>       (cons
>        (list 'lambda nil i)
>        funs))
> #+end_src

Yes, the binding of i is effective before the lambda
expression is "compiled" (whatever that means when you
are playing interpreter, but it still means something,
no?)

> Stefan would say that this accumulates a list of function expressions in
> FUNS, not a list of functions, strictly speaking.  Works only because
> the Lisp interpreter is made to be nice to everyone.

FWIW, I find those languages (the ones trying to be nice)
the most interesting. But they can be confusing, too :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Closures - do you understand them well?

[Michael Heerdegen]

tomas@tuxteam.de writes:

> > The expansion of (push `(lambda () ,i) funs) is
>                                      ^^^
> It was the "poor human's closure" in pre-lexical times,
> after all.

I wonder if that's a reason why lambda expressions where originally made
to be treated as functions.

> > #+begin_src emacs-lisp
> > (setq funs
> >       (cons
> >        (list 'lambda nil i)
> >        funs))
> > #+end_src
>
> Yes, the binding of i is effective before the lambda
> expression is "compiled" (whatever that means when you
> are playing interpreter, but it still means something,
> no?)

That's a somewhat quite meta question - but it very probably does mean
something (at least I know what you mean...)

Michael.

Re: [External] : Re: Closures - do you understand them well?

[Emanuel Berg]

Drew Adams wrote:

>>> "Don't Loop, Iterate"
>> 
>> What does that saying say?
>
> Just a pun:
>
> 1. "loop" and "iterate" are synonyms.

No, loop is in general, iterate is a special case of loops,
namely the incremental (e.g. a for loop) and/or thru
a data-structure (`dolist') ...

(while t (sleep-for 1))

is a loop but it doesn't iterate.

Or maybe I just got it all wrong at age 12 reading
"Learn C++ in 21 Days" so thank you for correcting me if
so ...

(to be honest that sounds logical 'cuz I still don't know C++)

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Closures - do you understand them well?

[Emanuel Berg]

Drew Adams wrote:

>>> Indeed, when I looked at it, I decided it was too much
>>> trouble figuring out how to change `cl-loop` to provide
>>> that behavior :-( IMO `cl-loop` is too complex for its
>>> own good.
>> 
>> All of Lisp is, and the reason is it's so simple to begin
>> with ...
>
> Most likely Stefan meant that the code implementing
> `cl-loop' is complex - even too complex for its own good
> (it's a bother to maintain/improve/correct).

Okay, maybe that complexity have something to do with the huge
scope of what you can do with that in so many ways?

And why I think it's cool with things that breaks the
pattern ...

I don't want a lot of Lisp, the Lisp I use, to be like that.

But it's fun, it feels like the land of total loop freedom
which I didn't know existed :)

In the computer books I read as a kid they always said the for
loop was much more modern an better than the while loop, the
do ... until loop, and wasn't there a 'wend' loop as well,
whatever that did.

I believed it then, now I'm so sure. The while loop is useful
and the for loop is useful, but, for specific datastructures,
there should be specific loops, one shouldn't, IMO, iterate
that manually with the for-increment-i unless there is some
extra spanner one wants to throw somewhere in particular which
the standard datastructure-loop function doesn't do ...

The result of that will also be a nice set of often use
datastructures, all with associated, trusted loop functions
people can the compete to optimize. So it's de facto
standardization which leads to, yeah, better code and better
software I guess.

So yeah, in a way cl-loop in all it's scope is like the
opposite of that, instead of nicely dedicated it can do
everything - including being dedicated because it has that as
well - but again, I'm a maximalist, I don't want what's best
or makes the most sense, well I want that as well, because
I want EVERYTHING!

> But it's also the case that for a _user_ the syntax
> of `(cl-)loop' is complex - certainly much more
> complex than the usual Lisp syntax.  You essentially
> have to learn another language - `loop' - to use it.

Maybe an exaggeration but in a matter of speaking, yes, in
particular the 'with i = 0' looks like, I don't now, some
dialect of SQL?

> That's one of the arguments in favor of using macro
> `iterate' instead: its syntax is more "lispy".

`cl-loop' is, or can be, much less lispy, yes.

> Users can of course learn the language of `loop',
> just as they can learn the language of Unix `find',
> and just as they can learn the language (patterns)
> of regular expressions, or those of `pcase'.

With `cl-loop' it's too much, no doubt about it, still it's
there so why reduce it if it's useful. I would stop using it
if all of Elisp was like that but it isn't so hey, relax.
`cl-loop is cool B) It is much more powerful (expressive) than
the C or C++ for loop in ways that I do understand, and
I don't understand all if (cl-lib), by far :)

> Nevertheless, it remains true that a second/separate
> language is introduced. That can present advantages (code
> can often look like simple English commands), but it also
> presents disadvantages.

Indeed, keep Lisp the old way, no more `cl-loop' style stuff and
where it appear, don't put everything in one function, is what
I wouldn't do!

> It's largely a question of taste/style, and
> "Des goûts

Interesting so "circumflex DNC" (line 83) in the Linux VT:

  https://dataswamp.org/~incal/conf/vt/remap.inc

> (The rule behind Occam's taste & color preferences, perhaps,
> but a good rule of thumb nevertheless. The taste question
> comes in when deciding just which multiplying is
> "unnecessary".)

Whaat???

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

tomas wrote:

> Just at macro expansion (aka compile) time

Side question, how many 'times' are there and what,
at least conceptually, happens in each? Figure or ascii
diagram, anyone?

> Welcome to Lisp's multiple personality :-)

Explain :-) :-(

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

On Thu, Dec 22, 2022 at 05:00:25AM +0100, Emanuel Berg wrote:
> tomas wrote:
> 
> > Just at macro expansion (aka compile) time
> 
> Side question, how many 'times' are there and what,
> at least conceptually, happens in each? Figure or ascii
> diagram, anyone?

It's turtles [1] all the way down! Since you have eval,
you are empowered to compile at run time and thus have
a compile time in there. Still, there are (at least) two
phases (at least whenever you have anything more than
"just" a naive interpreter), one where your "source" is
being analysed to find opportunities ("this variable is
always bound to 3 here [2] (or to the identity fun, or...)
so we can roll that constant into the compile product"),
and the run time, where that product is "run".

> > Welcome to Lisp's multiple personality :-)
> 
> Explain :-) :-(

Other languages (e.g. C) have clearly distinct sub-languages
to do things at compile time (C, again, has CPP, which is
clearly distinct from C proper). In Lisp, you can use Lisp
at compile time (the language you use to transform your
source is Lisp), but at the same time, your environment is
completely different at those two phases. Some go as far
as to say that they are different languages. Typically there
are devices to control that (in Emacs Lisp `eval-when-compile'
and `eval-and-compile', I'll let more knowledgeable folks
chime in and complete that list).

Cheers

[1] https://en.wikipedia.org/wiki/It%27s_turtles_all_the_way_down
[2] That's why lexical is easier for optimising compilers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Mutation - do you understand it really?

[Emanuel Berg]

Michael Heerdegen wrote:

>> Immutable variables are just a lot better behaved (and
>> easier to work with for the compiler, which is why
>> GCC/LLVM/... use an SSA conversion to replace those nasty
>> mutable variables with immutable ones).
>
> The original question used `setq'. I presented it to
> a friend that knows lambda calculus and Haskell very well,
> but Lisp not at all.

A mathematician who has learned one computer language
(Haskell) and pretends to be unable to/takes pride in not ever
doing it again?

> When he asked "what does this setq do" and I gave an answer
> he was shocked

Okay, but what did you say then?

> that these bindings are mutual in Lisp and his reaction was
> like "WTF?...Ok, then...".

First the variables are mutable, now the bindings are
mutual ...

Symbols can have values, that value can change so it's
mutable, but the symbol can be assigned something else (or
nothing), so that's mutable as well. It's all mutable, that's
what you mean?

So when you compile you don't do that, instead generate a new
one for each time the binding is reassociated?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Iteration macros (was: [External] : Re: Closures - do you understand them well?)

[Emanuel Berg]

Stefan Monnier via Users list for the GNU Emacs text editor wrote:

>> But it's also the case that for a _user_ the syntax of
>> `(cl-)loop' is complex - certainly much more complex than
>> the usual Lisp syntax. You essentially have to learn
>> another language - `loop' - to use it.
>
> That's what I meant, actually, yes.
>
> I'm not opposed to introducing specialized sublanguages
> (after all, I've done that myself for `pcase`, `setf`, and
> `bindat`, and I'm not opposed to things like `rx` or `peg`)

On the contrary, they are cool, perhaps one shouldn't have one
language completely made up of them, but here and there they
are interesting for sure, add spice ...

> but I think what annoys me in `cl-loop` is that you cannot
> understand each subelement independently because the effect
> of each element often depends on the presence/absence of
> other elements or the place where it appears, ...: it's not
> just a separate language but that language is not modular
> (and hence in my view is a bad language design).

But `cl-loop' has cool features as well, the multiple for
loops "for" example ...

Can't we have a `leap' (the Advanced Elisp to-the-Point Loop)
that implements all loops known to programming but without the
redundancy/intertanglement of keywords that in practice is
a problem using `cl-loop' and understanding it's
documentation ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Iteration macros (was: [External] : Re: Closures - do you understand them well?)

[Emanuel Berg]

Drew Adams wrote:

>>> But it's also the case that for a _user_ the syntax of
>>> `(cl-)loop' is complex - certainly much more complex than
>>> the usual Lisp syntax. You essentially have to learn
>>> another language - `loop' - to use it.
>> 
>> That's what I meant, actually, yes.
>> 
>> I'm not opposed to introducing specialized sublanguages
>> (after all, I've done that myself for `pcase`, `setf`, and
>> `bindat`, and I'm not opposed to things like `rx` or
>> `peg`), but I think what annoys me in `cl-loop` is that you
>> cannot understand each subelement independently because the
>> effect of each element often depends on the
>> presence/absence of other elements or the place where it
>> appears, ...: it's not just a separate language but that
>> language is not modular (and hence in my view is a bad
>> language design).
>
> 100% agreement.

But first add/do everything, then reduce ... is a method.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Eric Abrahamsen wrote:

> But I've always found multiple layers of indirection
> confusing (recursion/TCO and nested macros included)

TCO = Tail-Call Optimization, so you can use recursion without
blowing up the stack ...

But recursion isn't confusing, it makes for elegant/textbook
programs, it's not really a layer of indirection, is it?

Nested macros tho are like the definition of that (extra
layer/indirection) since instead of

you program -> a program -> that does stuff

it is

you program -> a program -> that expands into a program ->
that does stuff

It makes the programming language programmable ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

tomas wrote:

> It was the "poor human's closure" in pre-lexical times,
> after all.

What are the differences, are we talking closures as in
lexical-let and defun(s)?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>>> The expansion of (push `(lambda () ,i) funs) is
>>                                      ^^^
>> It was the "poor human's closure" in pre-lexical times,
>> after all.
>
> I wonder if that's a reason why lambda expressions where
> originally made to be treated as functions.

Why should the not be, aren't they anonymous functions?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

tomas wrote:

>> Side question, how many 'times' are there and what, at
>> least conceptually, happens in each? Figure or ascii
>> diagram, anyone?
>
> It's turtles [1] all the way down! Since you have eval, you
> are empowered to compile at run time and thus have a compile
> time in there. Still, there are (at least) two phases (at
> least whenever you have anything more than "just" a naive
> interpreter), one where your "source" is being analysed to
> find opportunities ("this variable is always bound to 3 here
> [2] (or to the identity fun, or...) so we can roll that
> constant into the compile product"), and the run time, where
> that product is "run".
>
>>> Welcome to Lisp's multiple personality :-)
>> 
>> Explain :-) :-(
>
> Other languages (e.g. C) have clearly distinct sub-languages
> to do things at compile time (C, again, has CPP, which is
> clearly distinct from C proper). In Lisp, you can use Lisp
> at compile time (the language you use to transform your
> source is Lisp), but at the same time, your environment is
> completely different at those two phases. Some go as far as
> to say that they are different languages. Typically there
> are devices to control that (in Emacs Lisp
> `eval-when-compile' and `eval-and-compile', I'll let more
> knowledgeable folks chime in and complete that list).

The byte-compiler is written in Elisp and it transforms Elisp
including performing various optimization efforts know to the
world of compilers, then the resulting transformed/optimized
Elisp is or can be run independently.

The environment are different since first it's the environment
of the byte-compiler, after that it's the environment of the
compiled program ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> Michael Heerdegen wrote:
>
> >>> The expansion of (push `(lambda () ,i) funs) is
> >>                                      ^^^
> >> It was the "poor human's closure" in pre-lexical times,
> >> after all.
> >
> > I wonder if that's a reason why lambda expressions where
> > originally made to be treated as functions.
>
> Why should the not be, aren't they anonymous functions?

Note I mean the expression, not the values they eval to.

The (unevaluated) expressions are just lists.  As values they are
"quoted lambdas" with all of the problems they come with.  The Elisp
interpreter still accepts them as function values.

Michael.

Re: Mutation - do you understand it really?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> Symbols can have values, that value can change so it's
> mutable, but the symbol can be assigned something else (or
> nothing), so that's mutable as well. It's all mutable, that's
> what you mean?
>
> So when you compile you don't do that, instead generate a new
> one for each time the binding is reassociated?

In Haskell you can't change variable bindings like in Lisp.  You
program in a completely different style.  Read an introduction into
Haskell if you are interested, then you'll understand the reaction.

Michael.

RE: [External] : Re: Mutation - do you understand it really?

[Drew Adams]

> In Haskell you can't change variable bindings like in Lisp.  You
> program in a completely different style.  Read an introduction into
> Haskell if you are interested, then you'll understand the reaction.

Yes - _completely_ different style, behavior, and
meaning.

This isn't specific to Haskell.  But sure, that's
a good example of a popular programming language
that's _purely_ functional.

(We should just say "functional", but that term
has come to encompass even languages such as Lisp,
and it pretty much just means languages that
emphasize expression evaluation, with nesting and
return values.)

Languages that are purely functional (or purely
logic(al)) use variables only in the _math_ sense:
the value of a variable doesn't "vary".  Variables
aren't memory locations that are assigned values
(more than once).  In a given definitional context
a variable has the same value everywhere.  Same
thing for function names.

That, however, with the proviso that at the _top_
level such languages typically _do_ treat function
names, and sometimes variable names, dynamically,
i.e., as dynamic variables - variables in the
traditional programming (assignment) sense: memory
locations to be filled and refilled.  E.g., you
can redefine a function.  There's no `let' at the
top level, within which everything is defined.

Note that Lisp - even "pure" lisp: only lexically
scoped etc. - isn't a purely functional language.
The mere presence of `quote' makes it referentially
opaque.  And even applicative evaluation order is a
problem - not what you want in a useful functional
language, in general.

Lisp isn't lambda calculus or combinatory logic.
And it doesn't pretend to be.  Its impurity isn't
shamefaced. ;-)  Lisp is, well, loosey-goosey.

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

> The (unevaluated) expressions are just lists. As values they
> are "quoted lambdas" with all of the problems they come
> with. The Elisp interpreter still accepts them as
> function values.

Okay, why, for technical reasons?

And what values are the correct function values then?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> Michael Heerdegen wrote:
>
> > The (unevaluated) expressions are just lists. As values they
> > are "quoted lambdas" with all of the problems they come
> > with. The Elisp interpreter still accepts them as
> > function values.
>
> Okay, why, for technical reasons?

Is your question why they are (still) accepted?  I'm not an expert, but
I'll try to give an answer.

In the dynamical binding dialect, lambda expressions were
self-evaluating expressions: the function value was identical (`equal')
to the evaluated expression (when interpreting uncompiled code).  See
the definition of `lambda' in subr.el.

That fact could be exploited to build function values with certain
properties in a copy-and-paste like style (using backquote expressions,
typically).  "Poor-mans closures" already had been mentioned.

Old code still uses such techniques so I guess it would break too
much of such stuff if such values would not be supported any more.

Oh, and of course, since the Emacs interpreter has to support both
dialects at the same time (you can use packages with lexical binding
enabled and disabled in one and the same session), the interpreter has
to support such values anyway since they are valid in one dialect.
AFAIU, conceptually the type of binding is a property of the code - the
interpreter just can run both types of code.


> And what values are the correct function values then?

Lexical closures use a different representation - see

  (info "(elisp) Closures")

The actual structure is not interesting for programming any more.  And
in compiled code or natively compiled code function values are not such
nicely explorable lists anyway.

Michael.

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

> AFAIU, conceptually the type of binding is a property of the
> code - the interpreter just can run both types of code.

I think lexical/static is much better than dynamic in general,
and was thinking about the other discussion where dynamic
typing resulted in much slower compiled code compared with
static typing, so what about the binding dialect/variable
scope, how does that (lexical vs dynamic) affect the execution
speed of the compiled code?

Obviously both Elisp and CL has both ...

Maybe some optimizations to be made with lexical/static
because of the increased enclosure (ha), modularity
and predictability?

>> And what values are the correct function values then?
>
> Lexical closures use a different representation - see
>
>   (info "(elisp) Closures")
>
> The actual structure is not interesting for programming
> any more.

Are we talking lexical let-closures? Why not?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On Sun, Feb 26, 2023 at 01:45:56PM +0100, Emanuel Berg wrote:

[...]

> Maybe some optimizations to be made with lexical/static
> because of the increased enclosure (ha), modularity
> and predictability?

That's the biggest part of it, yes: the compiler has a chance
of knowing what's "going on" and whether "someone else" will
be able to change a variable's value "behind its back" or
not.

Cheers
- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> >> And what values are the correct function values then?
> >
> > Lexical closures use a different representation - see
> >
> >   (info "(elisp) Closures")
> >
> > The actual structure is not interesting for programming
> > any more.
>
> Are we talking lexical let-closures? Why not?

Unless I still misunderstand your question: Lisp is a high-level
language, not Assembler, so like you don't modify the bytes of byte-code
functions by hand, you also don't edit the internal representation of
closures.

Michael.

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>> Are we talking lexical let-closures? Why not?
>
> Unless I still misunderstand your question: Lisp is
> a high-level language, not Assembler, so like you don't
> modify the bytes of byte-code functions by hand, you also
> don't edit the internal representation of closures.

??? Assembler?

It seems to be some big misunderstanding here, yes :)

I simply mean this:

(let ((...))
  (defun () ...) )

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > > The actual structure is not interesting for programming
> > > any more.
> >
> > Are we talking lexical let-closures? Why not?
>
> Unless I still misunderstand your question: Lisp is a high-level
> language, not Assembler, so like you don't modify the bytes of byte-code
> functions by hand, you also don't edit the internal representation of
> closures.

Maybe I should mention oclosures where you can access variables with
accessor functions "from the outside".  But this is also a high-level
thing, the internal structure of oclosures is not important and an
implementation like for "normal" function values.

Micha.

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>>> Are we talking lexical let-closures? Why not?
>>
>> Unless I still misunderstand your question: Lisp is
>> a high-level language, not Assembler, so like you don't
>> modify the bytes of byte-code functions by hand, you also
>> don't edit the internal representation of closures.
>
> Maybe I should mention oclosures where you can access
> variables with accessor functions "from the outside".
> But this is also a high-level thing, the internal structure
> of oclosures is not important and an implementation like for
> "normal" function values.

https://www.iro.umontreal.ca/~monnier/oclosure.pdf

For some reason I'm not attracted to that idea like I was
top-level lexical let-closures, i.e.

(let ((...))
  (defun (...) ...) )

but I should take a look at that document and not just make up
my mind, alltho that works most of the time by now since
I trust my intuition which is a trained skill like any other.

Okay, so when are oclosures used?

When they are useful, right?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

>>>> The actual structure is not interesting for programming
>>>> any more.
>>>
>>> Are we talking lexical let-closures? Why not?
>>
>> Unless I still misunderstand your question: Lisp is
>> a high-level language, not Assembler, so like you don't
>> modify the bytes of byte-code functions by hand, you also
>> don't edit the internal representation of closures.
>
> Maybe I should mention oclosures where you can access
> variables with accessor functions "from the outside".
> But this is also a high-level thing, the internal structure
> of oclosures is not important and an implementation like for
> "normal" function values.

Open closures BTW, isn't that a contradiction in terms,
even? :)

Smells too much OO but maybe they were there before the OO
paradigm ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

Michael Heerdegen wrote:

> Maybe I should mention oclosures where you can access
> variables with accessor functions "from the outside".
> But this is also a high-level thing, the internal structure
> of oclosures is not important and an implementation like for
> "normal" function values.

Here is the abstract:

  While folklore teaches us that closures and objects are two
  sides of the same coin, they remain quite different in
  practice, most notably because closures are opaque, the only
  supported operation being to call them.

  In this article we discuss a few cases where we need
  functions to be less opaque, and propose to satisfy this
  need by extending our beloved 𝜆 so as to expose as sorts of
  record fields some of the variables it captures. These open
  closures are close relatives of CLOS’s funcallable objects
  as well as of the function objects of traditional
  object-oriented languages like Java, except that they are
  functions made to behave like objects rather than the
  reverse. We present the design and implementation of such
  a feature in the context of Emacs Lisp. [1]

There is a list of use case, they seem to have a lot to do
with really specific situations and fringe cases were one
would previous hack one's way thru it each time, and not rely
on ideomatic, clean code, so now with oclosures that's
much easier.

Good, but if so it's maybe nothing indispensible for the Joe
Emacs Hacker, whereas the top-level lexical let-closures are
one or two small steps forward with the three use cases I have
identified (no/less global varaibles, share variables between
functions, preserved variable values between function calls).

But I'm all for the esoteric stuff as well so it's all good.

[1] https://www.iro.umontreal.ca/~monnier/oclosure.pdf

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Emanuel Berg]

> https://www.iro.umontreal.ca/~monnier/oclosure.pdf

A better link is

  https://zenodo.org/record/6228797

here we see SEO on the basis of the URL isn't a thing of the
past, at least not for very unfrequent searches.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Closures - do you understand them well?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 02 Mar 2023 21:48:54 +0100
> 
> > https://www.iro.umontreal.ca/~monnier/oclosure.pdf
> 
> A better link is
> 
>   https://zenodo.org/record/6228797

A much better link, since the former gives you 404.

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> Michael Heerdegen wrote:
>
> >> Are we talking lexical let-closures? Why not?
> >
> > Unless I still misunderstand your question: Lisp is
> > a high-level language, not Assembler, so like you don't
> > modify the bytes of byte-code functions by hand, you also
> > don't edit the internal representation of closures.
>
> ??? Assembler?
>
> It seems to be some big misunderstanding here, yes :)
>
> I simply mean this:
>
> (let ((...))
>   (defun () ...) )

Yes, I'm aware that this is what you are thinking about now.  But I
think you diverged a bit from the question I had answered in this
subthread.  I had answered something different.

This answer was about (please go some messages up...) your question
about why we don't use to change function values (destructively) as we
once did with lambda forms before lexical binding existed.

Michael.

Re: Closures - do you understand them well?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> A better link is
>
>   https://zenodo.org/record/6228797
>
> here we see SEO on the basis of the URL isn't a thing of the
> past, at least not for very unfrequent searches.

They are now also described in the manual.

If you look at the Emacs sources you'll see that there are a few quite
simple uses (e.g. keyboard macros).

Note that my intention was not to advertise oclosures to you, I just
wanted to mention them because we were speaking (at least I was
speaking) about modification of functions.


Michael.