Why are there two dolist?

Why are there two dolist?

[Lennart Borgman]

And shouldn't they say that you do not have to intern/declare VAR?

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 12:14 AM, Lennart
Borgman<lennart.borgman@gmail.com> wrote:
> And shouldn't they say that you do not have to intern/declare VAR?

- There is one definition in subr.el and one in cl-macro.el. (I think
I said this before.)

- And I meant it should say that the variable is let bound inside
dolist, ie there is a (let ((VAR ...) inside the defmacro dolist. That
is not clear to from the doc string. The word "bound" there could
equally well mean that (setq VAR ...) was used.

Re: Why are there two dolist?

[David Kastrup]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Wed, Aug 12, 2009 at 12:14 AM, Lennart
> Borgman<lennart.borgman@gmail.com> wrote:
>> And shouldn't they say that you do not have to intern/declare VAR?
>
> - There is one definition in subr.el and one in cl-macro.el. (I think
> I said this before.)
>
> - And I meant it should say that the variable is let bound inside
> dolist, ie there is a (let ((VAR ...) inside the defmacro dolist. That
> is not clear to from the doc string. The word "bound" there could
> equally well mean that (setq VAR ...) was used.

No.  Binding and setting a variable are fundamentally different things
and not interchangeable.  While there is a small bit of overlap in
current Emacs (because of its dynamic binding implementation), with
regard to language idioms the concepts are clearly separate, and with a
lexically binding Lisp variant (which we might get some day) the
semantics are even more separated.

-- 
David Kastrup

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 10:45 AM, David Kastrup<dak@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Wed, Aug 12, 2009 at 12:14 AM, Lennart
>> Borgman<lennart.borgman@gmail.com> wrote:
>>> And shouldn't they say that you do not have to intern/declare VAR?
>>
>> - There is one definition in subr.el and one in cl-macro.el. (I think
>> I said this before.)
>>
>> - And I meant it should say that the variable is let bound inside
>> dolist, ie there is a (let ((VAR ...) inside the defmacro dolist. That
>> is not clear to from the doc string. The word "bound" there could
>> equally well mean that (setq VAR ...) was used.
>
> No.  Binding and setting a variable are fundamentally different things
> and not interchangeable.  While there is a small bit of overlap in
> current Emacs (because of its dynamic binding implementation), with
> regard to language idioms the concepts are clearly separate, and with a
> lexically binding Lisp variant (which we might get some day) the
> semantics are even more separated.


Ok, but it not pretty obvious when you got for example `boundp'. So
even if the concepts are clearly separate then the use of the names
does not seem to be that.

And that was actually what I asked about.

Hm, there is really no easy way to be correct if you have taken some
steps aside.

Re: Why are there two dolist?

[Miles Bader]

You mean the "subr.el", and "cl-macs.el" versions?

The latter is an extended version of the former, supporting extra stuff
to be (kinda) CL compatible (e.g., an implicit block around the body).

Most people will never see the cl-macs version; those who _should_ see
it will, because they'll have done (require 'cl) first; since the
cl-macs version supports the basic subr.el usage, it's OK if some code
that doesn't need the extra features ends up se

> And shouldn't they say that you do not have to intern/declare VAR?

It says that it binds VAR (like let), which implies that you don't need
to declare it.  I don't know why it should say anything about interning,
as that's not a concept most code needs to care about.

-Miles

-- 
Egotist, n. A person of low taste, more interested in himself than in me.

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 12:37 AM, Miles Bader<miles@gnu.org> wrote:
> You mean the "subr.el", and "cl-macs.el" versions?

Hi Miles,
Thanks for replying.

Yes.


> The latter is an extended version of the former, supporting extra stuff
> to be (kinda) CL compatible (e.g., an implicit block around the body).
>
> Most people will never see the cl-macs version; those who _should_ see
> it will, because they'll have done (require 'cl) first; since the
> cl-macs version supports the basic subr.el usage, it's OK if some code
> that doesn't need the extra features ends up se


Does not this open to subtle bugs? Would it not be better to move the
cl-macs.el version to subr.el?


>> And shouldn't they say that you do not have to intern/declare VAR?
>
> It says that it binds VAR (like let), which implies that you don't need
> to declare it.  I don't know why it should say anything about interning,
> as that's not a concept most code needs to care about.


Yes, I can see now that it "implies" that, but... - you have to speak
lisp to understand it, or? (Saying "intern" was a bad idea by me.)

There is probably something I did not notice in the elisp manual about
the word "bound"? At least I misunderstood/forgot this.


> -Miles
>
> --
> Egotist, n. A person of low taste, more interested in himself than in me.
>

Re: Why are there two dolist?

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
>> Most people will never see the cl-macs version; those who _should_ see
>> it will, because they'll have done (require 'cl) first; since the
>> cl-macs version supports the basic subr.el usage, it's OK if some code
>> that doesn't need the extra features ends up se
>
> Does not this open to subtle bugs?

I think it's such a simple macro (at least the basic subr.el version is) that
this isn't such a big problem.

> Would it not be better to move the cl-macs.el version to subr.el?

I haven't looked at the code (recently), but because elisp doesn't
natively support lexical blocks, I suspect the cl-macs version either
(1) adds unnecessary runtime overhead, or (2) drags in lots of big and
crufty compile-time analysis code (cl.el has lots of this) to optimize
away the block when not used.

-miles

-- 
Religion, n. A daughter of Hope and Fear, explaining to Ignorance the nature
of the Unknowable.

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 12:51 AM, Miles Bader<miles@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>>> Most people will never see the cl-macs version; those who _should_ see
>>> it will, because they'll have done (require 'cl) first; since the
>>> cl-macs version supports the basic subr.el usage, it's OK if some code
>>> that doesn't need the extra features ends up se
>>
>> Does not this open to subtle bugs?
>
> I think it's such a simple macro (at least the basic subr.el version is) that
> this isn't such a big problem.
>
>> Would it not be better to move the cl-macs.el version to subr.el?
>
> I haven't looked at the code (recently), but because elisp doesn't
> natively support lexical blocks, I suspect the cl-macs version either
> (1) adds unnecessary runtime overhead, or (2) drags in lots of big and
> crufty compile-time analysis code (cl.el has lots of this) to optimize
> away the block when not used.


Yes, but I wonder whether it does something useful at all. Someone who
understands lisp better could maybe tell.

Maybe the CL version should go away instead? It creates a lexically
scoped block with a name nil. The explanation in (defmacro block ...)
does however not say what is lexically scoped so I have no clue. (And
I do not know if return-from mentioned there can work with a nil
name.)

However if if means that one dolist version makes VAR lexically scoped
and the other does not, then the difference is big. And one of them
should leave.


> -miles
>
> --
> Religion, n. A daughter of Hope and Fear, explaining to Ignorance the nature
> of the Unknowable.
>

Re: Why are there two dolist?

[Harald Hanche-Olsen]

+ Lennart Borgman <lennart.borgman@gmail.com>:

> On Wed, Aug 12, 2009 at 12:51 AM, Miles Bader<miles@gnu.org> wrote:
> > I haven't looked at the code (recently), but because elisp doesn't
> > natively support lexical blocks, I suspect the cl-macs version either
> > (1) adds unnecessary runtime overhead, or (2) drags in lots of big and
> > crufty compile-time analysis code (cl.el has lots of this) to optimize
> > away the block when not used.

Actually, the cl-macs version looks pretty simple to me.

> Maybe the CL version should go away instead? It creates a lexically
> scoped block with a name nil. The explanation in (defmacro block ...)
> does however not say what is lexically scoped so I have no clue.

AFAICT, the only thing lexically scoped is the name of the block.
In other words, (return-from NAME) will return from the innermost
block named NAME that lexically contains the return-from form. I think
this is pretty well explained in the block docstring?

> (And I do not know if return-from mentioned there can work with a
> nil name.)

(return-from nil) is perfectly good CL, so it ought to work.

> However if if means that one dolist version makes VAR lexically scoped
> and the other does not, then the difference is big.

But it doesn't mean that, as a simple experiment bears out. I think
the only difference between the two is the block, and that will not
affect any elisp code that is unaware of block and return-from.

- Harald

Re: Why are there two dolist?

[Miles Bader]

Harald Hanche-Olsen <hanche@math.ntnu.no> writes:
>> > I haven't looked at the code (recently), but because elisp doesn't
>> > natively support lexical blocks, I suspect the cl-macs version either
>> > (1) adds unnecessary runtime overhead, or (2) drags in lots of big and
>> > crufty compile-time analysis code (cl.el has lots of this) to optimize
>> > away the block when not used.
>
> Actually, the cl-macs version looks pretty simple to me.

Right, I checked -- it actually just incurs run-time overhead... :/

Anyway, the two-dolists are not the prettiest thing in the world, but
they work fine, and there's a reason for having them.

-miles

-- 
Do not taunt Happy Fun Ball.

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 1:34 AM, Miles Bader<miles@gnu.org> wrote:
> Harald Hanche-Olsen <hanche@math.ntnu.no> writes:
>>> > I haven't looked at the code (recently), but because elisp doesn't
>>> > natively support lexical blocks, I suspect the cl-macs version either
>>> > (1) adds unnecessary runtime overhead, or (2) drags in lots of big and
>>> > crufty compile-time analysis code (cl.el has lots of this) to optimize
>>> > away the block when not used.
>>
>> Actually, the cl-macs version looks pretty simple to me.
>
> Right, I checked -- it actually just incurs run-time overhead... :/
>
> Anyway, the two-dolists are not the prettiest thing in the world, but
> they work fine, and there's a reason for having them.


How about a notion in the doc string of them on the differences? And
how to get the correct version during compilation? (If that is
possible...)

Re: Why are there two dolist?

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
>> Anyway, the two-dolists are not the prettiest thing in the world, but
>> they work fine, and there's a reason for having them.
>
> And how to get the correct version during compilation? (If that is
> possible...)

If you depend on cl.el extensions, then use (eval-when-compile (require 'cl)).

Otherwise, don't worry about it.

-Miles

-- 
"Don't just question authority,
Don't forget to question me."
-- Jello Biafra

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 1:30 AM, Harald Hanche-Olsen<hanche@math.ntnu.no> wrote:

>> Maybe the CL version should go away instead? It creates a lexically
>> scoped block with a name nil. The explanation in (defmacro block ...)
>> does however not say what is lexically scoped so I have no clue.
>
> AFAICT, the only thing lexically scoped is the name of the block.
> In other words, (return-from NAME) will return from the innermost
> block named NAME that lexically contains the return-from form. I think
> this is pretty well explained in the block docstring?


Thanks, but I do not understand all you wrote here. Could (return-from
NAME) be inside a function in dolist or does it have to be just inside
dolist so to say?


>> (And I do not know if return-from mentioned there can work with a
>> nil name.)
>
> (return-from nil) is perfectly good CL, so it ought to work.


Thanks.


>> However if if means that one dolist version makes VAR lexically scoped
>> and the other does not, then the difference is big.
>
> But it doesn't mean that, as a simple experiment bears out. I think
> the only difference between the two is the block, and that will not
> affect any elisp code that is unaware of block and return-from.


If I understand you correctly the only difference is if (return-from
nil) goes to dolist or maybe outside of that?

But that is of course a big difference too.


> - Harald
>
>
>

Re: Why are there two dolist?

[Harald Hanche-Olsen]

+ Lennart Borgman <lennart.borgman@gmail.com>:

> Thanks, but I do not understand all you wrote here. Could (return-from
> NAME) be inside a function in dolist or does it have to be just inside
> dolist so to say?

Hmm. I was writing some code to demonstrate my point, but found to my
surprise that the following snippet

(block nil
  (flet ((foo (x) (if (eq x 'b)
		      (return-from nil 'foo-done))))
    (list 'from-dolist
	  (dolist (var '(a b c) 'dolist-done)
	    (foo var)))))

which returns foo-done in a proper CL, returns (from-dolist foo-done)
in elisp with the cl package loaded. This seems wrong.

In CL, flet declares a lexically local function binding. Since the
declaration of foo here is lexically outside the dolist, the
return-from inside the body of foo should not see the implicit block
named nil in the dolist, and that is what happens in CL. In elisp,
however, that does not work as I expected. I'm confused.

- Harald

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 2:27 AM, Harald Hanche-Olsen<hanche@math.ntnu.no> wrote:
> + Lennart Borgman <lennart.borgman@gmail.com>:
>
>> Thanks, but I do not understand all you wrote here. Could (return-from
>> NAME) be inside a function in dolist or does it have to be just inside
>> dolist so to say?
>
> Hmm. I was writing some code to demonstrate my point, but found to my
> surprise that the following snippet
>
> (block nil
>  (flet ((foo (x) (if (eq x 'b)
>                      (return-from nil 'foo-done))))
>    (list 'from-dolist
>          (dolist (var '(a b c) 'dolist-done)
>            (foo var)))))
>
> which returns foo-done in a proper CL, returns (from-dolist foo-done)
> in elisp with the cl package loaded. This seems wrong.
>
> In CL, flet declares a lexically local function binding. Since the
> declaration of foo here is lexically outside the dolist, the
> return-from inside the body of foo should not see the implicit block
> named nil in the dolist, and that is what happens in CL. In elisp,
> however, that does not work as I expected. I'm confused.


I do not understand your code above, but is not the conclusion that
the cl-macs.el version does not do what it is supposed to do? What
value does it have then? (Perhaps it can be fixed, I don't know.)

Re: Why are there two dolist?

[Miles Bader]

Lennart Borgman <lennart.borgman@gmail.com> writes:
> I do not understand your code above, but is not the conclusion that
> the cl-macs.el version does not do what it is supposed to do? What
> value does it have then? (Perhaps it can be fixed, I don't know.)

It need not work perfectly to have value.

I don't know whether CL return-from is defined to work across
function-call boundaries, but the vast majority of uses are far simpler,
and the cl.el version should work fine in those cases.

[cl.el is not anywhere close to being truly CL-compatible in general]

-Miles

-- 
Opportunity, n. A favorable occasion for grasping a disappointment.

Re: Why are there two dolist?

[Lennart Borgman]

On Wed, Aug 12, 2009 at 2:54 AM, Miles Bader<miles@gnu.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>> I do not understand your code above, but is not the conclusion that
>> the cl-macs.el version does not do what it is supposed to do? What
>> value does it have then? (Perhaps it can be fixed, I don't know.)
>
> It need not work perfectly to have value.


That is very true. But there is more reasons to wonder what positive
values it have since it seems not to do what it is supposed to to, at
least not here.

It has both negative and positive values I suppose. But it is worth
noting them. Putting a note on them so this is not forgotten. The
example Harald gave should be a good note to put there.

Maybe nothing more has to be done, but I think this would be good.
Otherwise we are perhaps just wasting time.

Re: Why are there two dolist?

[Harald Hanche-Olsen]

+ Lennart Borgman <lennart.borgman@gmail.com>:

> I do not understand your code above, but is not the conclusion that
> the cl-macs.el version does not do what it is supposed to do?

Yeah, at least as far as the lexically-scoped claim goes.

> What value does it have then?

At least in simpler usage, it does provide a block that return-from
can use for a non-local exit, so it has some value.

Actually, the bug seems to be in block, not in dolist. My example was
perhaps overly complex. Here is a simpler example, using only two
nested blocks, both named b:

(block b
  (flet ((f (x) (return-from b x)))
    (list 'inner-block-returned (block b (f 42)))))

In common lisp, the innermost call (f 42) executes a return-from that
is lexically inside the outer block but not inside the inner one, so
the value 42 is returned from the outer block. In elisp, on the other
hand, the return-from inside foo causes the inner block to return 42,
so the flet, and hence the outer block, returns instead the list
(inner-block-returned 42).

This seems to me to flatly contradict what is said about lexical
scoping in the docstring of block.

Oh, but wait: Looking at the code for block, I begin to suspect that
byte compilation might make a difference. And indeed it does:

(defun foo ()
  (block b
    (flet ((f (x) (return-from b x)))
      (list 'inner-block-returned (block b (f 'bar))))))

(foo) ; => (inner-block-returned bar)
(byte-compile 'foo)
(foo) ; => bar

So byte-compiling a block produces the expected result.

Oh well, I guess this is just one more wart on the cl package. (It has
a few, but I find it useful anyhow.) I still think that inadvertently
using the cl version of dolist cannot possibly harm any elisp code
that does not otherwise rely on cl.

- Harald