bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Jari Aalto]

Package: emacs
Version: 23.2+1-4
Severity: serious

The current definition of dolist in subr.el breaks lot of things that rely
on the standard 'dolist' behavior:

    $ emacs -Q --debug-init --batch --eval "(dolist (elt '(1 2)) (return elt))"
    Symbol's function definition is void: return

    $ macs -Q -l cl-macs --debug-init --batch --eval "(dolist (elt '(1 2)) (return elt))"
    <no errors>

Please update subr.el::dolist in respect to 'return'.

Jari

-- System Information
Debian Release: squeeze/sid
  APT Prefers testing
  APT policy: (990, testing) (500, unstable) (1, experimental)
Architecture: amd64
Kernel: Linux picasso 2.6.32-5-amd64 #1 SMP Fri Sep 17 21:50:19 UTC 2010 x86_64 GNU/Linux
Locale: LANG=en_DK.UTF-8

-- Versions of packages `emacs depends on'.
Depends:
emacs23         23.2+1-4        GNU Emacs is the extensible self-documenting 
emacs23-lucid   23.2+1-4        GNU Emacs is the extensible self-documenting 
emacs23-nox     23.2+1-4        GNU Emacs is the extensible self-documenting 

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Stefan Monnier]

> The current definition of dolist in subr.el breaks lot of things that rely
> on the standard 'dolist' behavior:

>     $ emacs -Q --debug-init --batch --eval "(dolist (elt '(1 2)) (return elt))"
>     Symbol's function definition is void: return

By standard, you mean "CL", right?  If you want CL behavior, then do
(require 'cl), and it will then work.


        Stefan

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Lennart Borgman]

On Mon, Nov 15, 2010 at 6:22 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> The current definition of dolist in subr.el breaks lot of things that rely
>> on the standard 'dolist' behavior:
>
>>     $ emacs -Q --debug-init --batch --eval "(dolist (elt '(1 2)) (return elt))"
>>     Symbol's function definition is void: return
>
> By standard, you mean "CL", right?  If you want CL behavior, then do
> (require 'cl), and it will then work.


So this is one of the reasons you should do

   (eval-when-compile (require 'cl))

Or? ;-)

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Stefan Monnier]

>>> The current definition of dolist in subr.el breaks lot of things that rely
>>> on the standard 'dolist' behavior:
>> 
>>>     $ emacs -Q --debug-init --batch --eval "(dolist (elt '(1 2)) (return elt))"
>>>     Symbol's function definition is void: return
>> 
>> By standard, you mean "CL", right?  If you want CL behavior, then do
>> (require 'cl), and it will then work.


> So this is one of the reasons you should do

>    (eval-when-compile (require 'cl))

> Or? ;-)

Don't know about "should", but it can be one of the cases where you
might want to do that, yes.  Of course, you can also change your code to
use catch&throw instead (which is what's used by `return' under the
hood, AFAIK).


        Stefan

bug#7408: closed (Re: bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return)

[jari]

On 2010-11-15 17:33, GNU bug Tracking System wrote:
| Your bug report
| 
| #7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return
| 
| which was filed against the emacs package, has been closed.
| 
| The explanation is attached below, along with your original report.
| If you require more details, please reply to 7408@debbugs.gnu.org.

The 'dolist' has existed long before subr.el::dolist was written. Code
that uses dolist may expect "standard" CL behavior of, because that is
what has been previously available in Emacs.

I do not find it appropriate to close the bug report, because the
problem still exists:

	There are two definitions of dolist, which are not compatible
	in current Emacs

Jari

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Glenn Morris]

> 	There are two definitions of dolist, which are not compatible
> 	in current Emacs

subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
Therefore this cannot be a major issue in practice.

bug#7408: Linux patchutils: Development of the project?

[Jari Aalto]

Glenn Morris <rgm <at> gnu.org>
> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
> Therefore this cannot be a major issue in practice.

In my book it is still a bug, no matter how many years ago this bug were
introduced:

    - To have two different implementations of same function.
    - To not be able to rely on uniform behavior.

Jari

bug#7408: Linux patchutils: Development of the project?

[Lennart Borgman]

On Tue, Nov 16, 2010 at 10:25 PM, Jari Aalto <jari.aalto@cante.net> wrote:
> Glenn Morris <rgm <at> gnu.org>
>> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
>> Therefore this cannot be a major issue in practice.
>
> In my book it is still a bug, no matter how many years ago this bug were
> introduced:
>
>    - To have two different implementations of same function.
>    - To not be able to rely on uniform behavior.

IMO subtle differences are among the worst bugs. It can take a long
time to discover what is wrong and you often guess wrong (especially
if you are not aware of how difficult it can be).

And the subtle difference often shows up in complicated situations.

Maybe the cl version should be renamed?

bug#7408: Linux patchutils: Development of the project?

[Stefan Monnier]

>> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
>> Therefore this cannot be a major issue in practice.
> In my book it is still a bug, no matter how many years ago this bug were
> introduced:

>     - To have two different implementations of same function.
>     - To not be able to rely on uniform behavior.

You look at it the wrong way: the problem is not with dolist, it's that
you use `return' which is a special form that's not defined in standard
Elisp.  After (require 'cl), `return' gets defined and things work as
you expect.


        Stefan

bug#7408: Linux patchutils: Development of the project?

[jari]

On 2010-11-16 17:21, Stefan Monnier wrote:
| >> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
| >> Therefore this cannot be a major issue in practice.
| > In my book it is still a bug, no matter how many years ago this bug were
| > introduced:
| 
| >     a: - To have two different implementations of same function.
| >     b: - To not be able to rely on uniform behavior.
| 
| You look at it the wrong way: the problem is not with dolist, it's that
| you use `return' which is a special form that's not defined in standard
| Elisp.

As I understand it, the fact that there are forms like these nowadays:

   unless
   when
   ...
   dolist

was because people used these constructs from CL so many time that
they were "re-introduced" directly in subr.el in order to prevent
loading CL.

However re-implementation of CL equivalent of 'dolist' did not meet
all the criteria. The 'return' has been integral part of 'dolist'
since the start.

| After (require 'cl), `return' gets defined and things work as you
| expect.

The implementation in subr.el::dolist does not have any support for
'return'.

It is not that 'return' gets defined, but what happens is that loading
CL clobbers that current definition pf 'dolist' by overwiting it with
its own implementation:

     Two different implementation, that are not compatible.

    $  diff -u  1.subr.el  2.cl-macs.el | grep -vE ";|declare"

    --- 1.subr.el	2010-11-17 06:31:53.000000000 +0200
    +++ 2.cl-macs.el	2010-11-17 06:32:31.000000000 +0200
    @@ -1,19 +1,15 @@
     (defmacro dolist (spec &rest body)
       "Loop over a list.
    -Evaluate BODY with VAR bound to each car from LIST, in turn.
    +Evaluate BODY with VAR bound to each `car' from LIST, in turn.
     Then evaluate RESULT to get return value, default nil.

     \(fn (VAR LIST [RESULT]) BODY...)"
    -  (let ((temp '--dolist-tail--))
    -    `(let ((,temp ,(nth 1 spec))
    -	   ,(car spec))
    -       (while ,temp
    -	 (setq ,(car spec) (car ,temp))
    -	 ,@body
    -	 (setq ,temp (cdr ,temp)))
    -       ,@(if (cdr (cdr spec))
    -	     `((setq ,(car spec) nil) ,@(cdr (cdr spec)))))))
    +  (let ((temp (make-symbol "--cl-dolist-temp--")))
>>  +    (list 'block nil
    +	  (list* 'let (list (list temp (nth 1 spec)) (car spec))
    +		 (list* 'while temp (list 'setq (car spec) (list 'car temp))
    +			(append body (list (list 'setq temp
    +						 (list 'cdr temp)))))
    +		 (if (cdr (cdr spec))
    +		     (cons (list 'setq (car spec) nil) (cdr (cdr spec)))
    +		   '(nil))))))

Naturally, loading CL also defines other things. But the fundamental
problem is two incompatible dolist implementations.

A possible solution:

  - Make subr.el::dolist repect return-block (nil-block)
  - have 'return' autoloaded

If user uses:

  - Plain dolist, nothing more is loaded (subr.el is used)
  - "return", it triggers autoloading CL (or parts of it).

Jari

bug#7408: Linux patchutils: Development of the project?

[Glenn Morris]

What does any of this have to do with "Linux patchutils", BTW?

bug#7408: Linux patchutils: Development of the project?

[Štěpán Němec]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
>>> Therefore this cannot be a major issue in practice.
>> In my book it is still a bug, no matter how many years ago this bug were
>> introduced:
>
>>     - To have two different implementations of same function.
>>     - To not be able to rely on uniform behavior.
>
> You look at it the wrong way: the problem is not with dolist, it's that
> you use `return' which is a special form that's not defined in standard
> Elisp.  After (require 'cl), `return' gets defined and things work as
> you expect.

While I don't share Jari's complaint in general (i.e., the approach "if
you only want the subr.el dolist, you don't care if it's really CL
dolist you end up calling; if you want the CL dolist, you require 'cl
and are done with it" has served me acceptably well), I've always
wondered what the hell was the person adding another function of the
same name thinking. To have a library that clobbers existing definitions
is a no no even outside Emacs, let alone in the core.

Is the explanation (I'm not familiar with the history) that at the time
cl.el was added there was no dolist in core Emacs, so there was no
perceived need to call it dolist* as in other similar cases (mapcar*,
defun* etc)? (In that case my sincere disdain would go for the person
who introduced dolist into subr.el later without addressing the naming
clash.)

Same with `dotimes'.

  Štěpán

bug#7408: Linux patchutils: Development of the project?

[Stefan Monnier]

> The 'return' has been integral part of 'dolist' since the start.

Who cares?

> | After (require 'cl), `return' gets defined and things work as you
> | expect.

> The implementation in subr.el::dolist does not have any support for
> 'return'.

Again, who cares?  If you want to use `return' you need to require the
package that defines it.

> It is not that 'return' gets defined, but what happens is that loading
> CL clobbers that current definition pf 'dolist' by overwiting it with
> its own implementation:

And?  Why would you care?  Without looking at the source code, can you
tell the difference?

> Naturally, loading CL also defines other things. But the fundamental
> problem is two incompatible dolist implementations.

You still haven't shown any evidence of incompatibility.

Show me a piece of code which would work with CL's dolist (but without
CL's return) and yet doesn't work with subr.el's dolist.


        Stefan

bug#7408: Linux patchutils: Development of the project?

[Stefan Monnier]

> Is the explanation (I'm not familiar with the history) that at the time
> cl.el was added there was no dolist in core Emacs, so there was no
> perceived need to call it dolist* as in other similar cases (mapcar*,
> defun* etc)? (In that case my sincere disdain would go for the person
> who introduced dolist into subr.el later without addressing the naming
> clash.)

AFAIK subr.el's dolist and dotimes are 100% compatible with
CL's definition.  Of course, if you want to call `return' in there,
you'll need to define `return', which is only provided by CL so you need
to (require 'cl).

I'd reconsider it only if someone can show me reasonable code that works
without error both with CL loaded and without CL loaded and yet returns
different results.


        Stefan

bug#7408: Linux patchutils: Development of the project?

[martin rudalics]

 > ..., I've always
 > wondered what the hell was the person adding another function of the
 > same name thinking. To have a library that clobbers existing definitions
 > is a no no even outside Emacs, let alone in the core.
 >
 > Is the explanation (I'm not familiar with the history) that at the time
 > cl.el was added there was no dolist in core Emacs, so there was no
 > perceived need to call it dolist* as in other similar cases (mapcar*,
 > defun* etc)? (In that case my sincere disdain would go for the person
 > who introduced dolist into subr.el later without addressing the naming
 > clash.)
 >
 > Same with `dotimes'.

Strongly seconded.  For my opinion on this see

http://lists.gnu.org/archive/html/emacs-devel/2007-09/msg01582.html

martin

bug#7408: Linux patchutils: Development of the project?

[MON KEY]

> Show me a piece of code which would work with CL's dolist (but without
> CL's return) and yet doesn't work with subr.el's dolist.

Well, FWIW following is the toplevel/main interface to my elisp
implementation of Steel Bank Common Lisp's `map1' which in turn is
that implementation's high-level driver for its portable Common Lisp
mapping procedures.

Among other things, my port to elisp required adding a cath/throw
inside of a dolist b/c Elisp's `dolist' doesn't have `return'.

Indeed, I was as surprised by the same `dolist' disconnect which Jari
is currently poking at. What I found troubling was the process
required to locate the locus of this disconnect...  In so much as
Elisp code transitions silently between the two versions of dolist it
should do transparently.

BTW - the impetus for this routine was w/re incorporating more cl seq
fun into core but without the keyword overhead...

;;; :COURTESY SBCL :FILE sbcl/src/code/list.lisp
(defun mon-map1 (fun-designator original-arglists accumulate take-car)
  (unless
      (functionp fun-designator)
    (error (concat ":FUNCTION `mon-map1' "
                   "-- arg FUN-DESIGNATOR does not satisfy `functionp'")))
  (let* ((mmp1-arg-lsts (mon-copy-list-mac original-arglists))
         (mmp1-rtn-list (list nil))
         (mmp1-tmp mmp1-rtn-list))
    (do ((mmp1-rslt nil)
         (mmp1-args '() '()))
        ((catch 'is-null ;; :ADDED
           (dolist (mmp1-thrw mmp1-arg-lsts nil)
             (when (null mmp1-thrw) ;; :WAS (return t)))
               (throw 'is-null t))))
         (if accumulate
             (cdr mmp1-rtn-list)
           (car original-arglists)))
      (do ((mmp1-arg-l mmp1-arg-lsts (cdr mmp1-arg-l)))
          ((null mmp1-arg-l))
        (push (if take-car (caar mmp1-arg-l) (car mmp1-arg-l)) mmp1-args)
        (setf (car mmp1-arg-l) (cdar mmp1-arg-l)))
      (setq mmp1-rslt
            (apply fun-designator (nreverse mmp1-args)))
      (case accumulate
        (:nconc (setq mmp1-tmp (last (nconc mmp1-tmp mmp1-rslt))))
        (:list (setcdr mmp1-tmp (list mmp1-rslt))
               (setq mmp1-tmp (cdr mmp1-tmp)))))))