bug#7227: re-search-forward goes infinite loop with dash inside []

bug#7227: re-search-forward goes infinite loop with dash inside []

[Jari Aalto]

Package: emacs
Version: 23.2+1-4
Severity: normal

Executing the following code causes Emacs to go into infinite loop.
Simply run it behind the last paren with C-x C-e

(re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")
                        ==========

If the dash is taken awy from the "[a-z0-9.-]", this code does not cause
infinite loop:

(re-search-forward "^\\([a-z0-9.]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")

TEST ROWS FOR THE REGEXPS:

row-one 1234 rest of line

row2    1234 rest of line

TEST SETUP

emacs -Q --no-site-file
<copy the re-search-forward lisp statements into buffer>
<copy the test rows into buffer>
<Execute lisp statements>

-- System Information
Debian Release: squeeze/sid
  APT Prefers testing
  APT policy: (990, testing) (500, unstable) (1, experimental)
Architecture: amd64
Kernel: Linux picasso 2.6.32-5-amd64 #1 SMP Fri Sep 17 21:50:19 UTC 2010 x86_64 GNU/Linux
Locale: LANG=en_DK.UTF-8

-- Versions of packages `emacs depends on'.
Depends:
emacs23         23.2+1-4        GNU Emacs is the extensible self-documenting
emacs23-lucid   23.2+1-4        GNU Emacs is the extensible self-documenting
emacs23-nox     23.2+1-4        GNU Emacs is the extensible self-documenting

bug#7227: re-search-forward goes infinite loop with dash inside []

[Andreas Schwab]

Jari Aalto <jari.aalto@cante.net> writes:

> Executing the following code causes Emacs to go into infinite loop.
> Simply run it behind the last paren with C-x C-e
>
> (re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")

Works fine here.

Note that nested repetitions like \([...]+\)+ are bad in any case.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#7227: re-search-forward goes infinite loop with dash inside []

[Andreas Röhler]

Am 16.10.2010 11:16, schrieb Jari Aalto:
> Package: emacs Version: 23.2+1-4 Severity: normal
>
> Executing the following code causes Emacs to go into infinite loop.
> Simply run it behind the last paren with C-x C-e
>
> (re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\)
> +\\([a-z].*\\)") ==========
>
> If the dash is taken awy from the "[a-z0-9.-]", this code does not
> cause infinite loop:
>
> (re-search-forward "^\\([a-z0-9.]+\\)+[ \t]+\\([0-9]+\\)
> +\\([a-z].*\\)")
>
> TEST ROWS FOR THE REGEXPS:
>
> row-one 1234 rest of line
>
> row2    1234 rest of line
>
> TEST SETUP
>
> emacs -Q --no-site-file <copy the re-search-forward lisp statements
> into buffer> <copy the test rows into buffer> <Execute lisp
> statements>
>
> -- System Information Debian Release: squeeze/sid APT Prefers
> testing APT policy: (990, testing) (500, unstable) (1, experimental)
> Architecture: amd64 Kernel: Linux picasso 2.6.32-5-amd64 #1 SMP Fri
> Sep 17 21:50:19 UTC 2010 x86_64 GNU/Linux Locale: LANG=en_DK.UTF-8
>
> -- Versions of packages `emacs depends on'. Depends: emacs23
> 23.2+1-4        GNU Emacs is the extensible self-documenting
> emacs23-lucid   23.2+1-4        GNU Emacs is the extensible
> self-documenting emacs23-nox     23.2+1-4        GNU Emacs is the
> extensible self-documenting
>
>
>
>

Hi,

can't reproduce this with

GNU Emacs 23.1.1 (i586-suse-linux-gnu, GTK+ Version 2.20.1) of 2010-07-05

in scratch buffer I get an error immediatly:

Debugger entered--Lisp error: (search-failed "^\\([a-z0-9.-]+\\)+[
]+\\([0-9]+\\) +\\([a-z].*\\)")
   re-search-forward("^\\([a-z0-9.-]+\\)+[ 	]+\\([0-9]+\\) +\\([a-z].*\\)")
   eval((re-search-forward "^\\([a-z0-9.-]+\\)+[ 	]+\\([0-9]+\\)
+\\([a-z].*\\)"))
   eval-last-sexp-1(nil)
   eval-last-sexp(nil)
   call-interactively(eval-last-sexp nil nil)

Thanks for tiny-tools BTW.

Andreas

--
https://code.launchpad.net/~a-roehler/python-mode/python-mode-components
https://code.launchpad.net/s-x-emacs-werkstatt/

bug#7227: Close bts:gnu

[Jari Aalto]

Reason for close:

Not a bug. In non-matching case, the regexp engine pumps up; thus "Loop
effect" feel.

bug#7227: re-search-forward goes infinite loop with dash inside []

[Jari Aalto]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Jari Aalto <jari.aalto@cante.net> writes:
>
>> Executing the following code causes Emacs to go into infinite loop.
>> Simply run it behind the last paren with C-x C-e
>>
>> (re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")
>
> Works fine here.

Hm. The test case needs adjustment:

;; (1) Run C-x C-e, works:

(re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")
row2    1234 rest of line

;; (2) but C-x C-e below causes a loop

(re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")
------------------------------------------------------------------------
row2    1234 rest of line

> Note that nested repetitions like \([...]+\)+ are bad in any case.

This was only a test. The anchor "^" should help, and I assume the first
'[a-z0-9.-]+' should be greedy, so it shouldn't take that long to
resolve.

Jari

bug#7227: re-search-forward goes infinite loop with dash inside []

[Andreas Schwab]

Jari Aalto <jari.aalto@cante.net> writes:

> Hm. The test case needs adjustment:
>
> ;; (1) Run C-x C-e, works:
>
> (re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")
> row2    1234 rest of line
>
> ;; (2) but C-x C-e below causes a loop
>
> (re-search-forward "^\\([a-z0-9.-]+\\)+[ \t]+\\([0-9]+\\) +\\([a-z].*\\)")
> ------------------------------------------------------------------------
> row2    1234 rest of line
>
>> Note that nested repetitions like \([...]+\)+ are bad in any case.
>
> This was only a test. The anchor "^" should help, and I assume the first
> '[a-z0-9.-]+' should be greedy, so it shouldn't take that long to
> resolve.

Not in the non-matching case.  When the match against the dash line is
tried you get quadratic behaviour due to explosion of the search tree.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#7408: Linux patchutils: Development of the project?

[Jari Aalto]

Glenn Morris <rgm <at> gnu.org>
> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
> Therefore this cannot be a major issue in practice.

In my book it is still a bug, no matter how many years ago this bug were
introduced:

    - To have two different implementations of same function.
    - To not be able to rely on uniform behavior.

Jari

bug#7408: Linux patchutils: Development of the project?

[Lennart Borgman]

On Tue, Nov 16, 2010 at 10:25 PM, Jari Aalto <jari.aalto@cante.net> wrote:
> Glenn Morris <rgm <at> gnu.org>
>> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
>> Therefore this cannot be a major issue in practice.
>
> In my book it is still a bug, no matter how many years ago this bug were
> introduced:
>
>    - To have two different implementations of same function.
>    - To not be able to rely on uniform behavior.

IMO subtle differences are among the worst bugs. It can take a long
time to discover what is wrong and you often guess wrong (especially
if you are not aware of how difficult it can be).

And the subtle difference often shows up in complicated situations.

Maybe the cl version should be renamed?

bug#7408: Linux patchutils: Development of the project?

[Stefan Monnier]

>> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
>> Therefore this cannot be a major issue in practice.
> In my book it is still a bug, no matter how many years ago this bug were
> introduced:

>     - To have two different implementations of same function.
>     - To not be able to rely on uniform behavior.

You look at it the wrong way: the problem is not with dolist, it's that
you use `return' which is a special form that's not defined in standard
Elisp.  After (require 'cl), `return' gets defined and things work as
you expect.


        Stefan

bug#7408: Linux patchutils: Development of the project?

[jari]

On 2010-11-16 17:21, Stefan Monnier wrote:
| >> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
| >> Therefore this cannot be a major issue in practice.
| > In my book it is still a bug, no matter how many years ago this bug were
| > introduced:
| 
| >     a: - To have two different implementations of same function.
| >     b: - To not be able to rely on uniform behavior.
| 
| You look at it the wrong way: the problem is not with dolist, it's that
| you use `return' which is a special form that's not defined in standard
| Elisp.

As I understand it, the fact that there are forms like these nowadays:

   unless
   when
   ...
   dolist

was because people used these constructs from CL so many time that
they were "re-introduced" directly in subr.el in order to prevent
loading CL.

However re-implementation of CL equivalent of 'dolist' did not meet
all the criteria. The 'return' has been integral part of 'dolist'
since the start.

| After (require 'cl), `return' gets defined and things work as you
| expect.

The implementation in subr.el::dolist does not have any support for
'return'.

It is not that 'return' gets defined, but what happens is that loading
CL clobbers that current definition pf 'dolist' by overwiting it with
its own implementation:

     Two different implementation, that are not compatible.

    $  diff -u  1.subr.el  2.cl-macs.el | grep -vE ";|declare"

    --- 1.subr.el	2010-11-17 06:31:53.000000000 +0200
    +++ 2.cl-macs.el	2010-11-17 06:32:31.000000000 +0200
    @@ -1,19 +1,15 @@
     (defmacro dolist (spec &rest body)
       "Loop over a list.
    -Evaluate BODY with VAR bound to each car from LIST, in turn.
    +Evaluate BODY with VAR bound to each `car' from LIST, in turn.
     Then evaluate RESULT to get return value, default nil.

     \(fn (VAR LIST [RESULT]) BODY...)"
    -  (let ((temp '--dolist-tail--))
    -    `(let ((,temp ,(nth 1 spec))
    -	   ,(car spec))
    -       (while ,temp
    -	 (setq ,(car spec) (car ,temp))
    -	 ,@body
    -	 (setq ,temp (cdr ,temp)))
    -       ,@(if (cdr (cdr spec))
    -	     `((setq ,(car spec) nil) ,@(cdr (cdr spec)))))))
    +  (let ((temp (make-symbol "--cl-dolist-temp--")))
>>  +    (list 'block nil
    +	  (list* 'let (list (list temp (nth 1 spec)) (car spec))
    +		 (list* 'while temp (list 'setq (car spec) (list 'car temp))
    +			(append body (list (list 'setq temp
    +						 (list 'cdr temp)))))
    +		 (if (cdr (cdr spec))
    +		     (cons (list 'setq (car spec) nil) (cdr (cdr spec)))
    +		   '(nil))))))

Naturally, loading CL also defines other things. But the fundamental
problem is two incompatible dolist implementations.

A possible solution:

  - Make subr.el::dolist repect return-block (nil-block)
  - have 'return' autoloaded

If user uses:

  - Plain dolist, nothing more is loaded (subr.el is used)
  - "return", it triggers autoloading CL (or parts of it).

Jari

bug#7408: Linux patchutils: Development of the project?

[Glenn Morris]

What does any of this have to do with "Linux patchutils", BTW?

bug#7408: Linux patchutils: Development of the project?

[Stefan Monnier]

> The 'return' has been integral part of 'dolist' since the start.

Who cares?

> | After (require 'cl), `return' gets defined and things work as you
> | expect.

> The implementation in subr.el::dolist does not have any support for
> 'return'.

Again, who cares?  If you want to use `return' you need to require the
package that defines it.

> It is not that 'return' gets defined, but what happens is that loading
> CL clobbers that current definition pf 'dolist' by overwiting it with
> its own implementation:

And?  Why would you care?  Without looking at the source code, can you
tell the difference?

> Naturally, loading CL also defines other things. But the fundamental
> problem is two incompatible dolist implementations.

You still haven't shown any evidence of incompatibility.

Show me a piece of code which would work with CL's dolist (but without
CL's return) and yet doesn't work with subr.el's dolist.


        Stefan

bug#7408: Linux patchutils: Development of the project?

[Štěpán Němec]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> subr.el has had a dolist definition since at least Emacs 21.x; ie 9 years.
>>> Therefore this cannot be a major issue in practice.
>> In my book it is still a bug, no matter how many years ago this bug were
>> introduced:
>
>>     - To have two different implementations of same function.
>>     - To not be able to rely on uniform behavior.
>
> You look at it the wrong way: the problem is not with dolist, it's that
> you use `return' which is a special form that's not defined in standard
> Elisp.  After (require 'cl), `return' gets defined and things work as
> you expect.

While I don't share Jari's complaint in general (i.e., the approach "if
you only want the subr.el dolist, you don't care if it's really CL
dolist you end up calling; if you want the CL dolist, you require 'cl
and are done with it" has served me acceptably well), I've always
wondered what the hell was the person adding another function of the
same name thinking. To have a library that clobbers existing definitions
is a no no even outside Emacs, let alone in the core.

Is the explanation (I'm not familiar with the history) that at the time
cl.el was added there was no dolist in core Emacs, so there was no
perceived need to call it dolist* as in other similar cases (mapcar*,
defun* etc)? (In that case my sincere disdain would go for the person
who introduced dolist into subr.el later without addressing the naming
clash.)

Same with `dotimes'.

  Štěpán

bug#7408: Linux patchutils: Development of the project?

[Stefan Monnier]

> Is the explanation (I'm not familiar with the history) that at the time
> cl.el was added there was no dolist in core Emacs, so there was no
> perceived need to call it dolist* as in other similar cases (mapcar*,
> defun* etc)? (In that case my sincere disdain would go for the person
> who introduced dolist into subr.el later without addressing the naming
> clash.)

AFAIK subr.el's dolist and dotimes are 100% compatible with
CL's definition.  Of course, if you want to call `return' in there,
you'll need to define `return', which is only provided by CL so you need
to (require 'cl).

I'd reconsider it only if someone can show me reasonable code that works
without error both with CL loaded and without CL loaded and yet returns
different results.


        Stefan

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Jari Aalto]

2010-11-17 15:39 Stefan Monnier <monnier@IRO.UMontreal.CA>:
>> Is the explanation (I'm not familiar with the history) that at the time
>> cl.el was added there was no dolist in core Emacs, so there was no
>> perceived need to call it dolist* as in other similar cases (mapcar*,
>> defun* etc)? (In that case my sincere disdain would go for the person
>> who introduced dolist into subr.el later without addressing the naming
>> clash.)
>
> AFAIK subr.el's dolist and dotimes are 100% compatible with
> CL's definition.  Of course, if you want to call `return' in there,
> you'll need to define `return', which is only provided by CL so you need
> to (require 'cl).
>
2010-11-17 15:39 Stefan Monnier <monnier@IRO.UMontreal.CA>:
>> The 'return' has been integral part of 'dolist' since the start.
>
> Who cares?

There happens to be people that care.

> Show me a piece of code which would work with CL's dolist (but without
> CL's return) and yet doesn't work with subr.el's dolist.

As per request:

    - "if you want to call `return' in there, you'll need to define `return"
    - "dolist and dotimes are 100% compatible with CL's definition."

Jari

    $ emacs -Q

    (progn
      (autoload 'return "cl-macs" nil nil 'macro)
      (dolist (elt '(1 2))
        (return elt)))

    Debugger entered--Lisp error: (no-catch --cl-block-nil-- 1)
      cl-block-throw(--cl-block-nil-- 1)
      (return-from nil elt)
      (return elt)
      (while --dolist-tail-- (setq elt (car --dolist-tail--)) (return elt) (setq --dolist-tail-- (cdr --dolist-tail--)))
      (let ((--dolist-tail-- ...) elt) (while --dolist-tail-- (setq elt ...) (return elt) (setq --dolist-tail-- ...)))
      (dolist (elt (quote ...)) (return elt))
      (progn (autoload (quote return) "cl-macs" nil nil (quote macro)) (dolist (elt ...) (return elt)))
      eval((progn (autoload (quote return) "cl-macs" nil nil (quote macro)) (dolist (elt ...) (return elt))))
      eval-last-sexp-1(nil)
      eval-last-sexp(nil)
      call-interactively(eval-last-sexp nil nil)

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Stefan Monnier]

>     $ emacs -Q

>     (progn
>       (autoload 'return "cl-macs" nil nil 'macro)
>       (dolist (elt '(1 2))
>         (return elt)))

This causes cl-macs to be run at an unexpected time.  I.e. it's ruled
out for being a contrived example.  E.g. I'd be *really* surprised if it
were an example you bumped into before this discussion.


        Stefan

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[jari]

On 2010-11-21 00:45, Stefan Monnier wrote:
| >     $ emacs -Q
| 
| >     (progn
| >       (autoload 'return "cl-macs" nil nil 'macro)
| >       (dolist (elt '(1 2))
| >         (return elt)))
| 
| This causes cl-macs to be run at an unexpected time.  I.e. it's ruled
| out for being a contrived example. 

Is the above code not valid? Does it not do what it is supposed to do;
to define `return' when it will be used for the first time?

| E.g. I'd be *really* surprised if it were an example you bumped into
| before this discussion.

Surprise or not, it demonstrates the defiency of claimed:

    - "dolist and dotimes are 100% compatible with CL's definition."

The question is not about "missing return", but differing dolist
implementation in subr.el

Jari

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Eli Zaretskii]

> Date: Sun, 21 Nov 2010 11:08:48 +0200
> From: jari <jari.aalto@cante.net>
> Cc: 7408@debbugs.gnu.org
> 
> On 2010-11-21 00:45, Stefan Monnier wrote:
> | >     $ emacs -Q
> | 
> | >     (progn
> | >       (autoload 'return "cl-macs" nil nil 'macro)
> | >       (dolist (elt '(1 2))
> | >         (return elt)))
> | 
> | This causes cl-macs to be run at an unexpected time.  I.e. it's ruled
> | out for being a contrived example. 
> 
> Is the above code not valid?

Breaking news: you can easily hang yourself with a rope made of valid
ELisp code.

bug#7408: 23.2.1 dolist -- subr.el and cl-macs differ with nil-block return

[Stefan Monnier]

> | >     $ emacs -Q
> | >     (progn
> | >       (autoload 'return "cl-macs" nil nil 'macro)
> | >       (dolist (elt '(1 2))
> | >         (return elt)))
> | This causes cl-macs to be run at an unexpected time.  I.e. it's ruled
> | out for being a contrived example. 
> Is the above code not valid?

Doesn't matter: lots of vlid Elisp code doesn't do what you'd expect.

> Does it not do what it is supposed to do;

Apparently it doesn't for you.

> to define `return' when it will be used for the first time?

The normal way to use CL features like `return' is with (require 'cl)
somewhere at the top-level.  Anything else is poor code that needs to
be improved.

> | E.g. I'd be *really* surprised if it were an example you bumped into
> | before this discussion.
> Surprise or not, it demonstrates the defiency of claimed:
>     - "dolist and dotimes are 100% compatible with CL's definition."

You don't need to teach me this.  I know full well that 100%
compatibility between two pieces of code in Elisp is *never* true,
unless the two are `eq'.


        Stefan

bug#7408: Linux patchutils: Development of the project?

[martin rudalics]

 > ..., I've always
 > wondered what the hell was the person adding another function of the
 > same name thinking. To have a library that clobbers existing definitions
 > is a no no even outside Emacs, let alone in the core.
 >
 > Is the explanation (I'm not familiar with the history) that at the time
 > cl.el was added there was no dolist in core Emacs, so there was no
 > perceived need to call it dolist* as in other similar cases (mapcar*,
 > defun* etc)? (In that case my sincere disdain would go for the person
 > who introduced dolist into subr.el later without addressing the naming
 > clash.)
 >
 > Same with `dotimes'.

Strongly seconded.  For my opinion on this see

http://lists.gnu.org/archive/html/emacs-devel/2007-09/msg01582.html

martin