match-string debugging problem

match-string debugging problem

[Stephen Berman]

There seems to be something about match-string that I don't
understand.  Here is an example of the kind of code I'm working with:

(defvar mystring1 "+++++ ")
(defvar mystring2 " ~~~~~")
(defun mystring-list ()
  (interactive)
  (with-current-buffer (get-buffer-create "*test*")
    (switch-to-buffer "*test*")
    (dotimes (num 5)
      (insert mystring1 "test" (int-to-string (1+ num)) mystring2 "\n"))
    (goto-char (point-min))
    (let ((mystring-list ()))
      (while (re-search-forward
	      (concat "^" (regexp-quote mystring1) "\\(.+\\)"
		      (regexp-quote mystring2) "$")
	      (point-max) t)
	(setq mystring-list (append (list (match-string 1)) mystring-list)))
      (insert "\n")
      (setq mystring-list (reverse mystring-list))
      (dolist (elt mystring-list)
	(insert elt " ")))))

After evalling this code and typing `M-x mystring-list', buffer *test*
consists of these lines:

+++++ test1 ~~~~~
+++++ test2 ~~~~~
+++++ test3 ~~~~~
+++++ test4 ~~~~~
+++++ test5 ~~~~~
test1 test2 test3 test4 test5 

The last line indicates that match-string correctly matches the
strings that build mystring-list.  But when I step through the code
with edebug, match-string always returns nil and a wrong-type-argument
error is raised at the insert (since nil is not char-or-string-p).
(Edebug isn't the problem: evalling first the regexp search code in
*test* and then (match-string 1) also returns nil.)  Because of this
I'm having a hard time debugging other code that uses match-string.
Can someone explain what's going on?

Steve Berman

Re: match-string debugging problem

[Pascal Bourguignon]

Stephen Berman <Stephen.Berman@gmx.net> writes:

> There seems to be something about match-string that I don't
> understand.  Here is an example of the kind of code I'm working with:
> 
> (defvar mystring1 "+++++ ")
> (defvar mystring2 " ~~~~~")
> (defun mystring-list ()
>   (interactive)
>   (with-current-buffer (get-buffer-create "*test*")
>     (switch-to-buffer "*test*")
>     (dotimes (num 5)
>       (insert mystring1 "test" (int-to-string (1+ num)) mystring2 "\n"))
>     (goto-char (point-min))
>     (let ((mystring-list ()))
>       (while (re-search-forward
> 	      (concat "^" (regexp-quote mystring1) "\\(.+\\)"
> 		      (regexp-quote mystring2) "$")
> 	      (point-max) t)
> 	(setq mystring-list (append (list (match-string 1)) mystring-list)))
>       (insert "\n")
>       (setq mystring-list (reverse mystring-list))
>       (dolist (elt mystring-list)
> 	(insert elt " ")))))
> 
> After evalling this code and typing `M-x mystring-list', buffer *test*
> consists of these lines:
> 
> +++++ test1 ~~~~~
> +++++ test2 ~~~~~
> +++++ test3 ~~~~~
> +++++ test4 ~~~~~
> +++++ test5 ~~~~~
> test1 test2 test3 test4 test5 
> 
> The last line indicates that match-string correctly matches the
> strings that build mystring-list.  But when I step through the code
> with edebug, match-string always returns nil and a wrong-type-argument
> error is raised at the insert (since nil is not char-or-string-p).
> (Edebug isn't the problem: evalling first the regexp search code in
> *test* and then (match-string 1) also returns nil.)  Because of this
> I'm having a hard time debugging other code that uses match-string.
> Can someone explain what's going on?

re-search-forward uses global state (buffer, matched range, etc) as
match-data, to communicate with match-string.  When you're debugging,
this global state is switched or modified.  One could consider it a
bug in the debugger.

See: match-data
     save-match-data   
     save-excursion
     save-buffer

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
Until real software engineering is developed, the next best practice
is to develop with a dynamic system that has extreme late binding in
all aspects. The first system to really do this in an important way
is Lisp. -- Alan Kay

Re: match-string debugging problem

[Stephen Berman]

On 10 Mar 2005 18:06:52 +0100 Pascal Bourguignon <spam@mouse-potato.com> wrote:

> Stephen Berman <Stephen.Berman@gmx.net> writes:
>
[...]
>>     (let ((mystring-list ()))
>>       (while (re-search-forward
>> 	      (concat "^" (regexp-quote mystring1) "\\(.+\\)"
>> 		      (regexp-quote mystring2) "$")
>> 	      (point-max) t)
>> 	(setq mystring-list (append (list (match-string 1)) mystring-list)))
>>       (insert "\n")
>>       (setq mystring-list (reverse mystring-list))
>>       (dolist (elt mystring-list)
>> 	(insert elt " ")))))
>> 
>> After evalling this code and typing `M-x mystring-list', buffer *test*
>> consists of these lines:
>> 
>> +++++ test1 ~~~~~
>> +++++ test2 ~~~~~
>> +++++ test3 ~~~~~
>> +++++ test4 ~~~~~
>> +++++ test5 ~~~~~
>> test1 test2 test3 test4 test5 
>> 
>> The last line indicates that match-string correctly matches the
>> strings that build mystring-list.  But when I step through the code
>> with edebug, match-string always returns nil and a wrong-type-argument
>> error is raised at the insert (since nil is not char-or-string-p).
>> (Edebug isn't the problem: evalling first the regexp search code in
>> *test* and then (match-string 1) also returns nil.)  Because of this
>> I'm having a hard time debugging other code that uses match-string.
>> Can someone explain what's going on?
>
> re-search-forward uses global state (buffer, matched range, etc) as
> match-data, to communicate with match-string.  When you're debugging,
> this global state is switched or modified.  One could consider it a
> bug in the debugger.
>
> See: match-data
>      save-match-data   
>      save-excursion
>      save-buffer

How do you know about this state difference?  I couldn't find any
mention of it in the documentation of the functions you mention, nor
elsewhere in the Elisp manual (CVS version).  I glanced through the
comments in search.c but didn't see anything obviously (to me)
relevant.  If it's defined in the C source code I'd appreciate a
pointer, even though I probably wouldn't be able to understand it at
present.

In any case, what I did find is that I can in fact track match-string
by explicitly entering the ordinary Lisp debugger at that point.  So
putting `(debug)' before `(match-string 1)' in the above code stops
execution there and stepping into the function by pressing `d' returns
the matched string.  So if there's a bug, then it's only in Edebug.  I
couldn't find any relevant discussion in the Edebug section of the
manual.  So at the very least, the Elisp documentation could be more
explicit on this issue.

Steve Berman

Re: match-string debugging problem

[Pascal Bourguignon]

Stephen Berman <Stephen.Berman@gmx.net> writes:

> On 10 Mar 2005 18:06:52 +0100 Pascal Bourguignon <spam@mouse-potato.com> wrote:
> 
> > Stephen Berman <Stephen.Berman@gmx.net> writes:
> >
> [...]
> >>     (let ((mystring-list ()))
> >>       (while (re-search-forward
> >> 	      (concat "^" (regexp-quote mystring1) "\\(.+\\)"
> >> 		      (regexp-quote mystring2) "$")
> >> 	      (point-max) t)
> >> 	(setq mystring-list (append (list (match-string 1)) mystring-list)))
> >>       (insert "\n")
> >>       (setq mystring-list (reverse mystring-list))
> >>       (dolist (elt mystring-list)
> >> 	(insert elt " ")))))
> >> 
> >> After evalling this code and typing `M-x mystring-list', buffer *test*
> >> consists of these lines:
> >> 
> >> +++++ test1 ~~~~~
> >> +++++ test2 ~~~~~
> >> +++++ test3 ~~~~~
> >> +++++ test4 ~~~~~
> >> +++++ test5 ~~~~~
> >> test1 test2 test3 test4 test5 
> >> 
> >> The last line indicates that match-string correctly matches the
> >> strings that build mystring-list.  But when I step through the code
> >> with edebug, match-string always returns nil and a wrong-type-argument
> >> error is raised at the insert (since nil is not char-or-string-p).
> >> (Edebug isn't the problem: evalling first the regexp search code in
> >> *test* and then (match-string 1) also returns nil.)  Because of this
> >> I'm having a hard time debugging other code that uses match-string.
> >> Can someone explain what's going on?
> >
> > re-search-forward uses global state (buffer, matched range, etc) as
> > match-data, to communicate with match-string.  When you're debugging,
> > this global state is switched or modified.  One could consider it a
> > bug in the debugger.
> >
> > See: match-data
> >      save-match-data   
> >      save-excursion
> >      save-buffer
> 
> How do you know about this state difference?  

Infered from the doc of re-search-forward, match-string and save-match-data.

> I couldn't find any
> mention of it in the documentation of the functions you mention, nor
> elsewhere in the Elisp manual (CVS version).  I glanced through the
> comments in search.c but didn't see anything obviously (to me)
> relevant.  If it's defined in the C source code I'd appreciate a
> pointer, even though I probably wouldn't be able to understand it at
> present.
> 
> In any case, what I did find is that I can in fact track match-string
> by explicitly entering the ordinary Lisp debugger at that point.  So
> putting `(debug)' before `(match-string 1)' in the above code stops
> execution there and stepping into the function by pressing `d' returns
> the matched string.  So if there's a bug, then it's only in Edebug.  I
> couldn't find any relevant discussion in the Edebug section of the
> manual.  So at the very least, the Elisp documentation could be more
> explicit on this issue.


My guess is that either:

- Edebug matches regexp itself, and therefore erases the match-data, 
  (a save-match-data would be in order in Edebug), or

- Edebug changes the current buffer, and therefore when it executes 
  (match-string 1), this functions refers the match-data of the Edebug 
  buffer instead of that of the original buffer
  (a save-excursion would be in order in Edebug).

I'd bet for the former, since save-excursion is more commonly used
than save-match-data... (save-excursion calls save-buffer).


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
The rule for today:
Touch my tail, I shred your hand.
New rule tomorrow.

Re: match-string debugging problem

[Stephen Berman]

On 11 Mar 2005 00:40:14 +0100 Pascal Bourguignon <spam@mouse-potato.com> wrote:

> Stephen Berman <Stephen.Berman@gmx.net> writes:
>
>> [...]
>> >>     (let ((mystring-list ()))
>> >>       (while (re-search-forward
>> >> 	      (concat "^" (regexp-quote mystring1) "\\(.+\\)"
>> >> 		      (regexp-quote mystring2) "$")
>> >> 	      (point-max) t)
>> >> 	(setq mystring-list (append (list (match-string 1)) mystring-list)))
>> >>       (insert "\n")
>> >>       (setq mystring-list (reverse mystring-list))
>> >>       (dolist (elt mystring-list)
>> >> 	(insert elt " ")))))
>> 
>> [...]
>> In any case, what I did find is that I can in fact track match-string
>> by explicitly entering the ordinary Lisp debugger at that point.  So
>> putting `(debug)' before `(match-string 1)' in the above code stops
>> execution there and stepping into the function by pressing `d' returns
>> the matched string.  So if there's a bug, then it's only in Edebug.  I
>> couldn't find any relevant discussion in the Edebug section of the
>> manual.  So at the very least, the Elisp documentation could be more
>> explicit on this issue.
>
>
> My guess is that either:
>
> - Edebug matches regexp itself, and therefore erases the match-data, 
>   (a save-match-data would be in order in Edebug), or
>
> - Edebug changes the current buffer, and therefore when it executes 
>   (match-string 1), this functions refers the match-data of the Edebug 
>   buffer instead of that of the original buffer
>   (a save-excursion would be in order in Edebug).
>
> I'd bet for the former, since save-excursion is more commonly used
> than save-match-data... (save-excursion calls save-buffer).

I'm going to file a bug report to see if someone can confirm or refute
your conjecture.  Even if there's no code bug, I think the failure to
mention this behavior is at least a doc bug.

Steve Berman

Re: match-string debugging problem

[Alan Wehmann]

Stephen Berman <Stephen.Berman@gmx.net> writes:

> There seems to be something about match-string that I don't
> understand.  Here is an example of the kind of code I'm working with:
>
> (defvar mystring1 "+++++ ")
> (defvar mystring2 " ~~~~~")
> (defun mystring-list ()
>   (interactive)
>   (with-current-buffer (get-buffer-create "*test*")
>     (switch-to-buffer "*test*")
>     (dotimes (num 5)
>       (insert mystring1 "test" (int-to-string (1+ num)) mystring2 "\n"))
>     (goto-char (point-min))
>     (let ((mystring-list ()))
>       (while (re-search-forward
> 	      (concat "^" (regexp-quote mystring1) "\\(.+\\)"
> 		      (regexp-quote mystring2) "$")
> 	      (point-max) t)
> 	(setq mystring-list (append (list (match-string 1)) mystring-list)))
>       (insert "\n")
>       (setq mystring-list (reverse mystring-list))
>       (dolist (elt mystring-list)
> 	(insert elt " ")))))
>
> After evalling this code and typing `M-x mystring-list', buffer *test*
> consists of these lines:
>
> +++++ test1 ~~~~~
> +++++ test2 ~~~~~
> +++++ test3 ~~~~~
> +++++ test4 ~~~~~
> +++++ test5 ~~~~~
> test1 test2 test3 test4 test5 
>
> The last line indicates that match-string correctly matches the
> strings that build mystring-list.  But when I step through the code
> with edebug, match-string always returns nil and a wrong-type-argument
> error is raised at the insert (since nil is not char-or-string-p).
> (Edebug isn't the problem: evalling first the regexp search code in
> *test* and then (match-string 1) also returns nil.)  Because of this
> I'm having a hard time debugging other code that uses match-string.
> Can someone explain what's going on?
>
> Steve Berman
>
>
>
I tried your example, with 

GNU Emacs 21.2.2 (sparc-sun-solaris2.8, X toolkit) of 2005-02-10 on
gax

and also with

XEmacs 21.4 (patch 15) "Security Through Obscurity" [Lucid]
(sparc-sun-solaris2.8, Mule) of Fri Feb 11 2005 on gax

When I put a breakpoint in Edebug after

(match-string 1)

I get the strings I expect to see.  Also, a breakpoint after 'elt' in

	(insert elt " ")

behaves as I would expect & not as you describe.
-- 
Alan Wehmann
wehmann(removespam)@fnal.gov

Re: match-string debugging problem

[Stephen Berman]

On Sun, 13 Mar 2005 15:52:24 -0600 Alan Wehmann <wehmann@fnal.gov> wrote:

> Stephen Berman <Stephen.Berman@gmx.net> writes:
>
>> There seems to be something about match-string that I don't
>> understand.  Here is an example of the kind of code I'm working with:
>> [...]
>> The last line indicates that match-string correctly matches the
>> strings that build mystring-list.  But when I step through the code
>> with edebug, match-string always returns nil and a wrong-type-argument
>> error is raised at the insert (since nil is not char-or-string-p).
>> [...]
>>
> I tried your example, with 
>
> GNU Emacs 21.2.2 (sparc-sun-solaris2.8, X toolkit) of 2005-02-10 on
> gax
>
> and also with
>
> XEmacs 21.4 (patch 15) "Security Through Obscurity" [Lucid]
> (sparc-sun-solaris2.8, Mule) of Fri Feb 11 2005 on gax
>
> When I put a breakpoint in Edebug after
>
> (match-string 1)
>
> I get the strings I expect to see.  Also, a breakpoint after 'elt' in
>
> 	(insert elt " ")
>
> behaves as I would expect & not as you describe.

Yes, the example was a red herring, because I had neglected to test it
with the default Emacs (using the -q command line option -- I thought
I had done so, but misremembered).  It turns out the bug wasn't in
Emacs but in tabbar.el <http://sourceforge.net/projects/emhacks/>,
which I load from my init-file.  The bug has now been fixed in the
current CVS version of tabbar.el.

Steve Berman