bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Miguel V. S. Frasson]

Programming an Emacs lisp program that uses match-data, debugging pieces
by hand, I realized that managing matchs was a nightmare.  At first I
thought that navigation commands like C-a or C-M-f were messing
match-data (as one could think they use searching).  It could be.  But
for sure, that very handy help line that shows function arguments are
messing match data, making difficult to program Emacs lisp.

How to reproduce:

1) emacs -Q

2) type or paste the following code on *scratch* buffer
   (must be a lisp programming mode):

(string-match "f" "foo")
(match-string 0 "foo")

Normal behavior

3) Go to end of string-match line, evaluate with C-x C-e
   C-n   (or down) so that point goes to end of 2nd line
   *Very quickly* (do not let help for match-string appear), type
   C-b C-f   (or left right)
   and evaluate:   C-x C-e
   I get correct result "f"

Now the bug

4) Go to end of string-match line, evaluate with C-x C-e
   C-n    (getting to end of 2nd line)
   C-b    (or left)
   ...wait minibuffer function help for match-string show...
   C-f    (or right)
   C-x C-e
   I get ERROR 'args-out-of-range "foo" 15 21'

What I observed:

Steps 3) and 4) are (almost) identical except for time spent between C-b
and C-f commands. Error is probably due to a match-data change, probably
caused by minibuffer function help mechanism.

What I expect:

No unnecessary side-effects like change match-data should happen while
simply navigating through code.  Lisp modes should protect searches on
background with save-match-data, because it makes a nightmare to
evaluate code by hand.



In GNU Emacs 25.3.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.29)
 of 2018-04-19 built on lgw01-amd64-039
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description:    Ubuntu 18.04.1 LTS

Configured using:
 'configure --build=x86_64-linux-gnu --prefix=/usr
 '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
 '--infodir=${prefix}/share/info' --sysconfdir=/etc --localstatedir=/var
 --disable-silent-rules '--libdir=${prefix}/lib/x86_64-linux-gnu'
 '--libexecdir=${prefix}/lib/x86_64-linux-gnu' --disable-maintainer-mode
 --disable-dependency-tracking --prefix=/usr --sharedstatedir=/var/lib
 --program-suffix=25 --with-modules --with-x=yes --with-x-toolkit=gtk3
 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs25-0g8uTI/emacs25-25.3~1.gite0284ab=.
-fstack-protector-strong
 -Wformat -Werror=format-security -no-pie' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro
 -no-pie''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES

Important settings:
  value of $LANG: pt_BR.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
View mode: type C-h for help, h for commands, q to quit.
Mark saved where search started
next-line: End of buffer
Mark set
next-line: End of buffer
Quit
"foo"
0 (#o0, #x0, ?\C-@)
"f" [4 times]
Entering debugger...
Quit

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils goto-addr
thingatpt noutline outline easy-mmode view misearch multi-isearch
jka-compr find-func cl-extra help-fns help-mode easymenu cl-loaddefs
pcase cl-lib debug time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 91561 9006)
 (symbols 48 20270 0)
 (miscs 40 67 207)
 (strings 32 15835 4067)
 (string-bytes 1 452038)
 (vectors 16 12415)
 (vector-slots 8 437769 6180)
 (floats 8 173 356)
 (intervals 56 313 13)
 (buffers 976 20))


-- 
Miguel Vinicius Santini Frasson
mvsfrasson@gmail.com

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Philipp Stephani]

Am So., 27. Jan. 2019 um 00:44 Uhr schrieb Miguel V. S. Frasson
<mvsfrasson@gmail.com>:
>
> Programming an Emacs lisp program that uses match-data, debugging pieces
> by hand, I realized that managing matchs was a nightmare.  At first I
> thought that navigation commands like C-a or C-M-f were messing
> match-data (as one could think they use searching).  It could be.  But
> for sure, that very handy help line that shows function arguments are
> messing match data, making difficult to program Emacs lisp.
>
> How to reproduce:
>
> 1) emacs -Q
>
> 2) type or paste the following code on *scratch* buffer
>    (must be a lisp programming mode):
>
> (string-match "f" "foo")
> (match-string 0 "foo")
>
> Normal behavior
>
> 3) Go to end of string-match line, evaluate with C-x C-e
>    C-n   (or down) so that point goes to end of 2nd line
>    *Very quickly* (do not let help for match-string appear), type
>    C-b C-f   (or left right)
>    and evaluate:   C-x C-e
>    I get correct result "f"
>
> Now the bug
>
> 4) Go to end of string-match line, evaluate with C-x C-e
>    C-n    (getting to end of 2nd line)
>    C-b    (or left)
>    ...wait minibuffer function help for match-string show...
>    C-f    (or right)
>    C-x C-e
>    I get ERROR 'args-out-of-range "foo" 15 21'
>
> What I observed:
>
> Steps 3) and 4) are (almost) identical except for time spent between C-b
> and C-f commands. Error is probably due to a match-data change, probably
> caused by minibuffer function help mechanism.
>
> What I expect:
>
> No unnecessary side-effects like change match-data should happen while
> simply navigating through code.  Lisp modes should protect searches on
> background with save-match-data, because it makes a nightmare to
> evaluate code by hand.
>

Any function is allowed to change the match data, see
https://www.gnu.org/software/emacs/manual/html_node/elisp/Match-Data.html:
"Notice that all functions are allowed to overwrite the match data
unless they're explicitly documented not to do so.".
In general you almost always want to immediately bind the match
results to variables, like so:

(when (string-match "f" "foo")
  (let ((match (match-string 0 "foo")))
    ...
    match))

Evaluating the entire 'when' form will then work as intended.

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Stefan Kangas]

tags 34214 + notabug
thanks

Philipp Stephani <p.stephani2@gmail.com> writes:

>> Programming an Emacs lisp program that uses match-data, debugging pieces
>> by hand, I realized that managing matchs was a nightmare.  At first I
>> thought that navigation commands like C-a or C-M-f were messing
>> match-data (as one could think they use searching).  It could be.  But
>> for sure, that very handy help line that shows function arguments are
>> messing match data, making difficult to program Emacs lisp.
[...]
>> What I expect:
>>
>> No unnecessary side-effects like change match-data should happen while
>> simply navigating through code.  Lisp modes should protect searches on
>> background with save-match-data, because it makes a nightmare to
>> evaluate code by hand.
>>
>
> Any function is allowed to change the match data, see
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Match-Data.html:
> "Notice that all functions are allowed to overwrite the match data
> unless they're explicitly documented not to do so.".
> In general you almost always want to immediately bind the match
> results to variables, like so:
>
> (when (string-match "f" "foo")
>   (let ((match (match-string 0 "foo")))
>     ...
>     match))
>
> Evaluating the entire 'when' form will then work as intended.

Agreed, I don't think there's a bug here.  This is just how this works,
and is documented to work.

Any other opinions?

Best regards,
Stefan Kangas

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Stefan Kangas]

[Please use "Reply to all" so that your replies get recorded in the bug
 tracker.]

"Miguel V. S. Frasson" <mvsfrasson@gmail.com> writes:

> Hi.
>
> The "documented" behavior is in Elisp Reference, but not in doc-strings of
> functions that rely on match data. So they are not so easily spotted by
> non-experienced users.
>
> This bug teached me a lesson, but it took me a lot of time to realize how
> volatile match-data is, changed even by a helper mode like eldoc.
>
> IMO it is so easy to avoid interference into user experience in this case,
> adding convenience, just by saving match data inside eldoc...
>
> Should a helper mode "confuse" non-experienced users because it could rely
> on "documented" behavior? If so, why does Emacs have disabled commands, if
> they are also documented?
>
> Best regards
>
> Miguel

Actually, you may have a point regarding eldoc, if it's clobbering match
data just by moving around the buffer.  I guess it would be nice to
avoid that.

Let's see if anyone else has something to say here.

Best regards,
Stefan Kangas

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Miguel V. S. Frasson]

[-- Attachment #1: Type: text/plain, Size: 2416 bytes --]

Hi.

The "documented" behavior is in Elisp Reference, but not in doc-strings of
functions that rely on match data. So they are not so easily spotted by
non-experienced users.

This bug teached me a lesson, because it took me a lot of time to realize
how volatile match-data is, changed even by a helper mode like eldoc.

IMO it is so easy to avoid interference into user experience in this case,
adding convenience, just by saving match data inside eldoc...

Should a helper mode be able to "confuse" non-experienced users because it
could rely on "documented" behavior? If so, why does Emacs have disabled
commands, if they are also documented?

Best regards

Miguel

Em qui., 13 de ago. de 2020 às 08:37, Stefan Kangas <stefan@marxist.se>
escreveu:

> tags 34214 + notabug
> thanks
>
> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> >> Programming an Emacs lisp program that uses match-data, debugging pieces
> >> by hand, I realized that managing matchs was a nightmare.  At first I
> >> thought that navigation commands like C-a or C-M-f were messing
> >> match-data (as one could think they use searching).  It could be.  But
> >> for sure, that very handy help line that shows function arguments are
> >> messing match data, making difficult to program Emacs lisp.
> [...]
> >> What I expect:
> >>
> >> No unnecessary side-effects like change match-data should happen while
> >> simply navigating through code.  Lisp modes should protect searches on
> >> background with save-match-data, because it makes a nightmare to
> >> evaluate code by hand.
> >>
> >
> > Any function is allowed to change the match data, see
> >
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Match-Data.html:
> > "Notice that all functions are allowed to overwrite the match data
> > unless they're explicitly documented not to do so.".
> > In general you almost always want to immediately bind the match
> > results to variables, like so:
> >
> > (when (string-match "f" "foo")
> >   (let ((match (match-string 0 "foo")))
> >     ...
> >     match))
> >
> > Evaluating the entire 'when' form will then work as intended.
>
> Agreed, I don't think there's a bug here.  This is just how this works,
> and is documented to work.
>
> Any other opinions?
>
> Best regards,
> Stefan Kangas
>


-- 
Miguel Vinicius Santini Frasson
mvsfrasson@gmail.com

[-- Attachment #2: Type: text/html, Size: 3481 bytes --]

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Stefan Kangas]

tags 34214 - notabug
thanks

"Miguel V. S. Frasson" <mvsfrasson@gmail.com> writes:

> The "documented" behavior is in Elisp Reference, but not in
> doc-strings of functions that rely on match data. So they are not so
> easily spotted by non-experienced users.

I agree that this could be documented in `match-string'.

> IMO it is so easy to avoid interference into user experience in this
> case, adding convenience, just by saving match data inside eldoc...
>
> Should a helper mode be able to "confuse" non-experienced users
> because it could rely on "documented" behavior? If so, why does Emacs
> have disabled commands, if they are also documented?

I have no strong opinion here.  Maybe you're right.

Anyone else?

bug#34214: 25.3; minibuffer function help in lisp modes changes match-data

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

>> The "documented" behavior is in Elisp Reference, but not in
>> doc-strings of functions that rely on match data. So they are not so
>> easily spotted by non-experienced users.
>
> I agree that this could be documented in `match-string'.

I've now done so in Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no