bug#31586: 27.0.50; `frame-title-format' doesn't save match data

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Philipp]

emacs -Q -eval "(setq frame-title-format '(\"\" (:eval (string-match \".\" \"a\"))))"

Then go to the first character in the scratch buffer (M-<), and run

M-% a RET b RET

The first time you attempt to replace something, Emacs will signal an
error:

perform-replace: Args out of range: #<buffer *scratch*>, 0, 1

Backtrace:

Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
  buffer-substring-no-properties(0 1)
  perform-replace("a" "b" t nil nil nil nil nil nil nil nil)
  query-replace("a" "b" nil nil nil nil nil)
  funcall-interactively(query-replace "a" "b" nil nil nil nil nil)
  call-interactively(query-replace nil nil)
  command-execute(query-replace)

Apparently the `string-match' in `frame-title-format' has overwritten
the match data.  This worked in Emacs 24.5.  It breaks the informal
contract for the match data, namely that any code is free to change it,
and code that relies on the match data staying intact needs to protect
it.


In GNU Emacs 27.0.50 (build 69, x86_64-apple-darwin17.4.0, NS appkit-1561.20 Version 10.13.3 (Build 17D102))
 of 2018-05-22 built on p
Repository revision: 02f303d75f876517b7802f787413cbb418203315
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.3

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --with-modules --without-pop --with-mailutils
 --enable-gcc-warnings=yes --enable-checking
 --enable-check-lisp-object-type 'CFLAGS=-ggdb3 -O0''

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS
JSON

Important settings:
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date elec-pair
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote kqueue cocoa ns
multi-tty make-network-process emacs)

Memory information:
((conses 16 204577 6502)
 (symbols 48 20009 1)
 (miscs 40 56 152)
 (strings 32 28860 1923)
 (string-bytes 1 773241)
 (vectors 16 35283)
 (vector-slots 8 721708 13474)
 (floats 8 51 65)
 (intervals 56 210 0)
 (buffers 992 11))

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Eli Zaretskii]

> From: Philipp <p.stephani2@gmail.com>
> Date: Thu, 24 May 2018 23:55:57 +0200
> 
> emacs -Q -eval "(setq frame-title-format '(\"\" (:eval (string-match \".\" \"a\"))))"
> 
> Then go to the first character in the scratch buffer (M-<), and run
> 
> M-% a RET b RET
> 
> The first time you attempt to replace something, Emacs will signal an
> error:
> 
> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1

And this is an Emacs bug because...?  The :eval expression is yours,
so it's IMO your responsibility to protect it as needed.

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Stefan Monnier]

> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
>
> Backtrace:
>
> Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
>   buffer-substring-no-properties(0 1)
>   perform-replace("a" "b" t nil nil nil nil nil nil nil nil)
>   query-replace("a" "b" nil nil nil nil nil)
>   funcall-interactively(query-replace "a" "b" nil nil nil nil nil)
>   call-interactively(query-replace nil nil)
>   command-execute(query-replace)

FWIW, I think this qualifies as a bug in query-replace: Elisp code
should never presume that the match-data is preserved across something
like sit-for, read-char, or any other function which can run process
filters, redisplay, timers, or contains a yield-point.


        Stefan

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Sat, 26 May 2018 16:58:56 -0400
> Cc: 31586@debbugs.gnu.org
> 
> > perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
> >
> > Backtrace:
> >
> > Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
> >   buffer-substring-no-properties(0 1)
> >   perform-replace("a" "b" t nil nil nil nil nil nil nil nil)
> >   query-replace("a" "b" nil nil nil nil nil)
> >   funcall-interactively(query-replace "a" "b" nil nil nil nil nil)
> >   call-interactively(query-replace nil nil)
> >   command-execute(query-replace)
> 
> FWIW, I think this qualifies as a bug in query-replace: Elisp code
> should never presume that the match-data is preserved across something
> like sit-for, read-char, or any other function which can run process
> filters, redisplay, timers, or contains a yield-point.

Is this practical?  We have any number of hooks, advices, and other
means to make arbitrary Lisp run almost off any function call.  Given
that redisplay can be entered by such Lisp by calling 'redisplay' or
'message' or one of the other functions you mentioned, your suggestion
would mean we need to save-match-data around any call to any
function.  That would make our code very cluttered, indeed.

My POV is that using :eval is intrinsically tricky, and whoever does
that should take the necessary precautions.

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Stefan Monnier]

>> FWIW, I think this qualifies as a bug in query-replace: Elisp code
>> should never presume that the match-data is preserved across something
>> like sit-for, read-char, or any other function which can run process
>> filters, redisplay, timers, or contains a yield-point.
> Is this practical?  We have any number of hooks, advices, and other
> means to make arbitrary Lisp run almost off any function call.  Given
> that redisplay can be entered by such Lisp by calling 'redisplay' or
> 'message' or one of the other functions you mentioned, your suggestion
> would mean we need to save-match-data around any call to any
> function.  That would make our code very cluttered, indeed.

That's how we've lived so far, except that the need for save-match-data
is not around "any" call, but only around "any call except for <...>"
where <...> is the set of "primitive enough" functions.  The main
problem so far is that this set is not formally defined (and also that
the byte-compiler doesn't warn you if you use a function outside of this
set without wrapping with save-match-data), but other than that it works
well in practice, because in 99% there is *very* little code executed
between a regexp match and the use of the match-data.

[ Side question: while `message` does cause a form of redisplay, IIUC it
  doesn't cause a *real* redisplay in the sense that it won't recompute
  mode-lines, frame-titles, nor will it run jit-lock, IOW it won't run
  elisp code.  ]

> My POV is that using :eval is intrinsically tricky, and whoever does
> that should take the necessary precautions.

I think it would be preferable to save the match-data around the whole
redisplay than have each :eval do it.

More to the point: AFAICT in the problem at hand, between the
regexp-match and the call to buffer-substring-no-properties, process
filters can be executed, so it's not just the match data which could be
changed, but the whole buffer's contents, so save-match-data around
the :eval call will just patch over one particular instance of a more
general problem, I think.

This said, having looked at the code this time, the bug is not quite as
clear as I thought: perform-replace does already save&restore the match
data, as evidenced by:

		  (setq key (read-event))
		  ;; Necessary in case something happens during
		  ;; read-event that clobbers the match data.
		  (set-match-data real-match-data)

But it does it in a fairly complex way, so the exact problem is hard
to pinpoint.  If someone can understand what replace-match-data really
does, maybe they can figure out the origin of the problem.


        Stefan

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Drew Adams]

> >> Elisp code
> >> should never presume that the match-data is preserved across something
> >> like sit-for, read-char, or any other function which can run process
> >> filters, redisplay, timers, or contains a yield-point.
> > Is this practical?  We have any number of hooks, advices, and other
> > means to make arbitrary Lisp run almost off any function call.  Given
> > that redisplay can be entered by such Lisp by calling 'redisplay' or
> > 'message' or one of the other functions you mentioned, your suggestion
> > would mean we need to save-match-data around any call to any
> > function.  That would make our code very cluttered, indeed.
> 
> That's how we've lived so far, except that the need for save-match-data
> is not around "any" call, but only around "any call except for <...>"
> where <...> is the set of "primitive enough" functions.  The main
> problem so far is that this set is not formally defined (and also that
> the byte-compiler doesn't warn you if you use a function outside of this
> set without wrapping with save-match-data), but other than that it works
> well in practice, because in 99% there is *very* little code executed
> between a regexp match and the use of the match-data.

Would it make sense to "formalize" this a bit, by having an
explicit such list of the functions (those "primitive enough"
to never, or perhaps hardly ever, need wrapping with `s-m-d')?

Even if such a list were not consulted by any code (and it
could be, presumably, for some control somewhere), it might
at least help developers and users by letting them know what
the story is.

Just a thought.

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Stefan Monnier]

> Would it make sense to "formalize" this a bit, by having an
> explicit such list of the functions (those "primitive enough"
> to never, or perhaps hardly ever, need wrapping with `s-m-d')?

Yes!


        Stefan

bug#31586:

[Binbin YE]

[-- Attachment #1: Type: text/plain, Size: 171 bytes --]

Experiencing some similar issue and there is an discussion and findings on

https://github.com/syl20bnr/spacemacs/issues/9700

Do you think it is a related issue?

Cheers

[-- Attachment #2: Type: text/html, Size: 535 bytes --]

bug#31586:

[Eli Zaretskii]

> From: Binbin YE <phantom2501@gmail.com>
> Date: Mon, 30 Jul 2018 13:34:45 +0900
> 
> Experiencing some similar issue and there is an discussion and findings on 
> 
> https://github.com/syl20bnr/spacemacs/issues/9700
> 
> Do you think it is a related issue?

It could be, yes.

bug#33697: 26.1; file-truename messes with match data

[Lars Ingebrigtsen]

Philipp <p.stephani2@gmail.com> writes:

> emacs -Q -eval "(setq frame-title-format '(\"\" (:eval (string-match \".\" \"a\"))))"
>
> Then go to the first character in the scratch buffer (M-<), and run
>
> M-% a RET b RET
>
> The first time you attempt to replace something, Emacs will signal an
> error:
>
> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
>
> Backtrace:
>
> Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
>   buffer-substring-no-properties(0 1)

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

I can reproduce this in Emacs 26.1, but not in Emacs 28.1, so I guess
this has been fixed in the years since it was reported, and I'm closing
this bug report.  If you're still seeing the problem in recent Emacs
versions, please respond to the debbugs address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31586: bug#33697: 26.1; file-truename messes with match data

[Juri Linkov]

>> M-% a RET b RET
>>
>> The first time you attempt to replace something, Emacs will signal an
>> error:
>>
>> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
>>
>> Backtrace:
>>
>> Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
>>   buffer-substring-no-properties(0 1)
>
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
>
> I can reproduce this in Emacs 26.1, but not in Emacs 28.1, so I guess
> this has been fixed in the years since it was reported, and I'm closing
> this bug report.  If you're still seeing the problem in recent Emacs
> versions, please respond to the debbugs address and we'll reopen.

This was fixed in bug#36328.