bug#31586: 27.0.50; `frame-title-format' doesn't save match data

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Philipp]

emacs -Q -eval "(setq frame-title-format '(\"\" (:eval (string-match \".\" \"a\"))))"

Then go to the first character in the scratch buffer (M-<), and run

M-% a RET b RET

The first time you attempt to replace something, Emacs will signal an
error:

perform-replace: Args out of range: #<buffer *scratch*>, 0, 1

Backtrace:

Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
  buffer-substring-no-properties(0 1)
  perform-replace("a" "b" t nil nil nil nil nil nil nil nil)
  query-replace("a" "b" nil nil nil nil nil)
  funcall-interactively(query-replace "a" "b" nil nil nil nil nil)
  call-interactively(query-replace nil nil)
  command-execute(query-replace)

Apparently the `string-match' in `frame-title-format' has overwritten
the match data.  This worked in Emacs 24.5.  It breaks the informal
contract for the match data, namely that any code is free to change it,
and code that relies on the match data staying intact needs to protect
it.


In GNU Emacs 27.0.50 (build 69, x86_64-apple-darwin17.4.0, NS appkit-1561.20 Version 10.13.3 (Build 17D102))
 of 2018-05-22 built on p
Repository revision: 02f303d75f876517b7802f787413cbb418203315
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.3

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --with-modules --without-pop --with-mailutils
 --enable-gcc-warnings=yes --enable-checking
 --enable-check-lisp-object-type 'CFLAGS=-ggdb3 -O0''

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS
JSON

Important settings:
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date elec-pair
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote kqueue cocoa ns
multi-tty make-network-process emacs)

Memory information:
((conses 16 204577 6502)
 (symbols 48 20009 1)
 (miscs 40 56 152)
 (strings 32 28860 1923)
 (string-bytes 1 773241)
 (vectors 16 35283)
 (vector-slots 8 721708 13474)
 (floats 8 51 65)
 (intervals 56 210 0)
 (buffers 992 11))

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Eli Zaretskii]

> From: Philipp <p.stephani2@gmail.com>
> Date: Thu, 24 May 2018 23:55:57 +0200
> 
> emacs -Q -eval "(setq frame-title-format '(\"\" (:eval (string-match \".\" \"a\"))))"
> 
> Then go to the first character in the scratch buffer (M-<), and run
> 
> M-% a RET b RET
> 
> The first time you attempt to replace something, Emacs will signal an
> error:
> 
> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1

And this is an Emacs bug because...?  The :eval expression is yours,
so it's IMO your responsibility to protect it as needed.

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Stefan Monnier]

> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
>
> Backtrace:
>
> Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
>   buffer-substring-no-properties(0 1)
>   perform-replace("a" "b" t nil nil nil nil nil nil nil nil)
>   query-replace("a" "b" nil nil nil nil nil)
>   funcall-interactively(query-replace "a" "b" nil nil nil nil nil)
>   call-interactively(query-replace nil nil)
>   command-execute(query-replace)

FWIW, I think this qualifies as a bug in query-replace: Elisp code
should never presume that the match-data is preserved across something
like sit-for, read-char, or any other function which can run process
filters, redisplay, timers, or contains a yield-point.


        Stefan

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Sat, 26 May 2018 16:58:56 -0400
> Cc: 31586@debbugs.gnu.org
> 
> > perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
> >
> > Backtrace:
> >
> > Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
> >   buffer-substring-no-properties(0 1)
> >   perform-replace("a" "b" t nil nil nil nil nil nil nil nil)
> >   query-replace("a" "b" nil nil nil nil nil)
> >   funcall-interactively(query-replace "a" "b" nil nil nil nil nil)
> >   call-interactively(query-replace nil nil)
> >   command-execute(query-replace)
> 
> FWIW, I think this qualifies as a bug in query-replace: Elisp code
> should never presume that the match-data is preserved across something
> like sit-for, read-char, or any other function which can run process
> filters, redisplay, timers, or contains a yield-point.

Is this practical?  We have any number of hooks, advices, and other
means to make arbitrary Lisp run almost off any function call.  Given
that redisplay can be entered by such Lisp by calling 'redisplay' or
'message' or one of the other functions you mentioned, your suggestion
would mean we need to save-match-data around any call to any
function.  That would make our code very cluttered, indeed.

My POV is that using :eval is intrinsically tricky, and whoever does
that should take the necessary precautions.

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Stefan Monnier]

>> FWIW, I think this qualifies as a bug in query-replace: Elisp code
>> should never presume that the match-data is preserved across something
>> like sit-for, read-char, or any other function which can run process
>> filters, redisplay, timers, or contains a yield-point.
> Is this practical?  We have any number of hooks, advices, and other
> means to make arbitrary Lisp run almost off any function call.  Given
> that redisplay can be entered by such Lisp by calling 'redisplay' or
> 'message' or one of the other functions you mentioned, your suggestion
> would mean we need to save-match-data around any call to any
> function.  That would make our code very cluttered, indeed.

That's how we've lived so far, except that the need for save-match-data
is not around "any" call, but only around "any call except for <...>"
where <...> is the set of "primitive enough" functions.  The main
problem so far is that this set is not formally defined (and also that
the byte-compiler doesn't warn you if you use a function outside of this
set without wrapping with save-match-data), but other than that it works
well in practice, because in 99% there is *very* little code executed
between a regexp match and the use of the match-data.

[ Side question: while `message` does cause a form of redisplay, IIUC it
  doesn't cause a *real* redisplay in the sense that it won't recompute
  mode-lines, frame-titles, nor will it run jit-lock, IOW it won't run
  elisp code.  ]

> My POV is that using :eval is intrinsically tricky, and whoever does
> that should take the necessary precautions.

I think it would be preferable to save the match-data around the whole
redisplay than have each :eval do it.

More to the point: AFAICT in the problem at hand, between the
regexp-match and the call to buffer-substring-no-properties, process
filters can be executed, so it's not just the match data which could be
changed, but the whole buffer's contents, so save-match-data around
the :eval call will just patch over one particular instance of a more
general problem, I think.

This said, having looked at the code this time, the bug is not quite as
clear as I thought: perform-replace does already save&restore the match
data, as evidenced by:

		  (setq key (read-event))
		  ;; Necessary in case something happens during
		  ;; read-event that clobbers the match data.
		  (set-match-data real-match-data)

But it does it in a fairly complex way, so the exact problem is hard
to pinpoint.  If someone can understand what replace-match-data really
does, maybe they can figure out the origin of the problem.


        Stefan

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Drew Adams]

> >> Elisp code
> >> should never presume that the match-data is preserved across something
> >> like sit-for, read-char, or any other function which can run process
> >> filters, redisplay, timers, or contains a yield-point.
> > Is this practical?  We have any number of hooks, advices, and other
> > means to make arbitrary Lisp run almost off any function call.  Given
> > that redisplay can be entered by such Lisp by calling 'redisplay' or
> > 'message' or one of the other functions you mentioned, your suggestion
> > would mean we need to save-match-data around any call to any
> > function.  That would make our code very cluttered, indeed.
> 
> That's how we've lived so far, except that the need for save-match-data
> is not around "any" call, but only around "any call except for <...>"
> where <...> is the set of "primitive enough" functions.  The main
> problem so far is that this set is not formally defined (and also that
> the byte-compiler doesn't warn you if you use a function outside of this
> set without wrapping with save-match-data), but other than that it works
> well in practice, because in 99% there is *very* little code executed
> between a regexp match and the use of the match-data.

Would it make sense to "formalize" this a bit, by having an
explicit such list of the functions (those "primitive enough"
to never, or perhaps hardly ever, need wrapping with `s-m-d')?

Even if such a list were not consulted by any code (and it
could be, presumably, for some control somewhere), it might
at least help developers and users by letting them know what
the story is.

Just a thought.

bug#31586: 27.0.50; `frame-title-format' doesn't save match data

[Stefan Monnier]

> Would it make sense to "formalize" this a bit, by having an
> explicit such list of the functions (those "primitive enough"
> to never, or perhaps hardly ever, need wrapping with `s-m-d')?

Yes!


        Stefan

bug#31586:

[Binbin YE]

[-- Attachment #1: Type: text/plain, Size: 171 bytes --]

Experiencing some similar issue and there is an discussion and findings on

https://github.com/syl20bnr/spacemacs/issues/9700

Do you think it is a related issue?

Cheers

[-- Attachment #2: Type: text/html, Size: 535 bytes --]

bug#31586:

[Eli Zaretskii]

> From: Binbin YE <phantom2501@gmail.com>
> Date: Mon, 30 Jul 2018 13:34:45 +0900
> 
> Experiencing some similar issue and there is an discussion and findings on 
> 
> https://github.com/syl20bnr/spacemacs/issues/9700
> 
> Do you think it is a related issue?

It could be, yes.

bug#33697: 26.1; file-truename messes with match data

[Kévin Le Gouguec]

Hi,

I set frame-title-format so that it displays the basename of the current
project's root folder for the active buffer.  However, my $HOME is under
version control; I do not want this particular "project" to show up in
the frame's title.

Here is the code that I use:

    (require 'subr-x)

    (defun my/project-name ()
      (when-let* ((project (project-current))
                  (root (car (project-roots project))))
        (when (not (file-equal-p root "~"))
          (file-name-nondirectory (string-trim-right root "/")))))

    (setq frame-title-format
          '(:eval
            (let ((project (my/project-name)))
              (concat (when project (format "[%s] " project)) "%b"))))

After adding the above to my .emacs, I started experiencing weird
failures with query-replace.  I managed to reproduce this with emacs -Q:

- evaluate the above
- move to the beginning of *scratch*
- M-% a RET b RET
⇒ perform-replace: Args out of range: #<buffer *scratch*>, 0, 2
  (point has moved after the first match)

I dug a bit and file-equal-p calls file-truename, whose first (cond …)
ends up calling string-match.  Although I did not delve much longer and
could be wrong, I imagine that query-replace finds the first match,
moves point to it, leaves the minibuffer; this causes frame-title-format
to be evaluated; file-equal-p calls file-truename, the global match data
is updated, then perform-replace gets confused about what to do.

Wrapping the (cond …) with (save-match-data …) solves the issue for me
(as does wrapping my/project-name).  I do not know if this is the
preferred way to address this; I do not even know if this is a "bug" (as
in, nobody said file-equal-p does *not* modify match data).


Thank you for your time.



In GNU Emacs 26.1 (build 1, x86_64-suse-linux-gnu, X toolkit, Xaw3d scroll bars)
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
Recent messages:
Loading /home/peniblec/.emacs-custom.el (source)...
Loading diff-hl-flydiff...done
Loading elec-pair...done
Loading page-break-lines...done
Loading icomplete...done
Loading mb-depth...done
Loading /home/peniblec/.emacs-custom.el (source)...done
Global key bindings for ‘rg’ enabled with prefix: C-c s
Loading quail/latin-ltx...done
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --disable-build-details --without-pop --with-mailutils
 --without-hesiod --with-gameuser=:games --with-kerberos
 --with-kerberos5 --with-file-notification=inotify --with-modules
 --enable-autodepend --prefix=/usr --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --localstatedir=/var
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --enable-locallisppath=/usr/share/emacs/26.1/site-lisp:/usr/share/emacs/site-lisp
 --with-x --with-xim --with-sound --with-xpm --with-jpeg --with-tiff
 --with-gif --with-png --with-rsvg --with-dbus --with-xft --without-gpm
 --with-x-toolkit=lucid --with-toolkit-scroll-bars
 --x-includes=/usr/include --x-libraries=/usr/lib64:/usr/share/X11
 --without-libotf --without-m17n-flt --build=x86_64-suse-linux
 'CFLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong
 -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection
 -g -D_GNU_SOURCE -DGDK_DISABLE_DEPRECATION_WARNINGS
 -DGLIB_DISABLE_DEPRECATION_WARNINGS -pipe -Wno-pointer-sign
 -Wno-unused-variable -Wno-unused-label -fno-optimize-sibling-calls
 -fno-PIE -DSYSTEM_PURESIZE_EXTRA=55000 -DSITELOAD_PURESIZE_EXTRA=10000
 ' LDFLAGS=-Wl,-O2'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS
LUCID X11 MODULES THREADS LIBSYSTEMD

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  minibuffer-depth-indicate-mode: t
  icomplete-mode: t
  global-page-break-lines-mode: t
  page-break-lines-mode: t
  electric-pair-mode: t
  diff-hl-flydiff-mode: t
  global-diff-hl-mode: t
  diff-auto-refine-mode: t
  show-paren-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/peniblec/.emacs.d/elpa/soap-client-3.1.5/soap-inspect hides /usr/share/emacs/26.1/lisp/net/soap-inspect
/home/peniblec/.emacs.d/elpa/soap-client-3.1.5/soap-client hides /usr/share/emacs/26.1/lisp/net/soap-client

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml mml-sec epa derived epg gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils vc-git vc-bzr vc-src vc-sccs vc-svn vc-cvs
vc-rcs project delight advice eighters-theme quail cl-extra help-mode rg
rg-ibuffer rg-result wgrep-rg wgrep s rg-header rg-compat subr-x
ibuf-ext ibuffer ibuffer-loaddefs grep compile comint ansi-color ring
edmacro kmacro disp-table mb-depth icomplete page-break-lines elec-pair
diff-hl-flydiff diff diff-hl vc-dir ewoc vc vc-dispatcher diff-mode
easy-mmode cus-start pcase cus-load finder-inf info package easymenu
epg-config url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib paren ispell time-date mule-util
delsel lpr tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 174446 11163)
 (symbols 48 27255 1)
 (miscs 40 122 131)
 (strings 32 52826 2044)
 (string-bytes 1 1447365)
 (vectors 16 26324)
 (vector-slots 8 633140 10178)
 (floats 8 76 334)
 (intervals 56 323 0)
 (buffers 992 12))

bug#33697: 26.1; file-truename messes with match data

[Eli Zaretskii]

merge 33697 31586
thanks

> From: kevin.legouguec@gmail.com (Kévin Le Gouguec)
> Date: Mon, 10 Dec 2018 22:02:09 +0100
> 
> I set frame-title-format so that it displays the basename of the current
> project's root folder for the active buffer.  However, my $HOME is under
> version control; I do not want this particular "project" to show up in
> the frame's title.
> 
> Here is the code that I use:
> 
>     (require 'subr-x)
> 
>     (defun my/project-name ()
>       (when-let* ((project (project-current))
>                   (root (car (project-roots project))))
>         (when (not (file-equal-p root "~"))
>           (file-name-nondirectory (string-trim-right root "/")))))
> 
>     (setq frame-title-format
>           '(:eval
>             (let ((project (my/project-name)))
>               (concat (when project (format "[%s] " project)) "%b"))))
> 
> After adding the above to my .emacs, I started experiencing weird
> failures with query-replace.  I managed to reproduce this with emacs -Q:
> 
> - evaluate the above
> - move to the beginning of *scratch*
> - M-% a RET b RET
> ⇒ perform-replace: Args out of range: #<buffer *scratch*>, 0, 2
>   (point has moved after the first match)

This is bug#31586, please see the discussions there.

bug#33697: 26.1; file-truename messes with match data

[Kévin Le Gouguec]

My bad, I assumed this would be a no-brainer and did not bother to
search the bug database.  I will be more diligent next time.

I would argue that this is different, since my code does not explicitly
call string-match; however bug#31586 seems to consider the bigger
picture, so I guess it makes more sense to focus on that one.

Thank you for pointing me towards this discussion!

bug#33697: 26.1; file-truename messes with match data

[Lars Ingebrigtsen]

Philipp <p.stephani2@gmail.com> writes:

> emacs -Q -eval "(setq frame-title-format '(\"\" (:eval (string-match \".\" \"a\"))))"
>
> Then go to the first character in the scratch buffer (M-<), and run
>
> M-% a RET b RET
>
> The first time you attempt to replace something, Emacs will signal an
> error:
>
> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
>
> Backtrace:
>
> Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
>   buffer-substring-no-properties(0 1)

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

I can reproduce this in Emacs 26.1, but not in Emacs 28.1, so I guess
this has been fixed in the years since it was reported, and I'm closing
this bug report.  If you're still seeing the problem in recent Emacs
versions, please respond to the debbugs address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31586: bug#33697: 26.1; file-truename messes with match data

[Juri Linkov]

>> M-% a RET b RET
>>
>> The first time you attempt to replace something, Emacs will signal an
>> error:
>>
>> perform-replace: Args out of range: #<buffer *scratch*>, 0, 1
>>
>> Backtrace:
>>
>> Debugger entered--Lisp error: (args-out-of-range #<buffer *scratch*> 0 1)
>>   buffer-substring-no-properties(0 1)
>
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
>
> I can reproduce this in Emacs 26.1, but not in Emacs 28.1, so I guess
> this has been fixed in the years since it was reported, and I'm closing
> this bug report.  If you're still seeing the problem in recent Emacs
> versions, please respond to the debbugs address and we'll reopen.

This was fixed in bug#36328.