bug#65347: 29.1; Underscore in query replace prevents case-matching

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Spencer Baugh]

1. emacs -Q
2. (insert "Foo_bar")
3. M-% foo RET quux RET
4. Observe the buffer contains Quux_bar
5. Undo, so the buffer contains Foo_bar again
6. M-% foo_bar RET quux RET
7. Observe the buffer contains quux

Expected behavior:
7. Observe the buffer contains Quux

It seems to me that this would match the docstring better:

  Replacement transfers the case pattern of the old text to the
  new text, if both ‘case-fold-search’ and ‘case-replace’ are
  non-nil and FROM-STRING has no uppercase letters.


In GNU Emacs 29.1 (build 4, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.15.12, Xaw scroll bars) of 2023-08-08 built on

Repository revision: ea63217c1c85f1ca4f1f22b9c4781167de6cf00d
Repository branch: my-emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Rocky Linux 8.8 (Green Obsidian)

Configured using:
 'configure --config-cache --with-x-toolkit=lucid
 --with-gif=ifavailable'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty make-network-process emacs)

Memory information:
((conses 16 64542 9889)
 (symbols 48 9474 0)
 (strings 32 23737 1993)
 (string-bytes 1 692637)
 (vectors 16 9301)
 (vector-slots 8 148671 13049)
 (floats 8 34 25)
 (intervals 56 241 0)
 (buffers 976 10))

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Wed, 16 Aug 2023 17:27:34 -0400
> 
> 
> 1. emacs -Q
> 2. (insert "Foo_bar")
> 3. M-% foo RET quux RET
> 4. Observe the buffer contains Quux_bar
> 5. Undo, so the buffer contains Foo_bar again
> 6. M-% foo_bar RET quux RET
> 7. Observe the buffer contains quux
> 
> Expected behavior:
> 7. Observe the buffer contains Quux
> 
> It seems to me that this would match the docstring better:
> 
>   Replacement transfers the case pattern of the old text to the
>   new text, if both ‘case-fold-search’ and ‘case-replace’ are
>   non-nil and FROM-STRING has no uppercase letters.

This is a documentation issue: the complex heuristics used by
replace-match to decide whether and how to "transfer the case pattern"
depend on whether _all_ the words in the replaced text use the same
case pattern.  In your example, Foo_bar is 2 words (what is a "word"
is determined by the buffer's syntax table), and only one of them is
capitalized.  So Emacs does not capitalize the replacement.  Try doing
the same with Foo_Bar, and you will see Quux in the replacement.

These subtleties come up from time to time, so I think it's high time
we documented that (in the hope this will stop this from popping up),
so I've now done so on the emacs-29 branch.

I think this is all we should do here, as the behavior is correct and
justified, has been like that since time immemoriam, and we should not
change it, therefore.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Wed, 16 Aug 2023 17:27:34 -0400
>> 
>> 
>> 1. emacs -Q
>> 2. (insert "Foo_bar")
>> 3. M-% foo RET quux RET
>> 4. Observe the buffer contains Quux_bar
>> 5. Undo, so the buffer contains Foo_bar again
>> 6. M-% foo_bar RET quux RET
>> 7. Observe the buffer contains quux
>> 
>> Expected behavior:
>> 7. Observe the buffer contains Quux
>> 
>> It seems to me that this would match the docstring better:
>> 
>>   Replacement transfers the case pattern of the old text to the
>>   new text, if both ‘case-fold-search’ and ‘case-replace’ are
>>   non-nil and FROM-STRING has no uppercase letters.
>
> This is a documentation issue: the complex heuristics used by
> replace-match to decide whether and how to "transfer the case pattern"
> depend on whether _all_ the words in the replaced text use the same
> case pattern.  In your example, Foo_bar is 2 words (what is a "word"
> is determined by the buffer's syntax table), and only one of them is
> capitalized.  So Emacs does not capitalize the replacement.  Try doing
> the same with Foo_Bar, and you will see Quux in the replacement.
>
> These subtleties come up from time to time, so I think it's high time
> we documented that (in the hope this will stop this from popping up),
> so I've now done so on the emacs-29 branch.
>
> I think this is all we should do here, as the behavior is correct and
> justified, has been like that since time immemoriam, and we should not
> change it, therefore.

Ah, that makes sense!  I have no further complaint then and we should be
able to close the bug.

Although, a further relevant question: is there some way to make _ not
be a word separator, in this specific context?  Without changing the
syntax table of the buffer overall?  So that foo_bar is interpreted as
one word?

Maybe some kind of escaping?

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 65347@debbugs.gnu.org
> Date: Thu, 17 Aug 2023 08:42:35 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > These subtleties come up from time to time, so I think it's high time
> > we documented that (in the hope this will stop this from popping up),
> > so I've now done so on the emacs-29 branch.
> >
> > I think this is all we should do here, as the behavior is correct and
> > justified, has been like that since time immemoriam, and we should not
> > change it, therefore.
> 
> Ah, that makes sense!  I have no further complaint then and we should be
> able to close the bug.

Done now.

> Although, a further relevant question: is there some way to make _ not
> be a word separator, in this specific context?  Without changing the
> syntax table of the buffer overall?  So that foo_bar is interpreted as
> one word?
> 
> Maybe some kind of escaping?

I'm not aware of any such way.  The code explicitly checks the syntax
to be 'word' or not 'word'.  But maybe Stefan has some insights?

You could, of course, make a wrapper command that would modify the
syntax table before invoking the real thing.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Although, a further relevant question: is there some way to make _ not
> be a word separator, in this specific context?  Without changing the
> syntax table of the buffer overall?  So that foo_bar is interpreted as
> one word?

There's `superword-mode`, but I suspect that query-replace does not
honor it in this case.  Could be considered as a bug in
`superword-mode`.


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  65347@debbugs.gnu.org
> Date: Thu, 17 Aug 2023 22:04:53 -0400
> 
> > Although, a further relevant question: is there some way to make _ not
> > be a word separator, in this specific context?  Without changing the
> > syntax table of the buffer overall?  So that foo_bar is interpreted as
> > one word?
> 
> There's `superword-mode`, but I suspect that query-replace does not
> honor it in this case.  Could be considered as a bug in
> `superword-mode`.

Actually, I think it's a bug in replace-match: it doesn't honor
find-word-boundary-function-table.  I think it should.  Patches
welcome.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  65347@debbugs.gnu.org
>> Date: Thu, 17 Aug 2023 22:04:53 -0400
>> 
>> > Although, a further relevant question: is there some way to make _ not
>> > be a word separator, in this specific context?  Without changing the
>> > syntax table of the buffer overall?  So that foo_bar is interpreted as
>> > one word?
>> 
>> There's `superword-mode`, but I suspect that query-replace does not
>> honor it in this case.  Could be considered as a bug in
>> `superword-mode`.
>
> Actually, I think it's a bug in replace-match: it doesn't honor
> find-word-boundary-function-table.  I think it should.  Patches
> welcome.

I suppose this is a bit of an XY problem - the real thing I (speaking
for the user who reported this to me) want is to have case-matching for
symbols instead of words.

e.g. if I have Foo-do-action and I do a replace of foo-do-action with
bar-do-action, I want the replacement to be Bar-do-action.

That seems like a reasonable customization... I could add support for
this to replace-match - it would just be a matter of teaching it to
conditionally use symbol boundaries instead of word boundaries.

Alternatively, much more interestingly, the case-matching could *always*
detect case patterns both for symbols and for words.  And if there's no
case pattern for words, but there is a case pattern for symbols, apply
that case pattern to symbols in the replacement.  We could even turn
that on by default.  Does that sound like a good change?  I can make
that change if it sounds desirable.

Alternatively, much more simply, maybe the case-matching could just
detect if the first word is capitalized, and then also capitalize the
first word in the replacement?  That would work for this specific usage.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  65347@debbugs.gnu.org
> Date: Fri, 18 Aug 2023 12:30:48 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> There's `superword-mode`, but I suspect that query-replace does not
> >> honor it in this case.  Could be considered as a bug in
> >> `superword-mode`.
> >
> > Actually, I think it's a bug in replace-match: it doesn't honor
> > find-word-boundary-function-table.  I think it should.  Patches
> > welcome.
> 
> I suppose this is a bit of an XY problem - the real thing I (speaking
> for the user who reported this to me) want is to have case-matching for
> symbols instead of words.

We look at this from different angles.  As an Emacs maintainer, I
would like to see every place that deals with words to honor
find-word-boundary-function-table, instead of inventing its own ideas
about where a word begins and where it ends.

> e.g. if I have Foo-do-action and I do a replace of foo-do-action with
> bar-do-action, I want the replacement to be Bar-do-action.

Here, you look at the issue from a very narrow perspective: of words
separated by '-' or '_', because that just happens to be the case that
you bumped into.  But that is just one very particular use case; words
can be separated by a lot more characters.  I would actually begin
considering the simpler case of "Foo do action".

> Alternatively, much more simply, maybe the case-matching could just
> detect if the first word is capitalized, and then also capitalize the
> first word in the replacement?  That would work for this specific usage.

Given enough user options, we could support that as optional behavior.
But one thing is clear: we cannot simultaneously support all of the
possible interpretations of "capitalization pattern".  And another
thing is clear: this heuristic is confusing and rarely kept in mind as
it is, so imagine what will happen if we add half a dozen options to
support its variants...

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  65347@debbugs.gnu.org
>> Date: Fri, 18 Aug 2023 12:30:48 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> There's `superword-mode`, but I suspect that query-replace does not
>> >> honor it in this case.  Could be considered as a bug in
>> >> `superword-mode`.
>> >
>> > Actually, I think it's a bug in replace-match: it doesn't honor
>> > find-word-boundary-function-table.  I think it should.  Patches
>> > welcome.
>> 
>> I suppose this is a bit of an XY problem - the real thing I (speaking
>> for the user who reported this to me) want is to have case-matching for
>> symbols instead of words.
>
> We look at this from different angles.  As an Emacs maintainer, I
> would like to see every place that deals with words to honor
> find-word-boundary-function-table, instead of inventing its own ideas
> about where a word begins and where it ends.

Understandable, I want this too of course.  Better support for case
matching in words would be good.

>> e.g. if I have Foo-do-action and I do a replace of foo-do-action with
>> bar-do-action, I want the replacement to be Bar-do-action.
>
> Here, you look at the issue from a very narrow perspective: of words
> separated by '-' or '_', because that just happens to be the case that
> you bumped into.  But that is just one very particular use case; words
> can be separated by a lot more characters.  I would actually begin
> considering the simpler case of "Foo do action".

But no, this use case very specifically is about replacing symbols
(sexps) while matching case.  There's a body of code which frequently
uses the same symbol with different capitalization (both as a lowercase
variable name and an uppercase type name) and I want to do replacements
which match case.

I don't much care about the word boundary or superword-mode, because I'm
not dealing with words - I'm very deliberately dealing with symbols.

>> Alternatively, much more simply, maybe the case-matching could just
>> detect if the first word is capitalized, and then also capitalize the
>> first word in the replacement?  That would work for this specific usage.
>
> Given enough user options, we could support that as optional behavior.
> But one thing is clear: we cannot simultaneously support all of the
> possible interpretations of "capitalization pattern".  And another
> thing is clear: this heuristic is confusing and rarely kept in mind as
> it is, so imagine what will happen if we add half a dozen options to
> support its variants...

Sure.  This idea is very hacky anyway.

What about the other thing I proposed?  That's what I'm more interested
in, because this is a problem of replacing symbols.  And it's much more
elegant.  Repeated here:

>>Alternatively, much more interestingly, the case-matching could *always*
>>detect case patterns both for symbols and for words.  And if there's no
>>case pattern for words, but there is a case pattern for symbols, apply
>>that case pattern to symbols in the replacement.  We could even turn
>>that on by default.  Does that sound like a good change?  I can make
>>that change if it sounds desirable.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 65347@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Fri, 18 Aug 2023 14:31:46 -0400
> 
> > Here, you look at the issue from a very narrow perspective: of words
> > separated by '-' or '_', because that just happens to be the case that
> > you bumped into.  But that is just one very particular use case; words
> > can be separated by a lot more characters.  I would actually begin
> > considering the simpler case of "Foo do action".
> 
> But no, this use case very specifically is about replacing symbols
> (sexps) while matching case.

What are "symbols" in this context?

> I don't much care about the word boundary or superword-mode, because I'm
> not dealing with words - I'm very deliberately dealing with symbols.

Capitalization issues with program code are conceptually different
from those with human-readable text.  You are basically talking about
refactoring, not about text replacement.  So the use cases that are of
interest to you are not well supported by query-replace, because it
doesn't target them.  It could well mean that you will need a custom
replace-match function.  Insisting on replace-match to support these
cases is not necessarily wise, from where I stand.

> What about the other thing I proposed?  That's what I'm more interested
> in, because this is a problem of replacing symbols.  And it's much more
> elegant.  Repeated here:
> 
> >>Alternatively, much more interestingly, the case-matching could *always*
> >>detect case patterns both for symbols and for words.  And if there's no
> >>case pattern for words, but there is a case pattern for symbols, apply
> >>that case pattern to symbols in the replacement.  We could even turn
> >>that on by default.  Does that sound like a good change?  I can make
> >>that change if it sounds desirable.

I don't understand the proposal.  Again, what are "symbols" in this
context, and how are they different from "words"?

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 65347@debbugs.gnu.org,  monnier@iro.umontreal.ca
>> Date: Fri, 18 Aug 2023 14:31:46 -0400
>> 
>> > Here, you look at the issue from a very narrow perspective: of words
>> > separated by '-' or '_', because that just happens to be the case that
>> > you bumped into.  But that is just one very particular use case; words
>> > can be separated by a lot more characters.  I would actually begin
>> > considering the simpler case of "Foo do action".
>> 
>> But no, this use case very specifically is about replacing symbols
>> (sexps) while matching case.
>
> What are "symbols" in this context?

Strings made up of symbol constituents as defined by the current
major-mode's syntax table.  The normal definition of symbols.

>> I don't much care about the word boundary or superword-mode, because I'm
>> not dealing with words - I'm very deliberately dealing with symbols.
>
> Capitalization issues with program code are conceptually different
> from those with human-readable text.  You are basically talking about
> refactoring, not about text replacement.  So the use cases that are of
> interest to you are not well supported by query-replace, because it
> doesn't target them.  It could well mean that you will need a custom
> replace-match function.  Insisting on replace-match to support these
> cases is not necessarily wise, from where I stand.

Hm, that's fair.  Although I would bet that the majority of usage of
query-replace is with program code, since the majority of Emacs usage is
with program code.  So features which make query-replace work better
with code are still useful.

Another feature that could support this would be to allow defining
multiple query/replacement pairs, and applying them together across the
file or across multiple files, querying as we go.  Then "foo" could be
replaced with "bar" and "Foo" with "Bar".  That kind of simultaneous
replacement is something I've definitely wanted before.

>> What about the other thing I proposed?  That's what I'm more interested
>> in, because this is a problem of replacing symbols.  And it's much more
>> elegant.  Repeated here:
>> 
>> >>Alternatively, much more interestingly, the case-matching could *always*
>> >>detect case patterns both for symbols and for words.  And if there's no
>> >>case pattern for words, but there is a case pattern for symbols, apply
>> >>that case pattern to symbols in the replacement.  We could even turn
>> >>that on by default.  Does that sound like a good change?  I can make
>> >>that change if it sounds desirable.
>
> I don't understand the proposal.  Again, what are "symbols" in this
> context, and how are they different from "words"?

The normal definition of symbols in Emacs, as defined by the syntax
table.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> We look at this from different angles.  As an Emacs maintainer, I
>> would like to see every place that deals with words to honor
>> find-word-boundary-function-table, instead of inventing its own ideas
>> about where a word begins and where it ends.

Yes, that could be good in any case.  It would happen to fix the example
problem, tho only in the specific case where the user has enabled
`superword-mode`.

In my experience, there are enough slightly different circumstances that
it's going to be hard to cover all cases automatically.

Personally (as a user) the functionality I'd prefer is the following:
When the matched string does not have the same capitalization as the search
pattern, I'd like Emacs to guess what should be the capitalization of
the replacement and ask me to confirm its guess.

This solution would not satisfy the "automatically" part, but it would
succeed at the "cover all cases" part, only at the small cost of an
extra RET.


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 65347@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Fri, 18 Aug 2023 15:14:46 -0400
> 
> > What are "symbols" in this context?
> 
> Strings made up of symbol constituents as defined by the current
> major-mode's syntax table.  The normal definition of symbols.

Then you are talking about something very different from "words".

> > Capitalization issues with program code are conceptually different
> > from those with human-readable text.  You are basically talking about
> > refactoring, not about text replacement.  So the use cases that are of
> > interest to you are not well supported by query-replace, because it
> > doesn't target them.  It could well mean that you will need a custom
> > replace-match function.  Insisting on replace-match to support these
> > cases is not necessarily wise, from where I stand.
> 
> Hm, that's fair.  Although I would bet that the majority of usage of
> query-replace is with program code, since the majority of Emacs usage is
> with program code.

That's profoundly not true!  Emacs is used with human-readable text
not less, and maybe more, than with program source code.  I'm typing
this email in Emacs; I'm routinely making changes to our documentation
in Emacs -- these and others are all frequent text-editing activities.

> So features which make query-replace work better with code are still
> useful.

No, we need a real refactoring in Emacs!  Using M-% as poor-man's
refactoring is fine, but pretending that it _is_ refactoring, and
adding minor extensions to it that are motivated by refactoring, is
simply wrong!  It will likely complicate the text-oriented replacement
we have already, and will always fall short of decent refactoring
capabilities.

We should work on adding refactoring instead of tweaking M-% and M-*
in these directions.

> Another feature that could support this would be to allow defining
> multiple query/replacement pairs, and applying them together across the
> file or across multiple files, querying as we go.  Then "foo" could be
> replaced with "bar" and "Foo" with "Bar".  That kind of simultaneous
> replacement is something I've definitely wanted before.

IMO, this makes little or no sense in human-readable text; it does
make sense in the refactoring context.  So let's add refactoring
capabilities to Emacs, and leave M-% for text.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 65347@debbugs.gnu.org,  monnier@iro.umontreal.ca
>> Date: Fri, 18 Aug 2023 15:14:46 -0400
>> 
>> > What are "symbols" in this context?
>> 
>> Strings made up of symbol constituents as defined by the current
>> major-mode's syntax table.  The normal definition of symbols.
>
> Then you are talking about something very different from "words".
>
>> > Capitalization issues with program code are conceptually different
>> > from those with human-readable text.  You are basically talking about
>> > refactoring, not about text replacement.  So the use cases that are of
>> > interest to you are not well supported by query-replace, because it
>> > doesn't target them.  It could well mean that you will need a custom
>> > replace-match function.  Insisting on replace-match to support these
>> > cases is not necessarily wise, from where I stand.
>> 
>> Hm, that's fair.  Although I would bet that the majority of usage of
>> query-replace is with program code, since the majority of Emacs usage is
>> with program code.
>
> That's profoundly not true!  Emacs is used with human-readable text
> not less, and maybe more, than with program source code.  I'm typing
> this email in Emacs; I'm routinely making changes to our documentation
> in Emacs -- these and others are all frequent text-editing activities.
>
>> So features which make query-replace work better with code are still
>> useful.
>
> No, we need a real refactoring in Emacs!  Using M-% as poor-man's
> refactoring is fine, but pretending that it _is_ refactoring, and
> adding minor extensions to it that are motivated by refactoring, is
> simply wrong!  It will likely complicate the text-oriented replacement
> we have already, and will always fall short of decent refactoring
> capabilities.
>
> We should work on adding refactoring instead of tweaking M-% and M-*
> in these directions.

Okay, I'm convinced.  I'll defer this functionality to the future
refactoring support in Emacs, built with project.el and eglot no doubt.

>> Another feature that could support this would be to allow defining
>> multiple query/replacement pairs, and applying them together across the
>> file or across multiple files, querying as we go.  Then "foo" could be
>> replaced with "bar" and "Foo" with "Bar".  That kind of simultaneous
>> replacement is something I've definitely wanted before.
>
> IMO, this makes little or no sense in human-readable text; it does
> make sense in the refactoring context.  So let's add refactoring
> capabilities to Emacs, and leave M-% for text.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  65347@debbugs.gnu.org
> Date: Fri, 18 Aug 2023 15:20:21 -0400
> 
> In my experience, there are enough slightly different circumstances that
> it's going to be hard to cover all cases automatically.

Perhaps, but this could only be established by auditing each case one
by one.

> Personally (as a user) the functionality I'd prefer is the following:
> When the matched string does not have the same capitalization as the search
> pattern, I'd like Emacs to guess what should be the capitalization of
> the replacement and ask me to confirm its guess.

This already happens?  M-% shows you the replacement in the exact
capitalization it will insert, _after_ the guesswork.  And you are
already requested to confirm each replacement.  So what is missing?

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 65347@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Fri, 18 Aug 2023 17:00:12 -0400
> 
> > We should work on adding refactoring instead of tweaking M-% and M-*
> > in these directions.
> 
> Okay, I'm convinced.  I'll defer this functionality to the future
> refactoring support in Emacs, built with project.el and eglot no doubt.

Eglot can serve as the back-end, and it isn't the only one that comes
to mind (I sincerely hope that at least some of the simpler
refactoring jobs will not require an LSP, but could be done using
built-in capabilities).

But back-end is just one part of this.  We should IMO begin by our own
research into the UI parts of this: how does the user specify the
requested refactoring?  Several alternatives are possible, and we
should study them and decide what is best for Emacs.

Would someone please step forward and work on adding refactoring to
Emacs?

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Personally (as a user) the functionality I'd prefer is the following:
>> When the matched string does not have the same capitalization as the search
>> pattern, I'd like Emacs to guess what should be the capitalization of
>> the replacement and ask me to confirm its guess.
> This already happens?  M-% shows you the replacement in the exact
> capitalization it will insert, _after_ the guesswork.  And you are
> already requested to confirm each replacement.  So what is missing?

When the guesswork is wrong I find it cumbersome to fix it.
I'd like to be able to edit it right there and that the result
is remembered for the next time the same capitalization is found.


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: sbaugh@janestreet.com,  65347@debbugs.gnu.org
> Date: Sat, 19 Aug 2023 10:39:02 -0400
> 
> >> Personally (as a user) the functionality I'd prefer is the following:
> >> When the matched string does not have the same capitalization as the search
> >> pattern, I'd like Emacs to guess what should be the capitalization of
> >> the replacement and ask me to confirm its guess.
> > This already happens?  M-% shows you the replacement in the exact
> > capitalization it will insert, _after_ the guesswork.  And you are
> > already requested to confirm each replacement.  So what is missing?
> 
> When the guesswork is wrong I find it cumbersome to fix it.
> I'd like to be able to edit it right there and that the result
> is remembered for the next time the same capitalization is found.

If C-r is inconvenient, then I guess we need some minor variation of
the existing commands e and E?

And what do you mean by "remembered for the next time", when the only
difference is in letter-case?

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier [2023-08-19 10:39:02] wrote:
> When the guesswork is wrong I find it cumbersome to fix it.
> I'd like to be able to edit it right there and that the result
> is remembered for the next time the same capitalization is found.

Admittedly, another problem is that I'm usually looking at the buffer
text that's about to be changed, rather than looking at the minibuffer
showing me what the text will change to, so I usually notice only after
the fact that the change was wrong :-(

So maybe what I want is something like:

1. like we do now, highlight the text to be changed and ask y/n whether
   to make the change (I likely won't see the exact question in the
   minibuffer at that point).
2. if I say `y` but the capitalization of the change is the result of
   guesswork, stop right after making the change, highlight it, and ask
   again whether that was good, letting me edit it before continuing.

Similarly when I say `!` it should stop after every change that
encountered a not-yet-seen capitalization.


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: sbaugh@janestreet.com,  65347@debbugs.gnu.org
> Date: Sat, 19 Aug 2023 10:52:02 -0400
> 
> So maybe what I want is something like:
> 
> 1. like we do now, highlight the text to be changed and ask y/n whether
>    to make the change (I likely won't see the exact question in the
>    minibuffer at that point).
> 2. if I say `y` but the capitalization of the change is the result of
>    guesswork, stop right after making the change, highlight it, and ask
>    again whether that was good, letting me edit it before continuing.

That's what ',' does, no?

> Similarly when I say `!` it should stop after every change that
> encountered a not-yet-seen capitalization.

Define "not yet seen capitalization".

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> And what do you mean by "remembered for the next time", when the only
> difference is in letter-case?

When it finds "FOo" and converts it to "Bar" and I change it to "BAr" it
should remember that mapping such that next time it finds that exact
same capitalization "FOo" it should use "BAr" as replacement.


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Similarly when I say `!` it should stop after every change that
>> encountered a not-yet-seen capitalization.
>
> Define "not yet seen capitalization".

I'm replacing "foo" with "bar", so at the start we know for sure:

- "foo" should be replaced by "bar"
- that's it, anything else is guesswork

We find "Foo", it's a not-yet-seen capitalization.
We propose "Bar" (which the user may accept of modify).
At that point we know:

- "foo" should be replaced by "bar"
- "Foo" should be replaced by "Bar" (or whichever alternative the user
  provided instead).
- that's it, anything else is guesswork

When we find "fOo", it's a not-yet-seen capitalization, so we have to
ask the user again, ...


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: sbaugh@janestreet.com,  65347@debbugs.gnu.org
> Date: Sat, 19 Aug 2023 11:51:46 -0400
> 
> > And what do you mean by "remembered for the next time", when the only
> > difference is in letter-case?
> 
> When it finds "FOo" and converts it to "Bar" and I change it to "BAr" it
> should remember that mapping such that next time it finds that exact
> same capitalization "FOo" it should use "BAr" as replacement.

Is it really what happens in these cases?  IME, the capitalization
patterns, when they don't fit the heuristic, are inconsistent with one
another, and bumping into one such "non-standard" replacement doesn't
mean all those replacements after it will be necessarily
"non-standard", let alone in the same way.

IOW, I believe the best strategy is to keep replacing using the
default heuristic, and leave it to the user to detect the rare
deviations and fix them with ',' or 'C-r'.

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> When it finds "FOo" and converts it to "Bar" and I change it to "BAr" it
>> should remember that mapping such that next time it finds that exact
>> same capitalization "FOo" it should use "BAr" as replacement.
> Is it really what happens in these cases?  IME, the capitalization
> patterns, when they don't fit the heuristic, are inconsistent with one
> another, and bumping into one such "non-standard" replacement doesn't
> mean all those replacements after it will be necessarily
> "non-standard", let alone in the same way.

I guess YMMV, but my experience is that the mapping is usually fixed
(at least "per search&replace"), and in the OP's case it is as well.


        Stefan

bug#65347: 29.1; Underscore in query replace prevents case-matching

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Eglot can serve as the back-end, and it isn't the only one that comes
  > to mind (I sincerely hope that at least some of the simpler
  > refactoring jobs will not require an LSP, but could be done using
  > built-in capabilities).

It is ok if Emacs has facilities to use language servers,
but it would be a bad end for Emacs if it becomes dependent
on language servers for its important features.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)