bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Drew Adams]

emacs -Q

In *scratch*:

1. Duplicate the two lines of text, to get this:

;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with C-x C-f and enter text in its buffer.
;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with C-x C-f and enter text in its buffer.

2. Activate the region from before the `T' in the first `This' to after
   the `f' in the second `file', with point before the `T'.  Then `M-x
   rectangle-mark-mode'.

3. `C-M-%' and type `e.*t' for the regexp and `AA' for the replacement
   text.

I was expecting both occurrences of "eat" (inside "create") to be
candidates for replacement, since they are both within the rectangle and
they both match the regexp - but only the second occurrence is a
candidate for replacement.

(If you use regexp `e.t' instead, there is no such error.)

It seems that what is happening is that the regexp is being checked
against the full region, i.e., before the region is limited to the
rectangular portion.

Is that the design?  I think a user would expect the rectangle to define
the region of possible query-replacing, not just define a clipping area
from the full region of searching.  IOW, I think a user would expect the
rectangular limits to be established first, and then searching to be
limited to that rectangular space.

If nothing else, if this is the intended design then I think the doc
should be clear about it.  It should say, in that case, that matches are
sought throughout the full, non-rectangular region, and only those
matches that are wholly (?) within the rectangle are then retained as
possible matches.

My guess of what's happening is supported by this: If you do the same
thing (same region) but without using `rectangle-mark-mode' then you see
that rectangle mark mode just retains the matches for the normal,
non-rectangular region, that are wholly within the rectangle.

Here are the matches for the normal, non-rectangular region:

;; This buffer is for text that is not saved, and for Lisp evaluation.
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
;; To create a file, visit it with C-x C-f and enter text in its buffer.
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
;; This buffer is for text that is not saved, and for Lisp evaluation.
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
;; To create a file, visit it with C-x C-f and enter text in its buffer.
        ^^^

This should be documented, as it is not, I think, what someone expects
by searching a rectangular region (a region delimited by a rectangle).
I think someone would expect that the search domain is limited by the
rectangle, and then query-replace applies to matches within that domain.

That's very different from the current behavior, which is to leave the
search domain as the full region (undelimited by the rectangle) and then
filter out (remove) any matches from that that are not wholly within the
rectangle.



In GNU Emacs 25.2.1 (x86_64-w64-mingw32)
 of 2017-04-24 built on LAPHROAIG
Windowing system distributor 'Microsoft Corp.', version 6.1.7601
Configured using:
 'configure --without-dbus --without-compress-install 'CFLAGS=-O2
 -static -g3''

bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Drew Adams]

Clearly the current design of using a rectangular region to
limit replacing does not limit search for the replacement
commands in the same sense that the (normal) region limits
search.

The (normal) region limits the text that replacement search
tries to match, by bounding it.  The rectangular region keeps
the ordinary region as the domain of text that search tries
to match, and it filters the matches against that domain to
remove any matches that are not wholly within the limits of
the rectangle.

That's the design, and it's OK.  But it is misleading in
that we don't distinguish the two "region" behaviors.  It is
all too easy to expect that the rectangular region bounds
search in the same way that the ordinary region does.

To provide a different design, where the rectangular region
really limits the domain of text that we try to search,
would require some work.  It's OK that we stick to what we
have, at least for now, since it is anyway useful.

But I do think that users should be told what to expect (e.g.,
wrt regexp searching, in particular).  Currently there seems
to be NO doc about replacement over the rectangular region.

The code for this feature seems to have been added without
also providing doc and, as bug #27897 points out, it was
apparently only half-added.  Arg REGION-NONCONTIGUOUS-P
should also be added to other replacement commands, such as
`replace-string', `replace-regexp', `query-replace-regexp-eval',
and `map-query-replace-regexp'.  They too can benefit from
the new feature, and adding support for it is trivial (add
the optional arg and pass it to `perform-replace').

Please consider this bug report (#27896) to be a request
to add doc (in the Emacs manual) about replacing text over
a rectangular region.  That doc would be a good place to
point out that the rectangle limits do not limit the area
where searching is tried; they just provide post-matching
filtering to remove any matches that are not (wholly)
within the rectangle.

I think it's important to make this clear to users.  This
info should perhaps also be added to the doc strings of
the relevant commands.  (That's especially true if you
decide not to document this feature in the Emacs manual.)

> I think a user would expect the rectangle to define
> the region of possible query-replacing, not just define a clipping area
> from the full region of searching.  IOW, I think a user would expect the
> rectangular limits to be established first, and then searching to be
> limited to that rectangular space.
> 
> If nothing else, if this is the intended design then I think the doc
> should be clear about it.  It should say, in that case, that matches are
> sought throughout the full, non-rectangular region, and only those
> matches that are wholly (?) within the rectangle are then retained as
> possible matches.
> 
> My guess of what's happening is supported by this: If you do the same
> thing (same region) but without using `rectangle-mark-mode' then you see
> that rectangle mark mode just retains the matches for the normal,
> non-rectangular region, that are wholly within the rectangle.
> 
> Here are the matches for the normal, non-rectangular region:
> 
> ;; This buffer is for text that is not saved, and for Lisp evaluation.
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ;; To create a file, visit it with C-x C-f and enter text in its buffer.
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ;; This buffer is for text that is not saved, and for Lisp evaluation.
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ;; To create a file, visit it with C-x C-f and enter text in its buffer.
>         ^^^
> 
> This should be documented, as it is not, I think, what someone expects
> by searching a rectangular region (a region delimited by a rectangle).
> I think someone would expect that the search domain is limited by the
> rectangle, and then query-replace applies to matches within that domain.
> 
> That's very different from the current behavior, which is to leave the
> search domain as the full region (undelimited by the rectangle) and then
> filter out (remove) any matches from that that are not wholly within the
> rectangle.
> 
> 
> 
> In GNU Emacs 25.2.1 (x86_64-w64-mingw32)
>  of 2017-04-24 built on LAPHROAIG
> Windowing system distributor 'Microsoft Corp.', version 6.1.7601
> Configured using:
>  'configure --without-dbus --without-compress-install 'CFLAGS=-O2
>  -static -g3''
> 
> 
> 

bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Lars Ingebrigtsen]

Drew Adams <drew.adams@oracle.com> writes:

> The (normal) region limits the text that replacement search
> tries to match, by bounding it.  The rectangular region keeps
> the ordinary region as the domain of text that search tries
> to match, and it filters the matches against that domain to
> remove any matches that are not wholly within the limits of
> the rectangle.

Yup.  It's a pretty odd design decision, though -- I wonder whether it
this happened on purpose or whether that was just the simplest way to
implement this.

I've added Juri to the CCs; perhaps he can clarify.

(The problem is that if you ask to replace "foo.*bar", and you have

foo bar bar

then `query-replace' will match the entire line, and then filter out
that match (given that we have a rectangular region that covers just
"foo bar") and not replace anything.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Drew Adams]

Thanks for looking at this feature request.

> > The (normal) region limits the text that replacement search
> > tries to match, by bounding it.  The rectangular region keeps
> > the ordinary region as the domain of text that search tries
> > to match, and it filters the matches against that domain to
> > remove any matches that are not wholly within the limits of
> > the rectangle.
> 
> Yup.  It's a pretty odd design decision, though --
> I wonder whether it this happened on purpose or
> whether that was just the simplest way to implement this.

I'm pretty sure it was done this way just because
that's much simpler to do.  Filtering is after
the fact - after matching.  And matching respects
only the buffer restriction (`point-(min|max)').

In search+.el I make it possible (via an option)
to actually limit search to the active region.
That is, search matches cannot extend beyond the
region limits.

That feature required quite a few changes to the
Isearch code, even if most of them were simple.

(The general idea is to bind two vars for active
region begin and end and respecting the option,
and then replace calls to `point-(min|max)' by
those vars.  IOW, instead of limiting search to
the buffer restriction, limit it to the region.)

What this request is about amounts to doing
something similar, but doing that with all of the
begin and end bits of a noncontiguous region.

I haven't yet had the time (or motivation) to do
that in my code.

I think it would greatly help Emacs to have such
a feature, i.e., to truly treat the limits of each
zone of a noncontiguous region as search limits.

Now, a question arises as to whether, if we did
have such a feature, it would _also_ be good to
still have the behavior that we have now, i.e.
still be able to have a filter work on matches
for the full buffer, that is, to _not_ limit the
search tries to the region.

I'm guessing that it _would_ be good to conserve
the current behavior, i.e., to have a variable
that chooses which behavior to use.  But I don't
have a good idea of why I think both can be useful.

One reason to keep (also) the current behavior
might be backward compatibility, of course.  But
my gut whispers that there are other reasons.
I haven't really thought about it much.

> I've added Juri to the CCs; perhaps he can clarify.
> (The problem is that if you ask to replace "foo.*bar", and you have
>   foo bar bar
> then `query-replace' will match the entire line, and then filter out
> that match (given that we have a rectangular region that covers just
> "foo bar") and not replace anything.)

It's more important than that.  But yeah, that at
least illustrates one difference.

The point is this: the search boundaries are
currently `point-min' and `point-max', which
respect only narrowing.  If you want to limit
search to a region then you need to narrow to it.

With isearch+.el you can have search respect the
region limits without narrowing (so you can see
the surrounding text while searching - you can
even optionally dim that surrounding text).

And in this case the region bounds act just like
the buffer bounds: _matching_, itself, (not just
filtering) is limited to the region text.

Also with Isearch+ you can search a set of zones,
i.e., a noncontiguous region (while showing or
hiding the antizones).  But in this case, the
limits of those region pieces (zones) aren't used
for matching; they're used only for filtering.

IOW, with my code searching within zones has the
same fault that vanilla Isearch has for searching
within a region using a filter that restricts to
the region: it _matches_ beyond what should be
the search space, and _then it filters_ out any
matches that extend beyond the search space.

In a nutshell, the requested feature is to be
able (but not be required) to not only _filter_
out matches that go beyond zone limits but also
to not allow those matches in the first place.
That is, _limit matching_ itself to be within
such bounds.

(Note: a noncontiguous region need not be
rectangular.  This feature would apply to any
sequence of buffer zones.)

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Juri Linkov]

>> > The (normal) region limits the text that replacement search
>> > tries to match, by bounding it.  The rectangular region keeps
>> > the ordinary region as the domain of text that search tries
>> > to match, and it filters the matches against that domain to
>> > remove any matches that are not wholly within the limits of
>> > the rectangle.
>>
>> Yup.  It's a pretty odd design decision, though --
>> I wonder whether it this happened on purpose or
>> whether that was just the simplest way to implement this.
>
> I'm pretty sure it was done this way just because
> that's much simpler to do.

Indeed.  A proper implementation would require
too much changes in core code.  Just imagine
how to properly replace multi-line regexps in the
rectangular region using current search functions.

As a workaround, to imitate rectangular narrowing,
you can copy the rectangular region with C-x r k
to a temporary buffer, query-replace in it, and
paste the result back with C-x r y.

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> Indeed.  A proper implementation would require
> too much changes in core code.  Just imagine
> how to properly replace multi-line regexps in the
> rectangular region using current search functions.

Yes, that sounds tough...

But we could somewhat easily change `C-M-%' to be slightly less
surprising, and document its limitations.  That is, we say that no
multi-line matches are impossible in rectangle mode, and we implement it
by just narrowing to each successive line part, and executing the
query-replace-regexp once per line.  (With some magic to handle `!' etc,
I guess.)

Would that work?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Juri Linkov]

> But we could somewhat easily change `C-M-%' to be slightly less
> surprising, and document its limitations.  That is, we say that no
> multi-line matches are impossible in rectangle mode, and we implement it

The current limitation is inability to use some greedy
regexps in rectangle mode.  This is not much worse than
limiting query-replace to non-multi-line matches only.

> by just narrowing to each successive line part, and executing the
> query-replace-regexp once per line.  (With some magic to handle `!' etc,
> I guess.)

Actually not narrowing in the sense of narrow-to-region,
but temporarily setting the LIMIT arg of 'replace-search'
to the end of the region of the current line.

I invite anyone who uses the rectangle mode to try
different use cases to see whether this works well.
Maybe this also helps in cases like described in bug#6362.

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

>> But we could somewhat easily change `C-M-%' to be slightly less
>> surprising, and document its limitations.  That is, we say that no
>> multi-line matches are impossible in rectangle mode, and we implement it
>
> The current limitation is inability to use some greedy
> regexps in rectangle mode.  This is not much worse than
> limiting query-replace to non-multi-line matches only.

Aren't multi-line replacements already disallowed in rectangle mode by
the filtering function?

>> by just narrowing to each successive line part, and executing the
>> query-replace-regexp once per line.  (With some magic to handle `!' etc,
>> I guess.)
>
> Actually not narrowing in the sense of narrow-to-region,
> but temporarily setting the LIMIT arg of 'replace-search'
> to the end of the region of the current line.

Yup.

But I guess it's not that easy to get this to work sensibly -- it'd only
highlight matches in one line at a time, which would be odd.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Juri Linkov]

>>> But we could somewhat easily change `C-M-%' to be slightly less
>>> surprising, and document its limitations.  That is, we say that no
>>> multi-line matches are impossible in rectangle mode, and we implement it
>>
>> The current limitation is inability to use some greedy
>> regexps in rectangle mode.  This is not much worse than
>> limiting query-replace to non-multi-line matches only.
>
> Aren't multi-line replacements already disallowed in rectangle mode by
> the filtering function?

Except in a degenerate case when a rectangular region covers lines
completely from beginning to end.

>>> by just narrowing to each successive line part, and executing the
>>> query-replace-regexp once per line.  (With some magic to handle `!' etc,
>>> I guess.)
>>
>> Actually not narrowing in the sense of narrow-to-region,
>> but temporarily setting the LIMIT arg of 'replace-search'
>> to the end of the region of the current line.
>
> Yup.
>
> But I guess it's not that easy to get this to work sensibly -- it'd only
> highlight matches in one line at a time, which would be odd.

Maybe it's possible to add special handling for lazy-highlighting
as well, but unfortunately implementing all this takes too much time.

So this is a question of priorities: does it make sense to spend
much time implementing this.  Are there any users of this feature?

I mean the feature of regexp and multi-line replacements in rectangle mode,
because non-regexp single-line string replacements already work fine
in rectangle mode.

Or it's enough to explain the current limitation in the documentation.

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Drew Adams]

> So this is a question of priorities: does it make sense to spend
> much time implementing this.  Are there any users of this feature?
> 
> I mean the feature of regexp and multi-line replacements in rectangle
> mode, because non-regexp single-line string replacements already work 
> fine in rectangle mode.

I won't suggest what anyone should work on.
We all have limited time and (different)
limited interests.

I'll just say this: the ability for a user
to do this (search any noncontiguous region
properly) would greatly enhance Emacs, IMO.

It's powerful to be able to search zones
(aka noncontiguous regions in a general way,
and that includes limiting _matching_ to the
zones to be searched, not matching against
the whole buffer restriction and then
filtering.

However, this must _not_ be limited to the
assumption of a _rectangular_ noncontiguous
region.  That would defeat the purpose.
That's just one (albeit important) use case.

(And you can already see that from your
discussion of the only exception - for
rectangles - being the rare case when the
rectangle width includes whole lines.  The
case for arbitrary zones is very different.)

I've long had in mind doing this myself,
and started to do so more than once, but I
haven't gotten around to accomplishing it.

If someone does it (in a general way, with
no assumptions of rectangularity) then
great.  If not, that will be for some
indefinite future.

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Juri Linkov]

> I've long had in mind doing this myself,
> and started to do so more than once, but I
> haven't gotten around to accomplishing it.
>
> If someone does it (in a general way, with
> no assumptions of rectangularity) then
> great.  If not, that will be for some
> indefinite future.

In the most general case, a noncontiguous region
is just a set of subregions (or as you call this -
'zones') that for the commands that operate on it
should look as just one contiguous string.
How to implement this, I have no idea.

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Drew Adams]

> > I've long had in mind doing this myself,
> > and started to do so more than once, but I
> > haven't gotten around to accomplishing it.
> >
> > If someone does it (in a general way, with
> > no assumptions of rectangularity) then
> > great.  If not, that will be for some
> > indefinite future.
> 
> In the most general case, a noncontiguous region
> is just a set of subregions (or as you call this -
> 'zones') that for the commands that operate on it
> should look as just one contiguous string.

Well, no.  It depends on the command, of
course.  But _most_ commands will want to
treat a set of zones as just that.

They'll perform operations on one or more
of the zones - either separately or in
combination in some way.

Operations on zones need not even treat
_any_ (let alone all) of the zones as a
string.

> How to implement this, I have no idea.

My approach is to do, for each zone,
what I currently do for the active region
as a whole, which is what vanilla Emacs
does for the buffer as a whole.

IOW, instead of always using `point-min'
and `point-max' for _matching_, use the
limits of each zone.  IOW, just limit
matching to the search space.

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Juri Linkov]

> My approach is to do, for each zone,
> what I currently do for the active region
> as a whole, which is what vanilla Emacs
> does for the buffer as a whole.
>
> IOW, instead of always using `point-min'
> and `point-max' for _matching_, use the
> limits of each zone.  IOW, just limit
> matching to the search space.

Is this similar to how multiple major modes work?
i.e. https://www.emacswiki.org/emacs/MultipleModes

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Drew Adams]

> > My approach is to do, for each zone,
> > what I currently do for the active region
> > as a whole, which is what vanilla Emacs
> > does for the buffer as a whole.
> >
> > IOW, instead of always using `point-min'
> > and `point-max' for _matching_, use the
> > limits of each zone.  IOW, just limit
> > matching to the search space.
> 
> Is this similar to how multiple major modes work?
> i.e.
> https://urldefense.com/v3/__https://www.emacswiki.org/emacs/MultipleMod
> es__;!!GqivPVa7Brio!L1gKnae4wuMn0F1d58axUCx08NTPDOSKxu5AJGwkvZB2eQ04tfi
> 0Xcu6DJtQGsq_$

Maybe, I don't know.  I've never looked into
`mmm-mode' or MuMaMo or any other such.  I
have a vague recollection/impression that
they may rely on text or overlay properties
in some way.  There's also Alan Mackenzie's
proposal a couple of years back for "islands"
as a way of handling multiple major modes.

As to "how", I said nothing really about the
actual how, beyond saying that the general
idea is to handle zone limits analogously to
how we currently handle buffer limits.

I didn't say "analogously", but perhaps I
should have.  My own approach (started but
not finished, and I'd pretty much need to
start over if I resumed) did, yes, amount to
more than an analogy.

In isearch+.el I do what I described, in
order to treat the active region the same
way vanilla Isearch treats the whole buffer:
bind variables at search start to the region
limits, and then replace `point-(min|max)'
with those variable values when non-nil.

My thoughts and initial attempts to handle
zones (~noncontiguous region) were to do
the same thing: limit search to the limits
of each zone.

(I currently use a filter to limit search
that way, but filtering is after the fact.
What's needed is to limit search _matching_.)

bug#27896: [External] : Re: bug#27896: 25.2; `C-M-%' with `rectangle-mark-mode'

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> So this is a question of priorities: does it make sense to spend
> much time implementing this.  Are there any users of this feature?
>
> I mean the feature of regexp and multi-line replacements in rectangle mode,
> because non-regexp single-line string replacements already work fine
> in rectangle mode.
>
> Or it's enough to explain the current limitation in the documentation.

I don't think it's a particularly important bug to fix -- like you, I
think that the number of users here is pretty small.  (To say the
least.)

But I'm not sure whether this is something that should be documented,
either -- it looks like a bug to me, so just leaving it open until
somebody feels like fixing it is an option.

As for how, I don't know.  Perhaps marking regions as skippable somehow
would be an option?  That is, when doing rectangles, mark all the text
outside of the rectangle as skippable (temporarily).  I seem to remember
there being some discussion about this in general for the searching
functions...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no