emacs 22 - regular-expression isearch on spaces extremely lenient

emacs 22 - regular-expression isearch on spaces extremely lenient

[Ken Manheimer]

i recently started noticing that emacs 22 regular expression isearches
do not treat spaces exactly - any number of spaces in your search will
map to any number, same or different, of spaces in the target.

for example:
 - copy this message into an empty buffer
 - postiong the cursor at the beginning of the buffer
 - start an isearch (Ctrl-s)
 - toggle it to a regexp-isearch (Meta-r)
 - enter three spaces
 - enter an "s"
when i do this, the cursor winds up positioned on the "s" at the
beginning of the third word in this message, "started".

i'm starting to realize that this behavior has been in emacs 22 for at
least a while, and is not in emacs 21.  can anyone tell me whether or
not it's deliberate, and what the rationale is?  (i started writing
this up as a bug for emacs-devel, but looking into isearch-search and
seeing that `search-spaces-regexp' is specifically set to the value of
`search-whitespace-regexp' is making me suspect that the behavior is a
deliberate design feature.  ?)
--
ken manheimer
ken.manheimer@gmail.com
http://myriadicity.net

RE: emacs 22 - regular-expression isearch on spaces extremely lenient

[Drew Adams]

    i recently started noticing that emacs 22 regular expression isearches
    do not treat spaces exactly - any number of spaces in your search will
    map to any number, same or different, of spaces in the target.

    can anyone tell me whether or not it's deliberate, and what the
    rationale is?

(setq search-whitespace-regexp nil) will turn this off. When this is nil,
each space you type matches literally, against one space.

`search-whitespace-regexp' is, by default, "\\s-+", which searches for any
amount of whitespace when you type a space. This was introduced for regexp
search in Emacs 21, I believe. There is no such "magic-space" searching in
Emacs 20. Doc:

 If non-nil, regular expression to match a sequence of whitespace chars.
 This applies to regular expression incremental search.
 When you put a space or spaces in the incremental regexp, it stands for
 this, unless it is inside of a regexp construct such as [...] or *, + or ?.
 You might want to use something like "[ \t\r\n]+" instead.
 In the Customization buffer, that is `[' followed by a space,
 a tab, a carriage return (control-M), a newline, and `]+'.

The rationale was, I believe, that some users might want that: type space to
find any amount of whitespace, in particular, to find two words that are
separated by a newline.

There was talk of using this "magic-space" searching also for plain
incremental search in Emacs 22, but I don't think that was done.

FWIW, I agree with Miles on this - this is a misfeature, if turned on by
default. It should be off by default, and you should be able to turn it on
via a simple toggle during incremental search (regexp or plain).

Here is what I wrote 2005/02/06 to emacs-devel on this:

            > > sometimes the actual whitespace matters.
            > Right: in *regexp* search.
          while people generally expect regexp searches to be a bit
          fuzzy, they might expect a non-regexp search to be exact.
          Since the fuzzy whitespace matching often "looks" like normal
          matching (because the majority of whitespace is in fact a
          single space), it might take some time to see what's going on,
          resulting in some subtle errors. This is particularly true if
          one embeds a search inside a keyboard macro [which I often do].

    Plain (incremental) search should be a literal search. Regexp
    search should rigorously respect the regexp. People don't expect
    either to be fuzzy.

    The question is "Under what circumstances should typing a space be
    interpreted as wanting to search for any amount of whitespace?"

    This is unrelated to both plain search and regexp search. You
    might or might not want this _input effect_ with either plain
    or regexp search.

    This is akin to word search (as I think someone mentioned).
    Ultimately, a word search or a space-means-whitespace search is
    implemented with a regexp search - but the point in both cases
    is to provide a user-friendly way to do it, instead of requiring
    users to know about regexps.

    By default, neither `C-M-s' nor `C-s' should respect the
    user-friendly space-input feature. Or, rather, the default
    behavior of each should be determined by a user option - a la
    case-fold-search. And, regardless of the value of this option,
    you should be able to toggle space-means-whitespace
    searching from both `C-M-s' and `C-s', via a key sequence.

    The question then becomes how to toggle this space-means-whitespace
    searching?

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[Eric Hanchrow]

>>>>> "Drew" == Drew Adams <drew.adams@oracle.com> writes:

    Drew>     The question then becomes how to toggle this
    Drew>     space-means-whitespace searching?

I haven't the slightest idea how to toggle it, which of course is the
important question.  But I do have an idea: for regexp searches, if
while I'm typing my regexp, and I type a space, and this feature is
on, it'd be lovely if Emacs would simply insert the appropriate regexp
for whitespace -- such as \s-+ -- into the mini-buffer.  That way I
will learn a bit more about regexps.

-- 

            |\      _,,,---,,_
      ZZZzz /,`.-'`'    -.  ;-;;,_
           |,4-  ) )-,_. ,\ (  `'-'
          '---''(_/--'  `-'\_) fL

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[Kevin Rodgers]

Eric Hanchrow wrote:
>>>>>> "Drew" == Drew Adams <drew.adams@oracle.com> writes:
> 
>     Drew>     The question then becomes how to toggle this
>     Drew>     space-means-whitespace searching?
> 
> I haven't the slightest idea how to toggle it, which of course is the
> important question.  But I do have an idea: for regexp searches, if
> while I'm typing my regexp, and I type a space, and this feature is
> on, it'd be lovely if Emacs would simply insert the appropriate regexp
> for whitespace -- such as \s-+ -- into the mini-buffer.  That way I
> will learn a bit more about regexps.

Excellent suggestion!

-- 
Kevin Rodgers

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[Richard Stallman]

    > I haven't the slightest idea how to toggle it, which of course is the
    > important question.  But I do have an idea: for regexp searches, if
    > while I'm typing my regexp, and I type a space, and this feature is
    > on, it'd be lovely if Emacs would simply insert the appropriate regexp
    > for whitespace -- such as \s-+ -- into the mini-buffer.  That way I
    > will learn a bit more about regexps.

That could help clarify matters for some users, but it could be
confusing for other users.

That space in regexp i-search matches all whitespace is not a new
feature.  We have only changed the way it is implemented, to make
quoting spaces more reliable.  So I see no urgent need to change this.

But I would not mind adding an option to make SPC echo as some other
string in regexp i-search.  Then, after the release, we could
ask users to try turning it on and say whether they like it.

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[ken manheimer]

Drew Adams <drew.adams <at> oracle.com> writes:

> [Ken Manheimer wrote:]
>
>     i recently started noticing that emacs 22 regular expression isearches
>     do not treat spaces exactly - any number of spaces in your search will
>     map to any number, same or different, of spaces in the target.
> 
>     can anyone tell me whether or not it's deliberate, and what the
>     rationale is?
> 
> (setq search-whitespace-regexp nil) will turn this off. When this is nil,
> each space you type matches literally, against one space.

thanks!  i did infer that from a bit of investigation (in isearch-search), and
am kinda mortified by how unobvious it is.  i agree very strongly with your
analysis, and feel the feature is too elusive in too many ways - all the
questions i was asking: behaviorally ("wait a minute, does it really do that,
and if so, when?"), properness ("is it really supposed to do that?"),
control/inhibition (how the hell do i turn it off?!") - that it's a Real Bad
Idea.  oh well.

i agree that making the control obvious, and defaulting it to off, is a much
better way to make the feature available.  (i could even see having a transient
message in the minibuffer whenever someone types in a space, saying how to make
it a fuzzy space with Ctrl-Q.)

> `search-whitespace-regexp' is, by default, "\\s-+", which searches for any
> amount of whitespace when you type a space. This was introduced for regexp
> search in Emacs 21, I believe. There is no such "magic-space" searching in
> Emacs 20. Doc:
> 
>  If non-nil, regular expression to match a sequence of whitespace chars.
>  This applies to regular expression incremental search.
>  When you put a space or spaces in the incremental regexp, it stands for
>  this, unless it is inside of a regexp construct such as [...] or *, + or ?.
>  You might want to use something like "[ \t\r\n]+" instead.
>  In the Customization buffer, that is `[' followed by a space,
>  a tab, a carriage return (control-M), a newline, and `]+'.

i didn't notice anything about it in the documentation for isearch, though now
that i look at the section on regular expression search i see something about
it.  this is a pitfall of rich features - too much functionality to describe
adequately without overwhelming.

> The rationale was, I believe, that some users might want that: type space to
> find any amount of whitespace, in particular, to find two words that are
> separated by a newline.

i would have suggested using "\ " or "^Q " (which both currently now have the
opposite effect).

> There was talk of using this "magic-space" searching also for plain
> incremental search in Emacs 22, but I don't think that was done.

good.

> FWIW, I agree with Miles on this - this is a misfeature, if turned on by
> default. It should be off by default, and you should be able to turn it on
> via a simple toggle during incremental search (regexp or plain).

who's miles?

> Here is what I wrote 2005/02/06 to emacs-devel on this:
> 
>             > > sometimes the actual whitespace matters.
>             > Right: in *regexp* search.
>           while people generally expect regexp searches to be a bit
>           fuzzy, they might expect a non-regexp search to be exact.
>           Since the fuzzy whitespace matching often "looks" like normal
>           matching (because the majority of whitespace is in fact a
>           single space), it might take some time to see what's going on,
>           resulting in some subtle errors. This is particularly true if
>           one embeds a search inside a keyboard macro [which I often do].
> 
>     Plain (incremental) search should be a literal search. Regexp
>     search should rigorously respect the regexp. People don't expect
>     either to be fuzzy.
> 
>     The question is "Under what circumstances should typing a space be
>     interpreted as wanting to search for any amount of whitespace?"
> 
>     This is unrelated to both plain search and regexp search. You
>     might or might not want this _input effect_ with either plain
>     or regexp search.
> 
>     This is akin to word search (as I think someone mentioned).
>     Ultimately, a word search or a space-means-whitespace search is
>     implemented with a regexp search - but the point in both cases
>     is to provide a user-friendly way to do it, instead of requiring
>     users to know about regexps.
> 
>     By default, neither `C-M-s' nor `C-s' should respect the
>     user-friendly space-input feature. Or, rather, the default
>     behavior of each should be determined by a user option - a la
>     case-fold-search. And, regardless of the value of this option,
>     you should be able to toggle space-means-whitespace
>     searching from both `C-M-s' and `C-s', via a key sequence.
> 
>     The question then becomes how to toggle this space-means-whitespace
>     searching?

thanks much for the thorough answer - it's good to have the mystery settled,
even if i don't agree with the situation.  at least i now know various ways to
work around it...

(i guess it's too late to propose inverting the mode of the feature on
emacs-devel?  i'd just copy this posting over there, asking if it's worth
reopening the discussion.  i wouldn't want to do that, though, if the discussion
ended on a conclusive pronouncement that even dismissed revisiting in the event
of complaints...)

ken manheimer
http://myriadicity.net

RE: emacs 22 - regular-expression isearch on spacesextremely lenient

[Drew Adams]

    > FWIW, I agree with Miles on this - this is a misfeature, if
    turned on by
    > default. It should be off by default, and you should be able
    to turn it on
    > via a simple toggle during incremental search (regexp or plain).

    who's miles?

Miles Bader. He wrote this in reply to your original post:

    "Ken Manheimer" <ken.manheimer@gmail.com> writes:
    > i'm starting to realize that this behavior has been in emacs 22 for at
    > least a while, and is not in emacs 21.  can anyone tell me whether or
    > not it's deliberate, and what the rationale is?

    I'm pretty sure it's deliberate, though I can't give a good rationale
    for it (I think it's stupid and annoying behavior), nor find a NEWS
    entry for it....  However I recall people talking about it on
    emacs-devel as if it were a feature.

----------------

    (i guess it's too late to propose inverting the mode of the feature on
    emacs-devel?  i'd just copy this posting over there, asking if
    it's worth
    reopening the discussion.  i wouldn't want to do that, though,
    if the discussion
    ended on a conclusive pronouncement that even dismissed
    revisiting in the event
    of complaints...)

I would say to go ahead and say what you think to emacs-devel. If someone
finds that inappropriate, s?he will say so.

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[David Combs]

Emacs has been such a well incrementally-designed system (what,
35, 40 years?) that it's been a real pleasure to use.

What DWIM there has been of the type that when you discover
a piece, in your ordinary use of emacs, that your (well,
mine, anyway) reaction is NOT of the type:

 1:  God damn to hell -- WHY did those idiots make it do that!

, but rather of the type:

 2:  Incredible -- this thing has not only a doctor built in to
      it, but a world-class *mind-reader* as well!

Never once have I been confronted with some surprising behaviour
that I haven't felt that it was exactly what I wanted it to do.

------

This space-->whitespaces thing seems very, very different; I haven't
seen in this thread *anyone* who'd give a type-2 (above) opinion
about this "feature", nor can I even *imagine* any normal emacs-user
who would.

So how did this thing get included?

Are we all subject to whatever whim that occurs to the devel-people?

Does (gnu)=Emacs *belong* to just them -- such that whatever
*they* vote for becomes "law"?

How many thousands, or tens (hundreds?) of thousands of people
daily use emacs, in fact *rely* on emacs for 
their most important work?

Now, maybe it's infeasible to try to get a vote from that world-user-base;
but heck, aren't there a lot of people who read this newsgroup
at least once a week?

Why not set up a vote among all of *us* (yes, include the devel-people
too)?

Come up with a description of this "feature" that's acceptable
to both sides on this issue, and set up some kind of a computer-based
vote.

(To help avoidng vote-fraud, we could limit it to those who have
posted within, say, the last year or two -- and we'd suspect
funny-tricks if any voter appeared twice.)

-----

We really have to have some final hurdle that any controversial
feature must pass before it gets included -- *especially*
when it's not being defaulted "off".

I've been using emacs for *so* long (since 1980 with twenex-emacs,
rms on gnu jumping over the moon), even if not so expertly,
that this (by now) old dog's paws have a really hard time
switching now hard-wired habits and expectations.

Seems like as good a time as any to set up a better
procedure for (thus far) few "controversial" changes.

Just my two bits -- but I hope I'm not the only one
who feels like I do.


David

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[Tim X]

dkcombs@panix.com (David Combs) writes:

> Emacs has been such a well incrementally-designed system (what,
> 35, 40 years?) that it's been a real pleasure to use.
>
> What DWIM there has been of the type that when you discover
> a piece, in your ordinary use of emacs, that your (well,
> mine, anyway) reaction is NOT of the type:
>
>  1:  God damn to hell -- WHY did those idiots make it do that!
>
> , but rather of the type:
>
>  2:  Incredible -- this thing has not only a doctor built in to
>       it, but a world-class *mind-reader* as well!
>
> Never once have I been confronted with some surprising behaviour
> that I haven't felt that it was exactly what I wanted it to do.
>
> ------
>
> This space-->whitespaces thing seems very, very different; I haven't
> seen in this thread *anyone* who'd give a type-2 (above) opinion
> about this "feature", nor can I even *imagine* any normal emacs-user
> who would.
>
> So how did this thing get included?
>
> Are we all subject to whatever whim that occurs to the devel-people?
>
> Does (gnu)=Emacs *belong* to just them -- such that whatever
> *they* vote for becomes "law"?
>
> How many thousands, or tens (hundreds?) of thousands of people
> daily use emacs, in fact *rely* on emacs for 
> their most important work?
>
> Now, maybe it's infeasible to try to get a vote from that world-user-base;
> but heck, aren't there a lot of people who read this newsgroup
> at least once a week?
>
> Why not set up a vote among all of *us* (yes, include the devel-people
> too)?
>
> Come up with a description of this "feature" that's acceptable
> to both sides on this issue, and set up some kind of a computer-based
> vote.
>
> (To help avoidng vote-fraud, we could limit it to those who have
> posted within, say, the last year or two -- and we'd suspect
> funny-tricks if any voter appeared twice.)
>
> -----
>
> We really have to have some final hurdle that any controversial
> feature must pass before it gets included -- *especially*
> when it's not being defaulted "off".
>
> I've been using emacs for *so* long (since 1980 with twenex-emacs,
> rms on gnu jumping over the moon), even if not so expertly,
> that this (by now) old dog's paws have a really hard time
> switching now hard-wired habits and expectations.
>
> Seems like as good a time as any to set up a better
> procedure for (thus far) few "controversial" changes.
>
> Just my two bits -- but I hope I'm not the only one
> who feels like I do.
>

Just a couple of comments....

While I can appreciate where your coming from, I am not as convinced
your suggestions are very practicle. As it is, a lot of debate tends
to happen on the devel list prior to major changes - I'd be concerned
widening that debate would result in even more difficulty in arriving
at a consensus. One often sighted complaint about emacs is that it is
so slow to change (something I actually like). Widening debate and
input is only going to make that slower.

I'm also not sure that introducing a vote on features would be a good
thing. We see far too many posts to this group which contain
complaints regarding some feature and all too often, those complaints
are based on either ignorance or lack of familiarity with either the
emacs philosophy or the emacs way of approaching a problem. It takes a
while before you really begin to appreciate the power of emacs and I
cannot see how we wold isolate those who just want emacs to be more
like what they are already use to and those experienced enough to
fully understand why some features or operations are the way they are. 

One thing I've always liked about emacs has been that generally, when
an existing feature changes in some substantial way, there is a
relatively simple way of getting the old behavior bac. Is this not the
case with the new RE behavior in emacs 22?

I saw a post on this list only a few weeks ago which outlined some of
the benefits of the new RE behavior. Unfortunately, I cannot remember
what they were, but at the time, they seemed to make sense. Without
wanting to sound rude, are you certain your disatisfaction isn't
merely the result of change and that the change may actually have some
advantages and not be as 'ad hoc' as you seem to feel? having lurked
on the develop list for a while at one time, I don't recall seeing any
new feature or change of an existing feature which wasn't debated and
argued over extensively. 

My gut feeling is that if anyone has strong enough feelings regarding
how emacs should change/evolve, they really should just participate on
the devel process rather than possibly muddying the waters with a lot
of opinion regarding how things should be done by people who are most
likely largely uninformed regarding the internals, history and
limitations inherent in the code of any system with as long a history
as emacs. 

just my 2 cnets

Tim

-- 
tcross (at) rapttech dot com dot au

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[David Kastrup]

dkcombs@panix.com (David Combs) writes:

> So how did this thing get included?
>
> Are we all subject to whatever whim that occurs to the devel-people?

And gets accepted by a majority of them.  You can easily join the
development list and discuss things as well if you want to: it does
not require you becoming a developer.

> Does (gnu)=Emacs *belong* to just them -- such that whatever *they*
> vote for becomes "law"?

You can be one of "them" if you want to.  And yes, the directions that
Emacs takes are solely determined by those who communicate with the
developers.  How could it be otherwise?

> How many thousands, or tens (hundreds?) of thousands of people daily
> use emacs, in fact *rely* on emacs for their most important work?

Well, tough.  If they decide not to participate in the discussions
about what to do and what not, they have to take what is getting
handed down to them.  If they are afraid of what this could be, they
can just stay with their favorite old version.

> Now, maybe it's infeasible to try to get a vote from that
> world-user-base; but heck, aren't there a lot of people who read
> this newsgroup at least once a week?
>
> Why not set up a vote among all of *us* (yes, include the
> devel-people too)?

Because then the devel-people would be in a minority.  They are those
with a clue about the internals of Emacs, and they are those who will
have to implement whatever changes are made to Emacs.  Nobody pays
them a dime (at least most of them).  So why should they let others,
who are too lazy to get themselves with the development process at
all, dictate what direction they are going to take Emacs to?

Yes, Emacs goes where those developing it take it.  There is no other
way.  Read the developer lists for a while and voice your opinion
whenever you think your opinion important.

> (To help avoidng vote-fraud, we could limit it to those who have
> posted within, say, the last year or two -- and we'd suspect
> funny-tricks if any voter appeared twice.)
>
> We really have to have some final hurdle that any controversial
> feature must pass before it gets included -- *especially* when it's
> not being defaulted "off".

Who is this "we" you are talking of?

> I've been using emacs for *so* long (since 1980 with twenex-emacs,
> rms on gnu jumping over the moon), even if not so expertly, that
> this (by now) old dog's paws have a really hard time switching now
> hard-wired habits and expectations.
>
> Seems like as good a time as any to set up a better
> procedure for (thus far) few "controversial" changes.
>
> Just my two bits -- but I hope I'm not the only one
> who feels like I do.

You are free to join the discussions among the developers.  It does
not get better than that.  Because ultimately the developers take
Emacs where they decide they want it to go.  Feel free to stay with an
old version, or to switch to a different editor.  But I doubt you'll
find that the development situation with a different editor will
provide more opportunity for making yourself heard.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[Miles Bader]

"Ken Manheimer" <ken.manheimer@gmail.com> writes:
> i'm starting to realize that this behavior has been in emacs 22 for at
> least a while, and is not in emacs 21.  can anyone tell me whether or
> not it's deliberate, and what the rationale is?

I'm pretty sure it's deliberate, though I can't give a good rationale
for it (I think it's stupid and annoying behavior), nor find a NEWS
entry for it....  However I recall people talking about it on
emacs-devel as if it were a feature.

-Miles
-- 
We live, as we dream -- alone....

Re: emacs 22 - regular-expression isearch on spaces extremely lenient

[don provan]

"Drew Adams" <drew.adams@oracle.com> writes:

> (setq search-whitespace-regexp nil) will turn this off. When this is nil,
> each space you type matches literally, against one space.

Thanks! This always annoyed me, but I've never seen a mention of the
control variable to turn it off.

>  If non-nil, regular expression to match a sequence of whitespace chars.
>  This applies to regular expression incremental search.

I think the biggest problem with it is the inconsistency between
incremental search and elisp regexp searches. Literally just yesterday
I had to debug a problem caused because my practice regexp at the
command line did what I wanted with a space, but when I used the same
string in a functional search, it didn't. I've seen the documentation
for this feature, but wherever it was, it wasn't clear to me until now
that it was specific to incremental regexp searches rather than a
general feature of all regexp searches.

-don

RE: emacs 22 - regular-expression isearch on spaces extremely lenient

[Drew Adams]

Fwding to emacs-devel. They are the ones who can change this behavior or
perhaps make the user option move visible somehow.

    To: help-gnu-emacs@gnu.org
    > (setq search-whitespace-regexp nil) will turn this off. When
    > this is nil, each space you type matches literally, against
    > one space.

    Thanks! This always annoyed me, but I've never seen a mention of the
    control variable to turn it off.

    >  If non-nil, regular expression to match a sequence of
    >  whitespace chars. This applies to regular expression
    >  incremental search.

    I think the biggest problem with it is the inconsistency between
    incremental search and elisp regexp searches. Literally just yesterday
    I had to debug a problem caused because my practice regexp at the
    command line did what I wanted with a space, but when I used the same
    string in a functional search, it didn't. I've seen the documentation
    for this feature, but wherever it was, it wasn't clear to me until now
    that it was specific to incremental regexp searches rather than a
    general feature of all regexp searches.

    -don