lax matching is not a great default behavior

lax matching is not a great default behavior

[Drew Adams]

This has been discussed somewhat, but I don't think there was
any actual proposal to change the behavior.  So here goes.

Does it still make sense to make search and replacement
use lax matching, i.e., fold stuff, by default?

I don't think it does.  I think that users, especially
new users, would be less confused if Emacs defaulted to
literal searching - no whitespace, case, "character",
or other folding by default.

Folding features and their combinations and customizations
will only increase in the future (at least I hope so -
that's a good thing).  The starting point that is the
least ambiguous, least surprising, and least difficult for
a user to discover and understand is zero folding: literal
search.

I realize that this would change the longstanding practice
of having letter-case folding be default.  I think that
choice made sense back when most programming languages and
OS's did not have a letter-case difference.  But I don't
think it is the best choice for the default behavior now.

Same for whitespace folding, though that is not longstanding.

Even more so for "character" folding, i.e., ignoring
diacriticals and differences among quote marks etc. (and
some other ad hoc equivalences).  Such classes might be
clear to those who are familiar with them, and this folding
feature can certainly be useful.  But I don't think it is
the right choice for the default behavior.

Literal search is dead simple, plain, obvious, boring, clear.

Anyone who wants to go further, and fold whitespace, for
instance, can easily ask Emacs how to do that and find out
that there is a simple toggle key for it.  Starting with
whitespace folding turned on, on the other hand, can be
confusing, particularly for programmers, who often care
about which whitespace chars, and how many, they are
searching for.

Similarly for case fold.  Let's start with WYTIWYSF: what
you type is what you search for.  Anyone who wants to
abstract from letter case can easily ask Emacs how to do
that and find out that a simple toggle key suffices.

(Yes, there is also the "clever" complication of automatic
case-fold-search change when an upper-case char is present.
I don't propose any change in that regard now, but I do
think that its time is also soon up.  Not so useful as a
default behavior, IMO.  And I don't think it was a great
idea to copy the same cleverness for "character" folding.
But this is a different story.)

It's likely I won't have more to say about this.  I think
the pros & cons are pretty straightforward.  It's more a
question of choosing than arguing reasons at this point,
I expect.

Re: lax matching is not a great default behavior

[Andreas Schwab]

Drew Adams <drew.adams@oracle.com> writes:

> I realize that this would change the longstanding practice
> of having letter-case folding be default.  I think that
> choice made sense back when most programming languages and
> OS's did not have a letter-case difference.  But I don't
> think it is the best choice for the default behavior now.

Case folding makes a lot of sense when searching in plain text, as is
the case in this sentence when searching for "case".

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

RE: lax matching is not a great default behavior

[Drew Adams]

> > I realize that this would change the longstanding practice
> > of having letter-case folding be default.  I think that
> > choice made sense back when most programming languages and
> > OS's did not have a letter-case difference.  But I don't
> > think it is the best choice for the default behavior now.
> 
> Case folding makes a lot of sense when searching in plain text,
> as is the case in this sentence when searching for "case".

Case folding can make a lot of sense in some contexts, yes.

By "plain text" I'm guessing that you might really mean text
written in a natural language (sentences & such)?  In that
case (!), I think you are right.

And that is the behavior we provide, to start with, for Info,
for instance, regardless of a user's customized value of
`case-fold-search'.  And that's TRT.

Maybe that is the main use case (!) we should consider, for
the general default behavior.  Dunno.  Still, the simplicity
of literal search (no guesswork) argues in its favor.

Re: lax matching is not a great default behavior

[Andreas Röhler]

Am 28.11.2015 um 09:29 schrieb Andreas Schwab:
> Drew Adams <drew.adams@oracle.com> writes:
>
>> I realize that this would change the longstanding practice
>> of having letter-case folding be default.  I think that
>> choice made sense back when most programming languages and
>> OS's did not have a letter-case difference.  But I don't
>> think it is the best choice for the default behavior now.
>
> Case folding makes a lot of sense when searching in plain text, as is
> the case in this sentence when searching for "case".
>
> Andreas.
>

Notion resp. implementation of case-folding is unclear outside ASCII and 
related.

There must not be anything to be folded in a buffer. Folding is a notion 
which makes sense in special cases only - which are common in ASCII 
world, but not a reason for a default.

New users should not be bothered with this vast area. GNU Linux is 
case-sensitive and Emacs should start by default like that.

While specific language modes, like SQL, might or should change that.

Re: lax matching is not a great default behavior

[Andreas Schwab]

Andreas Röhler <andreas.roehler@online.de> writes:

> There must not be anything to be folded in a buffer. Folding is a notion
> which makes sense in special cases only - which are common in ASCII world,
> but not a reason for a default.

Case folding has nothing to do with ASCII.  Latin, Cyrillic, Greek,
Hebrew, Arabic, a lot of scripts have the concept of letter case, or
different forms for the same letter.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: lax matching is not a great default behavior

[Yuri Khan]

On Tue, Dec 1, 2015 at 4:38 PM, Andreas Schwab <schwab@linux-m68k.org> wrote:

> Case folding has nothing to do with ASCII.  Latin, Cyrillic, Greek,
> Hebrew, Arabic, a lot of scripts have the concept of letter case, or
> different forms for the same letter.

Also, Japanese has no notion of case, but hiragana vs katakana
distinction is very similar to lower vs upper case.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Tue, 1 Dec 2015 16:42:01 +0600
> Cc: John Wiegley <jwiegley@gmail.com>,
> 	Andreas Röhler <andreas.roehler@online.de>,
> 	Drew Adams <drew.adams@oracle.com>, Emacs developers <emacs-devel@gnu.org>
> 
> > Case folding has nothing to do with ASCII.  Latin, Cyrillic, Greek,
> > Hebrew, Arabic, a lot of scripts have the concept of letter case, or
> > different forms for the same letter.
> 
> Also, Japanese has no notion of case, but hiragana vs katakana
> distinction is very similar to lower vs upper case.

While all of the above is true, case-folding in Emacs only folds
characters where the UCD defines case pairs.  AFAIK, this includes
neither Hebrew and Arabic, nor Katakana and Hiragana.  See
characters.el for the full story.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Tue, 1 Dec 2015 10:23:09 +0100
> Cc: John Wiegley <jwiegley@gmail.com>, Andreas Schwab <schwab@linux-m68k.org>,
> 	Drew Adams <drew.adams@oracle.com>
> 
> Notion resp. implementation of case-folding is unclear outside ASCII and related.

What is "related" to ASCII?  Do you mean Latin characters?  If so, the
notion of case-folding is clearly well beyond these: Cyrillic and
Greek characters are 2 obvious examples.

Moreover, the Unicode Character Database has other scripts that have
"small" and "capital" variants of letters: Armenian, Georgian, Coptic,
and a few old scripts.  Also, many symbols have small and capital
variants.  IOW, this attribute is in no way limited to ASCII and Latin
scripts.

Scripts that don't support case variants are not affected by
case-folding at all in Emacs.

> There must not be anything to be folded in a buffer. Folding is a notion which makes sense in special cases only - which are common in ASCII world, but not a reason for a default.

As mentioned before, this is contrary to a long-standing tradition in
Emacs: searches are case-insensitive by default, and not only for
ASCII characters.  Emacs is primarily a programmer's editor, and most
programming languages use ASCII characters for program source.  So by
your own logic above, case-folding should make a lot of sense in many
Emacs buffers.

> New users should not be bothered with this vast area.

I don't think they are bothered.  The defaults work seamlessly, no
bother is expected or required.

> GNU Linux is case-sensitive and Emacs should start by default like that.

I don't think the fact that Linux filesystems are case-sensitive is
relevant to searching buffer text for strings.  They are two entirely
unrelated features.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Fri, 27 Nov 2015 21:04:54 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> 
> This has been discussed somewhat, but I don't think there was
> any actual proposal to change the behavior.  So here goes.
> 
> Does it still make sense to make search and replacement
> use lax matching, i.e., fold stuff, by default?

I will repeat up front what I've already said numerous time in other
similar discussions: arguing about defaults in Emacs is largely a
waste of time.  It usually fails to produce any real effect except
tremendous loss of time and energy.  In all the years I've been
involved in Emacs development, I remember only one case of such a
discussion that ended up in real changes: the changes in default
colors of faces in Emacs 21.  But that was when the development team
was very small and everyone on it had the same perspective.  (It still
took a lot of time and argument.)

Given the ease with which you can change those defaults, the defaults
are almost irrelevant.  They are only significant to newcomers.
However, none of those who participate in such arguments is a
newcomer, so the only people for whom this matters are not here to
voice their opinions.

That said...

> I don't think it does.  I think that users, especially
> new users, would be less confused if Emacs defaulted to
> literal searching - no whitespace, case, "character",
> or other folding by default.

Most (if not all) other programs out there do fold by default, both
the letter-case and the equivalent characters.  So I don't think you
are right about newbie expectations.  Emacs follows (or rather
precedes, I think) what other similar programs do, so its defaults
should be more consistent with user expectations than a literal
search.

Re: lax matching is not a great default behavior

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I will repeat up front what I've already said numerous time in other similar
> discussions: arguing about defaults in Emacs is largely a waste of time. It
> usually fails to produce any real effect except tremendous loss of time and
> energy.

I agree with Eli that our development efforts are not well spent discussing
defaults already in the wild. For two major reasons:

 1. We don't actually know what newcomers find useful. There are different
    sorts of newcomers, with different expectations, so every decision is good
    for some and worse for others.

 2. "That ship has sailed." Everyone now used to the existing default would
    have to learn (a) why their Emacs just broke, and (b) how to fix it. And
    in many cases, the cause is far from the cure.

If a change in defaults can be backed by serious evidence that we made a huge
error in judgment in the past, I'd consider it. But if it's more of a shot in
the dark based on personal preferences, we should leave it alone.

John

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: lax matching is not a great default behavior

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The way to decide about changing the defaults is to poll the users.

People have recently made various changes in defaults without doing that.
For instance, making C-j in an isearch match spaces.  This change should
be reverted without delay.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, drew.adams@oracle.com, emacs-devel@gnu.org
> Date: Tue, 01 Dec 2015 09:40:55 -0500
> 
> The way to decide about changing the defaults is to poll the users.

Such a poll could only work if the behavior intended to become the
default is already available in released versions of Emacs, so users
could turn it on and try it.  This is not the case with character
folding, which is only available in development snapshots, and
actually is still in flux: it changes in non-trivial ways almost every
day.

If we are afraid users will hate this default, we can turn it off in
v25.1 and consider making it the default later.  Alternatively, we
could quickly release Emacs 25.2 with character folding turned off if
we see an outcry.  But polling at this time will not be efficient,
IMO.

> People have recently made various changes in defaults without doing that.
> For instance, making C-j in an isearch match spaces.  This change should
> be reverted without delay.

This was fixed quite some time ago.

Re: lax matching is not a great default behavior

[Mark Oteiza]

Eli Zaretskii <eliz@gnu.org> writes:
> If we are afraid users will hate this default, we can turn it off in
> v25.1 and consider making it the default later.  Alternatively, we
> could quickly release Emacs 25.2 with character folding turned off if
> we see an outcry.  But polling at this time will not be efficient,
> IMO.

I think the performance hit of char-folding is a good reason to disable
it. (Perhaps it's much worse for me since this emacs is not optimized.)

The fact that char-folding is so slow has bitten me many times when I
happen to isearch for a "long" string (rather, something that turns into
a large/slow regexp).  Emacs suddenly churning in isearch is surprising
and disruptive.

Sometimes this is just when I'm looking for a long symbol, in which case
I could `M-s _`--in any case, it is like having to work around
char-folding.

P.S. Regardless of the default, I think the variable
`search-default-regexp-mode' (or whatever variable ends up controlling
the behaviour) should be mentioned in NEWS.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: Mark Oteiza <mvoteiza@udel.edu>
> Date: Tue, 01 Dec 2015 13:49:46 -0500
> 
> I think the performance hit of char-folding is a good reason to disable
> it. (Perhaps it's much worse for me since this emacs is not optimized.)

Then turn it off in your sessions.  That's just the default, you don't
have to suffer if you don't like/need it.  (The release version is not
supposed to be unoptimized on end-users' machines, btw.)

> P.S. Regardless of the default, I think the variable
> `search-default-regexp-mode' (or whatever variable ends up controlling
> the behaviour) should be mentioned in NEWS.

It already is, has been there for quite some time.  But thanks for the
reminder anyway.

Re: lax matching is not a great default behavior

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mark Oteiza <mvoteiza@udel.edu>
>> Date: Tue, 01 Dec 2015 13:49:46 -0500
>> 
>> I think the performance hit of char-folding is a good reason to disable
>> it. (Perhaps it's much worse for me since this emacs is not optimized.)
>
> Then turn it off in your sessions.  That's just the default, you don't
> have to suffer if you don't like/need it.  (The release version is not
> supposed to be unoptimized on end-users' machines, btw.)

Optimization does not tend to make staggering differences.  There is
something to be said for Emacs developers trying to work with the
defaults and not just silently reconfigure them when they get in the way
of productivity not just because of old habits but because of actual
ergonomic or performance problems.

Developers are less likely to suffer from the "I guess it is supposed to
be that way" disease, so they in particular should think twice before
just reconfiguring.

-- 
David Kastrup

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Mark Oteiza <mvoteiza@udel.edu>,  emacs-devel@gnu.org
> Date: Tue, 01 Dec 2015 20:32:24 +0100
> 
> Optimization does not tend to make staggering differences.

In my measurements, an optimized binary is 2 to 3 times faster than an
unoptimized one.  That's something that is very visible in day-to-day
user experience.

> There is something to be said for Emacs developers trying to work
> with the defaults and not just silently reconfigure them when they
> get in the way of productivity not just because of old habits but
> because of actual ergonomic or performance problems.

I don't reconfigure them, FWIW.

> Developers are less likely to suffer from the "I guess it is supposed to
> be that way" disease, so they in particular should think twice before
> just reconfiguring.

Agreed.  But after thinking twice, there's no need to think the third
time.

Re: lax matching is not a great default behavior

[Mark Oteiza]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mark Oteiza <mvoteiza@udel.edu>
>> Date: Tue, 01 Dec 2015 13:49:46 -0500
>> 
>> I think the performance hit of char-folding is a good reason to disable
>> it. (Perhaps it's much worse for me since this emacs is not optimized.)
>
> Then turn it off in your sessions.  That's just the default, you don't
> have to suffer if you don't like/need it.

I thought the discussion was about the default value.

Re: lax matching is not a great default behavior

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

On 1 Dec 2015 6:49 pm, "Mark Oteiza" <mvoteiza@udel.edu> wrote:
> I think the performance hit of char-folding is a good reason to disable
> it. (Perhaps it's much worse for me since this emacs is not optimized.)

I honestly didn't notice a performance hit, and I compile without
optimizations. Is there a particular search-term/buffer-contents that lead
to your problem?

[-- Attachment #2: Type: text/html, Size: 482 bytes --]

Re: lax matching is not a great default behavior

[Mark Oteiza]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 1 Dec 2015 6:49 pm, "Mark Oteiza" <mvoteiza@udel.edu> wrote:
>> I think the performance hit of char-folding is a good reason to disable
>> it. (Perhaps it's much worse for me since this emacs is not optimized.)
>
> I honestly didn't notice a performance hit, and I compile without
> optimizations. Is there a particular search-term/buffer-contents that
> lead to your problem?

Example of a long symbol in elisp: try searching for
"tty-set-up-initial-frame-faces" in faces.el

Searching for "free software foundation" in NEWS is also very slow.

(Yes, I know symbol search and word search are things)

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: Mark Oteiza <mvoteiza@udel.edu>
> Date: Tue, 01 Dec 2015 14:51:48 -0500
> 
> Example of a long symbol in elisp: try searching for
> "tty-set-up-initial-frame-faces" in faces.el
> 
> Searching for "free software foundation" in NEWS is also very slow.

The first match is almost instantaneous here; it's the next one,
especially if it fails, is slow.

So it doesn't seem to be the search itself, it's something else that's
at work here.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Tue, 01 Dec 2015 22:01:04 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> The first match is almost instantaneous here; it's the next one,
> especially if it fails, is slow.
> 
> So it doesn't seem to be the search itself, it's something else that's
> at work here.

Seems to be lazy-highlight.  Set isearch-lazy-highlight to nil, and
Bob's your uncle.

Re: lax matching is not a great default behavior

[Artur Malabarba]

2015-12-01 20:04 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Tue, 01 Dec 2015 22:01:04 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>>
>> The first match is almost instantaneous here; it's the next one,
>> especially if it fails, is slow.
>>
>> So it doesn't seem to be the search itself, it's something else that's
>> at work here.
>
> Seems to be lazy-highlight.  Set isearch-lazy-highlight to nil, and
> Bob's your uncle.

If lazy-highlighting adds a lag before the user can move to the second
match, then we need to fix lazy-highlighting to not block input.

RE: lax matching is not a great default behavior

[Drew Adams]

> > Seems to be lazy-highlight.  Set isearch-lazy-highlight to nil, and
> > Bob's your uncle.
> 
> If lazy-highlighting adds a lag before the user can move to the second
> match, then we need to fix lazy-highlighting to not block input.

Change the value of `laxy-highlight-max-at-a-time' to 1, to see no
blocking (no competition with reading input).  If you have it set
to, say, 20, then up to 20 isearch matches will be lazy-highlighted
before an already pressed additional `C-s' shows its effect by
moving to the next search hit.

This is very easy to see with symmetric folding turned on.  Lazy
highlighting occurs in bursts of `laxy-highlight-max-at-a-time'
search-occurrence highlights, and after each burst ordinary
search processing continues.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Tue, 1 Dec 2015 23:31:18 +0000
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: Mark Oteiza <mvoteiza@udel.edu>, emacs-devel <emacs-devel@gnu.org>
> 
> > Seems to be lazy-highlight.  Set isearch-lazy-highlight to nil, and
> > Bob's your uncle.
> 
> If lazy-highlighting adds a lag before the user can move to the second
> match, then we need to fix lazy-highlighting to not block input.

Yes, but do you understand why it's a problem anyway?  Lazy hilighting
uses the same regexp search functions and the same regexp as the code
that find the first hit.  We are talking about 2 cases where there's
no other matches for that regexp.  So how come finding out that there
are no more matches is almost instantaneous, but looking for them as
part of lazy-highlighting is so slow?

Re: lax matching is not a great default behavior

[martin rudalics]

 >> If lazy-highlighting adds a lag before the user can move to the second
 >> match, then we need to fix lazy-highlighting to not block input.
 >
 > Yes, but do you understand why it's a problem anyway?  Lazy hilighting
 > uses the same regexp search functions and the same regexp as the code
 > that find the first hit.  We are talking about 2 cases where there's
 > no other matches for that regexp.  So how come finding out that there
 > are no more matches is almost instantaneous, but looking for them as
 > part of lazy-highlighting is so slow?

I'd blame the overlay mechanism.

martin

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Wed, 02 Dec 2015 09:23:17 +0100
> From: martin rudalics <rudalics@gmx.at>
> Cc: mvoteiza@udel.edu, emacs-devel@gnu.org
> 
>  >> If lazy-highlighting adds a lag before the user can move to the second
>  >> match, then we need to fix lazy-highlighting to not block input.
>  >
>  > Yes, but do you understand why it's a problem anyway?  Lazy hilighting
>  > uses the same regexp search functions and the same regexp as the code
>  > that find the first hit.  We are talking about 2 cases where there's
>  > no other matches for that regexp.  So how come finding out that there
>  > are no more matches is almost instantaneous, but looking for them as
>  > part of lazy-highlighting is so slow?
> 
> I'd blame the overlay mechanism.

It worked just fine for me 2 days ago.

Re: lax matching is not a great default behavior

[Artur Malabarba]

2015-12-02 3:37 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Tue, 1 Dec 2015 23:31:18 +0000
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>> Cc: Mark Oteiza <mvoteiza@udel.edu>, emacs-devel <emacs-devel@gnu.org>
>>
>> > Seems to be lazy-highlight.  Set isearch-lazy-highlight to nil, and
>> > Bob's your uncle.
>>
>> If lazy-highlighting adds a lag before the user can move to the second
>> match, then we need to fix lazy-highlighting to not block input.
>
> Yes, but do you understand why it's a problem anyway?  Lazy hilighting
> uses the same regexp search functions and the same regexp as the code
> that find the first hit.  We are talking about 2 cases where there's
> no other matches for that regexp.  So how come finding out that there
> are no more matches is almost instantaneous, but looking for them as
> part of lazy-highlighting is so slow?

Yes. lazy-highlight is definitely doing something wrong. If I can hit
C-s and instantly be told "no more matches", lazy-highlight shouldn't
be slow at all.
I just tested here and the same happens for me.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Wed, 2 Dec 2015 13:06:22 +0000
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: Mark Oteiza <mvoteiza@udel.edu>, emacs-devel <emacs-devel@gnu.org>
> 
> Yes. lazy-highlight is definitely doing something wrong. If I can hit
> C-s and instantly be told "no more matches", lazy-highlight shouldn't
> be slow at all.
> I just tested here and the same happens for me.

Moreover, a yesterday's morning build doesn't seem to exhibit the
problem, so it's something that was changed during the last 36 hours.
61a4b57, perhaps?

Which now raises a doubt whether this is indeed what annoyed Mark,
since I don't believe that impression was based on a single-day
experience.

Re: lax matching is not a great default behavior

[Artur Malabarba]

2015-12-02 13:44 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
>> Yes. lazy-highlight is definitely doing something wrong. If I can hit
>> C-s and instantly be told "no more matches", lazy-highlight shouldn't
>> be slow at all.
>> I just tested here and the same happens for me.
>
> Moreover, a yesterday's morning build doesn't seem to exhibit the
> problem, so it's something that was changed during the last 36 hours.
> 61a4b57, perhaps?

Yes. If I remove multi-char matching the lazy-highlight problem is gone too.

> Which now raises a doubt whether this is indeed what annoyed Mark,
> since I don't believe that impression was based on a single-day
> experience.

Indeed. Perhaps Mark can clarify for us exactly what slow behaviour he
had been experiencing.

RE: lax matching is not a great default behavior

[Drew Adams]

> > The first match is almost instantaneous here; it's the next one,
> > especially if it fails, is slow.
> >
> > So it doesn't seem to be the search itself, it's something else
> > that's at work here.
> 
> Seems to be lazy-highlight.  Set isearch-lazy-highlight to nil, and
> Bob's your uncle.

Of course then only one hit is found at a time.  Naturally that
makes a big difference (always, though it's generally not so
noticeable for non-folded searching).

The same is true, BTW, for the symmetric char folding I added.

It's the reason I have this in the Commentary:

;; Be aware that character-fold searching can be much slower when
;; symmetric - there are many more possibilities to search for.
;; If, for example, you search only for a single "e"-family
;; character then every "e" in the buffer is a search hit (which
;; means lazy-highlighting them all, by default).  Searching with
;; a longer search string is much faster.
;;
;; If you also use library `isearch+.el' then you can turn off lazy
;; highlighting using the toggle key `M-s h L'.  This can vastly
;; improve performance when character folding is symmetric.

Toggling lazy highlighting off makes even symmetric folding snappy,
and even for a one-char string such as é, which (symmetrically)
matches every "e"-like character.

Re: lax matching is not a great default behavior

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Such a poll could only work if the behavior intended to become the
  > default is already available in released versions of Emacs, so users
  > could turn it on and try it.  This is not the case with character
  > folding, which is only available in development snapshots, and
  > actually is still in flux: it changes in non-trivial ways almost every
  > day.

  > If we are afraid users will hate this default, we can turn it off in
  > v25.1 and consider making it the default later.

That seems like the right approach.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: lax matching is not a great default behavior

[Per Starbäck]

> If we are afraid users will hate this default, we can turn it off in
> v25.1 and consider making it the default later.

That would be good, together with introducing an entry for it in the
Options menu directly below (or above?) "Ignore case for search". This
deserves to be as visible as that option. Should it be "Ignore accents
for search"?

I think it's often best to make something just available first to get
more feedback on it before it's made the default. (One thing I expect
from that feedback is bug reports about how that option doesn't work
for some characters for some languages, as I've written about in
another thread, which will give an indication of whether the problem
is as serious as I made it out to be.)

> Alternatively, we
> could quickly release Emacs 25.2 with character folding turned off if
> we see an outcry.  But polling at this time will not be efficient,
> IMO.

Not at all as good! To "quickly release" something doesn't mean that
it is a quick change for users, who may keep using that version for a
long time.

RE: lax matching is not a great default behavior

[Drew Adams]

> > If we are afraid users will hate this default, we can turn it off in
> > v25.1 and consider making it the default later.
> 
> That would be good, together with introducing an entry for it in the
> Options menu directly below (or above?) "Ignore case for search". This
> deserves to be as visible as that option. Should it be "Ignore accents
> for search"?
> 
> I think it's often best to make something just available first to get
> more feedback on it before it's made the default...
> 
> > Alternatively, we could quickly release Emacs 25.2 with character
> > folding turned off if we see an outcry.  But polling at this time
> > will not be efficient, IMO.
> 
> Not at all as good! To "quickly release" something doesn't mean that
> it is a quick change for users, who may keep using that version for a
> long time.

100% agreed.

Re: lax matching is not a great default behavior

[Artur Malabarba]

Drew Adams <drew.adams@oracle.com> writes:

>> > If we are afraid users will hate this default, we can turn it off in
>> > v25.1 and consider making it the default later.
>>
>> I think it's often best to make something just available first to get
>> more feedback on it before it's made the default...
>> 
> 100% agreed.

I don't mind leaving this OFF by default in Emacs 25. So long as the
eventual goal is to have it ON by default (preferably in 26).

Per Starbäck <per@starback.se> writes:
>> That would be good, together with introducing an entry for it in the
>> Options menu directly below (or above?) "Ignore case for search". This
>> deserves to be as visible as that option. Should it be "Ignore accents
>> for search"?

Feel free to bring this up on a separate topic, Per. A more
user-friendly name for this feature is something that's been brought up
before and we're not going to reach anything as a side-note in the
current thread.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Thu, 3 Dec 2015 23:27:00 +0100
> From: Per Starbäck <per@starback.se>
> Cc: rms@gnu.org, jwiegley@gmail.com, Drew Adams <drew.adams@oracle.com>, 
> 	"emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> > If we are afraid users will hate this default, we can turn it off in
> > v25.1 and consider making it the default later.
> 
> That would be good

I see no real reasons yet for such a decision.  Character folding was
introduced with the explicit goal of giving users what the other
text-editing and word-processing environments provide, what they
therefore are expected to expect.  To revert that decision will take
more than just "I think it's wrong" kind of posts.

> Should it be "Ignore accents for search"?

No, because ignoring accents is just a small part of character
folding.  Please take a look at character-fold.el for the details.

> > Alternatively, we
> > could quickly release Emacs 25.2 with character folding turned off if
> > we see an outcry.  But polling at this time will not be efficient,
> > IMO.
> 
> Not at all as good! To "quickly release" something doesn't mean that
> it is a quick change for users, who may keep using that version for a
> long time.

If they are annoyed by a feature, they will upgrade quickly, I think.

Re: lax matching is not a great default behavior

[Per Starbäck]

>> > If we are afraid users will hate this default, we can turn it off in
>> > v25.1 and consider making it the default later.
>>
>> That would be good
>
> I see no real reasons yet for such a decision.  Character folding was
> introduced with the explicit goal of giving users what the other
> text-editing and word-processing environments provide, what they
> therefore are expected to expect.  To revert that decision will take
> more than just "I think it's wrong" kind of posts.

I didn't follow its creation, but I don't think users generally expect
that (yet). (I just checked searches in Gedit and Firefox where there
were no such features, at least not in the versions that are standard
in my operating system distribution.)

Not that I think that matters a lot. I think a good reason to
introduce character folding is because it's a good feature, simple as
that. But it needs to tried out more and get more feedback from
different locales before made into the default.

I may have missed something, but I have not read a single "I think
it's wrong" post. I've read that making the feature available to users
first will make it possible to have a poll before changing a default
that is a massive change, and I've read my own examples of how the
American-centered assumptions are just wrong in some situations. There
is probably more feedback of a similar kind. By enabling this feature
it will be possible to get that feedback, without the outcry that
comes with changing the default. By ironing out the wrinkles this will
be a welcome change when the default later is changed.

You have to realize that this is a *massive* change, even though it
may not feel so for someone who almost only writes in English.


>> Should it be "Ignore accents for search"?
>
> No, because ignoring accents is just a small part of character
> folding.  Please take a look at character-fold.el for the details.

I know, but it has to be called something. Do you have a better suggestion?

>> > Alternatively, we
>> > could quickly release Emacs 25.2 with character folding turned off if
>> > we see an outcry.  But polling at this time will not be efficient,
>> > IMO.
>>
>> Not at all as good! To "quickly release" something doesn't mean that
>> it is a quick change for users, who may keep using that version for a
>> long time.
>
> If they are annoyed by a feature, they will upgrade quickly, I think.

That kind of user will rather change their options themselves. I'm not
primarily talking about people installing Emacs themselves, but those
who use a version their system adminstrator or the OS distribution
provider installed for them.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Fri, 4 Dec 2015 10:33:14 +0100
> From: Per Starbäck <per@starback.se>
> Cc: rms@gnu.org, jwiegley@gmail.com, Drew Adams <drew.adams@oracle.com>, 
> 	"emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> > I see no real reasons yet for such a decision.  Character folding was
> > introduced with the explicit goal of giving users what the other
> > text-editing and word-processing environments provide, what they
> > therefore are expected to expect.  To revert that decision will take
> > more than just "I think it's wrong" kind of posts.
> 
> I didn't follow its creation, but I don't think users generally expect
> that (yet). (I just checked searches in Gedit and Firefox where there
> were no such features, at least not in the versions that are standard
> in my operating system distribution.)

Try more serious editing environments.  E.g., MS Word does that by
default.

> Not that I think that matters a lot. I think a good reason to
> introduce character folding is because it's a good feature, simple as
> that. But it needs to tried out more and get more feedback from
> different locales before made into the default.

The entire time interval between Nov 15 this year and until we release
Emacs 25.1 (which will take a few months, probably more than 6,
judging by past experience) is supposed to provide that feedback.  All
it takes to turn this off by default is changing the default value of
a single variable (and change a couple of places in the User Manual to
reflect that).  Once we decide to do that, it can be done very quickly
and easily.  We can do that a day before the release, if we want to.

OTOH, turning it off today means that it will get much less testing,
and therefore bugs related to it (like the one reported just today in
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22090) will most probably
remain hidden for who knows how long.

> I may have missed something, but I have not read a single "I think
> it's wrong" post.

Any post that doesn't explain why folding characters might be _wrong_
in _most_ situations is not providing any useful arguments for turning
off the default.  Most posts I've seen explained why their authors
don't like this feature.

> I've read that making the feature available to users
> first will make it possible to have a poll before changing a default
> that is a massive change, and I've read my own examples of how the
> American-centered assumptions are just wrong in some situations. There
> is probably more feedback of a similar kind. By enabling this feature
> it will be possible to get that feedback, without the outcry that
> comes with changing the default. By ironing out the wrinkles this will
> be a welcome change when the default later is changed.
> 
> You have to realize that this is a *massive* change, even though it
> may not feel so for someone who almost only writes in English.

I do realize it's a massive change.  And you are wrong assuming that I
almost only write in English (look at my locale), let alone that this
is some American-centered view (which would have dictated exactly the
opposite default).

In any case, introducing massive changes that are turned on by default
is nothing new in Emacs development.  Bidirectional display engine
introduced in Emacs 24.1 comes to mind; it certainly was much more
massive than this one.  And turning that one off was nowhere as simple
as turning character folding off, so the risk was much higher.  We did
it anyway, because we thought that was TRT to do, and because we
wanted any bugs and adverse side effects of that change found and
fixed before the release.  Likewise here.

> >> Should it be "Ignore accents for search"?
> >
> > No, because ignoring accents is just a small part of character
> > folding.  Please take a look at character-fold.el for the details.
> 
> I know, but it has to be called something. Do you have a better suggestion?

Either "Character Folding in Search" or maybe "Character Equivalence
in Search".  (I'm not good at finding short descriptive names.)

> >> > Alternatively, we
> >> > could quickly release Emacs 25.2 with character folding turned off if
> >> > we see an outcry.  But polling at this time will not be efficient,
> >> > IMO.
> >>
> >> Not at all as good! To "quickly release" something doesn't mean that
> >> it is a quick change for users, who may keep using that version for a
> >> long time.
> >
> > If they are annoyed by a feature, they will upgrade quickly, I think.
> 
> That kind of user will rather change their options themselves. I'm not
> primarily talking about people installing Emacs themselves, but those
> who use a version their system adminstrator or the OS distribution
> provider installed for them.

A misfeature that causes an outcry will prompt sysadmins to upgrade, I
think.

Re: lax matching is not a great default behavior

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 4 Dec 2015 10:33:14 +0100
>> From: Per Starbäck <per@starback.se>
>> Cc: rms@gnu.org, jwiegley@gmail.com, Drew Adams <drew.adams@oracle.com>, 
>> 	"emacs-devel@gnu.org" <emacs-devel@gnu.org>
>> 
>> > I see no real reasons yet for such a decision.  Character folding was
>> > introduced with the explicit goal of giving users what the other
>> > text-editing and word-processing environments provide, what they
>> > therefore are expected to expect.  To revert that decision will take
>> > more than just "I think it's wrong" kind of posts.
>> 
>> I didn't follow its creation, but I don't think users generally expect
>> that (yet). (I just checked searches in Gedit and Firefox where there
>> were no such features, at least not in the versions that are standard
>> in my operating system distribution.)
>
> Try more serious editing environments.  E.g., MS Word does that by
> default.

That's not editing but text processing.  So we are talking about
text-mode defaults here rather than general editing defaults.

-- 
David Kastrup

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 04 Dec 2015 11:57:45 +0100
> Cc: jwiegley@gmail.com, Per Starbäck <per@starback.se>,
> 	rms@gnu.org, drew.adams@oracle.com, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I didn't follow its creation, but I don't think users generally expect
> >> that (yet). (I just checked searches in Gedit and Firefox where there
> >> were no such features, at least not in the versions that are standard
> >> in my operating system distribution.)
> >
> > Try more serious editing environments.  E.g., MS Word does that by
> > default.
> 
> That's not editing but text processing.  So we are talking about
> text-mode defaults here rather than general editing defaults.

Making character folding the default in Text Mode and its descendants
is a possibility we should consider, yes.  Though we should also
consider the fact that prog-mode and its descendants are meant for
editing program sources, where comments and strings with
human-readable text are not uncommon.

Re: lax matching is not a great default behavior

[Per Starbäck]

>> > I see no real reasons yet for such a decision.  Character folding was
>> > introduced with the explicit goal of giving users what the other
>> > text-editing and word-processing environments provide, what they
>> > therefore are expected to expect.  To revert that decision will take
>> > more than just "I think it's wrong" kind of posts.
>>
>> I didn't follow its creation, but I don't think users generally expect
>> that (yet). (I just checked searches in Gedit and Firefox where there
>> were no such features, at least not in the versions that are standard
>> in my operating system distribution.)
>
> Try more serious editing environments.  E.g., MS Word does that by
> default.

I don't have access to MS Word, I don't think experiences from that
influences what users expect in an editor much, and I think it is
beside the point anyway. A good feature is good anyway.

> The entire time interval between Nov 15 this year and until we release
> Emacs 25.1 (which will take a few months, probably more than 6,
> judging by past experience) is supposed to provide that feedback.

The main feedback won't come until after it is available in a released version.
What we were talking about is what should be the default in the next
released version. Since you are instead talking about the default
*before release* we are maybe not in disagreement after all.

>> I may have missed something, but I have not read a single "I think
>> it's wrong" post.
>
> Any post that doesn't explain why folding characters might be _wrong_
> in _most_ situations is not providing any useful arguments for turning
> off the default.  Most posts I've seen explained why their authors
> don't like this feature.

Big problems are big even if they only affect 5% of the users. The
example I have given where this feature for Scandinavians is like
having a search for "I" find "J" is such a problem. That just isn't
acceptable, and will affect _most_ editing session for some, but of
course not _most_ situations for all users combined. Some adaptations
by language are needed to make this a good feature, and that won't be
there in place for the next release, and also we won't know which
adaptations are needed for other languages until more get to use this.

And this is strange. I haven't read a single such post, and yet it's
most posts you have seen. I wonder if you have read that into any post
by me.

> I do realize it's a massive change.  And you are wrong assuming that I
> almost only write in English (look at my locale), let alone that this
> is some American-centered view (which would have dictated exactly the
> opposite default).

I'm not assuming that (and know that it's not so). This is not
personal, but Emacs as a whole is American-centered. Its understanding
of the needs for other languages that don't use a Latin script has
earlier proved to be a better than its understanding of the needs for
other languages using a Latin script.
(And no, it certainly wouldn't. The harder you have to type áàẩããǎ the
better it is for you to be able to search for them easily when they
happen to be in a buffer.)

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Fri, 4 Dec 2015 13:04:24 +0100
> From: Per Starbäck <per@starback.se>
> Cc: John Wiegley <jwiegley@gmail.com>, rms@gnu.org, Drew Adams <drew.adams@oracle.com>, 
> 	"emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> > Try more serious editing environments.  E.g., MS Word does that by
> > default.
> 
> I don't have access to MS Word, I don't think experiences from that
> influences what users expect in an editor much, and I think it is
> beside the point anyway. A good feature is good anyway.

Given the sheer number of people who use (or have to use) Word, this
is hardly beside the point.

> > The entire time interval between Nov 15 this year and until we release
> > Emacs 25.1 (which will take a few months, probably more than 6,
> > judging by past experience) is supposed to provide that feedback.
> 
> The main feedback won't come until after it is available in a released version.

I disagree.  I've seen that happening during pretest many times.
Heck, even the discussion we are having now is part of that feedback.
And I don't expect it to be the last discussion about this.

> What we were talking about is what should be the default in the next
> released version. Since you are instead talking about the default
> *before release* we are maybe not in disagreement after all.

For now, the release is still so far away that any difference between
these 2 alternatives is indiscernible, IMO.

> >> I may have missed something, but I have not read a single "I think
> >> it's wrong" post.
> >
> > Any post that doesn't explain why folding characters might be _wrong_
> > in _most_ situations is not providing any useful arguments for turning
> > off the default.  Most posts I've seen explained why their authors
> > don't like this feature.
> 
> Big problems are big even if they only affect 5% of the users.

Yes, but we don't change the defaults on behalf of such a small
minority, except to prevent a disaster.  I see no disaster in this
case.

> The example I have given where this feature for Scandinavians is
> like having a search for "I" find "J" is such a problem.

Not "for Scandinavians", in text written in one of the Scandinavian
languages.  Right?

> That just isn't acceptable, and will affect _most_ editing session
> for some, but of course not _most_ situations for all users
> combined.

Then these users will customize their Emacs, and move on.

> Some adaptations by language are needed to make this a
> good feature, and that won't be there in place for the next release,
> and also we won't know which adaptations are needed for other
> languages until more get to use this.

Indeed.  But I don't see that as a sufficient reason to decide now
that the default should off for _everyone_.

> And this is strange. I haven't read a single such post, and yet it's
> most posts you have seen. I wonder if you have read that into any post
> by me.

I read everything that is posted to this list.

> Emacs as a whole is American-centered

Emacs ceased being American-centered long ago, around v20.1, I'd say,
if not earlier.

> Its understanding of the needs for other languages that don't use a
> Latin script has earlier proved to be a better than its
> understanding of the needs for other languages using a Latin script.

If you really think that, I'd encourage bug reports about any feature
or misfeature where this attitude is visible in practice.

> The harder you have to type áàẩããǎ the better it is for you to be
> able to search for them easily when they happen to be in a buffer.

There was at least one opposite view expressed in this discussion.
Which just tells us that one size will not fit all in the is regard,
there's nothing new.  But it tells nothing about which default is more
correct.  The default you propose has disadvantages, not just
advantages.

Re: lax matching is not a great default behavior

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> For now, the release is still so far away that any difference between these
> 2 alternatives is indiscernible, IMO.

Let's not assume that the next release is quite that far away.  I would like
to see it completed by June 2016, if that is possible.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Per Starbäck <per@starback.se>,  jwiegley@gmail.com,
>   rms@gnu.org,  drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Fri, 04 Dec 2015 10:47:14 -0700
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > For now, the release is still so far away that any difference between these
> > 2 alternatives is indiscernible, IMO.
> 
> Let's not assume that the next release is quite that far away.  I would like
> to see it completed by June 2016, if that is possible.

Yes.  I think I said 6 months elsewhere, so there's no contradiction.

RE: lax matching is not a great default behavior

[Drew Adams]

> > > If we are afraid users will hate this default, we can turn it
> > > off in v25.1 and consider making it the default later.
> >
> > That would be good

You snipped the rest of Per's point there, which makes a
difference, I think:

> > , together with introducing an entry for it in the Options
> > menu directly below (or above?) "Ignore case for search".
> > This deserves to be as visible as that option.

> I see no real reasons yet for such a decision. ... To revert
> that [other] decision will take more than just "I think it's
> wrong" kind of posts.

So, no real reasons have appeared yet for a decision in favor
of the (longstanding) default behavior that you don't agree with.

But a decision has already been made in favor of the default
you do agree with?  And now you're entertaining arguments to
"revert that decision"?  _Has_ such a decision to change the
default behavior in fact already been made?

> Character folding was introduced with the explicit goal of giving
> users what the other text-editing and word-processing environments
> provide, what they therefore are expected to expect.

So what?  So was CUA mode and lots of other features that are
not turned on by default.  Being introduced for such a reason
is not by itself a sufficient reason, let alone a "decision",
that it should immediately become the new default behavior.

> In any case, introducing massive changes that are turned on by default
> is nothing new in Emacs development.  Bidirectional display engine
> introduced in Emacs 24.1 comes to mind; it certainly was much more
> massive than this one.  And turning that one off was nowhere as simple
> as turning character folding off, so the risk was much higher.  We did
> it anyway, because we thought that was TRT to do, and because we
> wanted any bugs and adverse side effects of that change found and
> fixed before the release.  Likewise here.

Bidi did not noticeably affect users who do not edit bidi text.
Had it done so, it is not so clear that it would have been turned
on by default.

I think that a much better comparison is CUA mode.  You argue
that we should turn char folding on by default _because_ that's
what users of other editors are used to.  Most users of other
editors are used to CUA-like behavior too.  Yet we don't turn
that on by default (and I agree with that decision).

Turning char folding on by default might well be the best thing
to do at some point, but I see no reason to rush to that.

> > Should it be "Ignore accents for search"?
> 
> No, because ignoring accents is just a small part of character
> folding.  Please take a look at character-fold.el for the details.

Agreed.  And neither is it folding of diacriticals, because there
are also ad hoc foldings (e.g., quote marks).  And there will
likely be more to come.  It is, in fact, a hodge podge of foldings
- pretty much all of the various char foldings provided by Emacs
so far, except for letter case.

In such a situation, only a vague term such as "character folding"
or "miscellaneous character folding" can characterize it.  Until
we perhaps divide it into different folding groups, which each
can have a specific name that says something, we can only, I think,
give it such a catch-all name.  That's OK, IMO.

> > > Alternatively, we
> > > could quickly release Emacs 25.2 with character folding turned off if
> > > we see an outcry.  But polling at this time will not be efficient,
> > > IMO.
> >
> > Not at all as good! To "quickly release" something doesn't mean that
> > it is a quick change for users, who may keep using that version for a
> > long time.
> 
> If they are annoyed by a feature, they will upgrade quickly, I think.

What's the crying need to do it this way?  Why not leave it off by
default, for now?  Make it available, advertise it, put it in the
Options menu, give it a toggle key, and see how users like it.  It's
not hard to _later_ make it the default behavior.  What's the rush?

> Most posts I've seen explained why their authors don't like this
> feature.

Well I, for one, very much like this feature.  But I don't think
we should precipitously turn it on by default.

> turning it off today means that it will get much less testing,
> and therefore bugs related to it...will most probably remain
> hidden for who knows how long.

I seriously doubt that.  That sounds alarmist, to me.  It is
trivial to toggle it on, and I am _sure_, given its usefulness,
that it will get sufficiently used ("tested") after the release
that we can get a good idea of whether, later, we want to turn
it on by default.

My expectation, if we turn it off by default, is that users will
try it, like it, and possibly ask for it to become the default
behavior.  There is no reason to jump the gun on this.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Fri, 4 Dec 2015 07:55:34 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: jwiegley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> > > > If we are afraid users will hate this default, we can turn it
> > > > off in v25.1 and consider making it the default later.
> > >
> > > That would be good
> 
> You snipped the rest of Per's point there, which makes a
> difference, I think:
> 
> > > , together with introducing an entry for it in the Options
> > > menu directly below (or above?) "Ignore case for search".
> > > This deserves to be as visible as that option.

I don't see how that is relevant.  Menu options have very little
relevance to the defaults.  And yes, I agree that there should be such
an option in the menu bar's menus.

> _Has_ such a decision to change the default behavior in fact already
> been made?

Yes.  The code that makes it the default didn't just write itself.

> > Character folding was introduced with the explicit goal of giving
> > users what the other text-editing and word-processing environments
> > provide, what they therefore are expected to expect.
> 
> So what?  So was CUA mode and lots of other features that are
> not turned on by default.

Yes, there are examples to the contrary as well.  But that's besides
the point.  The point in that part of the discussion was the claim
that features that mean massive changes must never be made ON by
default.  To refute that, all I need is a single significant example
to the contrary.  Which is what I provided.

> Bidi did not noticeably affect users who do not edit bidi text.

Likewise with character folding: as long as you search text where no
equivalent characters exist, you will see no difference at all.

> I think that a much better comparison is CUA mode.  You argue
> that we should turn char folding on by default _because_ that's
> what users of other editors are used to.  Most users of other
> editors are used to CUA-like behavior too.  Yet we don't turn
> that on by default (and I agree with that decision).

I'm not going to start arguing about CUA.  I will just say that CUA
was problematic because it actively _conflicted_ with many basic Emacs
keybindings.  That was the single most important problem that
justified its being off by default.  There's no such incompatibility
in the case in point.

> > > Should it be "Ignore accents for search"?
> > 
> > No, because ignoring accents is just a small part of character
> > folding.  Please take a look at character-fold.el for the details.
> 
> Agreed.  And neither is it folding of diacriticals, because there
> are also ad hoc foldings (e.g., quote marks).  And there will
> likely be more to come.  It is, in fact, a hodge podge of foldings
> - pretty much all of the various char foldings provided by Emacs
> so far, except for letter case.

Actually, it's not a hodge-podge at all.  Barring any user-level
customizations, it can be formally defined (and has been defined
elsewhere) what is and what isn't folded.

> Why not leave it off by default, for now?

"Why not" is not a compelling argument, sorry.  It cannot win the "why
not" argument in the other direction.

> > turning it off today means that it will get much less testing,
> > and therefore bugs related to it...will most probably remain
> > hidden for who knows how long.
> 
> I seriously doubt that.  That sounds alarmist, to me.

This is in fact based on actual experience of testing new features in
Emacs, during several pretests of a few major releases.

> My expectation, if we turn it off by default, is that users will
> try it, like it, and possibly ask for it to become the default
> behavior.

OTOH, if we turn it off by default, users might not even find it or
know it exists for another 5 years.

RE: lax matching is not a great default behavior

[Drew Adams]

> > > No, because ignoring accents is just a small part of character
> > > folding.  Please take a look at character-fold.el for the details.
> >
> > Agreed.  And neither is it folding of diacriticals, because there
> > are also ad hoc foldings (e.g., quote marks).  And there will
> > likely be more to come.  It is, in fact, a hodge podge of foldings
> > - pretty much all of the various char foldings provided by Emacs
> > so far, except for letter case.
> 
> Actually, it's not a hodge-podge at all.  Barring any user-level
> customizations, it can be formally defined (and has been defined
> elsewhere) what is and what isn't folded.

Whether it is formally defined or not does not answer the
question about the name to use for Emacs users.

The behavior is a combination of diacritical folding and
some ad hoc foldings.  Do you have a _specific_ name for it,
even one coming from the formal definition?  And if so, is
that name a good one for Emacs users?

AFAICT, "character folding" is as good as we've come up
with, so far - not some specific kind of character folding.
And this is because the behavior is not so straightforward
as just folding diacriticals.

> > Why not leave it off by default, for now?
> 
> "Why not" is not a compelling argument, sorry.  It cannot
> win the "why not" argument in the other direction.

"Why change the default?" is precisely the question Emacs
dev generally asks itself.  That is, why not leave it
unchanged?  It is default change that should be argued
for.  No "compelling" argument for default change should
mean we leave the default alone.

The question should not be "Why not change the default?".

> > > turning it off today means that it will get much less testing,
> > > and therefore bugs related to it...will most probably remain
> > > hidden for who knows how long.
> >
> > I seriously doubt that.  That sounds alarmist, to me.
> 
> This is in fact based on actual experience of testing new 
> features in Emacs, during several pretests of a few major
> releases.

No one is arguing that it will get less testing during
pretest if you turn it on during pretest.  You are turning
things on their head.

The question is about the default for the release, not
whether it should be tested or turned on for pretests.

This has been stated more than once now by more than one
person.  But you keep giving the argument that turning it
on for pretesting is beneficial.  So it is.  So turn it
on for pretesting, to get more feedback, and off for
the release.

We will continue to get feedback after the release even
if it is turned off.  And later (e.g. for the following
release) we can make a better judgment than any that can
be made now or during the pretest for this release.

> > My expectation, if we turn it off by default, is that users will
> > try it, like it, and possibly ask for it to become the default
> > behavior.
> 
> OTOH, if we turn it off by default, users might not even find it or
> know it exists for another 5 years.

Wanna bet? ;-)

Re: lax matching is not a great default behavior

[David Kastrup]

Drew Adams <drew.adams@oracle.com> writes:

>> > > No, because ignoring accents is just a small part of character
>> > > folding.  Please take a look at character-fold.el for the details.
>> >
>> > Agreed.  And neither is it folding of diacriticals, because there
>> > are also ad hoc foldings (e.g., quote marks).  And there will
>> > likely be more to come.  It is, in fact, a hodge podge of foldings
>> > - pretty much all of the various char foldings provided by Emacs
>> > so far, except for letter case.
>> 
>> Actually, it's not a hodge-podge at all.  Barring any user-level
>> customizations, it can be formally defined (and has been defined
>> elsewhere) what is and what isn't folded.
>
> Whether it is formally defined or not does not answer the
> question about the name to use for Emacs users.
>
> The behavior is a combination of diacritical folding and
> some ad hoc foldings.  Do you have a _specific_ name for it,
> even one coming from the formal definition?  And if so, is
> that name a good one for Emacs users?
>
> AFAICT, "character folding" is as good as we've come up
> with, so far - not some specific kind of character folding.

How about "fuzzy matching"?

-- 
David Kastrup

Re: lax matching is not a great default behavior

[Artur Malabarba]

2015-12-04 22:16 GMT+00:00 David Kastrup <dak@gnu.org>:
>> AFAICT, "character folding" is as good as we've come up
>> with, so far - not some specific kind of character folding.
>
> How about "fuzzy matching"?

That means something else to most people: to be able to skip some
characters and still match.

Re: lax matching is not a great default behavior

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-12-04 22:16 GMT+00:00 David Kastrup <dak@gnu.org>:
>>> AFAICT, "character folding" is as good as we've come up
>>> with, so far - not some specific kind of character folding.
>>
>> How about "fuzzy matching"?
>
> That means something else to most people: to be able to skip some
> characters and still match.

Isn't that Soundex matching more or less?

-- 
David Kastrup

RE: lax matching is not a great default behavior

[Drew Adams]

> > AFAICT, "character folding" is as good as we've come up
> > with, so far - not some specific kind of character folding.
> 
> How about "fuzzy matching"?

Too fuzzy. ;-)  It means _less_ than character folding.

This is a case of folding (abstracting from) certain
characteristics of characters.

(I have no problem with our calling it character folding.
But if there is a more specific name for kind of character
folding we're doing here, let's hear it.)

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Fri, 4 Dec 2015 13:30:16 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: per@starback.se, jwiegley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> > > > No, because ignoring accents is just a small part of character
> > > > folding.  Please take a look at character-fold.el for the details.
> > >
> > > Agreed.  And neither is it folding of diacriticals, because there
> > > are also ad hoc foldings (e.g., quote marks).  And there will
> > > likely be more to come.  It is, in fact, a hodge podge of foldings
> > > - pretty much all of the various char foldings provided by Emacs
> > > so far, except for letter case.
> > 
> > Actually, it's not a hodge-podge at all.  Barring any user-level
> > customizations, it can be formally defined (and has been defined
> > elsewhere) what is and what isn't folded.
> 
> Whether it is formally defined or not does not answer the
> question about the name to use for Emacs users.

"Character folding" is the accepted terminology for this, we didn't
invent it.  Likewise "character sequence equivalence".

> > > Why not leave it off by default, for now?
> > 
> > "Why not" is not a compelling argument, sorry.  It cannot
> > win the "why not" argument in the other direction.
> 
> "Why change the default?" is precisely the question Emacs
> dev generally asks itself.

We are not changing the default.  We introduced a new feature, and
this discussion is whether that feature should or shouldn't be turned
on by default.  There's no previous default here.

> This has been stated more than once now by more than one
> person.  But you keep giving the argument that turning it
> on for pretesting is beneficial.  So it is.  So turn it
> on for pretesting, to get more feedback, and off for
> the release.

We have enough time to decide about the default for the release.
Hopefully, we will have more data then than we have now, and the
decision will be more informed one.

Discussions that led to changes in the defaults, was: lax matching is not a great default behavior

[Dmitry Gutov]

On 11/28/2015 10:44 AM, Eli Zaretskii wrote:
> In all the years I've been
> involved in Emacs development, I remember only one case of such a
> discussion that ended up in real changes: the changes in default
> colors of faces in Emacs 21.  But that was when the development team
> was very small and everyone on it had the same perspective.  (It still
> took a lot of time and argument.)

http://debbugs.gnu.org/20290 comes to mind, and I'm very happy with it.

Re: lax matching is not a great default behavior

[David Kastrup]

Drew Adams <drew.adams@oracle.com> writes:

> This has been discussed somewhat, but I don't think there was
> any actual proposal to change the behavior.  So here goes.
>
> Does it still make sense to make search and replacement
> use lax matching, i.e., fold stuff, by default?
>
> I don't think it does.  I think that users, especially
> new users, would be less confused if Emacs defaulted to
> literal searching - no whitespace, case, "character",
> or other folding by default.

For better or worse, our default fantasy "new user" is not one using a
computer for the first time but rather one using Emacs for the first
time.  Most other desktop applications with a search functionality do
case folding.

It makes sense in text modes, less so in programming modes, but there
are a number of programming languages which ignore case (partly because
they are older than useful encodings distinguishing upper- and lowercase
letters, and when ASCII terminals came around, people got tired of
writing and reading in all-caps).

-- 
David Kastrup

RE: lax matching is not a great default behavior

[Drew Adams]

> > > > Agreed.  And neither is it folding of diacriticals, because
> > > > there are also ad hoc foldings (e.g., quote marks).  And there
> > > > will likely be more to come.  It is, in fact, a hodge podge of
> > > > foldings - pretty much all of the various char foldings provided
> > > > by Emacs so far, except for letter case.
> > >
> > > Actually, it's not a hodge-podge at all.  Barring any user-level
> > > customizations, it can be formally defined (and has been defined
> > > elsewhere) what is and what isn't folded.
> >
> > Whether it is formally defined or not does not answer the
> > question about the name to use for Emacs users.
> 
> "Character folding" is the accepted terminology for this, we didn't
> invent it.  Likewise "character sequence equivalence".

I've already agreed (from the beginning) that "character
folding" is the right term for Emacs to use.  And that
speaking of character equivalences is also appropriate.

(There has been some talk of adding multi-character string
equivalences, but even if we match strings instead of just
chars, speaking of "character foldings" makes sense to me.)

I mentioned "ad hoc" character equivalences because I didn't
think that the quotation-mark equivalences we've added are
included in any of the Unicode equivalences (whether
"canonically equivalent" or "compatible").

Are you saying that they are so included?  And that the
equivalences that Emacs will use are _all_ of those defined
by Unicode?

If not, then I'd still say that Emacs does character folding,
but _some_ character folding; a certain kind of character
folding.  And AFAIK we don't have a specific term that
characterizes just the folding we do.  (Which is OK.)

And we _will_ have "user-level customizations" - user-defined
equivalence classes, in the future (I hope).  IOW, more ad hoc
foldings to come.  We will have our - Emacs's - character
folding, which won't map one-to-one onto Unicode equivalences.
(Unless I'm mistaken about the quote-mark equivalences, this
is already the case.)

But again, "character folding" is the best term I've heard
mentioned for what Emacs does.  We need not use it always in
exactly the same sense as Unicode.

> We are not changing the default.  We introduced a new feature, and
> this discussion is whether that feature should or shouldn't be turned
> on by default.  There's no previous default here.

Hm.  That sounds close to gobbledygook, to me.  Turned on by
default would mean a changed default behavior: the behavior
you get without doing anything (toggling, customizing, coding)
would be different, new, never seen before by Emacs users.

There was no such _choice_ before, so the "default" matching
behavior until Emacs 25 was the only matching behavior, but if
that's your point, in claiming that turning this new behavior
on from the outset would not be changing the default behavior,
then I'd say that being that pedantic is, well, a bit silly.

Yes, users will now have a choice.  Should they need to do
something (e.g. toggle) to get the new behavior or not?
That's the question.  You say no; I think yes, a priori -
unless there are some good reasons otherwise.

> We have enough time to decide about the default for the release.
> Hopefully, we will have more data then than we have now, and the
> decision will be more informed one.

Agreed.  And _then_ we can entertain reasons to change the
"default" (initial) behavior.

Re: lax matching is not a great default behavior

[Eli Zaretskii]

> Date: Sat, 5 Dec 2015 01:27:03 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: per@starback.se, jwiegley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> > > Whether it is formally defined or not does not answer the
> > > question about the name to use for Emacs users.
> > 
> > "Character folding" is the accepted terminology for this, we didn't
> > invent it.  Likewise "character sequence equivalence".
> 
> I've already agreed (from the beginning) that "character
> folding" is the right term for Emacs to use.  And that
> speaking of character equivalences is also appropriate.
> 
> (There has been some talk of adding multi-character string
> equivalences, but even if we match strings instead of just
> chars, speaking of "character foldings" makes sense to me.)

Yes, multi-character string equivalences are supported.

> I mentioned "ad hoc" character equivalences because I didn't
> think that the quotation-mark equivalences we've added are
> included in any of the Unicode equivalences (whether
> "canonically equivalent" or "compatible").

Indeed, we added equivalences for quote characters that are not
defined by Unicode database.  I think that these equivalences should
just be the initial value for the user-customizable part of the
feature.  And I don't think these few additions justify new
terminology, the existing one still describes even that.

RE: lax matching is not a great default behavior

[Drew Adams]

> > I mentioned "ad hoc" character equivalences because I didn't
> > think that the quotation-mark equivalences we've added are
> > included in any of the Unicode equivalences (whether
> > "canonically equivalent" or "compatible").
> 
> Indeed, we added equivalences for quote characters that are not
> defined by Unicode database.  I think that these equivalences should
> just be the initial value for the user-customizable part of the
> feature.  And I don't think these few additions justify new
> terminology, the existing one still describes even that.

Good.  We agree on both counts.  (1. The current, predefined
ad hoc equivalences should be an initial value for a user-defined
ad hoc equivalence group.  2. "Character folding" is fine for
describing all of this functionality.  Our use of the term
need not be limited to what the Unicode standard defines as
character folding.)

Wrt user-customizable: I would like to see (after 25.1, no
doubt) the design accommodate users easily defining their own
equivalence groups (not just a single defcustom for all ad hoc
equivalences).

And we can try to make it easy for them to [en|dis]able any set
of such equivalence groups selectively, including for different
contexts/modes.

Re: lax matching is not a great default behavior

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> Wrt user-customizable: I would like to see (after 25.1, no doubt) the design
> accommodate users easily defining their own equivalence groups (not just a
> single defcustom for all ad hoc equivalences).

> And we can try to make it easy for them to [en|dis]able any set of such
> equivalence groups selectively, including for different contexts/modes.

I would like to see this as well, post 25.1.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2