Upcoming loss of usability of Emacs source files and Emacs.

Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Emacs.

First, some definitions.

A @dfn{working character} is a character you use in everyday life - you
can type it easily on your keyboard for self-insert-command, and it is
displayed clearly and unambiguously on your screen.  As Emacs developers,
our working characters are basically ASCII, though each of us has his own
characters (e.g. ä, ö, ü, ß for me) which count as "private" working
characters.

By contrast, a @dfn{display character}, or @dfn{non-working character} is
a character which isn't a working character.  To insert it into a buffer,
you need to type its numeric code, or use (and remember) a C-x 8 ...
binding, or some other clumsy workaround.  It might or might not get
properly displayed on your screen.

Traditionally, Emacs sources have been written exclusively in working
characters (with the essential exception of some characters which stand
for themselves, such as some non-ASCII punctuation symbols used in
`sentence-end').  This makes editing our source files optimally
efficient.

Recently, there has been a move, primarily by Paul, to introduce
non-working characters (specifically, LEFT/RIGHT SINGLE QUOTATION MARKs,
or (less shoutily) "curly quotes") into our sources and *Help* buffers,
not just marginally, but routinely into all our doc strings and error
strings.

Paul and I have had an extensive discussion on many of the issues this
raises, in the thread for bug #20707.  I am against these changes for
several reasons: basically, they will make our sources more difficult to
read and edit.  I am not even sure of the motivation for the changes,
though I think it is mainly that some people find the appearance of 0x60
and 0x27 as quote marks unaesthetic in some display environments.

My main objection is there is no option to turn this new "feature" off.
Some users are going to dislike these changes, possible dislike them a
lot, but we are going to be forcing the curly quotes on them.  This is
going to create much dissatisfaction.  For comparison, the fringe was a
new feature in Emacs-21 which had no way of being disabled.  Many users
hated it, and were up in arms about it until this was fixed in Emacs-22.

My view is that the curly quotes should not replace 0x60 and 0x27 in our
sources, but that the option to display them as curly quotes should be
made available to those that want it.

As far as I am aware, there has been no poll to gather and analyse the
views of Emacs developers on these changes, much less one for Emacs
users.  This is a Bad Thing.

What do people think?

--
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Oleh Krehel]

Alan Mackenzie <acm@muc.de> writes:

> My view is that the curly quotes should not replace 0x60 and 0x27 in our
> sources, but that the option to display them as curly quotes should be
> made available to those that want it.

I completely agree. I often search sources using "`" and "'" in a
regex. Now it's hard to type that in.

Additionally, I actually like to see "`" and "'" displayed. So if there
was an option to turn off "‘" visually, I would do it in my config.

Currently, there are only 253 lines matching "‘", but they are already
touching some files that I care about. And more are to come, since even
the style guide was changed:

     ;;* When a documentation string refers to a Lisp symbol, write it as
     ;;  it would be printed (which usually means in lower case), with
     ;;  single-quotes around it.  For example: ‘lambda’.  There are two
     ;;  exceptions: write t and nil without single-quotes.  (For
     ;;  compatibility with an older Emacs style, quoting with ` and '
     ;;  also works, e.g., `lambda' is treated like ‘lambda’.)

Oleh

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Oleh Krehel <ohwoeowho@gmail.com> writes:

>> My view is that the curly quotes should not replace 0x60 and 0x27 in
>> our sources, but that the option to display them as curly quotes
>> should be made available to those that want it.
>
> I completely agree. I often search sources using "`" and "'" in a
> regex. Now it's hard to type that in.

I also like `foo' better because it's easy to type and looks good
(although that might be just a matter of habit).  But there are also
packages which exploit that convention.  For example, Gnus will
recognize that `forward-sexp' denotes a function, highlight it
appropriately, and make it clickable.  The `...' notation has only been
used for writing down variables and functions whereas single quotes
could also be used for prose which would at least give more false
positives if Gnus were adapted to the new convention.

> Additionally, I actually like to see "`" and "'" displayed. So if
> there was an option to turn off "‘" visually, I would do it in my
> config.

Ditto.  I was shocked when info started using single quotes and thought
it was an error at first.  It just looked wrong to me.

> ;;* When a documentation string refers to a Lisp symbol, write it as
> ;;  it would be printed (which usually means in lower case), with
> ;;  single-quotes around it.

But we're still referring to a function argument `foo' as FOO in
docstrings, no?

BTW, is there something better for inserting single quotes as using

  M-x insert-char LEFT SINGLE QUOTATION MARK RET
  M-x insert-char RIGHT SINGLE QUOTATION MARK RET

?  That's a littlebit inconvenient given that you have to do that quite
often.  (Yes, of course everybody has her own `quote-symbol-at-point'
function anyway.)

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Tassilo Horn wrote:
> But we're still referring to a function argument `foo' as FOO in
> docstrings, no?

Yes, there's no proposal to change that.

> is there something better for inserting single quotes as using
>
>    M-x insert-char LEFT SINGLE QUOTATION MARK RET
>    M-x insert-char RIGHT SINGLE QUOTATION MARK RET

I typically use Electric Quote mode, and type ` and ' which is pretty easy.  You 
can also type "C-x 8 [" and "C-x 8 ]".  If your Alt key works you can also type 
"A-[" and "A-]".   On my keyboard right now (Ubuntu 15.04) I can also type 
"Compose < '" and "Compose > '"; that's a Gnomeism, though, I believe.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Tassilo Horn wrote:
>> But we're still referring to a function argument `foo' as FOO in
>> docstrings, no?
>
> Yes, there's no proposal to change that.

I don't understand.  When I enable electric-quote-mode and type `hello'
in a docstring, I get ‘hello‘ instead.

Nico
-- 
Nicolas Petton
http://nicolas-petton.fr

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Nicolas Petton wrote:
> Paul Eggert<eggert@cs.ucla.edu>  writes:
>
>> >Tassilo Horn wrote:
>>> >>But we're still referring to a function argument `foo' as FOO in
>>> >>docstrings, no?
>> >
>> >Yes, there's no proposal to change that.
> I don't understand.  When I enable electric-quote-mode and type `hello'
> in a docstring, I get ‘hello‘ instead.

Tasillo was talking about function arguments, not about functions.  For example, 
when I type 'C-h f car RET' I see the following, and here the argument is still 
written as all-caps LIST without quotation of any sort.

-----
car is a built-in function in ‘C source code’.

(car LIST)

Return the car of LIST.  If arg is nil, return nil.
Error if arg is not nil and not a cons cell.  See also ‘car-safe’.

See Info node ‘(elisp)Cons Cells’ for a discussion of related basic
Lisp concepts such as car, cdr, cons cell and list.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Paul Eggert <eggert@cs.ucla.edu> writes:

>> But we're still referring to a function argument `foo' as FOO in
>> docstrings, no?
>
> Yes, there's no proposal to change that.

I was just wondering because the quoted text said to quote lisp symbols
with single quotes, and parameters are lisp symbols, too, or arent they?

>> is there something better for inserting single quotes as using
>>
>>    M-x insert-char LEFT SINGLE QUOTATION MARK RET
>>    M-x insert-char RIGHT SINGLE QUOTATION MARK RET
>
> I typically use Electric Quote mode, and type ` and ' which is pretty
> easy.

I just did `M-x electric-quote-mode RET' and `electric-quote-paragraph'
is t, and still nothing is rewritten.  Ah, it seems it's only active for
modes deriving from text-mode which message-mode apparently doesn't
do...

> You can also type "C-x 8 [" and "C-x 8 ]".

That works.

> If your Alt key works you can also type "A-[" and "A-]".

What does "Alt key works" mean?  Alt is Meta here, so typing Alt-AltGr-(
says M-[ is undefined.

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Robert Pluim]

Tassilo Horn <tsdh@gnu.org> writes:

>
> What does "Alt key works" mean?  Alt is Meta here, so typing Alt-AltGr-(
> says M-[ is undefined.

I think what Paul means is that Alt should send Alt. I have my left-Alt
configured as Meta, but my right-Alt configured as Alt, so I can type
single and double quotes without all the C-x 8 RET malarkey.

In xmodmap syntax:

clear mod1
clear mod3

keycode 133 = Meta_L
keycode 108 = Alt_R
add mod1 = Meta_L
add mod3 = Alt_R

Robert

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Robert Pluim <rpluim@gmail.com> writes:

>> What does "Alt key works" mean?  Alt is Meta here, so typing
>> Alt-AltGr-( says M-[ is undefined.
>
> I think what Paul means is that Alt should send Alt.  I have my
> left-Alt configured as Meta, but my right-Alt configured as Alt, so I
> can type single and double quotes without all the C-x 8 RET malarkey.

I don't even have a right Alt key, only AltGr which I need to be able to
type {[`@€´]} etc.

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Yuri Khan]

On Tue, Jun 16, 2015 at 6:35 PM, Tassilo Horn <tsdh@gnu.org> wrote:
> Robert Pluim <rpluim@gmail.com> writes:
>
>>> What does "Alt key works" mean?  Alt is Meta here, so typing
>>> Alt-AltGr-( says M-[ is undefined.
>>
>> I think what Paul means is that Alt should send Alt.  I have my
>> left-Alt configured as Meta, but my right-Alt configured as Alt, so I
>> can type single and double quotes without all the C-x 8 RET malarkey.
>
> I don't even have a right Alt key, only AltGr which I need to be able to
> type {[`@€´]} etc.

So extend that to be able to also enter “”‘’«»–—…÷×−±≠°§®©™.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Andy Moreton]

On Tue 16 Jun 2015, Tassilo Horn wrote:

> Paul Eggert <eggert@cs.ucla.edu> writes:
>
>>> But we're still referring to a function argument `foo' as FOO in
>>> docstrings, no?
>>
>> Yes, there's no proposal to change that.
>
> I was just wondering because the quoted text said to quote lisp symbols
> with single quotes, and parameters are lisp symbols, too, or arent they?
>
>>> is there something better for inserting single quotes as using
>>>
>>>    M-x insert-char LEFT SINGLE QUOTATION MARK RET
>>>    M-x insert-char RIGHT SINGLE QUOTATION MARK RET
>>
>> I typically use Electric Quote mode, and type ` and ' which is pretty
>> easy.
>
> I just did `M-x electric-quote-mode RET' and `electric-quote-paragraph'
> is t, and still nothing is rewritten.  Ah, it seems it's only active for
> modes deriving from text-mode which message-mode apparently doesn't
> do...
>
>> You can also type "C-x 8 [" and "C-x 8 ]".
>
> That works.
>
>> If your Alt key works you can also type "A-[" and "A-]".
>
> What does "Alt key works" mean?  Alt is Meta here, so typing Alt-AltGr-(
> says M-[ is undefined.

If you are using emacs on Windows, you need to check the values of:
    w32-alt-is-meta
    w32-lwindow-modifier
    w32-rwindow-modifier
    w32-apps-modifier
    w32-scroll-lock-modifier

The default mapping puts the ALT modifier on the Apps key.

    AndyM

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Tassilo Horn wrote:
> I was just wondering because the quoted text said to quote lisp symbols
> with single quotes, and parameters are lisp symbols, too, or arent they?

The intent is not to change how documentation refers to parameters; they're 
still all-caps only, with no quoting.  The wording for this part of the style 
conventions hasn't changed.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Oleh Krehel]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> My view is that the curly quotes should not replace 0x60 and 0x27 in our
>> sources, but that the option to display them as curly quotes should be
>> made available to those that want it.
>
> I completely agree. I often search sources using "`" and "'" in a
> regex. Now it's hard to type that in.
>
> Additionally, I actually like to see "`" and "'" displayed. So if there
> was an option to turn off "‘" visually, I would do it in my config.

As I saw the (subjectively) ugly new quotation chars already being used
in Emacs commit messages, I finally took the time to learn how display
tables work. So for all people who actually like how the old-style
quotation looks, this is how to replace visually all new-style quotation
with old-style:

    (let ((tbl (make-display-table)))
      (aset tbl 8220 (vector (make-glyph-code ?\" 'default)))
      (aset tbl 8221 (vector (make-glyph-code ?\" 'default)))
      (aset tbl 8216 (vector (make-glyph-code ?\` 'default)))
      (aset tbl 8217 (vector (make-glyph-code ?\' 'default)))
      (setq standard-display-table tbl))

While we're on the subject of modifying the source code purely for
aesthetics, I wanted to bring up the subject of Elisp indentation rules.
But then I saw http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20323, from
which it appears that the problem is solved for two months already
(although without a NEWS entry). Thanks to Dmitry for the change. There
are still issues of `lisp-indent-function' being inferior to
`common-lisp-indent-function' in some cases, but I can report them
later.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 22 Jun 2015 15:24:35 +0200
> Cc: emacs-devel@gnu.org
> 
>     (let ((tbl (make-display-table)))

Since standard-display-table could already exist, the above should be

  (let ((tbl (or standard-display-table (make-display-table))))

so as to avoid losing existing entries in the table made by the user
or one of the packages she uses.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[raman]

I agree strongly with you that the source code shouldn't get
contaminated by the display needs of    groups of users -- or be
dictated by some display convention that changes over time --- this all
feels like abandoning email headers in favor of coding them up with HTML
spans and font/color tags.

The trend you observe is especially disturbing in a long-lived project
like Emacs.
-- 

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> As far as I am aware, there has been no poll to gather and analyse
> the views of Emacs developers on these changes, much less one for
> Emacs users.

Of course not.  When was the last time you saw such a poll? ;-)

> This is a Bad Thing.

Yes.

> What do people think?

I've already made my position clear about this in the bug list.
But I'll state it one more time.

`...' is a very *good* way to set off inline symbols and other
code phrases from surrounding non-code text.  Those who think
it is somehow just an odd and imperfect, old-fashioned form of
*quoting* are sadly mistaken.  That is not what it is about.

'...' is used in ordinary English text, along with "...", for
normal quotation.  As a text editor (among other things), Emacs
should not interfere with this usage or make it difficult for
users or programs to parse it and manipulate it, by piling on
an alternative interpretation.  (Occam oblige.)

What is needed in any doc about software is a way to clearly
set off inline code phrases from surrounding, non-code text.
This demarking constitutes metadata that is different from
ordinary-text quoting.

When structured doc or markup is used, this is typically
accomplished using metadata that is provided by wrapping the
code phrases in XML (or similar) elements/markup:
<code>...</code>.

In Emacs, we want, if possible, a simple mechanism that lets
the text that contains the metadata (the "markup" text) to also
act as the text that the user interacts with directly - search
etc., but without the distraction of interacting with obvious
markup (<code> etc.).

`...' is simply an ingenious abbreviation for what is usually
handled more verbosely using constructs such as <code>...</code>.

As Alan points out, we want the metadata for this to be easy
and quick to type, and not to interfere with either appearance
or handling by program (including, but not limited to, Lisp).

For all Emacs purposes I am aware of `...' is a very *good*
invention.  It is a reasonable compromise (yes, like anything
else, it is not unambiguous in all contexts).  And it has
proven its worth in Emacs for 3 decades.

That one person (plus our dear leader, apparently) thinks
`...' is too "ugly" or too 1980s for his own use should not
be a reason for Emacs to continue down the rabbit hole it
has apparently been overzealously pushed into.

The argument that ` and ' used to look OK back in the 80s,
but fonts have changed so they are no longer symmetric,
really misses the point.

A delimiting pair of chars that is not confused with other
uses ([], (), {}, <>, '', "",...) is what is needed, and
`...' fits the bill well.  (Some other contexts use `` or '',
but like "", these have an obvious disadvantage.  Still, even
they would be preferable to '...'.)

This change, whether implemented (a) only for rendering
(appearance) or (b) at the base - actually using '...' in
the underlying text, is altogether misguided, IMHO.

Whether those originally responsible for `...' were aware of
all of its advantages as a means of setting off inline code,
I don't know.  But I thank them for it.  I hope that Emacs
will eventually come to its senses about this and appreciate
what a great gift `...' really is.

Instead of being ashamed of `...' as a black sheep, the Emacs
family should embrace it and be proud.  Especially, it should
understand how truly useful it is.  '...' for inline code is
a misguided, ugly hack, and in the long run not very workable.

Emacs still has important and exciting things to work on and
invent.  The '...' crusade is not one of them, IMHO.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 3994 bytes --]

+1 for what Alan has to say.
+1 for keeping ` and ' for greppability as Oleh mentioned
+1 to what Drew said; seeing commits related to curly braces is not very
exciting


--
Kaushal Modi

On Mon, Jun 15, 2015 at 11:23 AM, Drew Adams <drew.adams@oracle.com> wrote:

> > As far as I am aware, there has been no poll to gather and analyse
> > the views of Emacs developers on these changes, much less one for
> > Emacs users.
>
> Of course not.  When was the last time you saw such a poll? ;-)
>
> > This is a Bad Thing.
>
> Yes.
>
> > What do people think?
>
> I've already made my position clear about this in the bug list.
> But I'll state it one more time.
>
> `...' is a very *good* way to set off inline symbols and other
> code phrases from surrounding non-code text.  Those who think
> it is somehow just an odd and imperfect, old-fashioned form of
> *quoting* are sadly mistaken.  That is not what it is about.
>
> '...' is used in ordinary English text, along with "...", for
> normal quotation.  As a text editor (among other things), Emacs
> should not interfere with this usage or make it difficult for
> users or programs to parse it and manipulate it, by piling on
> an alternative interpretation.  (Occam oblige.)
>
> What is needed in any doc about software is a way to clearly
> set off inline code phrases from surrounding, non-code text.
> This demarking constitutes metadata that is different from
> ordinary-text quoting.
>
> When structured doc or markup is used, this is typically
> accomplished using metadata that is provided by wrapping the
> code phrases in XML (or similar) elements/markup:
> <code>...</code>.
>
> In Emacs, we want, if possible, a simple mechanism that lets
> the text that contains the metadata (the "markup" text) to also
> act as the text that the user interacts with directly - search
> etc., but without the distraction of interacting with obvious
> markup (<code> etc.).
>
> `...' is simply an ingenious abbreviation for what is usually
> handled more verbosely using constructs such as <code>...</code>.
>
> As Alan points out, we want the metadata for this to be easy
> and quick to type, and not to interfere with either appearance
> or handling by program (including, but not limited to, Lisp).
>
> For all Emacs purposes I am aware of `...' is a very *good*
> invention.  It is a reasonable compromise (yes, like anything
> else, it is not unambiguous in all contexts).  And it has
> proven its worth in Emacs for 3 decades.
>
> That one person (plus our dear leader, apparently) thinks
> `...' is too "ugly" or too 1980s for his own use should not
> be a reason for Emacs to continue down the rabbit hole it
> has apparently been overzealously pushed into.
>
> The argument that ` and ' used to look OK back in the 80s,
> but fonts have changed so they are no longer symmetric,
> really misses the point.
>
> A delimiting pair of chars that is not confused with other
> uses ([], (), {}, <>, '', "",...) is what is needed, and
> `...' fits the bill well.  (Some other contexts use `` or '',
> but like "", these have an obvious disadvantage.  Still, even
> they would be preferable to '...'.)
>
> This change, whether implemented (a) only for rendering
> (appearance) or (b) at the base - actually using '...' in
> the underlying text, is altogether misguided, IMHO.
>
> Whether those originally responsible for `...' were aware of
> all of its advantages as a means of setting off inline code,
> I don't know.  But I thank them for it.  I hope that Emacs
> will eventually come to its senses about this and appreciate
> what a great gift `...' really is.
>
> Instead of being ashamed of `...' as a black sheep, the Emacs
> family should embrace it and be proud.  Especially, it should
> understand how truly useful it is.  '...' for inline code is
> a misguided, ugly hack, and in the long run not very workable.
>
> Emacs still has important and exciting things to work on and
> invent.  The '...' crusade is not one of them, IMHO.
>
>

[-- Attachment #2: Type: text/html, Size: 5237 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 4225 bytes --]

*curly quotes (not braces)


--
Kaushal Modi

On Mon, Jun 15, 2015 at 11:30 AM, Kaushal <kaushal.modi@gmail.com> wrote:

> +1 for what Alan has to say.
> +1 for keeping ` and ' for greppability as Oleh mentioned
> +1 to what Drew said; seeing commits related to curly braces is not very
> exciting
>
>
> --
> Kaushal Modi
>
> On Mon, Jun 15, 2015 at 11:23 AM, Drew Adams <drew.adams@oracle.com>
> wrote:
>
>> > As far as I am aware, there has been no poll to gather and analyse
>> > the views of Emacs developers on these changes, much less one for
>> > Emacs users.
>>
>> Of course not.  When was the last time you saw such a poll? ;-)
>>
>> > This is a Bad Thing.
>>
>> Yes.
>>
>> > What do people think?
>>
>> I've already made my position clear about this in the bug list.
>> But I'll state it one more time.
>>
>> `...' is a very *good* way to set off inline symbols and other
>> code phrases from surrounding non-code text.  Those who think
>> it is somehow just an odd and imperfect, old-fashioned form of
>> *quoting* are sadly mistaken.  That is not what it is about.
>>
>> '...' is used in ordinary English text, along with "...", for
>> normal quotation.  As a text editor (among other things), Emacs
>> should not interfere with this usage or make it difficult for
>> users or programs to parse it and manipulate it, by piling on
>> an alternative interpretation.  (Occam oblige.)
>>
>> What is needed in any doc about software is a way to clearly
>> set off inline code phrases from surrounding, non-code text.
>> This demarking constitutes metadata that is different from
>> ordinary-text quoting.
>>
>> When structured doc or markup is used, this is typically
>> accomplished using metadata that is provided by wrapping the
>> code phrases in XML (or similar) elements/markup:
>> <code>...</code>.
>>
>> In Emacs, we want, if possible, a simple mechanism that lets
>> the text that contains the metadata (the "markup" text) to also
>> act as the text that the user interacts with directly - search
>> etc., but without the distraction of interacting with obvious
>> markup (<code> etc.).
>>
>> `...' is simply an ingenious abbreviation for what is usually
>> handled more verbosely using constructs such as <code>...</code>.
>>
>> As Alan points out, we want the metadata for this to be easy
>> and quick to type, and not to interfere with either appearance
>> or handling by program (including, but not limited to, Lisp).
>>
>> For all Emacs purposes I am aware of `...' is a very *good*
>> invention.  It is a reasonable compromise (yes, like anything
>> else, it is not unambiguous in all contexts).  And it has
>> proven its worth in Emacs for 3 decades.
>>
>> That one person (plus our dear leader, apparently) thinks
>> `...' is too "ugly" or too 1980s for his own use should not
>> be a reason for Emacs to continue down the rabbit hole it
>> has apparently been overzealously pushed into.
>>
>> The argument that ` and ' used to look OK back in the 80s,
>> but fonts have changed so they are no longer symmetric,
>> really misses the point.
>>
>> A delimiting pair of chars that is not confused with other
>> uses ([], (), {}, <>, '', "",...) is what is needed, and
>> `...' fits the bill well.  (Some other contexts use `` or '',
>> but like "", these have an obvious disadvantage.  Still, even
>> they would be preferable to '...'.)
>>
>> This change, whether implemented (a) only for rendering
>> (appearance) or (b) at the base - actually using '...' in
>> the underlying text, is altogether misguided, IMHO.
>>
>> Whether those originally responsible for `...' were aware of
>> all of its advantages as a means of setting off inline code,
>> I don't know.  But I thank them for it.  I hope that Emacs
>> will eventually come to its senses about this and appreciate
>> what a great gift `...' really is.
>>
>> Instead of being ashamed of `...' as a black sheep, the Emacs
>> family should embrace it and be proud.  Especially, it should
>> understand how truly useful it is.  '...' for inline code is
>> a misguided, ugly hack, and in the long run not very workable.
>>
>> Emacs still has important and exciting things to work on and
>> invent.  The '...' crusade is not one of them, IMHO.
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 5862 bytes --]

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Stephan Mueller]

[-- Attachment #1: Type: text/plain, Size: 4067 bytes --]

+1 also, even after reading and acknowledging the compelling article Ulrich referenced.

stephan();

Kaushal wrote:

+1 for what Alan has to say.
+1 for keeping ` and ' for greppability as Oleh mentioned
+1 to what Drew said; seeing commits related to curly quotes is not very exciting

On Mon, Jun 15, 2015 at 11:23 AM, Drew Adams <drew.adams@oracle.com<mailto:drew.adams@oracle.com>> wrote:
> As far as I am aware, there has been no poll to gather and analyse
> the views of Emacs developers on these changes, much less one for
> Emacs users.

Of course not.  When was the last time you saw such a poll? ;-)

> This is a Bad Thing.

Yes.

> What do people think?

I've already made my position clear about this in the bug list.
But I'll state it one more time.

`...' is a very *good* way to set off inline symbols and other
code phrases from surrounding non-code text.  Those who think
it is somehow just an odd and imperfect, old-fashioned form of
*quoting* are sadly mistaken.  That is not what it is about.

'...' is used in ordinary English text, along with "...", for
normal quotation.  As a text editor (among other things), Emacs
should not interfere with this usage or make it difficult for
users or programs to parse it and manipulate it, by piling on
an alternative interpretation.  (Occam oblige.)

What is needed in any doc about software is a way to clearly
set off inline code phrases from surrounding, non-code text.
This demarking constitutes metadata that is different from
ordinary-text quoting.

When structured doc or markup is used, this is typically
accomplished using metadata that is provided by wrapping the
code phrases in XML (or similar) elements/markup:
<code>...</code>.

In Emacs, we want, if possible, a simple mechanism that lets
the text that contains the metadata (the "markup" text) to also
act as the text that the user interacts with directly - search
etc., but without the distraction of interacting with obvious
markup (<code> etc.).

`...' is simply an ingenious abbreviation for what is usually
handled more verbosely using constructs such as <code>...</code>.

As Alan points out, we want the metadata for this to be easy
and quick to type, and not to interfere with either appearance
or handling by program (including, but not limited to, Lisp).

For all Emacs purposes I am aware of `...' is a very *good*
invention.  It is a reasonable compromise (yes, like anything
else, it is not unambiguous in all contexts).  And it has
proven its worth in Emacs for 3 decades.

That one person (plus our dear leader, apparently) thinks
`...' is too "ugly" or too 1980s for his own use should not
be a reason for Emacs to continue down the rabbit hole it
has apparently been overzealously pushed into.

The argument that ` and ' used to look OK back in the 80s,
but fonts have changed so they are no longer symmetric,
really misses the point.

A delimiting pair of chars that is not confused with other
uses ([], (), {}, <>, '', "",...) is what is needed, and
`...' fits the bill well.  (Some other contexts use `` or '',
but like "", these have an obvious disadvantage.  Still, even
they would be preferable to '...'.)

This change, whether implemented (a) only for rendering
(appearance) or (b) at the base - actually using '...' in
the underlying text, is altogether misguided, IMHO.

Whether those originally responsible for `...' were aware of
all of its advantages as a means of setting off inline code,
I don't know.  But I thank them for it.  I hope that Emacs
will eventually come to its senses about this and appreciate
what a great gift `...' really is.

Instead of being ashamed of `...' as a black sheep, the Emacs
family should embrace it and be proud.  Especially, it should
understand how truly useful it is.  '...' for inline code is
a misguided, ugly hack, and in the long run not very workable.

Emacs still has important and exciting things to work on and
invent.  The '...' crusade is not one of them, IMHO.



[-- Attachment #2: Type: text/html, Size: 7817 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Artur Malabarba]

> Recently, there has been a move, primarily by Paul, to introduce
> non-working characters (specifically, LEFT/RIGHT SINGLE QUOTATION MARKs,
> or (less shoutily) "curly quotes") into our sources and *Help* buffers,
> not just marginally, but routinely into all our doc strings and error
> strings.

Maybe I just haven't run into them, but I haven't seen curved quotes
inserted into the source of any docstrings. I keep writing them with `
and ', and they get displayed as curved quotes on the Help buffer
(which looks fine to me).

If it's happening, then I am against this. But, just to be clear, I am
very much in favor of *displaying* them as curved quotes on Help
buffers by default (which is what I thought we were doing now).

> What do people think?

I'm in favor of displaying quotes as curved by default. But I'm
against using them in the actual source (although they don't bother my
workflow).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/15/2015 06:38 PM, Artur Malabarba wrote:

> Maybe I just haven't run into them, but I haven't seen curved quotes
> inserted into the source of any docstrings.

Not yet, but Paul and Stefan have expressed interest in doing that too.

> If it's happening, then I am against this. But, just to be clear, I am
> very much in favor of *displaying* them as curved quotes on Help
> buffers by default (which is what I thought we were doing now).

Yes, that's a recent change I like as well.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> What do people think?
>
> I'm in favor of displaying quotes as curved by default. But I'm
> against using them in the actual source (although they don't bother my
> workflow).

I totally agree with you.

Cheers,
Nico
-- 
Nicolas Petton
http://nicolas-petton.fr

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Ulrich Mueller]

>>>>> On Mon, 15 Jun 2015, Alan Mackenzie wrote:

> My view is that the curly quotes should not replace 0x60 and 0x27 in our
> sources, but that the option to display them as curly quotes should be
> made available to those that want it.

> As far as I am aware, there has been no poll to gather and analyse the
> views of Emacs developers on these changes, much less one for Emacs
> users.  This is a Bad Thing.

> What do people think?

U+0060 shouldn't be used as a quote character, for the simple reason
that it isn't one. U+0060 is uniquely defined as the GRAVE ACCENT
character, and as such it doesn't form a pair with U+0027 APOSTROPHE.
(It forms a pair with U+00B4 ACUTE ACCENT: `´ but the latter is
outside of the ASCII range, and of course both are unsuitable as
quotes.)

So please get rid of U+0060 in all places where it is (ab)used as
a quote character rather sooner than later, and use ‘’ (U+2018 U+2019)
instead. And if an ASCII-only alternative is needed, it should be
either '' or "".

The following article (written 15 years ago) summarises the situation
well: http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

Ulrich

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Ulrich Mueller writes:

 > U+0060 shouldn't be used as a quote character, for the simple reason
 > that it isn't one. U+0060 is uniquely defined as the GRAVE ACCENT
 > character, and as such it doesn't form a pair with U+0027 APOSTROPHE.
 > (It forms a pair with U+00B4 ACUTE ACCENT: `´ but the latter is
 > outside of the ASCII range, and of course both are unsuitable as
 > quotes.)

And both of those characters are quite useless because they're not
combining characters.  (Note that proper usage for displaying an
isolated accent in Unicode is to accent a no-break space.)  In some
sense, they're compatibility characters that "shouldn't" appear in
modern (natural language) text at all.  (In fact, that's apparently
not entirely true.  U+00B4 is indeed treated as a compatibility
character for the sequence U+0020 U+0301, but U+0060 isn't.  Weird,
but I bet that's precisely because of usage of GRAVE ACCENT to
construct quotation marks in contexts like Emacs and TeX.)

So I have no problem in principle with Emacs coopting GRAVE ACCENT as
"informal" syntax for indicating symbols or code in documentation.

I'm mildly in favor of the change, but pedantic adherence to Unicode
isn't one of my reasons.

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> U+0060 shouldn't be used as a quote character, for the simple reason
> that it isn't one. U+0060 is uniquely defined as the GRAVE ACCENT
> character, and as such it doesn't form a pair with U+0027
> APOSTROPHE.
> (It forms a pair with U+00B4 ACUTE ACCENT: `´ but the latter is
> outside of the ASCII range, and of course both are unsuitable as
> quotes.)
> 
> So please get rid of U+0060 in all places where it is (ab)used as
> a quote character rather sooner than later, and use ‘’ (U+2018
> U+2019) instead. And if an ASCII-only alternative is needed, it
> should be either '' or "".

`...' is not quoting.  ` and ' are not used as quote chars.
That is apparently one of the misconceptions underlying this
misguided "reform".

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > (It forms a pair with U+00B4 ACUTE ACCENT: `´ but the latter is
  > outside of the ASCII range, and of course both are unsuitable as
  > quotes.)

Whether they are suitable as quotes in Emacs is for us to decide.

  > The following article (written 15 years ago) summarises the situation
  > well: http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

That article may provide pertinent information or arguments, but it
has no authority over us.  In the GNU Project, we do not consider
outside standards as decisions or commands; rather, they are part of
the context in which we consider what is the best thing to do.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Alan Mackenzie writes:
 > 
 > Hello, Emacs.
 > 
 > First, some definitions.
 > 
 > A @dfn{working character} is a character you use in everyday life - you
 > can type it easily on your keyboard for self-insert-command, and it is
 > displayed clearly and unambiguously on your screen.

You mean like 日本語Ωα→∪S, perhaps?  Indeed, I used all ofthose
characters today in my work, and although I can't easily type them
directly for self-insert-command, it really doesn't slow me down
compared to the gymnastics it would take me to type that in TeX.
`self-insert-command' is really a red herring, IMO.

 > By contrast, a @dfn{display character}, or @dfn{non-working
 > character} is a character which isn't a working character.  To
 > insert it into a buffer, you need to type its numeric code, or use
 > (and remember) a C-x 8 ...  binding, or some other clumsy
 > workaround.

For those of us who type Japanese, math, or music, efficient input
methods are not workarounds.  They're essential, and comfortable,
tools.  I certainly have sympathy for Dorothy and Toto from Kansas who
have been getting along with a working character set that fits on a
keyboard with only one shift function and because they have never
needed input methods before don't have them, but seriously, we can and
should get past that.  Most of the world cannot type their characters
with a reasonably-sized keyboard and a single shift function (in fact,
you need at least two, don't you?) and they need to remember how to
type their working set.  I guess for German you just type AltGr + a,
o, u, or s, which is sufficiently mnemonic, but that doesn't even work
for French which has accents acute and grave on several vowels.

To summarize, yes, requiring input methods is an imposition on you.
If users like you are as many as you believe, that's a big minus and
it should be considered in the decision.  But for most of the world,
it is indeed negligible (they already know how to use the input
methods and are used to learning new characters), and sufficient
benefits would justify imposing the burden on you and others.

 > It might or might not get properly displayed on your screen.

Uh, speak for yourself.  It will be displayed properly on my screen
(note, because of Japanese I can't use the old Linux console, I have
to use fbterm anyway).  And it is possible to do that on any *old*
*cheap* hardware nowadays -- systems that *can't* be configured to
display them properly aren't "old", they're valuable antiquities that
should be registered with UNESCO.  I'm sure your system, and RMS's,
can be configured to present them properly.  If they don't, it's
because you haven't configured your system to do so.

 > Paul and I have had an extensive discussion on many of the issues
 > this raises, in the thread for bug #20707.  I am against these
 > changes for several reasons: basically, they will make our sources
 > more difficult to read

I disagree.  With well-designed Unicode fonts, in principle these
changes will make Emacs sources easier to read.  Of course it's a
matter of style, and I've resisted such changes myself for decades
because I've found the advocates to be singularly lacking in taste.
But that is no longer true.  I don't always agree with the taste of
proponents of such changes, but I'd no longer claim they have none.

 > and edit.

That is not clear, though more plausible.  For a counterexample, if
use of U+0022 is restricted to string delimiter, while it is
considered better style to use U+201C and U+201D LEFT and RIGHT DOUBLE
QUOTATION MARK for quotations embedded in strings, I would suppose
that searching for strings and marking them would become easier, both
programatically and when writing quick functions and macros on the
fly.  Editability could go either way IMHO.

 > My main objection is there is no option to turn this new "feature"
 > off.

I don't think there should be.  Despite what I wrote above, I don't
see a need for this, it is going to be a PITA for quite a few
individuals (even if they're a small fraction), so I'm basically +0 on
it.  But if we're going to do it, it's a change in content style that
has semantic implications.  It can't be passed off as a display style
option like the fringe; a compromise would be worse than either extreme.

 > Some users are going to dislike these changes, possible dislike them a
 > lot, but we are going to be forcing the curly quotes on them.

Nobody is forced to accept anything.  If users don't like the current
distributed version of Emacs, they're not limited to it.  That's the
very definition of free software.  In your case, since you choose to
maintain packages that need to be compatible with current released and
development versions, that choice is painful.  But you aren't "forced"
to do anything.

And that blade cuts both ways, as you are clearly aware because you
ask for a poll.  Just as some users dislike changes, possibly a lot,
others dislike the status quo ante, possibly a lot.  Why should a few
reactionary curmudgeons :-) hold the progressives back?

 > As far as I am aware, there has been no poll to gather and analyse
 > the views of Emacs developers on these changes, much less one for
 > Emacs users.  This is a Bad Thing.

It used to be a Bad Thing[tm].  Now, modern VCS means that such
changes can be made and reverted quickly (although not always
effortlessly or without help of developers experienced in the
software), so rather than poll, just give them what you think they
need.  If they don't like it, you revert.  That method provides far
more accurate assessments of user sentiment than polling in advance,
and is less costly (unless you're stupid enough to make such a change
in a late beta).

Of course the proponents have to be cooperative.  I know that Paul has
a reasonably high opinion of his own opinions ;-), but I've also seen
him revert patches for reasons of "general user taste", and with very
little backtalk, on request.  I see no reason why the "commit and if
necessary revert" process won't work as described above in this case.

Regards,
Steve

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> Despite what I wrote above, I don't see a need for this, it is
> going to be a PITA for quite a few individuals (even if they're
> a small fraction), so I'm basically +0 on it.

Yippee!  For once I agree with an occurrence of Stephen's recurrent
"YAGNI" bray.  *No need* for this change.

Beyond that, this change is at best a blunder.  It should be reverted.
It is not just cosmetic.

> Just as some users dislike changes, possibly a lot, others dislike
> the status quo ante, possibly a lot.  Why should a few reactionary
> curmudgeons :-) hold the progressives back?

Demagogy & pandering, despite the smily.  Opposition to a particular
change, giving particular reasons, is not tantamount to knee-jerk,
reactionary opposition to change in general.  For shame.  We see this
kind of slight-of-hand charlatanism employed all too often - those
opposed to a purported innovation are branded as "afraid of change".

It is not about change vs no-change in general.  It is about this
change.

For the same reason, though a superficial poll (of readers here
or of Emacs users) can tell us something, good arguments for and
against are generally more helpful.

> It used to be a Bad Thing[tm].  Now, modern VCS means that such
> changes can be made and reverted quickly (although not always
> effortlessly or without help of developers experienced in the
> software), so rather than poll, just give them what you think they
> need.  If they don't like it, you revert.  That method provides far
> more accurate assessments of user sentiment than polling in advance,
> and is less costly (unless you're stupid enough to make such a
> change in a late beta).

I'm not convinced.  Momentum.  Inertia.

And there is no good reason *not* to bring such matters up for
discussion first (here, for example), and that can include
discussion with users.

> Of course the proponents have to be cooperative.  I know that Paul
> has a reasonably high opinion of his own opinions ;-), but I've also
> seen him revert patches for reasons of "general user taste", and with
> very little backtalk, on request.

Consider this one such request, by one user.

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Drew Adams writes:

 > Beyond that, this change is at best a blunder.

Speaking of demagogy.  *This* is demagogy.  No evidence has been
provided that the change is a blunder.  In fact, it can't be a
blunder: the advantages and disadvantages are mostly matters of taste
(including attitudes toward investment in specific kinds of tools).

 > It should be reverted.

Maybe.  We should make that decision based on having lots of users try
it.  The best way to try something like this is to install it and make
it default while we're still in beta.

 > It is not just cosmetic.

True.  So what?  Surely you don't propose we restrict future changes
in Emacs to cosmetic changes!

 > > Just as some users dislike changes, possibly a lot, others dislike
 > > the status quo ante, possibly a lot.  Why should a few reactionary
 > > curmudgeons :-) hold the progressives back?
 > 
 > Demagogy & pandering, despite the smily.

Wrong and wrong again.  Pointing out that users have likes and
dislikes and that they diverge radically is identifying the core issue
that needs to be addressed here.  And Alan *is* a reactionary and a
curmudgeon, quick to oppose changes he doesn't initiate himself and
well-set in his ways.  There's nothing inherently wrong with those
latter characteristics.  It's just that catering to Alan here indeed
does cost others an opportunity to improve their environments, and
offer that improvement to those in the middle.

 > For shame.  We see this kind of slight-of-hand charlatanism
 > employed all too often - those opposed to a purported innovation
 > are branded as "afraid of change".

Reactionaries in my experience are rarely "afraid" of change in the
sense of FUD.  No, they oppose change for objective, if often selfish,
reasons based on the *consequences* of change that they expect.

 > It is not about change vs no-change in general.  It is about this
 > change.

Which, based on the number of words you and Alan have devoted to
attacking it, you believe to be the end of the world.  That is an
overreaction, enough to justify the term "reactionary" with respect
to this specific issue.

 > For the same reason, though a superficial poll (of readers here
 > or of Emacs users) can tell us something, good arguments for and
 > against are generally more helpful.
 > 
 > > It used to be a Bad Thing[tm].  Now, modern VCS means that such
 > > changes can be made and reverted quickly (although not always
 > > effortlessly or without help of developers experienced in the
 > > software), so rather than poll, just give them what you think they
 > > need.  If they don't like it, you revert.  That method provides far
 > > more accurate assessments of user sentiment than polling in advance,
 > > and is less costly (unless you're stupid enough to make such a
 > > change in a late beta).
 > 
 > I'm not convinced.  Momentum.  Inertia.

Emacs has momentum = 0 in the world at large, and it's basically inert
in its development processes and strategic approach to the goals set
for it by RMS.  You think those are good things?  I don't.

 > And there is no good reason *not* to bring such matters up for
 > discussion first (here, for example), and that can include
 > discussion with users.

Sure there is.  The first good reason for saying nothing is fear of
attracting people who post long repetitive posts basically iterating
and reiterating their personal tastes.  The second good reason is that
most people don't know what they really want, sometimes when they're
quite sure about what they want.  I've not only seen this occur
repeatedly in others, I've experienced preference reversals in Emacs
usage myself several times.

CUA *is* a better default in the context of the world at large, active
regions are generally a better UI, typing deletes active region is
better UI.  I opposed, fairly violently in some cases, all of those,
based on various a priori arguments.  But once I had experienced them
I realized that they are more natural (well, the objective advantages
of CUA are real but tiny, the main advantage is conformance to
convention in the rest of the world, but the region changes provide a
more coherent model of text editing) and are at least as efficient as
the Emacs defaults were.  Of course there are still arguments against
each of those features, but I did change my mind.  And I have seen
others do so, also in the context of specific experience with features
they had opposed.  This change needs to be tested, not voted on a
priori.

 > > Of course the proponents have to be cooperative.  I know that Paul
 > > has a reasonably high opinion of his own opinions ;-), but I've also
 > > seen him revert patches for reasons of "general user taste", and with
 > > very little backtalk, on request.
 > 
 > Consider this one such request, by one user.

It's invalid in context of response to my post, because it's not based
on specific experience but rather on FUD.  That FUD may be based on
relevant general experience -- but given the human bias toward
opposing change, if specific experience can be gained cheaply,
arguments based on general experience should be ignored.

So can we all just shut up now, try the change, and let the leadership
decide whether anything actually broke based on user reports?

The one thing that remains to be said or not that is significant would
be for Paul to acknowledge willingness to back out the change
immediately on request from the project maintainers.  With that,
there's no reason I can see not to try it for a while.

Steve

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Stephen.

On Tue, Jun 16, 2015 at 09:50:29AM +0900, Stephen J. Turnbull wrote:

> And Alan *is* a reactionary ....

Yes.  It's a time consuming and thankless task, but somebody's got to do
it.

> .... and a curmudgeon, ....

I take exception to this.  It's distinctly unparliamentary language, it's
an uncalled for ad hominem.  Outside of free software, you don't even
know me.  Please look up the word's meaning if you need to, and then
don't use it again unless you really do mean it.

> .... quick to oppose changes he doesn't initiate himself ....

This is untrue.

> ....and well-set in his ways.  There's nothing inherently wrong with
> those latter characteristics.  It's just that catering to Alan here
> indeed does cost others an opportunity to improve their environments,
> and offer that improvement to those in the middle.

Emacs has options.  In fact, it might be truer to say Emacs _is_ options.
We are all able enough to make new behaviours and deviations from
standard behaviours optional.  The pertinent new behaviour is not
different in this respect.

[ .... ]

> Steve

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Alan Mackenzie writes:
 > Hello, Stephen.
 > 
 > On Tue, Jun 16, 2015 at 09:50:29AM +0900, Stephen J. Turnbull wrote:
 > 
 > > And Alan *is* a reactionary ....
 > 
 > Yes.  It's a time consuming and thankless task, but somebody's got
 > to do it.

I agree with everything you write here about reactionaries, and my
initial reaction also was reactionary.  Probably for much the same
reasons yours was and remains so, both in general and specific to the
issue of "curly quotes".

However, careful thought informed by 25+ years living in a non-ASCII
culture convinces me that it is time for Emacs to move on.  Move
carefully, yes, cf. Emanuel Berg's post and my reply.  But it's time.

 > > .... and a curmudgeon, ....
 > 
 > I take exception to this.  It's distinctly unparliamentary
 > language,

I wouldn't know.  I suspect if you grep the minutes of the English
Parliament you would find occasional use.  There are six uses in the
Congressional Record for the 113th Congress, three of which are in
articles titled "Tribute to Mr. X", and two in articles titled "IN
MEMORY OF EMANUEL RAYMOND LEWIS, LIBRARIAN EMERITUS OF THE U.S. HOUSE
OF REPRESENTATIVES".  It seems to me that that word puts you in
excellent company, and that's how I meant it, but since you feel
otherwise, I apologize and retract the word.

 > and then don't use it again unless you really do mean it.

Be assured, I won't call you that again.

 > Emacs has options.  In fact, it might be truer to say Emacs _is_
 > options.  We are all able enough to make new behaviours and
 > deviations from standard behaviours optional.  The pertinent new
 > behaviour is not different in this respect.

It is different in this respect.  It is a change in the way Emacs Lisp
programmers communicate with each other, and that is precisely why you
oppose it.  If it were intended to be purely cosmetic, or only used
inside of user strings, then we already have various modes that will
do that for those who want the behavior.  I very much doubt Paul would
have made those patches if this were about appearance only, but we'll
have to see what he says.  I can say I would be against it if it were
about appearance, since we can already have that.

Be warned: this battle has only just begun.[1]  It will soon shake
your windows and rattle your walls.[2]


Footnotes: 
[1]  That's not a threat.  I don't need a weatherman to tell which way
the wind's blowing in this community today.

[2]  Thanks to Bob D. for these words, too.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Stephen J. Turnbull wrote:
> The one thing that remains to be said or not that is significant would
> be for Paul to acknowledge willingness to back out the change
> immediately on request from the project maintainers.

Of course.  That should go without saying.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I think we should aim to represent Lisp code as ASCII so it can be
edited without an input method.

This is a separate question from how Emacs should represent quotations
when it displays doc strings and error messages.  If we indicate quotes
in doc strings and messages using a recognizable convention, the functions
involved can display them however we wish -- or however the user wishes.
It could display them as '...', or as `...', or as Unicode curly quotes
if the terminal can display Unicode curly quotes.

The reason I originally adopted `...' was to permit nested quotations.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> I think we should aim to represent Lisp code as ASCII so it can be
> edited without an input method.

That makes sense for Lisp code itself, but not so much for strings and comments 
that Lisp code contains.  Years ago we started putting non-ASCII characters into 
Lisp strings and comments, and this has worked well.  Going back and imposing an 
ASCII goal or requirement now would would be a maintenance pain for little 
benefit.  For example, it would be much harder to read and change this 
ASCII-only Lisp code:

   (string-match "[^a-zA-Z\u03B1-\u03C9\u0391-\u03A90-9']" name)

than to read and change what's in calc-explain-units-rec now:

   (string-match "[^a-zA-Zα-ωΑ-Ω0-9']" name)

Of course the fact that non-ASCII characters work in Lisp strings does not mean 
that we should necessarily change our source-code strings.  My point here is 
only that the goal of ASCII-only Lisp source files is now often more trouble 
than it's worth, and it should not be a determining factor when considering what 
source-code strings should look like.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think we should aim to represent Lisp code as ASCII so it can be
  > > edited without an input method.

  > That makes sense for Lisp code itself, but not so much for strings
  > and comments that Lisp code contains.

This principle should apply to text in strings, when that text is part
of the conventions of Lisp code.  For instance, how to quote a symbol
for a doc string or an error message.

  > Years ago we started putting non-ASCII characters into 
  > Lisp strings and comments, and this has worked well.

That's a different issue.  It is good to _permit_ putting non-ASCII
characters in strings and comments, as we do; but Lisp code (including
its normal conventions) shouldn't require use of such characters.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Emanuel Berg]

Richard Stallman <rms@gnu.org> writes:

>> Years ago we started putting non-ASCII characters
>> into Lisp strings and comments, and this has
>> worked well.
>
> That's a different issue. It is good to _permit_
> putting non-ASCII characters in strings and
> comments, as we do; but Lisp code (including its
> normal conventions) shouldn't require use of
> such characters.

I have been in a couple of discussions over this in
various places so I'm happy most (?) people think like
me at least here.

I agree everything (like this) should be "permitted",
but I don't see that big a difference between
Lisp code, strings, comments, docstrings, and so on.
If it is unpractical in one place, just because it
isn't as critical perhaps in some other place, it is
still as unpractical.

With computers, ASCII is the most practical thing, and
Unicode should be disencouraged. The Russians,
Chinese, even the noble Spaniards at this point have
absolutely no problem communicating computers in
English, and for this, ASCII is the most
practical way.

The only situation where Unicode (or "whatever" above
ASCII) should be used is when we have a person *using*
the computer, but the activity itself has absolutely
nothing to do with computers. Of course all the people
that have different alphabets or a couple of extra
chars here and there should use exactly that to
communicate, write, read...

But when it is computers, it is English, and it is
ASCII, and everyone will benefit from it, and making
it more advanced (supposedly) will not make for more
advanced computer users. So it is tough love, a bit
sad perhaps, but it is the truth what I can see.
To encourage computer-computer usage in other
languages seems to be tolerant and open-minded but it
boils down to doing a disfavor.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Emanuel Berg writes:

 > But when it is computers, it is English,

True.  There is very good reason to agree on one language to draw
keywords from[1], and similarly for one encoding for characters.  The
choice of English is due to backward compatibility, however -- if we
had chosen a Han language, all keywords and variables could be single
characters, removing the lexing problem from program analysis and
allowing some operation to be denoted implicitly by juxtaposition.[2]
(I don't know if that would be a good change, but it would be an
interesting experiment.  It's pretty much foreclosed for the next
several decades -- Chinese programmers and teachers of programmers,
who are used to English-derived programming languages, would resist
such a change, too!)

 > and it is ASCII,

This restriction, however, has only one justification, and that is
backward compatibility with systems incompetent to display non-ASCII
Unicode, and with humans who prefer not to learn the new conventions
and configure their systems to deal with them conveniently.  One of
the great ideas in Lisp was that parentheses would be the *only*
syntax.  Parentheses are asymmetric delimiters, making Lisp
exceedingly easy and efficient to parse, even from "inside" an
expression.  Then we add comments (with asymmetric delimiters, ";" and
"\n") which are similarly easy to deal with, and then strings -- and
there goes the neighborhood, because it's not possible to determine
which side of '"' is inside and which outside without parsing the
whole program *and* assuming the prefix is well-formed.

The world would be a better place if strings were delimited
asymmetrically.  With Unicode we can do that, and we could also make
some usages (such as the convention for displaying literal symbols in
documentation) prettier.  Since Emacs Lisp is a language that observes,
but does not promise to conform to, external standards, it *could* be a
leader here.

We (and our Lisp parsers) have already learned to deal with the
restriction to ASCII and several instances of symmetric delimiters, so
it's not a big improvement for us, but we would find it useful when
grepping, for example.  I suspect newbies would find it even more
useful, increasing readability.

 > To encourage computer-computer usage in other
 > languages seems to be tolerant and open-minded but it
 > boils down to doing a disfavor.

No, computer-computer usage uses *many* different languages, and we
invent new ones every day (usually called "protocols" in this
context).  It's computer-human usage that demands a single language.

Footnotes: 
[1]  And in fact for long-lived programs the same arguments apply to
docstrings and internal comments!  (And to some extent, developer
documentation in general, although it makes more sense to translate
manuals than internal documentation.)

[2]  Haskell and SmallTalk already do this to a great extent, of
course.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> It is good to _permit_ putting non-ASCII
> characters in strings and comments, as we do; but Lisp code (including
> its normal conventions) shouldn't require use of such characters.

This shouldn't be a problem.  The approach in Emacs master does not require the 
use of non-ASCII characters in source-code docstrings or diagnostics, and 
further changes along these lines should still support ASCII-only strings and 
comments containing quoted symbols that continue to work as before.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Paul Eggert <eggert@cs.ucla.edu> writes:

>> It is good to _permit_ putting non-ASCII characters in strings and
>> comments, as we do; but Lisp code (including its normal conventions)
>> shouldn't require use of such characters.
>
> This shouldn't be a problem.  The approach in Emacs master does not
> require the use of non-ASCII characters in source-code docstrings or
> diagnostics, and further changes along these lines should still
> support ASCII-only strings and comments containing quoted symbols that
> continue to work as before.

But with respect to the coding conventions Oleh cited, it is true that
the use of ‘foo-bar’ instead of `foo-bar' is at least encouraged for
lisp docstrings (and comments) although both will be displayed like the
former with describe-*, right?

To me, having two alternative ways to name lisp symbols in docstrings is
even worse than switching from `...' to ‘...’ in the first place [1].
I'd rather declare the former obsolete and make checkdoc bark at it to
get to some uniformity as soon as possible (which might mean in a bit
less than a decade).

[1] But I agree with Drew that the `...' in `foo-bar' isn't quoting but
    a markup which unambiguously says "foo-bar is a symbol."  In
    contrast, ‘foo-bar’ is ambiguous.  It could be a quote or a symbol.
    Well, of course there aren't too many quotes in lisp files so this
    problem probably isn't too relevant in practice.

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Tassilo Horn wrote:

> I'd rather declare the former obsolete

So would I, but there is some sentiment in this thread for supporting the old 
ASCII-only form.  And even if we want to obsolete the old form we'll need to 
support it for a while, to give users time to switch.

> ‘foo-bar’ is ambiguous.

Not in current Emacs.  It's a quoted symbol when it appears in a docstring, just 
as `foo-bar' is a quoted symbol.  Of course there are exceptions when the quotes 
are themselves escaped, but similar exceptions apply to ASCII quotes.

In practice, curved quotes are a bit less ambiguous than ASCII quotes, because 
one can easily use them to quote any ASCII-only Lisp code, and this is something 
that ASCII quotes can't do.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Paul Eggert <eggert@cs.ucla.edu> writes:

>> ‘foo-bar’ is ambiguous.
>
> Not in current Emacs.  It's a quoted symbol when it appears in a
> docstring, just as `foo-bar' is a quoted symbol.

Well, there could be

(defun foo-bar ()
  "Computes the Foo-Bar sequence.
For a description of the Foo-Bar sequence, see
‘Foo-Bar’, John Doe et al., Proceedings on Foo Bar, 2012."
  ...)

where ‘Foo-Bar’ is a real quote rather than a symbol markup.  Of
courses, such cases are probably rare but still there's a point in
having symbol markup separated from quoting.  (Well, ok, then the above
quote should simply be changed to “Foo-Bar” and we're good again.)

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Tassilo Horn writes:

 > Well, there could be
 > 
 > (defun foo-bar ()
 >   "Computes the Foo-Bar sequence.
 > For a description of the Foo-Bar sequence, see
 > ‘Foo-Bar’, John Doe et al., Proceedings on Foo Bar, 2012."
 >   ...)
 > 
 > where ‘Foo-Bar’ is a real quote rather than a symbol markup.  Of
 > courses, such cases are probably rare but still there's a point in
 > having symbol markup separated from quoting.  (Well, ok, then the
 > above quote should simply be changed to “Foo-Bar” and we're good
 > again.)

Use of double quotes in this context is the standard convention for
U.S. academics, so to that extent it's even less common than you
suggest.  Nevertheless, we should permit single quotes.  Then
\-escaping will allow you to specify that the quote is not
syntactically significant, as usual.

I don't really see a difference between "‘" and "`" for this purpose,
which is why I can only give lukewarm support to the proposal.  I
consider double quotes (for trings and the like) a much more important
direction.  But that's a much bigger step, which could easily prove
infeasible.

Paul's patch is a small step toward exploiting the possibilities for
improved syntax via Unicode that doesn't give us much, but if Alan,
Drew, and RMS are right, it isn't going to hurt anybody much while
we're trying it, either.  I expect this experiment to produce both
negative and positive surprises -- that's why we do experiments, after
all -- because the results occasionally surprise us!

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> [1] But I agree with Drew that the `...' in `foo-bar' isn't quoting but
>     a markup which unambiguously says "foo-bar is a symbol."  In
>     contrast, ‘foo-bar’ is ambiguous.  It could be a quote or a symbol.
>     Well, of course there aren't too many quotes in lisp files so this
>     problem probably isn't too relevant in practice.

As you noticed, indeed, ‘foo-bar’ can be ambiguous, but `foo-bar' can
*also* be ambiguous.  Actually, there are many more uses of quotes and
backtits in Elisp docstrings and comments which are not used for markup,
then there are of ‘ and ’.

To me, one of the motivations for the change (after the visual aspect) is
to have a *more* robust handling because it is *less* ambiguous (within
the context of Elisp).


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/17/2015 04:03 PM, Stefan Monnier wrote:

> To me, one of the motivations for the change (after the visual aspect) is
> to have a *more* robust handling because it is *less* ambiguous (within
> the context of Elisp).

That is probably true, but by that logic, using rare unicode characters 
is a great choice for any markup language. Yet, I don't see anybody 
doing that.

Another reason to use ‘‘ instead of anything else, seems to be that 
we're going to translate `' into those anyway. But that won't 
necessarily hold true forever: the quotes themselves aren't required.

Markdown, for instance, when rendered, only emphasizes code segments 
using a special tag, which translates into a different font 
face/color/etc. I don't see why we won't choose to do that, or allow 
users to customize that aspect.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> That is probably true, but by that logic, using rare unicode characters
> is a great choice for any markup language. Yet, I don't see anybody
> doing that.

Good point.  Instead they use some kind of escaping convention so as to
avoid ambiguity.  We could try that.


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/18/2015 07:24 AM, Stefan Monnier wrote:

> Good point.  Instead they use some kind of escaping convention so as to
> avoid ambiguity.  We could try that.

And we already do!

We use `\\=', as documented in `substitute-command-keys'.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

>> Good point.  Instead they use some kind of escaping convention so as to
>> avoid ambiguity.  We could try that.
> And we already do!
> We use `\\=', as documented in `substitute-command-keys'.

Several problems:
- it's butt-ugly.
- I can never remember it (why can't we use a standard quoting convention).
- The ambiguity shows up not only in the source docstring, but also in
  the *formatted* docstring after this quoting has been removed
  (e.g. regexp to recognize `...' xrefs and highlight them).
- Same ambiguity shows up in other formatted outputs (e.g. Info files).

IOW there are various places where `...' is used and some of them offer
some kind of quoting while others don't.


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Stefan Monnier wrote:
> Several problems:
> - it's butt-ugly.
> - I can never remember it (why can't we use a standard quoting convention).

It is indeed ugly and hard to remember.  Its main advantage is backward 
compatibility.  All the other quoting conventions I could think of would have 
caused new-style docstrings to not work in old-style Emacs.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/19/2015 05:39 AM, Stefan Monnier wrote:

> Several problems:
> - it's butt-ugly.

Yeah, I'm fine with changing it, if the backward compatibility is not a 
problem.

> - The ambiguity shows up not only in the source docstring, but also in
>    the *formatted* docstring after this quoting has been removed
>    (e.g. regexp to recognize `...' xrefs and highlight them).

Maybe we should have gone ahead with the original proposal, and handle 
the escaping in font-lock. Then `substitute-command-keys' would honor 
\\=, but won't remove it from the docstring.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> Maybe we should have gone ahead with the original proposal, and handle the
> escaping in font-lock. Then `substitute-command-keys' would honor \\=, but
> won't remove it from the docstring.

I was thinking more about substitute-command-keys removing the escape but
replacing it with text-properties.


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/19/2015 05:07 PM, Stefan Monnier wrote:

> I was thinking more about substitute-command-keys removing the escape but
> replacing it with text-properties.

Maybe. I think that would really call for a new function, though.

That aside, hardcoding the usage of a specific property might not be 
ideal, because then the buffer using the output can't change the 
presentation too easily.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

>> I was thinking more about substitute-command-keys removing the escape but
>> replacing it with text-properties.
> Maybe. I think that would really call for a new function, though.

Indeed.

> That aside, hardcoding the usage of a specific property might not be ideal,
> because then the buffer using the output can't change the presentation
> too easily.

I was thinking of an ad-hoc property rather than a predefined one.


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/19/2015 11:01 PM, Stefan Monnier wrote:

>> That aside, hardcoding the usage of a specific property might not be ideal,
>> because then the buffer using the output can't change the presentation
>> too easily.
>
> I was thinking of an ad-hoc property rather than a predefined one.

That seems non-ideal because the regexp engine has no way to check the 
text property on a quote character. Thus, we can't make a regexp that 
will match asd'asd in `asd\\'asd', if the escaped is handled in 
substitute-command-keys.

Although admittedly, this could be handled programmatically.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/28/2015 10:17 PM, Dmitry Gutov wrote:

> That seems non-ideal because the regexp engine has no way to check the
> text property on a quote character. Thus, we can't make a regexp that
> will match asd'asd in `asd\\'asd', if the escaped is handled in
> substitute-command-keys.
>
> Although admittedly, this could be handled programmatically.

Never mind, I just did that in a functional matcher.

And there's no reasonable way to get by with just a regexp with Paul's 
latest requirements.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/18/2015 07:24 AM, Stefan Monnier wrote:

> Good point.  Instead they use some kind of escaping convention so as to
> avoid ambiguity.  We could try that.

Could you maybe retract your support for using curlies in Elisp source 
code, then?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

>> Good point.  Instead they use some kind of escaping convention so as to
>> avoid ambiguity.  We could try that.
> Could you maybe retract your support for using curlies in Elisp source
> code, then?

I don't have a strong opinion on the markup used in the source code, as
long as we resolve the currently existing ambiguity (and as long as we
can render the result with symmetric quotes).


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/26/2015 10:26 PM, Stefan Monnier wrote:

> I don't have a strong opinion on the markup used in the source code, as
> long as we resolve the currently existing ambiguity (and as long as we
> can render the result with symmetric quotes).

What are the requirements?

Render in the Help buffer, or prettify them the source buffer too? The 
former is doable, the latter is inherently error-prone.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Dmitry Gutov wrote:
> Render in the Help buffer, or prettify them the source buffer too? The former is
> doable, the latter is inherently error-prone.

Yes, let's give up on the idea of prettifying them in source buffers.  Even if 
we could get it to mostly work, it'd be a recipe for confusion.

We do need to prettify them unambiguously in help buffers, in diagnostics, in 
menus (I just discovered this one), and probably in other places.  I haven't had 
time yet to investigate Dmitry's prototype for prettifying them in help buffers 
yet, but plan to do this soon.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> Render in the Help buffer, or prettify them the source buffer too?
> The former is doable, the latter is inherently error-prone.

Let's focus on the Help buffers first.  The prettification in source
buffers can be considered afterwards and will probably depend on the
solution chosen for the Help buffers (e.g. if we decide to use
curly-quotes in the source, then there won't be anything to do to
prettify the source code).


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/27/2015 05:08 AM, Stefan Monnier wrote:

> Let's focus on the Help buffers first.  The prettification in source
> buffers can be considered afterwards and will probably depend on the
> solution chosen for the Help buffers (e.g. if we decide to use
> curly-quotes in the source, then there won't be anything to do to
> prettify the source code).

Considering that if there's an escaping rule (and we've more or less 
agreed that there should be), the choice of using curly quotes in source 
buffers or not is orthogonal to the choice of the method of the escaping 
rule's application (font-lock or not).

So we'll only "have to" use the curlies in the source code if the 
escaping syntax (and its application) are determined not to be good enough.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> Considering that if there's an escaping rule (and we've more or less agreed
> that there should be), the choice of using curly quotes in source buffers or
> not is orthogonal to the choice of the method of the escaping rule's
> application (font-lock or not).

That's right.  So we should focus on the escape rules.


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/28/2015 08:00 AM, Stefan Monnier wrote:

> That's right.  So we should focus on the escape rules.

Maybe I should point out again that even in master we have an 
unambiguous escaping syntax for quotes ("\\=").

Personally, I think that it's good enough (and Paul apparently agrees).

The font-lock solution has other benefits; it could implement a 
different scheme (or delegate the escaping step to 
substitute-command-keys, which seems a fine idea now, actually). But it 
currently uses the syntax that's been requested: "\\~".

So, could you participate in this discussion? What's wrong with "\\="? 
What would you prefer to see use instead? Otherwise, what areas would 
you like to see improved? It's inherently a bikeshed issue, not a 
(particularly) technical one.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> Maybe I should point out again that even in master we have an unambiguous
> escaping syntax for quotes ("\\=").

Why can't we use just \ instead of \= ?


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/30/2015 04:50 PM, Stefan Monnier wrote:

> Why can't we use just \ instead of \= ?

We can (and that's what my first proposal did), but it'll require 
changing a certain amount of existing docstrings, and some of them will 
look less nice as a result. Such as substitute-command-keys one, where 
we'll have to escape all those backslashes. Adding a yet-another quoting 
method is not ideal either.

However, if we change \= to \ as the quoting method in 
substitute-command-keys as well, that will be less of a problem. 
Backward compatibility might be, but I've seen no evidence thus far, of 
\= being used anywhere outside of Emacs core.

http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00646.html

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Andreas Schwab]

Dmitry Gutov <dgutov@yandex.ru> writes:

> However, if we change \= to \ as the quoting method in
> substitute-command-keys as well, that will be less of a problem. Backward
> compatibility might be, but I've seen no evidence thus far, of \= being
> used anywhere outside of Emacs core.

But it will break doc strings that use \.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/30/2015 06:32 PM, Andreas Schwab wrote:

> But it will break doc strings that use \.

Indeed. They will need to be updated, and after that, will be broken in 
the previous Emacs versions.

So there's a tradeoff.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Dmitry Gutov wrote:
> However, if we change \= to \ as the quoting method in substitute-command-keys
> as well, that will be less of a problem. Backward compatibility might be, but
> I've seen no evidence thus far, of \= being used anywhere outside of Emacs core.

The problem with using \\ to quote in docstring sources is not that \\= is 
rarely used; it's that \\X is reasonably commonly used for lots of different 
values of X, and it's expected that \\X normally displays as \X.  If we decide 
to change the meaning of \\` and \\' (but not of \\ followed by other chars), 
then we'll have to change maybe 20 docstrings in the Emacs source code, and a 
few similar changes will no doubt be needed in 3rd-party packages.  This is why 
the current master uses \\= to quote quotes (as \\= has always worked so this is 
compatible with older Emacs).  And it's partly why I originally objected to \\` 
and \\'.

That being said, Stefan is right that \\` and \\' are shorter than \\=` and \\=' 
and in that sense are nicer.

A couple more downsides of \\` should be mentioned, though, before making this 
not-quite-compatible change.


First, \` and \' already have a special meaning in docstring sources (unrelated 
to this issue) and it will be confusing for people to read docstring source code 
like this:

"LISTIFIED is a list representing each topic header and body:

  \`(depth prefix text)'

or \`(depth prefix text bullet-plus)'"

if people know that backslash before grave accent means a grave accent, as in 
the above examples \` would still mean left quote, not grave accent.


Second and more important, here's an example of a docstring that would need to 
be changed:

"Face for characters displayed as sequences using `^' or `\\'."

Presumably this would be changed to:

"Face for characters displayed as sequences using `^' or `\\\\'."

But this would mean that we need to change the meaning of \\\\ in docstring 
sources too, and that's a bigger deal and would affect more than 20 or so 
docstrings.  For example, we'd need to change this docstring source:

"REGEXP must contain at least one parenthesized subexpression, typically
whitespace of the form \"\\\\(\\\\s-*\\\\)\"."

to this:

"REGEXP must contain at least one parenthesized subexpression, typically
whitespace of the form \"\\\\\\\\(\\\\\\\\s-*\\\\\\\\)\"."

I'm not sure this is a good idea.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> That being said, Stefan is right that \\` and \\' are shorter than \\=` and
> \\=' and in that sense are nicer.

Shorter is good, but to me the main problem is that I can never remember
this \= escaping (tho maybe after this longish discussion, I'll finally
remember it).  So if even I can't remember it, I can't expect other
people to remember it and even less to use it.

\ escaping OTOH is the standard escaping method.

> First, \` and \' already have a special meaning in docstring sources
> (unrelated to this issue)

I don't know which meaning you're referring to.

> and it will be confusing for people to read
> docstring source code like this:
>
> "LISTIFIED is a list representing each topic header and body:
>
>  \`(depth prefix text)'
>
> or \`(depth prefix text bullet-plus)'"

You lost me here.  Why would someone write the above docstring?
What's the intended meaning?

> Second and more important, here's an example of a docstring that would need
> to be changed:
> "Face for characters displayed as sequences using `^' or `\\'."
> Presumably this would be changed to:
> "Face for characters displayed as sequences using `^' or `\\\\'."
> But this would mean that we need to change the meaning of \\\\ in docstring
> sources too, and that's a bigger deal and would affect more than 20 or so
> docstrings.  For example, we'd need to change this docstring source:

Right, so the issue here is that we need to be able to escape the escape
char, and if we use \\ for that (which is the only choice if we want to
follow the standard escaping mechanism), then we have many places where
we'll have to double the backslashes (and those have already been
doubled for other layers of escaping).


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/30/2015 08:54 PM, Stefan Monnier wrote:

> Shorter is good, but to me the main problem is that I can never remember
> this \= escaping (tho maybe after this longish discussion, I'll finally
> remember it).  So if even I can't remember it, I can't expect other
> people to remember it and even less to use it.

I never knew about it until now, and now I'll probably remember it for a 
while, personally.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Stefan Monnier wrote:
>> \` and \' already have a special meaning in docstring sources
>> >(unrelated to this issue)
> I don't know which meaning you're referring to.

After looking into this I decided that \` and \' are typically mistakes.

I had thought that they may have sometimes have been inserted to avoid 
highlighting when viewing the sources.  For example, a source docstring like 
"...\`foo\'..." acts like "...`foo'..." except that when one is viewing the 
source the "foo" is not highlighted (and of course one sees the 
otherwise-unnecessary backslashes).  However, highlighting occurs anyway when 
the docstring is copied into a *Help* buffer, which means that in some sense 
it's misleading to try to suppress the source highlighting.

There appears to be no rhyme or reason to the few instances of backslash-quotes 
in strings, some of which are obviously just typos, so I took the liberty of 
eliding them in master commit 3213d7707026573ca425ba1c865b7fa1a8b46639.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

>> Why can't we use just \ instead of \= ?
> We can (and that's what my first proposal did), but it'll require
> changing a certain amount of existing docstrings,

Why?  [ Assuming we still accept "\=".  ]

> and some of them will look less nice as a result.

Do you have examples?

> Such as substitute-command-keys one, where we'll have to escape all
> those backslashes.

AFAICT they're already escaped (with \=), so that would not be worse, or
am I missing something?

> Adding a yet-another quoting method is not ideal either.

That's a downside, but \ is "the standard quoting method", so I think
overall it could be counted as removing an escaping method rather than
adding one.

> However, if we change \= to \ as the quoting method in
> substitute-command-keys as well, that will be less of
> a problem.

Hmm... yes, the idea is to use \ instead of \=, everywhere.

> Backward compatibility might be, but I've seen no evidence
> thus far, of \= being used anywhere outside of Emacs core.

We'd probably want to keep supporting \= for a while anyway.

>>>>> "Andreas" == Andreas Schwab <schwab@suse.de> writes:
> But it will break doc strings that use \.

Which case are you thinking of?


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/30/2015 08:41 PM, Stefan Monnier wrote:

>> We can (and that's what my first proposal did), but it'll require
>> changing a certain amount of existing docstrings,
>
> Why?  [ Assuming we still accept "\=".  ]

Simply because must \ occur more frequently than \=. Also, depending on 
how "we still accept" is handled, we might have to double the 
backslashes in some of occurrences of \=\= (e.g. if the \ escaping is 
handled in Lisp later).

> AFAICT they're already escaped (with \=), so that would not be worse, or
> am I missing something?

See above. With the my first proposed patch, at least, you'd have to 
change the current

thus, \\=\\=\\=\\= puts \\=\\= into the output,

to something like

thus, \\\\=\\=\\\\=\\= puts \\\\=\\= into the output,

>> Adding a yet-another quoting method is not ideal either.
>
> That's a downside, but \ is "the standard quoting method", so I think
> overall it could be counted as removing an escaping method rather than
> adding one.

If both are retained, even if they're both implemented in 
substitute-command-keys, I think they're bound to interact in 
interesting and unexpected ways.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Dmitry Gutov wrote:
>
> The font-lock solution has other benefits; it could implement a different scheme
> (or delegate the escaping step to substitute-command-keys, which seems a fine
> idea now, actually). But it currently uses the syntax that's been requested: "\\~".

I thought that font-lock couldn't use the same escape syntax that 
substitute-command-keys does, and that this was why you suggested \~ rather than 
\= -- i.e., so that one could use \= to escape characters that 
substitute-command-keys would otherwise interpret, and use \~ to escape 
characters that font-lock would otherwise interpret.

It would be better to use the one escape syntax, as it's one less thing to 
explain to Emacs users.  But how would it work?  I don't see how.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/30/2015 05:37 PM, Paul Eggert wrote:

> I thought that font-lock couldn't use the same escape syntax that
> substitute-command-keys does, and that this was why you suggested \~
> rather than \= -- i.e., so that one could use \= to escape characters
> that substitute-command-keys would otherwise interpret, and use \~ to
> escape characters that font-lock would otherwise interpret.

Yes. That was while before you asked to only consider paired quotes, and 
I hoped to avoid text property lookups in help--translate-quotes. It 
looks up `help-tilde-escaped' now.

> It would be better to use the one escape syntax, as it's one less thing
> to explain to Emacs users.  But how would it work?  I don't see how.

substitute-command-keys will add a `quoted' (or `escaped'? I'm fuzzy on 
the difference) text property to the characters that were escaped in its 
input. Thus translating "\\=`'" to #("`'" 0 1 (escaped t)) (if we're 
still using "\\=" there).

Then help--translate-quotes will check that property, instead of 
`help-tilde-escaped'.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Andy Moreton]

On Tue 30 Jun 2015, Paul Eggert wrote:

> Dmitry Gutov wrote:
>>
>> The font-lock solution has other benefits; it could implement a different scheme
>> (or delegate the escaping step to substitute-command-keys, which seems a fine
>> idea now, actually). But it currently uses the syntax that's been requested: "\\~".
>
> I thought that font-lock couldn't use the same escape syntax that
> substitute-command-keys does, and that this was why you suggested \~ rather
> than \= -- i.e., so that one could use \= to escape characters that
> substitute-command-keys would otherwise interpret, and use \~ to escape
> characters that font-lock would otherwise interpret.
>
> It would be better to use the one escape syntax, as it's one less thing to
> explain to Emacs users.  But how would it work?  I don't see how.

To go off on a tangent slightly, is there a good reason why the
documentation is a string containing escapes rather than a sexp using
the normal quoting mechanisms ?

    AndyM

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Andy Moreton wrote:
> is there a good reason why the
> documentation is a string containing escapes rather than a sexp using
> the normal quoting mechanisms ?

One good reason is inertia. :-)  There's a lot of code dealing with docstrings; 
e.g., they can be in C code as well as in Lisp, and are processed specially by 
source transformation.  That being said, how would the sexps work, and why would 
they be an improvement?  Can you give an example?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Andy Moreton]

On Wed 01 Jul 2015, Paul Eggert wrote:

> Andy Moreton wrote:
>> is there a good reason why the
>> documentation is a string containing escapes rather than a sexp using
>> the normal quoting mechanisms ?
>
> One good reason is inertia. :-)  There's a lot of code dealing with
> docstrings; e.g., they can be in C code as well as in Lisp, and are processed
> specially by source transformation.  That being said, how would the sexps
> work, and why would they be an improvement?  Can you give an example?

This was nothing more than idle musing. Doc strings contain markup for
quoted or interpolated expressions, which seem to match what quote and
backquote do in ordinary elisp.

    AndyM

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't have a strong opinion on the markup used in the source code, as
  > long as we resolve the currently existing ambiguity

Which "currently existing ambiguity" is this?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

>> I don't have a strong opinion on the markup used in the source code, as
>> long as we resolve the currently existing ambiguity
> Which "currently existing ambiguity" is this?

We can't currently reliably render the `...' markup as curly quotes
(e.g. via font-lock), because the `...' is ambiguous: we may find `...'
in places where the ` and the ' are not related, or where the real
closing quote is actually further (because the quoted text includes
a quote).


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We can't currently reliably render the `...' markup as curly quotes
  > (e.g. via font-lock), because the `...' is ambiguous:

I think we can, if we decide not to use them in doc strings around lists.
That way, `( will always be a backquote.
` will never occur before a symbol character
except when it is part of `...'.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> I think we can, if we decide not to use them in doc strings around lists.
> That way, `( will always be a backquote.
> ` will never occur before a symbol character
> except when it is part of `...'.

Unfortunately that's not true for the current Emacs docstrings.  For example, 
the docstring for verilog-auto-declare-nettype contains this:

Note using `default_nettype none isn't recommended practice;

Here the grave accent is part of Verilog syntax, and should not be displayed as 
a curved quote.  Of course this sort of thing is unusual, but the point is that 
exceptional cases like this do occur and we need an escape syntax to handle them.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Unfortunately that's not true for the current Emacs docstrings.  For example, 
  > the docstring for verilog-auto-declare-nettype contains this:

  > Note using `default_nettype none isn't recommended practice;

It is easy to change the rule to reject this as a quotation
because there is no close-quote on the same line.

I don't think we need a solution that is totally general
for everything we might possibly want to put in doc strings.
I'm not against one, if it is clean and painless.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
>    > Note using `default_nettype none isn't recommended practice;
>
> It is easy to change the rule to reject this as a quotation
> because there is no close-quote on the same line.

Yes there isn't.

That's a joke.  The point is that there is a close quote in the same line, in 
the word "isn't".  Of course that apostrophe is not intended to be a close 
quote, and we could alter the rules further to recognize it as not being a close 
quote, but whatever rules we end up with would inevitably be more complex and 
harder to document and quite possibly would still need exceptions.  Among other 
things, occasionally quotations do cross line boundaries in doc strings.

> I don't think we need a solution that is totally general
> for everything we might possibly want to put in doc strings.
> I'm not against one, if it is clean and painless.

We already have one in master that's reasonably clean.  Dmitry has proposed a 
different one that I plan to look at soon; if it works it should be even better. 
  So the problem is solvable.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes there isn't.

  > That's a joke.  The point is that there is a close quote in the same line, in 
  > the word "isn't".

I didn't notice that.  Oops!

  > > I don't think we need a solution that is totally general
  > > for everything we might possibly want to put in doc strings.
  > > I'm not against one, if it is clean and painless.

  > We already have one in master that's reasonably clean.

What is the solution that is installed now?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> What is the solution that is installed now?

When a grave accent and any matching apostrophe are copied from a docstring to a 
*Help* buffer, they are converted to left and right quotation marks according to 
user preference.  Characters can be escaped with \= to avoid conversion (this is 
a longstanding escape convention in docstrings).

Most existing docstrings don't need to be changed.  For example, in the source 
code the docstring for eval-after-load might contain this:

   "FORM can be an Elisp expression (in which case it's passed to `eval'),
   or a function (in which case it's passed to `funcall' with no arg)."

For this docstring if the user preference is curved quotes the *Help* buffer 
contains this:

   FORM can be an Elisp expression (in which case it's passed to ‘eval’),
   or a function (in which case it's passed to ‘funcall’ with no arg).

If the user preference is traditional, the *Help* buffer looks like this:

   FORM can be an Elisp expression (in which case it's passed to `eval'),
   or a function (in which case it's passed to `funcall' with no arg).


A few doc strings have unusual quoting and use \=X escape sequences to display a 
quoting character X without transformation.  For example, in the source code the 
docstring for texinfo-format-verb might contain this:

   "For example, @verb\{|@|\} results in @ and
   @verb\{+@'e?\\=`!\\=`+} results in @'e?\\=`!\\=`."

For this docstring the *Help* buffer contains the following, regardless of user 
preference for quoting, because the grave accents are escaped:

   For example, @verb{|@|} results in @ and
   @verb{+@'e?`!`+} results in @'e?`!`.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When a grave accent and any matching apostrophe are copied from a docstring to a 
  > *Help* buffer, they are converted to left and right quotation marks according to 
  > user preference.  Characters can be escaped with \= to avoid conversion (this is 
  > a longstanding escape convention in docstrings).

Oh, that's the feature you mean.  I'm fine with that.

  > A few doc strings have unusual quoting and use \=X escape sequences to display a 
  > quoting character X without transformation.

I have no complaints.  It is not a screw to edit, and it doesn't
need to be used often.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Good point.  Instead they use some kind of escaping convention so as
>>> to avoid ambiguity.  We could try that.
>> Could you maybe retract your support for using curlies in Elisp
>> source code, then?
>
> I don't have a strong opinion on the markup used in the source code,
> as long as we resolve the currently existing ambiguity (and as long as
> we can render the result with symmetric quotes).

After this discussion, I'm now pretty much indifferent on these changes
and if they should apply to display only or also to actual source code.
But if the main problem with `foo' is its ambiguity, and ‘foo’ is still
ambiguous (but less so), why not use something which is even less
ambiguous?

For example, I've seen uses of ⸢foo⸥ for marking up source code
snippets.  That seems pretty unambiguous (although you can never be
sure), and as long you can type it easily and display it however suits
you best, it's as good as anything else.

Bye,
Tassilo

BTW: On a related note: I don't get why *Help* buffers display quotes at
all when they are already displayed with a face that makes them
distinguishable from "normal" text and are clickable.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/28/2015 08:52 PM, Tassilo Horn wrote:

> For example, I've seen uses of ⸢foo⸥ for marking up source code
> snippets.  That seems pretty unambiguous (although you can never be
> sure), and as long you can type it easily and display it however suits
> you best, it's as good as anything else.

If we're prepared to change the markup radically, it would be better to 
use something like a tilde character, which is both ASCII and very rare 
in our docstrings.

> BTW: On a related note: I don't get why *Help* buffers display quotes at
> all when they are already displayed with a face that makes them
> distinguishable from "normal" text and are clickable.

Not the same thing. We only linkify the names of already-loaded (or 
autoloaded) functions and variables.

Whereas we also use the quotes for other cases, like key sequences (`C-h 
k') and arbitrary `symbols' (for example when describing possible values 
of a variable or a function argument).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 06/28/2015 08:52 PM, Tassilo Horn wrote:
>
>> For example, I've seen uses of ⸢foo⸥ for marking up source code
>> snippets.  That seems pretty unambiguous (although you can never be
>> sure), and as long you can type it easily and display it however
>> suits you best, it's as good as anything else.
>
> If we're prepared to change the markup radically, it would be better
> to use something like a tilde character, which is both ASCII and very
> rare in our docstrings.

That wasn't really meant as a suggestion.  I just wanted to ask why not
use some extremely rare paired characters if unambiguity is the main
point and we have some convenient input method anyhow.  That simply
typing a key which has the exact letter on it is even better is obvious.

>> BTW: On a related note: I don't get why *Help* buffers display quotes
>> at all when they are already displayed with a face that makes them
>> distinguishable from "normal" text and are clickable.
>
> Not the same thing.  We only linkify the names of already-loaded (or
> autoloaded) functions and variables.
>
> Whereas we also use the quotes for other cases, like key sequences
> (`C-h k') and arbitrary `symbols' (for example when describing
> possible values of a variable or a function argument).

Ok, right.  Basically, what I meant to say is that @kbd{} in texinfo and
`...' in elisp docstrings is markup which says that's a
keybinding/symbol/code snippet.  I don't see why this would imply that
it needs to be displayed with quote characters, it just needs to be
displayed as something "non-prose-y."

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/28/2015 11:28 PM, Tassilo Horn wrote:

> That wasn't really meant as a suggestion.  I just wanted to ask why not
> use some extremely rare paired characters if unambiguity is the main
> point and we have some convenient input method anyhow.  That simply
> typing a key which has the exact letter on it is even better is obvious.

I would understand it if it was meant as a sarcastic remark.

We don't have a convenient input method for ⸢ and ⸥ and I'd prefer not 
to have a need for it anyway (nor for similar characters).

> Ok, right.  Basically, what I meant to say is that @kbd{} in texinfo and
> `...' in elisp docstrings is markup which says that's a
> keybinding/symbol/code snippet.  I don't see why this would imply that
> it needs to be displayed with quote characters, it just needs to be
> displayed as something "non-prose-y."

Yes, we might want to at least provide an option to display them 
differently (that has been brought up before, by yours truly). Handling 
the quotes in font-lock should make implementing that easy enough.

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> That wasn't really meant as a suggestion.  I just wanted to ask why
> not use some extremely rare paired characters if unambiguity is the
> main point and we have some convenient input method anyhow.  That
> simply typing a key which has the exact letter on it is even better
> is obvious.

FWIW, a week or so ago I almost posed the same rhetorical question.

I was going to use this pair as an example [1]:

◄foobar►    BLACK (LEFT|RIGHT)-POINTING POINTER

Here it is, used for sexps and key descriptions we might show
in Lisp docs/help:

◄foobar►                = `foobar'
◄"foobar toto"►         = `"foobar toto"'
◄(foobar toto 42)►      = `(foobar toto 42)'
◄`(setq foo ',(bar t))► = ``(setq foo ',(bar t))'
◄M-x apropos-char►      = `M-x apropos-char'
◄C-x C-c►               = `C-x C-c'

Here are the same sexps shown with curly quotes:

‘foobar’                = `foobar'
‘"foobar toto"’         = `"foobar toto"'
‘(foobar toto 42)’      = `(foobar toto 42)'
‘`(setq foo ',(bar t))’ = ``(setq foo ',(bar t))'
‘M-x apropos-char’      = `M-x apropos-char'
‘C-x C-c’               = `C-x C-c'

Tassilo's point here, I think, and at least my point here, is
that this is *not at all about quotation* in the usual sense.
There is no reason to use quotation marks of any kind, which
can mess with their ordinary use as, well, er, um, quotation
marks.

Those who make claims to the effect that 99.999% of the world
uses curly quotes miss the point entirely, IMHO.  Of course
they do!  (Well, maybe not five 9s.)  And so do I.

But they use them for text *quotation*.  That's not what we
use `...' for.  We use it to set off code phrases from
surrounding text that talks about them.  It is our equivalent
of markup such as <code>...</code> and <kbd>...</kbd>.

Yes, of course, *any* mention instead of use is a form of
quotation in the larger, logical sense.  Quoting means pushing
up a metalanguage level, and vice versa.

But that's really beside the point here.  We have a context
where we want to distinguish code from text, including from
text quotations.  Emacs is, among other things, a general
text editor.

And we would like, if possible, the external form to be the
same as the internal form.  If we can avoid it, we would
prefer not to have <code>...</code> or similar be what is
written in a file and have that just be rendered using
highlighting or a special font - which is the typical
treatment.  We could live with that approach, no doubt, but
it is not what we would prefer.

Why?  Because we want to act on the internal form directly,
including interactively, in ways that other contexts do not,
or that do so only inelegantly, clumsily, or in a restricted
way.  We are spoiled, I guess, having had the simplicity of
Lisp syntax and simple `...' sexp mention for decades now.

It's true that you can use a good XML editor to both render
XML-based documents in a formatted way and let you search for
text that is within (or without) <code>...</code> elements.
But the UIs they provide, including for operations such as
search-&-replace and other modifications, are far too clunky
and limited for what we do.

Can we do what others do, having a different internal
representation from what gets rendered?  Of course.  And some
are arguing that that's the approach to take wrt curly quotes,
to paper over some of the warts.

Do we want to have to fiddle with two different levels that
way: internal vs rendered representations?  Of course not,
if we can reasonably avoid it.

Emacs is about Lisp for a reason.  And Lisp is about
representing programs and data the same way - for a reason.
If we can avoid such a translation and flipping between
representations, we would prefer it in general - direct
action, WYSIWYG.

So why do I say (as Tassilo also suggests, I think) that
◄foobar► is only a *rhetorical* suggestion?  Why don't I
really propose such a solution?  (And why didn't I bother
to mention it a week ago?)

For all of the Emacsy reasons people have so far given that
explain why curly quotes are a bad idea: trouble inserting,
searching, etc.  That is, for all of the _other_ reasons,
besides the reason that this is not about quotation.

What we should look for is both (1) a mirror pair of chars
that is not confusable with other uses of the same and
(2) a pair of chars that is simple to insert, search, etc. -
in general, simple to use in a rudimentary, plain-text
(perhaps even ASCII) context.

Criterion #1 rules out pairs such as curly quotes and
guillemets of all stripes, as well as [], (), {}, <> etc.

Criterion #2 rules out most non-ASCII Unicode chars.

(There is a third criterion that makes life easier: a pair
of *different* chars, not just using the same char twice:
'...', `...`, |...|, \...\, /.../, etc.  This is maybe not
essential, but it is handy.)

These two criteria work against each other to some extent.

To fit #1 we want a pair that is *rarely used* in typical
contexts, including documentation and other ordinary text
contexts (which make use of quotation).  But to fit #2 we
want a pair that must not be rare, because rare chars are
not easily typed, searched for, and otherwise used in even
a rudimentary plain-text context.

Maybe someone has a truly brilliant idea for this?

Note that we are in this dilemma, with these opposing
criteria, precisely because we have (till now) opted to
not use two different representations: internal and
external.  Sacrificing that simplicity and going to two
representations is one way to go, I guess, but it is not
one I favor.

`...' is the most brilliant idea I've seen so far for this.
Satisifies all of the criteria.  Simple; straightforward;
just works.  It ain't broke, and we shouldn't fix it.

But it has *one* strike against it: a few people find it
“ugly”, ‟ugly”, "ugly", ‘ugly’, ‟ugly„, 〝ugly〞, ❝ugly❞,
❛ugly❜, «ugly» !

Some of us don't find it so, but yes, beauty is in the eye
of the beholder.  Why someone's beauty concerns should trump
usefulness I don't know.  Mais on n'arrete pas le progres...

---
[1] Believe it or not, I scanned all of the Unicode chars for
a good example - a pair that works in terms of char width (at
least for the fonts I tried), one that has no particular other
use with which this use might be confused, and one that is not
easily confused with other pairs (e.g. various forms of brackets).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Drew Adams <drew.adams@oracle.com> writes:

> Tassilo's point here, I think, and at least my point here, is that
> this is *not at all about quotation* in the usual sense.

Yes, Drew, your points are pretty much what I've meant to say here, too.
Plus the additional point that if we're going to use something which is
better to type with some input aid anyway, then we could also go the
complete step and have a real Docstring Markup Language which would
allow to refer to foo-bar the function (not the variable), or to
insert-char the symbol (not the function).

Basically, with ‘...’ instead of `...' it might be a bit less ambiguous
that ... is something special but what kind of special thing ... is
still needs to be grasped from the context (and by the heuristics that
generate the *Help* buffers).  So with some new-style docstrings we can
solve the technical ambiguity (where does `(foo 'bar)' end?) and the
semantical ambiguity (does the author refer to foo-bar the function, the
variable, or the symbol?), too.

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Sun, 28 Jun 2015 16:05:06 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, rms@gnu.org, emacs-devel@gnu.org,
> 	Stefan Monnier <monnier@iro.umontreal.ca>, acm@muc.de, stephen@xemacs.org
> 
> What we should look for is both (1) a mirror pair of chars
> that is not confusable with other uses of the same and
> (2) a pair of chars that is simple to insert, search, etc. -
> in general, simple to use in a rudimentary, plain-text
> (perhaps even ASCII) context.

If you want mirror pairs, you should select only those pairs that are
mentioned in the BidiBrackets.txt file that is part of the Unicode
Character Database (UCD).  You can see them in Emacs by looking at
characters that have non-nil 'paired-bracket' property.

If you select any other pairs, they will look wrong in R2L text.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > ◄foobar►    BLACK (LEFT|RIGHT)-POINTING POINTER

These quotes display identically on my console, as diamonds,
and the input methods I use do not support them.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

>   > ◄foobar►    BLACK (LEFT|RIGHT)-POINTING POINTER
> 
> These quotes display identically on my console, as
> diamonds, and the input methods I use do not support them.
> 

Precisely.  That's why the suggestion to use them was rhetorical.

Just like curly quotes, which we have now apparently moved to,
alas, these chars are hard to input, search, use with other
programs, etc., as several people have pointed out clearly.

The point wrt these pointy chars was that (a) they suffer all
of the drawbacks of curly quotes, but (b) they at least are not
quote marks and are thus not confusable with text quotations.
Setting off inline code fragments is not quotation.

Because of (a), I do not suggest using such marks.

The argument here is that IF we were to pay the price of
opting for some Unicode chars to wrap inline code phrases etc.
THEN we should at least choose other Unicode chars that are
not quote chars.

We apparently have already chosen curly quotes, unfortunately.
That is the worst of the various possibilities, IMO.  Even
rare, crazy-pointy thingies are a better choice.  That's the
argument here.  If we shouldn't use these thingies - for the
reasons you gave and similar, then a fortiori we should not
use curly quotes either.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Tassilo.

On Sun, Jun 28, 2015 at 07:52:27PM +0200, Tassilo Horn wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >>> Good point.  Instead they use some kind of escaping convention so as
> >>> to avoid ambiguity.  We could try that.
> >> Could you maybe retract your support for using curlies in Elisp
> >> source code, then?

> > I don't have a strong opinion on the markup used in the source code,
> > as long as we resolve the currently existing ambiguity (and as long as
> > we can render the result with symmetric quotes).

> After this discussion, I'm now pretty much indifferent on these changes
> and if they should apply to display only or also to actual source code.
> But if the main problem with `foo' is its ambiguity, and ‘foo’ is still
> ambiguous (but less so), why not use something which is even less
> ambiguous?

Because it might not be typeable, and it might not be displayable.

> For example, I've seen uses of ⸢foo⸥ for marking up source code
> snippets.  That seems pretty unambiguous (although you can never be
> sure), ....

Whatever those quoting characters might be, they display on my terminal
as inverted question marks.  I haven't a clue what they might be, and it
would be a lot of work to find out.

> .... and as long you can type it easily and display it however suits
> you best, it's as good as anything else.

I agree.  That pretty much means ASCII.

> Bye,
> Tassilo

> BTW: On a related note: I don't get why *Help* buffers display quotes at
> all when they are already displayed with a face that makes them
> distinguishable from "normal" text and are clickable.

Help buffers traditionally display verbatim the text from the source
file (with the exception of explicit directives such as to display
bindings).  The face might not always be distinguishable from normal
text (e.g. font-lock might be disabled).  Anyhow, displaying the quotes
doesn't do any harm.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[chad]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]


> On 28 Jun 2015, at 12:27, Alan Mackenzie <acm@muc.de> wrote:
> 
>> For example, I've seen uses of ⸢foo⸥ for marking up source code
>> snippets.  That seems pretty unambiguous (although you can never be
>> sure), ....
> 
> Whatever those quoting characters might be, they display on my terminal
> as inverted question marks.  I haven't a clue what they might be, and it
> would be a lot of work to find out.

Curious: does `C-u C-x =' not work for some reason? 

It works fine for me even in a glass TTY, but I wasn't actually able
to find a nearby TTY that didn't display them properly with only 
several minutes effort.

~Chad

[-- Attachment #2: Type: text/html, Size: 4322 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Chad.

On Sun, Jun 28, 2015 at 12:54:17PM -0700, chad wrote:

> > On 28 Jun 2015, at 12:27, Alan Mackenzie <acm@muc.de> wrote:

> >> For example, I've seen uses of ⸢foo⸥ for marking up source code
> >> snippets.  That seems pretty unambiguous (although you can never be
> >> sure), ....

> > Whatever those quoting characters might be, they display on my terminal
> > as inverted question marks.  I haven't a clue what they might be, and it
> > would be a lot of work to find out.

> Curious: does `C-u C-x =' not work for some reason? 

Yes.  It doesn't work because I wasn't in Emacs.  I was in vim inside
mutt, composing an email.  I might have been looking at the source file
in less, or searching it with grep, or even manipulating it with sed or
awk.

Anytime you want to do something with such a file with such characters
is a pain.  Even inside Emacs, having to type  C-u C-x = just to find
out what a character is is a pain.

> It works fine for me even in a glass TTY, but I wasn't actually able
> to find a nearby TTY that didn't display them properly with only 
> several minutes effort.

Yes.  But that's only after knowing what it is you have to do.  A
typical Emacs user is going to have to start reading man pages, or
searching the web, or posting on help-gnu-emacs.  I'm not even convinced
a typical user knows about C-x =.

> ~Chad

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Alan Mackenzie <acm@muc.de> writes:

Hi Alan,

>> After this discussion, I'm now pretty much indifferent on these
>> changes and if they should apply to display only or also to actual
>> source code.  But if the main problem with `foo' is its ambiguity,
>> and ‘foo’ is still ambiguous (but less so), why not use something
>> which is even less ambiguous?
>
> Because it might not be typeable, and it might not be displayable.

With typing aids like electric-quote-mode any obscure quote/markup
characters could be typeable.

>> For example, I've seen uses of ⸢foo⸥ for marking up source code
>> snippets.  That seems pretty unambiguous (although you can never be
>> sure), ....
>
> Whatever those quoting characters might be, they display on my
> terminal as inverted question marks.  I haven't a clue what they might
> be, and it would be a lot of work to find out.

They don't need to be displayed as such.  For example, they could be
displayed with special faces with fallbacks for font-lock
disabled/missing terminal capabilities, etc.

The aim of the procedure is to be able to unambiguously refer to
functions/variables and write code snippets in docstrings.  Right now,
this is all done using `...' and in some cases, that doesn't work too
well because at least ` is not too uncommon to be part of snippets, too.
So then we might use ‘...’ which is less ambiguous.  ⸢...⸥ is completely
unambiguous with respect to what we currently have.

But the issue of how special parts of docstrings are enclosed is only
one kind of ambiguity we have.  For example, it would be very useful to
be able to refer to foo-mode the function and not the variable.  And my
bar-mode global bar-method variable could have a possible value
insert-char (the symbol, not the function).

>> BTW: On a related note: I don't get why *Help* buffers display quotes
>> at all when they are already displayed with a face that makes them
>> distinguishable from "normal" text and are clickable.
>
> Help buffers traditionally display verbatim the text from the source
> file (with the exception of explicit directives such as to display
> bindings).  The face might not always be distinguishable from normal
> text (e.g. font-lock might be disabled).  Anyhow, displaying the
> quotes doesn't do any harm.

No, and probably omitting them would break docstrings with ASCII tables
which assume that a cell with `foo' is displayed 5 chars wide.

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, I've seen uses of ⸢foo⸥ for marking up source code
  > snippets.

Those display as diamond shapes on my Linux console.

  > BTW: On a related note: I don't get why *Help* buffers display quotes at
  > all when they are already displayed with a face that makes them
  > distinguishable from "normal" text and are clickable.

Not all terminals can display the different face.
On those that can, your point may be valid.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Dmitry Gutov writes:
 > On 06/17/2015 04:03 PM, Stefan Monnier wrote:
 > 
 > > To me, one of the motivations for the change (after the visual
 > > aspect) is to have a *more* robust handling because it is *less*
 > > ambiguous (within the context of Elisp).
 > 
 > That is probably true, but by that logic, using rare unicode
 > characters is a great choice for any markup language.

You're missing the point by calling them "rare".  They are *not* rare;
they are in daily use by the 99.44% of the English-speaking population
that *doesn't* program.

Of course using a WYSIWYG word processor is a blatant admission that
you don't care about backward, forward, or cross-system compatibility
of your document, and *we* can't do that with programming languages.

The point is that these characters have better semantics from the
point of view of *new* programmers and even non-programmers.  Really,
isn't it time to start moving in that direction now that Unicode has
clearly won?

 > Yet, I don't see anybody  doing that.

APL.

Seriously, why would any language designer who cares about popularity
as such risk the wrath of the ASCII-capped lobby?  Emacs cares about
popularity of course, but as Drew and RMS are always at pains to point
out, we don't do things in Emacs because they're popular, we do them
because they are "right" (or, what is more important, because they're
directly freedom-enhancing), and hope to popularize TRT through Emacs.

 > Markdown, for instance, when rendered, only emphasizes code
 > segments using a special tag, which translates into a different
 > font face/color/etc. I don't see why we won't choose to do that, or
 > allow users to customize that aspect.

That would be insane.  Markdown (and ReST) do that because they, too,
need to deal with the ASCII-capped lobby, or at least they still did
when they were first developed a decade or so ago.  But *humans* don't
need tags, and programs are rapidly acquiring the ability to do
without.  I believe we should look forward to the day when that is the
norm, and *get there first*.

That same capability should do 90% of the job of disambiguating usage
of these *common* characters, and backslash-escaping should get 90% of
the rest.  It's still going to requiring people to change their
habits, and that is not going to make conservatives like Drew and Alan
happy (at first: if I didn't believe they'd find reason to come
around, I wouldn't propose it).  Can't make everybody happy all the
time.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/18/2015 07:29 AM, Stephen J. Turnbull wrote:

> You're missing the point by calling them "rare".  They are *not* rare;
> they are in daily use by the 99.44% of the English-speaking population
> that *doesn't* program.

Quotes are not that rare, but using a more rare characters would be even 
better, right? At least, by that logic.

> The point is that these characters have better semantics from the
> point of view of *new* programmers and even non-programmers.

Native-English speaking programmers who are into typography, maybe?

> APL.

Yep, not a markup language.

>   > Markdown, for instance, when rendered, only emphasizes code
>   > segments using a special tag, which translates into a different
>   > font face/color/etc. I don't see why we won't choose to do that, or
>   > allow users to customize that aspect.
>
> That would be insane.  Markdown (and ReST) do that because they, too,
> need to deal with the ASCII-capped lobby, or at least they still did
> when they were first developed a decade or so ago.

What's insane about only highlighting code segments with color?

> But *humans* don't
> need tags, and programs are rapidly acquiring the ability to do
> without.  I believe we should look forward to the day when that is the
> norm, and *get there first*.

I don't understand this response. I only mentioned tags because Markdown 
renders to HTML.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Your arguments are heated but not coherent.  Denigrating a significant
class of users with the term "lobby" does not make them less important
or less worthy of respect.  You can't refute a practical argument
by targeting an imaginary prejudice.

It is not pertinent what fraction of all computer users use non-ASCII
characters daily.  That doesn't tell us what fraction of Emacs users
use non-ASCII characters -- but that is not pertinent either.  You
implicitly presume that they are in favor of this change, but that is
not true.  Some Emacs users that use non-ASCII characters daily _in
Emacs_ will be inconvenienced by this change.

The situation is simple: using curly quotes in doc strings would be a
substantial inconvenience for many users _for no practical benefit_.

If it provided a practical benefit for other users, we would need to
consider how many gain how much (and what roles they have in Emacs
development), versus how many lose how much.  But since it doesn't,
that question does not arise.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Richard Stallman writes:

 > Your arguments are heated but not coherent.

You might try to refute them, rather than descending to mere name-
calling.  Especially in a post where you immediately accuse someone
else of name-calling:

 > Denigrating a significant class of users with the term "lobby" does
 > not make them less important or less worthy of respect.

No, the term "lobby" does not make anybody worthy of less respect.  A
lobby is simply a group of people with a common interest, who advocate
that interest to decision-makers.  Given that lobbying is one of your
most important activities, I don't see why you would associate it with
"denigration".

And I've already granted that the costs to those who *can* live with
just ASCII, and *don't* need input methods yet, matter.  I think
they're low enough to be worth paying, just as you think the sacrifice
of Japanese OCR is a cost that *I* should pay.  Such conflicts can't
be waved away, and each side will lobby for its own interest.

 > The situation is simple: using curly quotes in doc strings would be
 > a substantial inconvenience for many users _for no practical
 > benefit_.

And now you reveal *your* prejudice.  That's no help.  Paul claims
practical benefit; I agree -- and even Stefan seems to see at least a
possibility of practical benefit.

But both sides are just speculating about what most users will feel.
For reasons I've explained elsewhere, I believe only an experiment can
help determine more accurately what the mass of users will think.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Stephen.

On Fri, Jun 19, 2015 at 01:06:57PM +0900, Stephen J. Turnbull wrote:
> Richard Stallman writes:

>  > Your arguments are heated but not coherent.

> You might try to refute them, rather than descending to mere name-
> calling.  Especially in a post where you immediately accuse someone
> else of name-calling:

>  > Denigrating a significant class of users with the term "lobby" does
>  > not make them less important or less worthy of respect.

> No, the term "lobby" does not make anybody worthy of less respect.

Perhaps not, as such, but the phrase you used "... risk the wrath of the
ASCII-capped lobby?" sounds anything but respectful.

[ .... ]

> And I've already granted that the costs to those who *can* live with
> just ASCII, and *don't* need input methods yet, matter.  I think
> they're low enough to be worth paying, just as you think the sacrifice
> of Japanese OCR is a cost that *I* should pay.  Such conflicts can't
> be waved away, and each side will lobby for its own interest.

That's a red herring which has nothing to do with the current argument
about curly quotes.  The inconvenience of typing curly quotes is just as
much an inconvenience to those who use non-English keyboard layouts.  I
would imagine (correct me if I'm wrong) you use distinct keyboard
layouts for writing in English and Japanese.  I imagine also that
there's no key on the Japanese layout either for either of the curly
single quotes.

>  > The situation is simple: using curly quotes in doc strings would be
>  > a substantial inconvenience for many users _for no practical
>  > benefit_.

> And now you reveal *your* prejudice.  That's no help.  Paul claims
> practical benefit; I agree -- and even Stefan seems to see at least a
> possibility of practical benefit.

Richard meant what he wrote here.  Any benefits there may be are not
_practical_ ones.  The curly quotes are a pain to type.  There are no
practical benefits - nothing is made easier.  The putative benefits are
vague, aesthetic, about conforming with other people's expectations,
etc.

> But both sides are just speculating about what most users will feel.
> For reasons I've explained elsewhere, I believe only an experiment can
> help determine more accurately what the mass of users will think.

The current changes do not seem to have been made in the spirit of an
experiment.  Some of the changes (those to the elisp page "Documentation
Tips", for example, which assert that the standard quoting method is
already an "older convention") go well beyond the scope of
experimentation.  In a true experiment, comment and objections would be
actively encouraged at an early stage.  I don't think this has happened
wrt these curly quote changes.

The problem about such "experiments" which are not explicitly announced
is that people don't really notice them until it's "too late" to revert
them.  We were sleepwalking into accepting these curly quotes.  This was
not good.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Alan Mackenzie wrote:

> Any benefits there may be are not _practical_ ones.

Sure they are.  They make documentation strings easier to read, for people who 
are used to today's typical displays.  And there are other practical benefits, 
e.g., being able to cut and paste from help buffers that use modern styles.

> In a true experiment, comment and objections would be
> actively encouraged at an early stage.  I don't think this has happened
> wrt these curly quote changes.

You're quite mistaken.  The doc string fixes were proposed on 20 April in 
Bug#20385.  Comments were solicited and the fixes were improved over a period of 
several weeks, before the patch was installed on May 28.

In contrast, you installed commit 52c3946c872c8bd96508f74cdda5cbb90c664306, an 
equally large change to user behavior, and without first proposing it so that we 
could comment on it (surely you knew it would be controversial) and catch the 
obvious errors with it (even assuming we agreed with the idea).  This was 
another mistake, one that still needs some work.

I have obligations in the next few hours that prevent me from thinking about 
your long recent message on the topic, but I'll return to it as soon as I can.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Paul.

Sorry it's taken me some time to get around to answering your post.

On Fri, Jun 19, 2015 at 08:36:19AM -0700, Paul Eggert wrote:
> Alan Mackenzie wrote:

> > Any benefits there may be are not _practical_ ones.

> Sure they are.  They [curly quotes] make documentation strings easier
> to read, for people who are used to today's typical displays.

Is that really true?  Even if it is, the difference between curly and
ASCII quotes must be quite small in this respect, surely?

> And there are other practical benefits, e.g., being able to cut and
> paste from help buffers that use modern styles.

I haven't a clue what "help buffers that use modern styles" means.  One
can cut and paste from help buffers equally easily regardless of the
quoting style, if any, used in them.

> > In a true experiment, comment and objections would be actively
> > encouraged at an early stage.  I don't think this has happened wrt
> > these curly quote changes.

> You're quite mistaken.  The doc string fixes were proposed on 20 April
> in Bug#20385.  Comments were solicited and the fixes were improved
> over a period of several weeks, before the patch was installed on May
> 28.

I've just had a look at bug #20385.  I even posted to that discussion a
couple of times myself, yet I wasn't quite aware of the massive changes
you had in mind.  Possibly you weren't either when you started the
thread.

I don't think you were consciously trying to slip things through,
avoiding discussion, but if you had've been, doing precisely what you
did would have been an effective way of going about it.  Even the
subject "Support quoting 'like this' in doc strings", with nice gentle
fuzzy words like "support" instead of harder, more incisive words could
have been calculated not to draw too much attention.

You could have had a subject like "Supersede ASCII quoting convention in
doc strings", and most importantly, put it in emacs-devel where it would
have been much more prominent.  Note how much more vigorous has been the
debate in this thread than the one on the bug list.

> In contrast, you installed commit 52c3946c872c8bd96508f74cdda5cbb90c664306, an 
> equally large change to user behavior, ....

Not really.  I simply made the new behaviour optional, setting the
default to "off".  And I also drew attention to my change.

> .... and without first proposing it so that we could comment on it
> (surely you knew it would be controversial) ....

Yes, I guessed it would be a bit controversial.  But it was never really
intended to be much other than an emergency quick fix.  Having doc
strings mandatorily translated into curly quotes was truly
objectionable, and best fixed quickly.

> .... and catch the obvious errors with it (even assuming we agreed
> with the idea).

The whole idea of curly quotes in doc string is only now being fully
discussed.  As already noted, it could have been discussed much earlier.
You could even have implemented the option yourself, saving me (with
less familiarity in Emacs's C code) from having to do it.

> This was another mistake, one that still needs some work.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Alan Mackenzie wrote:
> You could have had a subject like "Supersede ASCII quoting convention in
> doc strings"

That would have been misleading, as the patch did not supersede ASCII quoting. 
ASCII quoting still works, and is still supported.

>> In contrast, you installed commit 52c3946c872c8bd96508f74cdda5cbb90c664306, an
>> equally large change to user behavior, ....
>
> Not really.  I simply made the new behaviour optional

No, your patch changed the default behavior for *Help* buffers.  The new default 
negatively affected the UI, in that it made *Help* buffers sometimes quote one 
way, sometimes another, inconsistently.  And more often than not the help 
buffers quoted with grave accent, which is the style we're changing the default 
away from.  All this was considerably more than just implementing an option.

 > I haven't a clue what "help buffers that use modern styles" means.

I meant help buffers that quote with curved single quotes rather than with grave 
accent and apostrophe.

> the difference between curly and
> ASCII quotes must be quite small in this respect

This thread is about a relatively minor change overall, yes.  Molehills and 
mountains do come to mind.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Paul.

On Tue, Jun 23, 2015 at 03:47:51PM -0700, Paul Eggert wrote:
> Alan Mackenzie wrote:
> > You could have had a subject like "Supersede ASCII quoting convention in
> > doc strings"

> That would have been misleading, as the patch did not supersede ASCII
> quoting.  ASCII quoting still works, and is still supported.

It would not have been misleading.  Your aim is precisely to supersede
ASCII quoting.  The very moment curly quoting starts appearing in source
files is when ASCII quoting will stop working properly.

> >> In contrast, you installed commit 52c3946c872c8bd96508f74cdda5cbb90c664306, an
> >> equally large change to user behavior, ....

> > Not really.  I simply made the new behaviour optional

> No, your patch changed the default behavior for *Help* buffers.

Paul, that's a really objectionable thing for you to do.  You snipped
the second half of my sentence ", setting the default to "off".", then
proceded to violently agree with what you'd snipped.  Must you?

> The new default negatively affected the UI, in that it made *Help*
> buffers sometimes quote one way, sometimes another, inconsistently.

Rubbish.  The default I set left quote characters untranslated, as has
been the case in Emacs for several decades.  That is utterly consistent.
The point is, that with this new option, users can chose how they want
to see quotes, something which wasn't possible in the days and weeks
before.

[ .... ]

> All this was considerably more than just implementing an option.

No.  It was implementing an option, changing the default, precisely as I
stated.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

>>> You could have had a subject like "Supersede ASCII quoting convention in
>>> doc strings"
>
>> That would have been misleading, as the patch did not supersede ASCII
>> quoting.  ASCII quoting still works, and is still supported.
>
> It would not have been misleading.  Your aim is precisely to supersede
> ASCII quoting.

It's true that I'd prefer we adopt a simpler system that uses docstrings largely 
unchanged.  But this approach was objected to vociferously, so the patch in 
question did not implement it.  It would have been misleading to label the patch 
as doing what I'd prefer.  Instead, the patch was labeled to do what it actually 
did, which is what labels are supposed to do.

> The very moment curly quoting starts appearing in source
> files is when ASCII quoting will stop working properly.

No, ASCII quoting still works properly, even though curved quotes are in source 
files now.  There were a few curved quotes even in Emacs 24.5 source files, and 
the sky did not fall.

>>>> In contrast, you installed commit 52c3946c872c8bd96508f74cdda5cbb90c664306, an
>>>> equally large change to user behavior, ....
>
>>> Not really.  I simply made the new behaviour optional
>
>> No, your patch changed the default behavior for *Help* buffers.
>
> Paul, that's a really objectionable thing for you to do.  You snipped
> the second half of my sentence ", setting the default to "off".",

I had written "you installed commit 52c3946c872c8bd96508f74cdda5cbb90c664306, an 
equally large change to user behavior".  You then objected "Not really.  I 
simply made the new behaviour optional, setting the default to 'off'."  Your 
objection was an attempt to minimize the scope of your change, even though it 
was indeed an equally large change to user behavior.  The "Not really" in your 
objection was misleading.  In hindsight I should have mentioned all this in a 
longer comment, and I should have included more of your quote, and my apologies 
if my attempt to be brief misled anybody.  I hope this further comment has made 
the main thrust more clear.

> The point is, that with this new option, users can chose how they want
> to see quotes,

Yes, there seems to be reasonable agreement that users should have an option to 
see quotes traditionally `like this' or in a curved-quote style ‘like this’. As 
users do have that option in the current master, perhaps we should move on to 
the next topic.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, ASCII quoting still works properly, even though curved quotes are in source 
  > files now.

Using curly quotes in the Lisp source files is a cause of confusion.
We should use only ASCII quotes in the sources, and change them in
display for users who want that.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > Perhaps not, as such, but the phrase you used "... risk the wrath
 > of the ASCII-capped lobby?" sounds anything but respectful.

I don't respect the ASCII-capped lobby.  They constitute a tiny
minority of humanity which has been getting in the way of bringing
sane computing to 7 billion people for 5 decades now.  I find the
argument that "*I* am ASCII-capped, so *we* shouldn't simplify and
disambiguate by using Unicode" especially distressing in the context
of Emacs.

 > That's a red herring which has nothing to do with the current
 > argument about curly quotes.  The inconvenience of typing curly
 > quotes is just as much an inconvenience to those who use
 > non-English keyboard layouts.

That's nonsense.  Emacs users learn *hundreds* of keychords.  Learning
a few more to be able to distinguish between ASCII GRAVE ACCENT and
Unicode LEFT SINGLE QUOTATION MARK is not a huge inconvenience.  If
you don't need to do so, then simply map grave to the quotation mark
and you're done.

 > I would imagine (correct me if I'm wrong) you use distinct keyboard
 > layouts for writing in English and Japanese.  I imagine also that
 > there's no key on the Japanese layout either for either of the
 > curly single quotes.

Japanese (and Chinese) mostly just use the layout of whatever keyboard
is in front of them, except that it's convenient to have labelled keys
for mode-switch commands for word processing users.  The process of
entering Japanese text has several levels for most programming users:

1.  Keystrokes.  These are ASCII keystrokes.
2.  Keystroke pairs.  Japanese phonetic writing is by syllable, not
    character.  Each syllable is mapped to a pair of ASCII keys.
3.  Dictionary lookup of phonetically entered words, resulting in a
    menu.
    Morphological and grammatical filtering to narrow the lookup
    results.
4.  Selection from the menu.
5.  Confirmation of the entered text.

There do exist phonetic Japanese keyboard layouts, but almost nobody
except a few professional typists use them.  Excel formulas and such
are easier to type in ASCII.  Chinese is similar; as far as I know
Chinese have no commonly used phonetic layout, they just use ASCII,
but that's just a guess based on casual conversation with Chinese grad
students.  When Koreans use Han ideographs, the process is similar.
When they don't, Hangul are composed characters but the process
entirely algorithmic.  I don't know if there are Hangul layouts in
common use but I suspect it's more common than in Japan for several
reasons.

Bottom line: about 1.3 billion people write languages where the common
practice is to type multiple ASCII characters per "native character".

Most of the rest of the world has to switch layouts to get
"self-insert-command" behavior for both ASCII and native characters.

 > Richard meant what he wrote here.

Of course he did.  The question is whether he has any experience with
using input methods other than self-insert-command.  I will bet "no",
and that his reaction is pre-judgment without enough relevant
experience.

 >  Any benefits there may be are not _practical_ ones.  The curly
 > quotes are a pain to type.

That's a fixable bug, but not in the use of curly quotes themselves,
but rather in the Emacs input system.

 > There are no practical benefits - nothing is made easier.

Wrong.  You evidently skipped over certain parts of Paul's and
Stefan's posts where they describe objective benefits they expect.
(You are welcome to disagree with them, presenting evidence to the
contrary, but not ignore them entirely as you are doing here.)  And
your deprecation of readability and beauty *in sources as well as in
help buffers* is unfair to those of us to whom it matters.  You don't
have to agree, but to me better readability and more beauty to
not-yet-programmers are quite "practical" in education.

 > In a true experiment, comment and objections would be actively
 > encouraged at an early stage.

On this subject, you can spell it "comment and objections", but in my
experience what you get is "prejudice and hysteria".  I see no reason
to change that assessment for what has happened here.  I've been
through these battles a number of times, and the comment and objection
stage is always just a waste of time and bandwidth.  Nobody ever
changes their mind based on so-called "rational argument"; the only
actual result that ever happens is that implementation is always
delayed, and often proponents give up entirely for a while.  Which
means nothing is learned.  Experiments, on the other hand, do produce
changes in position.  Sometimes on the part of proponents, sometimes
on the part of opponents.

One reason you perceive these changes to be "not experimental in
spirit" is because 90% or more of your programming life has been spent
in an environment (no VC or CVS or Subversion) where reverting a
change is practically a pain in the butt, as well as socially
difficult.  You've seen people fight reversion of their changes, and
win because they refuse and it's pragmatically hard enough that nobody
else is willing to do it.  We need to get past that and create an
environment where experiments are more frequent (and more frequently
reverted).  It's possible to do that now, although some social changes
will be needed too.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

On 22 June 2015 at 09:44, Stephen J. Turnbull <turnbull@sk.tsukuba.ac.jp>
wrote:

On this subject, you can spell it "comment and objections", but in my
> experience what you get is "prejudice and hysteria".  I see no reason
> to change that assessment for what has happened here.  I've been
> through these battles a number of times, and the comment and objection
> stage is always just a waste of time and bandwidth.  Nobody ever
> changes their mind based on so-called "rational argument"; the only
> actual result that ever happens is that implementation is always
> delayed, and often proponents give up entirely for a while.


For what it's worth, reading through this discussion has changed my mind. I
went into this sceptical of the change, but I'm now convinced it's the
right thing. I just updated gnu-apl-mode to use the curly quotes instead of
the old style.

I'm assuming that the existence of the curly quotes will not screw anything
up (as in: the application will still work and not crash in your face) in
older versions of Emacs. Can anyone confirm this?

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1566 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Elias Mårtenson wrote:
> I'm assuming that the existence of the curly quotes will not screw anything
> up (as in: the application will still work and not crash in your face) in
> older versions of Emacs. Can anyone confirm this?

That's correct.  Curved single quotes in docstrings do not crash older Emacs. 
They're treated as ordinary characters, much as curved double quotes or accented 
letters in words like "Bahá'í".  The main downside in older Emacs is that 
symbols quoted via curved quotes are not highlighted or otherwise treated as 
special in *Help* buffers.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > On this subject, you can spell it "comment and objections", but in my
  > experience what you get is "prejudice and hysteria".

What we get from you is ad-hominem attacks, over and over.

  >   Nobody ever
  > changes their mind based on so-called "rational argument";

I changed my mind in this discussion, but venom for those you disagree
with never changes.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Richard Stallman writes:

 >   > On this subject, you can spell it "comment and objections", but in my
 >   > experience what you get is "prejudice and hysteria".
 > 
 > What we get from you is ad-hominem attacks, over and over.
 > 
 >   >   Nobody ever
 >   > changes their mind based on so-called "rational argument";
 > 
 > I changed my mind in this discussion, but venom for those you disagree
 > with never changes.

Actually, you are the one engaging in ad hominem attacks here, by
making statements about me personally while failing to address the
valid points I make at all.

On the other hand, neither of my passages you quote are ad hominem at
all, since they refer to my past experiences at which none of the
active participants in this thread were present.  And the claim about
venom is manifestly false: there is no venom in this post.

When did you change your mind in this discussion, and about what?  I
missed it: it seems to me you advocate the same conservative position
in the same words as when you first entered the discussion.

Regards,

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Actually, you are the one engaging in ad hominem attacks here, by
  > making statements about me personally

I am criticizing what you say.  That's not ad-hominem.
But you are spewing contempt at other people, calling them a
"lobby".

					  while failing to address the
  > valid points I make at all.

I saw nothing else worth responding to.

  > When did you change your mind in this discussion, and about what?

Reread my messages and you will see.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Richard Stallman writes:

 >   > Actually, you are the one engaging in ad hominem attacks here, by
 >   > making statements about me personally
 > 
 > I am criticizing what you say.  That's not ad-hominem.

No, you criticize *me*, writing:

    What we get from you is ad-hominem attacks, over and over.

    I changed my mind in this discussion, but venom for those you
    disagree with never changes.

That is most definitely ad hominem, and indeed you know it to be
untrue.  You have had a number of experiences of perfectly courteous
exchanges with me, as have many on emacs-devel, despite disagreement.

 > But you are spewing contempt at other people, calling them a
 > "lobby".

Only in their role of advocating that non-ASCII characters be
prohibited in contexts where I believe they would be useful.  I myself
am lobbying for experimentation with non-ASCII characters in Emacs
syntax.  I see nothing "contemptuous" in that word, and nothing in
several online dictionaries suggests that it is offensive.

 > > When did you change your mind in this discussion, and about what?
 > 
 > Reread my messages and you will see.

Why are you evading a simple question, and refusing to share
information you certainly have?  Rereading would be both tedious and
unreliable.  Because your posts are very short, and with minimal
quoting for context (and rarely any attribution of quotations), it can
be very difficult to establish context for your words.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Stephen.

On Wed, Jun 24, 2015 at 02:06:30AM +0900, Stephen J. Turnbull wrote:
> Richard Stallman writes:

>  >   > Actually, you are the one engaging in ad hominem attacks here, by
>  >   > making statements about me personally

>  > I am criticizing what you say.  That's not ad-hominem.

> No, you criticize *me*, writing:

>     What we get from you is ad-hominem attacks, over and over.

That is criticising what you've been writing.  And from my point of
view, Richard's criticism has some considerable merit.

>     I changed my mind in this discussion, but venom for those you
>     disagree with never changes.

> That is most definitely ad hominem, and indeed you know it to be
> untrue.

It is not an ad hominem, but strictly speaking, it is untrue.  But an
uncomfortable proportion of your posts do contain venom.

> You have had a number of experiences of perfectly courteous exchanges
> with me, as have many on emacs-devel, despite disagreement.

This is true.

>  > But you are spewing contempt at other people, calling them a
>  > "lobby".

> Only in their role of advocating that non-ASCII characters be
> prohibited in contexts where I believe they would be useful.

What you wrote, second time round, was "I don't respect the ASCII-capped
lobby".  That is disparaging, and an explicit expression of disrespect
for people whose views differ from your own.  "Spewing contempt at other
people" is a characterisation of this sentence, and other things you
have written, which has some merit.

> I myself am lobbying for experimentation with non-ASCII characters in
> Emacs syntax.  I see nothing "contemptuous" in that word, and nothing
> in several online dictionaries suggests that it is offensive.

Oh, come on, Stephen!  You know perfectly well that the offensiveness of
words has everything to do with their context, and I put it to you that
your phrase "the ASCII-capped lobby" was intended to be offensive, but
deniably so.  I, for one, find it offensive.

>  > > When did you change your mind in this discussion, and about what?

>  > Reread my messages and you will see.

> Why are you evading a simple question, and refusing to share
> information you certainly have?

I would guess because it would take Richard more time and effort than
it's worth.

> Rereading would be both tedious and unreliable.  Because your posts
> are very short, and with minimal quoting for context (and rarely any
> attribution of quotations), it can be very difficult to establish
> context for your words.

This is a fair criticism.

Now the whole point of this post, if you hadn't guessed, is to get you
to post in a more congenial manner, even when (especially when) you
disagree with whom you're writing to.  English is your native language,
and you're as skilled in its use as anybody here.  So please stop the ad
hominems, stop the venom, stop the disrespect and disparagement.  It
would make this list a more pleasant place.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > > No, you criticize *me*, writing:
 > 
 > >     What we get from you is ad-hominem attacks, over and over.
 > 
 > That is criticising what you've been writing.

With effort it can be interpreted that way, but you know as well as I
do that was an attack on me personally.  Criticism of what I wrote
would quote the actual attacks, but this merely asserts that I do so.
Then, the word "what" implies "ad hominem attacks" constitute the
entirety of what I post.

 > >     I changed my mind in this discussion, but venom for those you
 > >     disagree with never changes.
 > 
 > > That is most definitely ad hominem, and indeed you know it to be
 > > untrue.
 > 
 > It is not an ad hominem,

Of course it is.  "Venom ... never changes" is an implicit assertion
that such venom is a personal attribute, especially since he knows it
to be untrue.  That is ad hominem.  You are thinking of phrasing like
"is all too frequent", and in fact you used that kind of phrasing
yourself.  But Richard did not.

I'm very surprised that you accuse me below of exploiting "plausible
deniability" for using a word which cannot possibly be explained away
that way, while you don't recognize the genuine article when it's
pointed out to you, and you even defend it.

 > What you wrote, second time round, was "I don't respect the
 > ASCII-capped lobby".  That is disparaging, and an explicit
 > expression of disrespect for people whose views differ from your
 > own.  "Spewing contempt at ot her people" is a characterisation of
 > this sentence, and other things you have written, which has some
 > merit.

Indeed it does.  It's not nice, but it's not a matter of mere
disagreement on a technical change.  It's a moral stance (one which I
didn't recognize myself until you forced me to clarify to myself *why*
I chose to invent that phrase).  Note that the term is precisely
calculated to satirize one who argues that *others'* use of non-ASCII
characters is an imposition on *him* because he's "unable" to input
them, or because his system hasn't been yet configured to enable their
input.  Asking that others not use certain characters because I would
have to learn how to input them, no, I'm sorry, I can't do that and I
can't respect even the mention of that rationale.

There are, of course, good reasons for maintaining ASCII and English
as the primary character set and language of inspiration for
programming languages.  Backward compatibility, compatibility with
existing software (including common platforms, although Eli is on
treacherous ground since Windows is avowedly proprietary), pick the
most common language and standardize on it, etc.  Stick to those and
you're not a member of the ASCII-capped.

It's also reasonable to point out that for oneself, the system really
isn't good enough yet.  To advocate that the commit be pushed only if
effort is also devoted to improved input methods and display, that is
very reasonable.  And elsewhere I've discussed approaches to
"plain-text input methods" (like the electric quotes mode, but more
general) with Eli already.

 > > I myself am lobbying for experimentation with non-ASCII characters in
 > > Emacs syntax.  I see nothing "contemptuous" in that word, and nothing
 > > in several online dictionaries suggests that it is offensive.
 > 
 > Oh, come on, Stephen!  You know perfectly well that the offensiveness of
 > words has everything to do with their context, and I put it to you that
 > your phrase "the ASCII-capped lobby" was intended to be offensive, but
 > deniably so.  I, for one, find it offensive.

You were intended to, this time, as one of those who has used "I don't
know how to input curly quotes" as a reason to oppose Paul's change.
I never had *any* intent of denying that "ASCII-capped" is offensive,
and I'm insulted that you think I'm stupid enough to try denying it.

But I put it to *you* that the focus of discussion has been on the
word "lobby", which simply means "advocacy group".  Don't ask me why,
that was not my choice.

 > >  > > When did you change your mind in this discussion, and about what?
 > 
 > >  > Reread my messages and you will see.
 > 
 > > Why are you evading a simple question, and refusing to share
 > > information you certainly have?
 > 
 > I would guess because it would take Richard more time and effort than
 > it's worth.

Something like

    In my first post I opposed all use of curly quotes in docstrings,
    but later I decided it would be OK to have an option to display
    ASCII grave and apostrophe as single quotes if the user wants to.
    I can't take time to dig up message IDs, sorry, but I think I was
    replying to Paul Eggert.

would be entirely satisfactory.  It took me 3 minutes to write, and I
had to construct a plausible example so the length would be realistic.
I wouldn't be surprised if that example is true -- but it would take
me 30 minutes to confirm, I believe.  Considered as a peace offering,
I think Richard's 3 minutes would be amply repaid by reduction of
future acrimony, but he'll have to judge that.

 > > Rereading would be both tedious and unreliable.  Because your posts
 > > are very short, and with minimal quoting for context (and rarely any
 > > attribution of quotations), it can be very difficult to establish
 > > context for your words.
 > 
 > This is a fair criticism.

Thank you.

 > Now the whole point of this post, if you hadn't guessed, is to get you
 > to post in a more congenial manner, even when (especially when) you
 > disagree with whom you're writing to.  English is your native language,
 > and you're as skilled in its use as anybody here.  So please stop the ad
 > hominems,

I don't ever use ad hominem argument[1] as far as I know.  If you know
of examples, please show them to me so I can learn to recognize my
mistaken thinking or phrasing that elicits misunderstanding, and
correct it.  Gratuitous ad hominems, such as the C-word, OK, I'll stop
that.

I reserve judgment on "ASCII-capped" as long as "MS-DOG" and similar
epithets are acceptable and official terminology such as "Win32" is
not, as it is a moral issue.  I doubt I'll find occasion to use it
again soon, though, if that's any comfort.

 > stop the venom, stop the disrespect and disparagement.

That's a reasonable request.


Footnotes: 
[1]  Which is the fallacy in which one argues from the character of
the advocate to the truth or falsity of the claim.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
> Date: Mon, 22 Jun 2015 10:44:09 +0900
> Cc: eggert@cs.ucla.edu, rms@gnu.org, emacs-devel@gnu.org
> 
> I don't respect the ASCII-capped lobby.  They constitute a tiny
> minority of humanity which has been getting in the way of bringing
> sane computing to 7 billion people for 5 decades now.  I find the
> argument that "*I* am ASCII-capped, so *we* shouldn't simplify and
> disambiguate by using Unicode" especially distressing in the context
> of Emacs.

With all due respect, Stephen, you are bending the argument too much.
Truth is, the Unicode adoption in mainline software we are involved with
every day is very slow.  For example, there still are no good solutions for
bidi-aware text terminals, which means, inter alia, that gettext-style
translations to R2L languages for console programs are still in trouble, 15
years after the UBA was codified.  At least 400 million people worldwide
need that, but we are still not there.

The net result of this extremely slow progress is that many theoretical
niceties are exceedingly hard to have in practice, and you cannot rely on
them being available to J.R. Hacker next door.

> That's nonsense.  Emacs users learn *hundreds* of keychords.

We also use programs other than Emacs, and some of us (gasp!) use them from
the shell prompt at least sometimes.  So the ability to type these
characters conveniently is important, certainly not something to disregard
so easily as you do.

> Japanese (and Chinese) mostly just use the layout of whatever keyboard
> is in front of them, except that it's convenient to have labelled keys
> for mode-switch commands for word processing users.

So perhaps the issues being discussed are non-issues for Japanese and
Chinese (and users of other complex writing systems).  But they are still
issues for the rest of the world, there's no need to deny or dismiss that.

>  > Richard meant what he wrote here.

> Of course he did.  The question is whether he has any experience with
> using input methods other than self-insert-command.  I will bet "no",

I do use input methods.  It's still a (minor) nuisance compared to pressing
a single key.  So if my keyboard had keys for ‘..’ and the likes, it would
be better.  As things stand, I will need to use tricks, especially out of
Emacs, to type them.  It's less convenient.

> and that his reaction is pre-judgment without enough relevant
> experience.

"Just because you're paranoid doesn't mean they aren't after you."
Just because Some People™ might have prejudice against these quotes,
it doesn't mean their introduction won't make our lives less
convenient.

>  >  Any benefits there may be are not _practical_ ones.  The curly
>  > quotes are a pain to type.
> 
> That's a fixable bug, but not in the use of curly quotes themselves,
> but rather in the Emacs input system.

Alas, the world of software is not limited to Emacs.

> the only actual result that ever happens is that implementation is always
> delayed, and often proponents give up entirely for a while.  Which means
> nothing is learned.  Experiments, on the other hand, do produce changes
> in position.

For some of us, Emacs is a first-class tool for everyday's work.  Some
of us cannot easily afford conducting "experiments" with tools of such
importance, because we have to do something other than gathering
experience by the end of the day.  Please don't dismiss our chagrin so
easily, just because you might have more free time on your hands to
conduct experiments.  While reverting a bunch of commits (whose number
is growing by the hour, btw) might be relatively easy, those commits
affect everyone who is using Emacs, as reverting them, locally is
hardly a practical option.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > The net result of this extremely slow progress is that many
 > theoretical niceties

Availability of fonts with complete Latin repertoire and convenient
input methods are not theoretical niceties any more.  They are
practical realities on any platform produced in the last decade.

It's a shame about bidi, I'll grant.

 > > Emacs users learn *hundreds* of keychords.
 > 
 > We also use programs other than Emacs, and some of us (gasp!) use
 > them from the shell prompt at least sometimes.  So the ability to
 > type these characters conveniently is important, certainly not
 > something to disregard so easily as you do.

I *don't* disregard that ability.  I consider the input methods I use
in the GUI and in terminal windows on three operating systems to be
quite convenient given that I type many thousands of characters per
day, but only type grave a few times (and most of those are for
reStructuredText documentation and TeX).  YMMV, and I acknowledge
that, but even you feel it necessary to admit that the "nuisance" is
"(minor)".

 > > and that his reaction is pre-judgment without enough relevant
 > > experience.
 > 
 > "Just because you're paranoid doesn't mean they aren't after you."
 > Just because Some People™ might have prejudice against these quotes,
 > it doesn't mean their introduction won't make our lives less
 > convenient.

That is completely out of context.  In the part you quote here, I was
arguing against a priori judgments for the whole project and all its
users, without any data except introspection to back them up.  Not
that inconvenience for the opponents was zero to ten decimal places.

 > >  >  Any benefits there may be are not _practical_ ones.  The curly
 > >  > quotes are a pain to type.
 > > 
 > > That's a fixable bug, but not in the use of curly quotes themselves,
 > > but rather in the Emacs input system.
 > 
 > Alas, the world of software is not limited to Emacs.

Again, out of context.  The complaint I was responding to was
specifically about Emacs input methods, primarily C-x 8 IIRC.

But guess what?  AFAICT, the rest of the software world doesn't have
these problems.  People are typing scores of odd characters in email
to me all the time.  And not just Japanese, but good ol' boys and
girls from the U S of A.  How do they manage that, I wonder?

 > For some of us, Emacs is a first-class tool for everyday's work.
 > Some of us cannot easily afford conducting "experiments" with tools
 > of such importance, because we have to do something other than
 > gathering experience by the end of the day.

Last I heard, Emacs 24 was the stable version, and it doesn't have
these commits in it, and never will.  Conducting experiments on people
(volunteers) while they are doing real work is the very definition of
"beta test".  If you don't like that, you shouldn't be using a beta
Emacs for everyday work.

Of course it's a matter of degree, some experiments are more
disruptive than others, and beta testers don't sign up for constant
churn.  This change hardly qualifies as hugely disruptive.  It went
unnoticed when initially committed and announced.  Surely issues like
frequent bootstrapping and debugging out-and-out build breakage, not
to mention various bugs, are far greater interruptions to daily work.

 > Please don't dismiss our chagrin so easily, just because you might
 > have more free time on your hands to conduct experiments.

Stop with the ad hominems, please.  It's not my alleged free time that
is relevant.

It's that I see benefits to third parties that *you* ignore, perhaps
because you perceive them to be "small".  But I believe them to be
potentially large.  I would like to see this small, initial experiment
performed to see if the disruption is as large and the benefits as
small as you, Richard, Alan, and Drew believe them to be.  If so, it
should be reverted and further experiments will have to be delayed
another decade or so.

 > While reverting a bunch of commits (whose number is growing by the
 > hour, btw) might be relatively easy, those commits affect everyone
 > who is using Emacs, as reverting them, locally is hardly a
 > practical option.

Huh?  People deal with exactly this problem all the time, for example
package maintainers in GNU/Linux distros.  Sure, it's a skill you have
to spend time developing, and I know that many people prefer not to
invest their time that way.  But "not practical" is too dismissive.
It *is* a practical option, used by many in some contexts.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: acm@muc.de,
>     eggert@cs.ucla.edu,
>     rms@gnu.org,
>     emacs-devel@gnu.org
> Date: Tue, 23 Jun 2015 03:32:01 +0900
> 
> Eli Zaretskii writes:
> 
>  > The net result of this extremely slow progress is that many
>  > theoretical niceties
> 
> Availability of fonts with complete Latin repertoire and convenient
> input methods are not theoretical niceties any more.  They are
> practical realities on any platform produced in the last decade.

Try the Windows console some day.

>  > > and that his reaction is pre-judgment without enough relevant
>  > > experience.
>  > 
>  > "Just because you're paranoid doesn't mean they aren't after you."
>  > Just because Some People™ might have prejudice against these quotes,
>  > it doesn't mean their introduction won't make our lives less
>  > convenient.
> 
> That is completely out of context.  In the part you quote here, I was
> arguing against a priori judgments for the whole project and all its
> users, without any data except introspection to back them up.  Not
> that inconvenience for the opponents was zero to ten decimal places.

I'm saying that the problems might be real even though those who speak
about them have prejudice, and therefore dismissing those problems
just because of that prejudice might be a mistake.

> But guess what?  AFAICT, the rest of the software world doesn't have
> these problems.  People are typing scores of odd characters in email
> to me all the time.  And not just Japanese, but good ol' boys and
> girls from the U S of A.  How do they manage that, I wonder?

My guess would be that the mail composing program inserts Unicode
quotes when the user types ASCII quotes.

> I see benefits to third parties that *you* ignore

I don't ignore them.  You will not see me objecting to these changes
in any of the relevant discussions.  I just think that we shouldn't
dismiss so easily the issues these changes bring with them.  IOW, we
should see this issue in its delicate balance.

>  > While reverting a bunch of commits (whose number is growing by the
>  > hour, btw) might be relatively easy, those commits affect everyone
>  > who is using Emacs, as reverting them, locally is hardly a
>  > practical option.
> 
> Huh?  People deal with exactly this problem all the time, for example
> package maintainers in GNU/Linux distros.  Sure, it's a skill you have
> to spend time developing, and I know that many people prefer not to
> invest their time that way.  But "not practical" is too dismissive.
> It *is* a practical option, used by many in some contexts.

It is not practical for me, as it will take too much time, time that I
don't have.  I prefer that we did these changes right the first time
-- as we have been doing recently.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Try the Windows console some day.

I have, and Japanese has worked fine for two decades AFAICR.  This
includes various quotation marks, which have been accessible from
MSIME since Windows 9x at the latest.  That might be specific to the
Japanese localization though, since you can't do anything without
input methods in Japan.  So tell me about Hebrew and/or Latin-X?

 > > That is completely out of context.  In the part you quote here, I
 > > was arguing against a priori judgments for the whole project and
 > > all its users, without any data except introspection to back them
 > > up.  Not that inconvenience for the opponents was zero to ten
 > > decimal places.
 > 
 > I'm saying that the problems might be real even though those who speak
 > about them have prejudice, and therefore dismissing those problems
 > just because of that prejudice might be a mistake.

I know what you're saying, and I've acknowledged that the problems are
real regardless of who is reporting them, several times.  Nor did I
anywhere imply that prejudice held by a speaker invalidates her
statement.  That's argumentum ad hominem, a well-known fallacy.  I
argued that prejudice creates bias, and is invalid as reasoning, and
that instead we should seek facts on which to base valid reasoning.

So please don't tell me that I'm dismissing your problems while
quoting my denial that I was dismissing them and the explanation of
what I *was* doing.

 > > But guess what?  AFAICT, the rest of the software world doesn't have
 > > these problems.  People are typing scores of odd characters in email
 > > to me all the time.  And not just Japanese, but good ol' boys and
 > > girls from the U S of A.  How do they manage that, I wonder?
 > 
 > My guess would be that the mail composing program inserts Unicode
 > quotes when the user types ASCII quotes.

It's not just quotation marks.  It's foreign words spelled correctly,
ellipses, the occasional math symbol (the lemniscate (infinity) is
popular among the more flaky of my correspondents), emoticons, and
various other symbols (dice, playing card suit symbols, enclosed
characters such as circled numerals).  There are too many of them to
excuse with "smart quotes"; users are using "input methods" of some
sort for these characters that don't exist on their keyboards.  Many
people *are* willing to pay the cost of inputting these characters;
they apparently do like them.

OK, so maybe these methods don't exist on the Windows console or are
far more annoying than their GUI or email counterparts, and so those
characters would be excessively painful to type for you in some
contexts.  That can surely be fixed (at some cost, eg, using Emacs
shell mode, which we already know you strongly oppose paying).  On the
other hand, Emacs's conservatism in these matters makes Emacs less
attractive to new developers.  How much?  I don't know, you don't
know, Alan doesn't know, Paul doesn't know, and RMS doesn't know.
Let's try loosening up and find out, no?

 > > I see benefits to third parties that *you* ignore
 > 
 > I don't ignore them.  You will not see me objecting to these changes
 > in any of the relevant discussions.

You don't bother to mention that you're aware of benefits, or what
they are, when you allege that I'm biased against admitting there are
costs.  As far as I can tell from the words you post, you are indeed
opposed to the changes, describing only costs at every turn.  I can't
know what you're thinking but not posting, sorry.

 > I just think that we shouldn't dismiss so easily the issues these
 > changes bring with them.  IOW, we should see this issue in its
 > delicate balance.

You're telling *me*?  Eli, it's the opponents like RMS, Alan, and
Drew who are dismissing all possible benefits without even trying the
change for long enough to confirm they can't adjust to the cost.  Alan
even admits that his opposition is "reactionary".[1]

Just like you, I don't have a strong position on the change in
question (I like it personally, but am unsure it's good for Emacs "in
its delicate balance").  All *I* am advocating is an experiment *in a
beta series*, and actually getting data on user preferences *informed
by user experience with the change* rather than pushing our biases on
them, or polling for user biases.  The idea that good user interfaces
can be derived exclusively from a priori principles was abandoned by
professionals a couple decades ago, except in organizations like
Emacs.[2]  UI changes need testing in practical use (beta) and tweaking
(and sometimes reversion) according to the results.

It's certainly true that I've engaged in hyperbole.  People
(especially the hyperlogical types who hang out here) sometimes *do*
change their opinions based on verbal reasoning.  User opinions in the
absence of experience *do* have a certain amount of validity.
Nevertheless, the strong propensity is to resist changing opinions,
and for there to be potentially large bias when asking users what they
think about *prospective* changes.  Shouldn't we take action to get
facts to augment reasoning, and to reduce bias in information about
user preference?

It's also true that I've expressed an *opinion* that the costs to a
relative few are small compared to potential benefits for many.
Obviously, the opponents disagree about the balance.  When opinions
differ, shouldn't we go get some actual facts?

 > It is not practical for me, as it will take too much time, time
 > that I don't have.

"There you go again."  You and the opposition focus entirely on the
costs *to yourselves*.  Rarely do we have much idea of what the
benefits of any change we don't propose ourselves might be.  Shouldn't
we find out, if there are developers who clearly feel there are net
benefits and would do the work (including reversion if so decided by
the maintainer)?

 > I prefer that we did these changes right the first time

Don't we all?  But your apparent belief that others can avoid these
"mistakes" without substantial effort from you is unrealistic.  I'm
sure Paul thought he was doing the right thing at the time.  The only
way for him to know what you think is the right way is for you to
review.  You didn't, so you face the cost of making changes now.
That's not "wrong", that's just the unpleasant set of alternatives
that are actually available to us: review or revise.

Every release is an experiment, containing new features that some
users, few or many, won't like.  Emacs has had a few fiascos among its
new features, and tends to be *very* conservative as a result.  I
propose that it's a lot cheaper to do the experiments when Emacs is
still in beta, and so we don't have to be so conservative.

Of course it's possible to go too far.  What's the appropriate
balance?  I don't know.  I think *this* experiment is a very
conservative one.  Try it and see, *then* oppose the feature if
appropriate.  If that works (that is, provides useful information
leading to consensus, which might be to keep or to revert the
feature), more progressive experiments may be justified.


Footnotes: 
[1]  Note: I agree with Alan that having some reactionaries around is
a *good* thing, that "somebody has to do it".  I don't think we should
open up the whole Unicode repertoire to "regularize" syntax at each
developer's whim.  Each new character introduced needs to be
justified.  Simply stating that principle isn't enough -- somebody
(the "reactionary") is needed to make sure that the discussion takes
place.

[2]  Emacs is an organization of professionals, even if it can't
afford to pay them market rates.  And it certainly can't afford to pay
HCI experts.  It just has its reasons for avoiding the experiments
needed to get good information about usability of UI changes, and so
falls back on "logical thought about the changes".  Disclaimer: I
believe that those reasons are valid, but they are weaker today than
ever before, and the conservatism should be relaxed.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/23/2015 06:16 AM, Stephen J. Turnbull wrote:

> It's also true that I've expressed an *opinion* that the costs to a
> relative few are small compared to potential benefits for many.
> Obviously, the opponents disagree about the balance.  When opinions
> differ, shouldn't we go get some actual facts?

Someone should also clearly state the benefits, and to what they are 
attributed.

There's not too much value in having curly quotes appear directly in the 
source file. Maybe they'll appeal to aesthetics of some subset of Elisp 
programmers, whereas a lot of others will just be surprised. We could 
also reach the same effect with font-lock, with extra flexibility, and 
without the need to change the substitute-command-keys API (and support 
it forever thereafter).

As Andreas mentioned before, not every native English speaker is 
accustomed to them, and certainly not most of the people for whom 
English is not a native language.

> Of course it's possible to go too far.  What's the appropriate
> balance?  I don't know.  I think *this* experiment is a very
> conservative one.  Try it and see, *then* oppose the feature if
> appropriate.  If that works (that is, provides useful information
> leading to consensus, which might be to keep or to revert the
> feature), more progressive experiments may be justified.

Anyone interested in "trying" it can check out the current Emacs sources 
and type curly quotes in docstrings to their heart's content. There's 
not much to see or evaluate there.

> [2]  Emacs is an organization of professionals, even if it can't
> afford to pay them market rates.  And it certainly can't afford to pay
> HCI experts.  It just has its reasons for avoiding the experiments
> needed to get good information about usability of UI changes, and so
> falls back on "logical thought about the changes".  Disclaimer: I
> believe that those reasons are valid, but they are weaker today than
> ever before, and the conservatism should be relaxed.

It would take quite a bit of conceit to assume that Emacs can make 
better choices WRT to docstring markup syntax than virtually every 
programming language and documentation format out there.

We're not even competing with other editors here.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 1:40 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 06/23/2015 06:16 AM, Stephen J. Turnbull wrote:

> As Andreas mentioned before, not every native English speaker is accustomed
> to them, and certainly not most of the people for whom English is not a
> native language.

It’s probably not worth noting, but that can’t possible be true.  Is
someone who sees ‘‘’ going to wonder what that symbol means, whereas
if they see ‘'’ they’ll go “aha!”?  Most languages have quotes in some
form or other and they vary wildly between languages, but I have a
very hard time understanding how anyone, native English reader/writer
or not, could be confused by typesetter’s quotation and not by “ASCII
quotation”.  Given that almost all professionally printed literature
uses real quotation marks, I’d assume that those who are learning
English will also be given the benefit of them (where native English
readers will have seen them in books since they were little).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Oleh Krehel]

Nikolai Weibull <now@disu.se> writes:

> On Tue, Jun 23, 2015 at 1:40 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>> On 06/23/2015 06:16 AM, Stephen J. Turnbull wrote:
>
>> As Andreas mentioned before, not every native English speaker is accustomed
>> to them, and certainly not most of the people for whom English is not a
>> native language.
>
> It’s probably not worth noting, but that can’t possible be true.  Is
> someone who sees ‘‘’ going to wonder what that symbol means, whereas
> if they see ‘'’ they’ll go “aha!”?

From my perspective (English isn't my mother-tongue), it's totally true.

Honestly, it's very weird to see "It’s" written by you, when I'm used to
seeing "It's" in all electronic media (which accounts for 100% of my
English usage). The only place where I think "It’s" is appropriate
instead of "It's" is a LaTeX-typeset PDF document: not HTML, not email,
not social media, not Info, and certainly not source code. And even in
LaTeX ASCII is used to typeset these characters for the simple reason
that they are present on 95% of the keyboards with an English layout.

Same thing goes for “aha!”. I actually remember seeing these type of
quotes (very rarely) even before I have learned Emacs. And I recall
being super-annoyed that my regular quotes that I had just learned how
to insert with my keyboard looked different. At that point, I had no
idea about ASCII or Unicode, or programming, and knew only the basics of
computer interaction, such as using the keyboard and the clipboard.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 3:22 PM, Oleh Krehel <ohwoeowho@gmail.com> wrote:
> Nikolai Weibull <now@disu.se> writes:
>
>> On Tue, Jun 23, 2015 at 1:40 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>>> On 06/23/2015 06:16 AM, Stephen J. Turnbull wrote:
>>
>>> As Andreas mentioned before, not every native English speaker is accustomed
>>> to them, and certainly not most of the people for whom English is not a
>>> native language.
>>
>> It’s probably not worth noting, but that can’t possible be true.  Is
>> someone who sees ‘‘’ going to wonder what that symbol means, whereas
>> if they see ‘'’ they’ll go “aha!”?
>
> From my perspective (English isn't my mother-tongue), it's totally true.
>
> Honestly, it's very weird to see "It’s" written by you, when I'm used to
> seeing "It's" in all electronic media (which accounts for 100% of my
> English usage).

As you yourself point out, this is due to you being a person who’s
only used English in a medium that for very long was severely limited
in expressive power (ASCII and ISO-8859-5?).  It doesn’t mean that
it’s true for most people who aren’t native English readers/writers.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Nikolai.

On Tue, Jun 23, 2015 at 08:22:58PM +0200, Nikolai Weibull wrote:
> On Tue, Jun 23, 2015 at 3:22 PM, Oleh Krehel <ohwoeowho@gmail.com> wrote:

> > Honestly, it's very weird to see "It’s" written by you, when I'm used to
> > seeing "It's" in all electronic media (which accounts for 100% of my
> > English usage).

> As you yourself point out, this is due to you being a person who’s
> only used English in a medium that for very long was severely limited
> in expressive power (ASCII and ISO-8859-5?).

ASCII and ISO-8859-1 are not even slightly limited in expressive power.
Anything you can say in English can be expressed efficiently in these
character sets.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 9:29 PM, Alan Mackenzie <acm@muc.de> wrote:
> Hello, Nikolai.
>
> On Tue, Jun 23, 2015 at 08:22:58PM +0200, Nikolai Weibull wrote:
>> On Tue, Jun 23, 2015 at 3:22 PM, Oleh Krehel <ohwoeowho@gmail.com> wrote:
>
>> > Honestly, it's very weird to see "It’s" written by you, when I'm used to
>> > seeing "It's" in all electronic media (which accounts for 100% of my
>> > English usage).

>> As you yourself point out, this is due to you being a person who’s
>> only used English in a medium that for very long was severely limited
>> in expressive power (ASCII and ISO-8859-5?).

> ASCII and ISO-8859-1 are not even slightly limited in expressive power.
> Anything you can say in English can be expressed efficiently in these
> character sets.

That was *-5, not *-1.  My point was that it’s severely limited in
expressing anything outside of what’s required for English, and even
then it’s been very restrictive.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/23/2015 04:02 PM, Nikolai Weibull wrote:

> It’s probably not worth noting, but that can’t possible be true.  Is
> someone who sees ‘‘’ going to wonder what that symbol means, whereas
> if they see ‘'’ they’ll go “aha!”?

Not confused, but probably a bit weirded out. Pretty much what Oleh said.

> Given that almost all professionally printed literature
> uses real quotation marks, I’d assume that those who are learning
> English will also be given the benefit of them (where native English
> readers will have seen them in books since they were little).

Speaking of printed literature, I've just opened a random O'Reilly's 
book. While curly double-quotes appear a lot, the closest thing to 
single quotes were apostrophes (which look the same, but serve a 
different purpose).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 4:03 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 06/23/2015 04:02 PM, Nikolai Weibull wrote:
>
>> It’s probably not worth noting, but that can’t possible be true.  Is
>> someone who sees ‘‘’ going to wonder what that symbol means, whereas
>> if they see ‘'’ they’ll go “aha!”?
>
> Not confused, but probably a bit weirded out. Pretty much what Oleh said.

Reading a German text where they „quote like this“ looks really weird
to an American or a Swede as well.  But we’re not even talking about
differing quotation styles.  We’re talking about the same symbols
(modulo the different code points), just rendered either as a nice
piece of wrapping around the quoted text, or as four fencepost that
keep the quoted text from walking on the lawn.

>> Given that almost all professionally printed literature
>> uses real quotation marks, I’d assume that those who are learning
>> English will also be given the benefit of them (where native English
>> readers will have seen them in books since they were little).

> Speaking of printed literature, I've just opened a random O'Reilly's book.
> While curly double-quotes appear a lot, the closest thing to single quotes
> were apostrophes (which look the same, but serve a different purpose).

That’s because the book was written in American English, not English
English.  In American English, single quotes are used when quotes are
nested.  It’s (generally) the other way around in English English.
Regarding apostrophes, they /are/ the same, even though they server
different purposes.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/23/2015 09:39 PM, Nikolai Weibull wrote:

> Reading a German text where they „quote like this“ looks really weird
> to an American or a Swede as well.  But we’re not even talking about
> differing quotation styles.  We’re talking about the same symbols
> (modulo the different code points), just rendered either as a nice
> piece of wrapping around the quoted text, or as four fencepost that
> keep the quoted text from walking on the lawn.

You say "nice", I say "weird". In source code, at least.

The fact that I'd have to enable electric-quote-mode and see Emacs 
insert different quotes than those I've just typed, doesn't help either. 
This might be the main reason for my opposition.

> That’s because the book was written in American English, not English
> English.  In American English, single quotes are used when quotes are
> nested.

FWIW, Emacs codebase leans toward American Engligh. And it's also the 
dialect of English most prevalent in the world, for better or worse.

 > It’s (generally) the other way around in English English.

I suppose I haven't read that many books in UK English. Lewis Carroll 
seems to disagree, too:

https://en.wikipedia.org/wiki/File:Alice's_Adventures_Under_Ground_-_Lewis_Carroll_-_British_Library_Add_MS_46700_f45v.jpg

> Regarding apostrophes, they /are/ the same, even though they server
> different purposes.

Looks are not that important. If anything, only seeing "single quotes" 
used as apostrophes can leave an impression that they're never used as 
quotation marks.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 9:04 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 06/23/2015 09:39 PM, Nikolai Weibull wrote:

>> It’s (generally) the other way around in English English.
>
> I suppose I haven't read that many books in UK English. Lewis Carroll seems
> to disagree, too:
>
> https://en.wikipedia.org/wiki/File:Alice's_Adventures_Under_Ground_-_Lewis_Carroll_-_British_Library_Add_MS_46700_f45v.jpg

Hence the “(generally)”.  Oh, and the 150 years since that book was
published may have had an influence on the matter.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW, Emacs codebase leans toward American Engligh. And it's also the 
  > dialect of English most prevalent in the world, for better or worse.

US English is best to use in Emacs, for that reason.  For consistency's sake,
it would be cleaner to convert all British English to US English.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Tue, 23 Jun 2015 15:02:56 +0200
> From: Nikolai Weibull <now@disu.se>
> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, Eli Zaretskii <eliz@gnu.org>, acm@muc.de, eggert@cs.ucla.edu, 
> 	rms@gnu.org, Emacs Developers <emacs-devel@gnu.org>
> 
> almost all professionally printed literature uses real quotation
> marks

That's true, but most of them look quite differently from  “..”, at
least with most monospaced fonts.  So the issue is not as easy or
simple as you seem to imply.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 5:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 23 Jun 2015 15:02:56 +0200
>> From: Nikolai Weibull <now@disu.se>
>> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, Eli Zaretskii <eliz@gnu.org>, acm@muc.de, eggert@cs.ucla.edu,
>>       rms@gnu.org, Emacs Developers <emacs-devel@gnu.org>
>>
>> almost all professionally printed literature uses real quotation
>> marks

> That's true, but most of them look quite differently from  “..”, at
> least with most monospaced fonts.  So the issue is not as easy or
> simple as you seem to imply.

Different from what?  Do you mean that most mono-spaced fonts don’t
have nice-looking quotation marks?  Mine (DejaVu Sans Mono) does, but
even so, I’m not quite sure what that has to do with anything.  Do you
mean that most users of Emacs who read their e-mail in a mono-spaced
font may be confused by their font’s poor rendering of real quotation
marks?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Tue, 23 Jun 2015 20:17:49 +0200
> From: Nikolai Weibull <now@disu.se>
> Cc: Brief Busters <dgutov@yandex.ru>, Stephen Turnbull <stephen@xemacs.org>, acm@muc.de, eggert@cs.ucla.edu, 
> 	rms@gnu.org, Emacs Developers <emacs-devel@gnu.org>
> 
> On Tue, Jun 23, 2015 at 5:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Tue, 23 Jun 2015 15:02:56 +0200
> >> From: Nikolai Weibull <now@disu.se>
> >> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, Eli Zaretskii <eliz@gnu.org>, acm@muc.de, eggert@cs.ucla.edu,
> >>       rms@gnu.org, Emacs Developers <emacs-devel@gnu.org>
> >>
> >> almost all professionally printed literature uses real quotation
> >> marks
> 
> > That's true, but most of them look quite differently from  “..”, at
> > least with most monospaced fonts.  So the issue is not as easy or
> > simple as you seem to imply.
> 
> Different from what?  Do you mean that most mono-spaced fonts don’t
> have nice-looking quotation marks?

They are different from what one sees in "almost all professionally
printed literature".  Moreover, they look different from what Word
inserts as part of its "smart quotes" feature, because its default
fonts are not monospaced.

> Mine (DejaVu Sans Mono) does

No, it doesn't (I'm looking at it).  Its curved quotes look very much
like tilted straight lines.  You want to see curved, compare with
Symbola or Lucida Console.

> Do you mean that most users of Emacs who read their e-mail in a
> mono-spaced font may be confused by their font’s poor rendering of
> real quotation marks?

No, I mean that seeing these quotes in scientific and professional
literature won't help, because they are typeset in a very different
typeface there.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 8:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> On Tue, Jun 23, 2015 at 5:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> >> almost all professionally printed literature uses real quotation
>> >> marks
>>
>> > That's true, but most of them look quite differently from  “..”, at
>> > least with most monospaced fonts.  So the issue is not as easy or
>> > simple as you seem to imply.
>>
>> Different from what?  Do you mean that most mono-spaced fonts don’t
>> have nice-looking quotation marks?
>
> They are different from what one sees in "almost all professionally
> printed literature".  Moreover, they look different from what Word
> inserts as part of its "smart quotes" feature, because its default
> fonts are not monospaced.
>
>> Mine (DejaVu Sans Mono) does
>
> No, it doesn't (I'm looking at it).  Its curved quotes look very much
> like tilted straight lines.

That’s a matter of opinion.  While it doesn’t curve like, for example,
Computer Modern Roman or, as you point out, Symbola or Lucida Console,
they hardly look like tilted straight lines.  See Tahoma for such a
rendition.  Either way, I don’t agree at all with the point your
trying to make.  The letter ‘a’ can, for example, be rendered in two
very different ways and we still manage to read it as the same letter.

> You want to see curved, compare with
> Symbola or Lucida Console.

So if a user happens to use these fonts they’ll be fine?

>> Do you mean that most users of Emacs who read their e-mail in a
>> mono-spaced font may be confused by their font’s poor rendering of
>> real quotation marks?
>
> No, I mean that seeing these quotes in scientific and professional
> literature won't help, because they are typeset in a very different
> typeface there.

So they’re expected to be able to read but not be able to see past the
minute differences between two symbols (which are their own mirror
images, so really only one) in two different typefaces?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Tue, 23 Jun 2015 21:05:42 +0200
> From: Nikolai Weibull <now@disu.se>
> Cc: Brief Busters <dgutov@yandex.ru>, Stephen Turnbull <stephen@xemacs.org>, acm@muc.de, eggert@cs.ucla.edu, 
> 	rms@gnu.org, Emacs Developers <emacs-devel@gnu.org>
> 
> > You want to see curved, compare with
> > Symbola or Lucida Console.
> 
> So if a user happens to use these fonts they’ll be fine?

No.

> >> Do you mean that most users of Emacs who read their e-mail in a
> >> mono-spaced font may be confused by their font’s poor rendering of
> >> real quotation marks?
> >
> > No, I mean that seeing these quotes in scientific and professional
> > literature won't help, because they are typeset in a very different
> > typeface there.
> 
> So they’re expected to be able to read but not be able to see past the
> minute differences between two symbols (which are their own mirror
> images, so really only one) in two different typefaces?

They shouldn't be expected to be able to distinguish between
confusingly similar, but subtly different characters.  Let me give you
a few examples:

 "
 “
 ”
 ‟
 〞
 〟
 “
 „

Can you tell which one is which just by looking at them?  (Assuming
your fonts are set to display all of them, that is.)

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 9:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 23 Jun 2015 21:05:42 +0200
>> From: Nikolai Weibull <now@disu.se>
>> Cc: Brief Busters <dgutov@yandex.ru>, Stephen Turnbull <stephen@xemacs.org>, acm@muc.de, eggert@cs.ucla.edu,
>>       rms@gnu.org, Emacs Developers <emacs-devel@gnu.org>
>>
>> > You want to see curved, compare with
>> > Symbola or Lucida Console.

>> So if a user happens to use these fonts they’ll be fine?

> No.

I don’t see why.  I thought that the lack of curvature was why the
user would be confused in regard to DejaVu Sans Mono, but then they
shouldn’t be confused if they use a mono-spaced font with enough
curvature.  But I’ll drop it, as I feel that this is just getting
confrontational for no good reason.

>> >> Do you mean that most users of Emacs who read their e-mail in a
>> >> mono-spaced font may be confused by their font’s poor rendering of
>> >> real quotation marks?
>> >
>> > No, I mean that seeing these quotes in scientific and professional
>> > literature won't help, because they are typeset in a very different
>> > typeface there.
>>
>> So they’re expected to be able to read but not be able to see past the
>> minute differences between two symbols (which are their own mirror
>> images, so really only one) in two different typefaces?
>
> They shouldn't be expected to be able to distinguish between
> confusingly similar, but subtly different characters.

Which is probably why most people don’t.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Marcin Borkowski]

On 2015-06-23, at 21:21, Eli Zaretskii <eliz@gnu.org> wrote:

> They shouldn't be expected to be able to distinguish between
> confusingly similar, but subtly different characters.  Let me give you
> a few examples:
>
>  "
>  “
>  ”
>  ‟
>  〞
>  〟
>  “
>  „

1. My personal opinion is: I prefer `...', but I could live with ‘...’
provided that the input and search problems will be solved in
a satisfactory way and that I could find a font which renders them
better than the default font in my Emacs (Ubuntu Mono), where the
Unicode quotes look terrible.  (I guess that when I finally get rid of
this pile of crap which Ubuntu has become, it will get better
automatically...)

2. I would like to point out to all Unicode fanboys (no offence, please,
I'm also a fanboy, though definitely not of Unicode) that AFAIK (though
I can't find any source now) there is a bug in Unicode: U+201C should
really be split into two characters, "LEFT DOUBLE QUOTATION MARK" and
"GERMAN RIGHT DOUBLE QUOTATION MARK" (see
e.g. https://en.wikipedia.org/wiki/Quotation_mark#German_.28Germany_and_Austria.29).
While their shape is identical, their bounding boxes should be mirror
images of each other.

So please don't tell me that Unicode "solves the problem of nice
quotes".  It does, but at the same time it introduces problems of its
own.  (A funny one: see http://unicodelookup.com/#math and try to guess
what happened to 0x1D455, or "mathematical italic small h".  A stupid
one: U+FE18, with a typo in the name.)  In fact, Unicode seems to be
fundamentally broken by design, since it identifies characters by
numbers instead of names.  This basically excludes any language with an
infinite[1] alphabet.

[1] "infinite" in a practical sense, not a theoretical one, of course.
It is closer to "potential infinity".  I mean here a language where it
is legal to create new characters on the fly, when needed, provided they
are combined from some basic shapes according to the rules.

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Matthew Carter]

Marcin Borkowski <mbork@mbork.pl> writes:

> On 2015-06-23, at 21:21, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> They shouldn't be expected to be able to distinguish between
>> confusingly similar, but subtly different characters.  Let me give you
>> a few examples:
>>
>>  "
>>  “
>>  ”
>>  ‟
>>  〞
>>  〟
>>  “
>>  „
>
> 1. My personal opinion is: I prefer `...', but I could live with ‘...’
> provided that the input and search problems will be solved in
> a satisfactory way and that I could find a font which renders them
> better than the default font in my Emacs (Ubuntu Mono), where the
> Unicode quotes look terrible.  (I guess that when I finally get rid of
> this pile of crap which Ubuntu has become, it will get better
> automatically...)
>
> 2. I would like to point out to all Unicode fanboys (no offence, please,
> I'm also a fanboy, though definitely not of Unicode) that AFAIK (though
> I can't find any source now) there is a bug in Unicode: U+201C should
> really be split into two characters, "LEFT DOUBLE QUOTATION MARK" and
> "GERMAN RIGHT DOUBLE QUOTATION MARK" (see
> e.g. https://en.wikipedia.org/wiki/Quotation_mark#German_.28Germany_and_Austria.29).
> While their shape is identical, their bounding boxes should be mirror
> images of each other.
>
> So please don't tell me that Unicode "solves the problem of nice
> quotes".  It does, but at the same time it introduces problems of its
> own.  (A funny one: see http://unicodelookup.com/#math and try to guess
> what happened to 0x1D455, or "mathematical italic small h".  A stupid
> one: U+FE18, with a typo in the name.)  In fact, Unicode seems to be
> fundamentally broken by design, since it identifies characters by
> numbers instead of names.  This basically excludes any language with an
> infinite[1] alphabet.
>
> [1] "infinite" in a practical sense, not a theoretical one, of course.
> It is closer to "potential infinity".  I mean here a language where it
> is legal to create new characters on the fly, when needed, provided they
> are combined from some basic shapes according to the rules.
>
> Best,

Having done a lot of web development work (where unicode and different
character sets between databases, web clients etc.) have caused many
headaches, I would hate to see something that should be kept to display
formats (rendered markdown, LaTeX, .odt files) in my primarily ASCII 
based terminal (emacs -nw mode).

It increases the difficulty in working with the characters (I'd always
prefer to type `...' than whatever is required to insert curlies) and it
will break some web based tutorials/guides where the writer/blogger
copies and pastes the unicode quotes out of some part of emacs and the
web portion fails on it (similar to users copying and pasting curly
quotes from Microsoft Word into various WYSIWYG editors).

In my Common Lisp personal projects I love taking advantage of the
non-ASCII character set that Common Lisp can use (ƒ, λ, α, ψ etc.), but
I would never expect someone else to maintain or touch said codebase.

-- 
Matthew Carter (m@ahungry.com)
http://ahungry.com

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Marcin Borkowski]

On 2015-06-23, at 22:30, Richard M. Stallman - Autoreply Message <rms-autoreply-control@gnu.org> wrote:

> I am not on vacation, but I am at the end of a long time delay. I am
> located somewhere on Earth, but as far as responding to email is concerned,
> I appear to be well outside the solar system.
>
> After your message arrives at gnu.org, I will collect it in my next batch of
> incoming mail, some time within the following 24 hours. [...]

Dr. Stallman, funny as your auto-message is, it's probably incorrect.
I'm not an astronomy buff myself, but I showed it to a friend with whom
I have a long tradition of sharing funny things found on the 'net, and
he pointed out that 24 light-hours is still within the Solar System,
somewhere in the inner Oort cloud.

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I have a long tradition of sharing funny things found on the 'net, and
  > he pointed out that 24 light-hours is still within the Solar System,
  > somewhere in the inner Oort cloud.

I will ask for it to be fixed.  Thanks.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> Someone should also clearly state the benefits, and to what they
> are attributed.

For me the main issue in source code's use of `...' is that
it's ambiguous.  Using curly quotes is one way to solve this issue (tho
it should come with some escaping mechanism to make sure that the very
rare other uses of curly quotes in source code aren't ambiguous).

We could also keep using `...' in source code with some escaping
mechanism (better and more complete than the one we currently have).

If we render using curly quotes, for aesthetic reasons, then using curly
quotes in the source code has also the advantage of eliminating magic
(font-lock or whatnot) between the two.


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/23/2015 04:34 PM, Stefan Monnier wrote:

> For me the main issue in source code's use of `...' is that
> it's ambiguous.  Using curly quotes is one way to solve this issue (tho
> it should come with some escaping mechanism to make sure that the very
> rare other uses of curly quotes in source code aren't ambiguous).

Indeed. We will need an unambiguous escaping mechanism either way.

> We could also keep using `...' in source code with some escaping
> mechanism (better and more complete than the one we currently have).

Would someone please just pick one? It seems the exact choice of this 
mechanism is irrelevant to the choice of which quotes to use.

The obvious approach is to use backslashes, but then you'll have to use 
four of them to escape a single quote character in a docstring.

> If we render using curly quotes, for aesthetic reasons, then using curly
> quotes in the source code has also the advantage of eliminating magic
> (font-lock or whatnot) between the two.

And then when we switch to a different rendering, we'll be stuck with 
the curlies anyway, for compatibility reasons.

A font-lock solution is nicer because it doesn't require any changes to 
the public Lisp API.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For me the main issue in source code's use of `...' is that
  > it's ambiguous.

It is ambiguous in some cases if parsed by a program.
No program needed to parse it until recent changes.

There are other easy ways to get rid of the ambiguity.
For instance, this

   To get out of the recursive edit, a command can throw to `exit' -- for
   instance `(throw 'exit nil)'.

can be written as

   To get out of the recursive edit, a command can throw to `exit':
      (throw 'exit nil)

or as

   To get out of the recursive edit, a command can throw to `exit'
   using `throw'.

I just looked at the main lisp directory and found no example
of a confusing doc string.  In lisp/emacs-lisp, the one
case in advice.el seems to be the only one.

Considering how few of these there are, ANY of these fixes would be an
easy solution.  Thus, it is better to fix them in local ways that
cause no trouble to anyone -- such as the two I've described above --
rather than make a global change.

That is especially true if the global change will cause some sort
of trouble.  I am not sure whether it will, but see my other message.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/24/2015 01:22 AM, Richard Stallman wrote:

> I just looked at the main lisp directory and found no example
> of a confusing doc string.  In lisp/emacs-lisp, the one
> case in advice.el seems to be the only one.

See the docstrings in macroexp-progn, macroexp-let* and macroexp-let2, 
for instance. Rewriting them like that would make them worse.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > See the docstrings in macroexp-progn, macroexp-let* and macroexp-let2, 
  > for instance.

They seem to be ok the way they are.  No change is needed, because
there is no problem.

Since they contain an unmatched backquote, a program to parse quotes
in doc strings might get confused by them.  But that problem is easy
to avoid: make the parser ignore backquotes that have no closing
singlequote.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/24/2015 05:55 PM, Richard Stallman wrote:

> Since they contain an unmatched backquote, a program to parse quotes
> in doc strings might get confused by them.

macroexp-let2 has several straight quotes later in the docstring. Why 
would we consider the first backtick unmatched?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > macroexp-let2 has several straight quotes later in the docstring. Why 
  > would we consider the first backtick unmatched?

I can't understand the doc string for macroexp-let2.
Can anyone explain to me what this macro is supposed
to do, and offer a concrete example of its use?

When I understand it, I can write a doc string for it that avoids
anything confusing in the quoting, and that is comprehensible.

At this point, all I can say is that V is a metasyntactic variable,
not literal Lisp code.  So it should be capitalized, not quoted.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> I can't understand the doc string for macroexp-let2.
> Can anyone explain to me what this macro is supposed
> to do, and offer a concrete example of its use?

It abstracts the form:

   (let ((sym (make-symbol)))
     `(let ((,sym ,exp))
        ... ,sym ... ,sym ...))

used in macros to make sure that EXP is executed exactly one (and at
a known place with the right variables in scope), instead of the naive

     `(... ,exp ... ,exp ...))

which risks running EXP too many times, or at the wrong time, or the
other naive

     `(let ((val ,exp))
        ... val ... val ...))

which risks name capture if the `val' variable appears elsewhere.

> At this point, all I can say is that V is a metasyntactic variable,
> not literal Lisp code.  So it should be capitalized, not quoted.

That's right,


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    (let ((sym (make-symbol)))
  >      `(let ((,sym ,exp))
  >         ... ,sym ... ,sym ...))

  > used in macros to make sure that EXP is executed exactly one (and at
  > a known place with the right variables in scope),

Ok.  But what are the arguments to macroexp-let2?
How would you call it, in that case?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

>> (let ((sym (make-symbol)))
>>   `(let ((,sym ,exp))
>>      ... ,sym ... ,sym ...))
>>
>> used in macros to make sure that EXP is executed exactly one (and at
>> a known place with the right variables in scope),
> Ok.  But what are the arguments to macroexp-let2?
> How would you call it, in that case?

I'd call it

 (macroexp-let2 nil sym exp
   `(... ,sym ... ,sym ...))


-- Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I propose this doc string for macroexp-let2.  What do you think?


Evaluate BODY with SYM bound to an expression for EXP's value.
The intended usage is that BODY generates an expression that
will refer to EXP's value multiple times, but will evaluate
EXP only once.  As BODY generates that expression, it should
use SYM to stand for the value of EXP.

If EXP is a simple, safe expression, then SYM's value is EXP itself.
Otherwise, SYM's value is a symbol which holds the value produced by
evaluating EXP.  The return value incorporates the value of BODY, plus
additional code to evaluate EXP once and save the result so SYM can
refer to it.

If BODY consists of multiple forms, they are all evaluated
but only the last one's value matters.

TEST is a predicate to determine whether EXP qualifies as simple and
safe; if TEST is nil, it defaults to `macroexp-const-p'.

Example:
 (macroexp-let2 nil foo EXP
   `(* ,foo ,foo))
generates an expression that evaluates EXP once,
then returns the square of that value.
You could do this with
  (let ((foovar EXP))
    (* foovar foovar))
but using `macroexp-let2' produces more efficient code in
cases where EXP is a constant.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> I propose this doc string for macroexp-let2.  What do you think?

I find it overly verbose for my taste, but if you like it, by all means,
install it,


        Stefan

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Here's a slightly revised doc string proposal for macroexp-let2.

It is a real pain for me to install anything due to my difficulties
with git.  Can someone please install this doc string for me?


Evaluate BODY with SYM bound to an expression for EXP's value.
The intended usage is that BODY generates an expression that
will refer to EXP's value multiple times, but will evaluate
EXP only once.  As BODY generates that expression, it should
use SYM to stand for the value of EXP.

If EXP is a simple, safe expression, then SYM's value is EXP itself.
Otherwise, SYM's value is a symbol which holds the value produced by
evaluating EXP.  The return value incorporates the value of BODY, plus
additional code to evaluate EXP once and save the result so SYM can
refer to it.

If BODY consists of multiple forms, they are all evaluated
but only the last one's value matters.

TEST is a predicate to determine whether EXP qualifies as simple and
safe; if TEST is nil, only constant expressions qualify.

Example:
 (macroexp-let2 nil foo EXP
   `(* ,foo ,foo))
generates an expression that evaluates EXP once,
then returns the square of that value.
You could do this with
  (let ((foovar EXP))
    (* foovar foovar))
but using `macroexp-let2' produces more efficient code in
cases where EXP is a constant.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> It is a real pain for me to install anything due to my difficulties
> with git.  Can someone please install this doc string for me?

Done as commit 5e3fde03b45877d3e30f859a14c10043e637aa63.

What difficulties are you having with git?  For example, does ‘git diff’ work 
for you?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I have edited a few files with changes I don't want to install yet,
and the idea of tackling git again to install a doc string discourages me.
It is sure to take me quite some time, and I have so much else to do.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

macroexp-let2 is written in a very strange way, computing
a value for ,var, then testing ,var to determine whether
to bind a variable to the value of EXP or use EXP directly.

If there is a reason for this, I don't see it -- so that reason
ought to be explained by a comment.

If there isn't a reason, why not write it in a clearer way?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Dmitry Gutov writes:

 > It would take quite a bit of conceit to assume that Emacs can make
 > better choices WRT to docstring markup syntax than virtually every
 > programming language and documentation format out there.

No, not at all.  To assume that Emacs can consistently do so *by
design* would be conceited.  To try new things, keep the ones that
work and abandon the others, is a strategy that is known to achieve
better choices in biology and business as well as in software.

 > We're not even competing with other editors here.

Exactly.  Emacs isn't competing.  It's doing what's right for Emacs.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/23/2015 06:36 PM, Stephen J. Turnbull wrote:

> No, not at all.  To assume that Emacs can consistently do so *by
> design* would be conceited.  To try new things, keep the ones that
> work and abandon the others, is a strategy that is known to achieve
> better choices in biology and business as well as in software.

Maybe. But you seem to be arguing in favor of a general notion (to keep 
experimenting), whereas I was under impression that this thread is about 
a specific issue.

It's a good recipe to continue talking past each other.

>   > We're not even competing with other editors here.
>
> Exactly.  Emacs isn't competing.  It's doing what's right for Emacs.

Exactly... not what I meant by the above sentence?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Dmitry Gutov writes:
 > On 06/23/2015 06:36 PM, Stephen J. Turnbull wrote:

 > Maybe. But you seem to be arguing in favor of a general notion (to keep 
 > experimenting),

Yes.  See Eli's responses in a separate subthread as well.

 > whereas I was under impression that this thread is about a specific
 > issue.

Some subthreads are, yes.  Others are not, they take the change that
inspired this thread as a specific example of a more general issue.

 > It's a good recipe to continue talking past each other.

It is, but people write about what they write about.  It's not just me.

 > >   > We're not even competing with other editors here.
 > >
 > > Exactly.  Emacs isn't competing.  It's doing what's right for Emacs.
 > 
 > Exactly... not what I meant by the above sentence?

You tell me.  You state a fact, I agree that it's true, "exactly".  I
take the lack of competition as an opportunity to do what's right, you
take it as a reason to stick to the status quo.  Opinions differ.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/24/2015 06:13 PM, Stephen J. Turnbull wrote:

> It is, but people write about what they write about.  It's not just me.

In this case it's as if you've ignored half of the message.

>   > >   > We're not even competing with other editors here.
>   > >
>   > > Exactly.  Emacs isn't competing.  It's doing what's right for Emacs.
>   >
>   > Exactly... not what I meant by the above sentence?
>
> You tell me.  You state a fact, I agree that it's true, "exactly".  I
> take the lack of competition as an opportunity to do what's right, you
> take it as a reason to stick to the status quo.  Opinions differ.

We *are* competing here. Not with Editors, but with other languages and 
documentation formats.

Of course, only if we're talking about using fancy quotes directly in 
Elisp docstrings, and not some other other arbitrary issue that comes to 
mind.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Dmitry Gutov writes:

 > We *are* competing here. Not with Editors, but with other languages and 
 > documentation formats.

I think Richard will tell you differently, that "competition" with
other editors, languages, or formats is never a reason for Emacs to do
something.  Cooperation (portability or compatibility) often is,
though.  But you'll have to ask him to get an authoritative statement.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Dmitry Gutov]

On 06/24/2015 09:55 PM, Stephen J. Turnbull wrote:

> I think Richard will tell you differently, that "competition" with
> other editors, languages, or formats is never a reason for Emacs to do
> something.  Cooperation (portability or compatibility) often is,
> though.  But you'll have to ask him to get an authoritative statement.

Again, you're holding on to the wrong word. Point is, we're trying to 
solve a minor problem that others have solved before.

Of course, the best way to do that is to approach the problem from an 
arbitrary angle, like asking a few students heretofore unfamiliar with 
Emacs (or programming). Or look at some kind of analog media.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Dmitry Gutov writes:
 > On 06/24/2015 09:55 PM, Stephen J. Turnbull wrote:
 > 
 > > I think Richard will tell you differently, that "competition" with
 > > other editors, languages, or formats is never a reason for Emacs to do
 > > something.  Cooperation (portability or compatibility) often is,
 > > though.  But you'll have to ask him to get an authoritative statement.
 > 
 > Again, you're holding on to the wrong word.

Then don't use it.  That's you talking past me, not the other way
around.

 > Point is, we're trying to solve a minor problem that others have
 > solved before.

Then post a pointer to those solutions or discussions.  You can't
assume your readers are so familiar with them that the word
"competition" will cause useful associations.

But AFAIK, no, this is not a problem that others have solved.  I've
seen the generic "shall we use Unicode characters as syntax" discussed
a few times, and to date the conclusion has always been "we can't
count on Unicode to be available, so we can't."  That's not a
solution, that's postponement.  And Emacs doesn't have that problem,
although apparently some platforms supported by Emacs do (specificly,
the Windows command line).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Turnbull has misrepresented my views again:

  > You're telling *me*?  Eli, it's the opponents like RMS, Alan, and
  > Drew who are dismissing all possible benefits without even trying the
  > change for long enough to confirm they can't adjust to the cost.

Please read my own messages to see what I have said about this at
various times during this discussion.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Richard Stallman writes:

 > Turnbull has misrepresented my views again:

Dr. Stallman, as far as I can tell, I did not misrepresent you.  I
simply summarized the import of your own words, as quoted below.

 >> You're telling *me*?  Eli, it's the opponents like RMS, Alan, and
 >> Drew who are dismissing all possible benefits without even trying
 >> the change for long enough to confirm they can't adjust to the
 >> cost.
 > 
 > Please read my own messages to see what I have said about this at
 > various times during this discussion.

For example, you wrote this, quoted in full except for the NSA-baiting
header and .signature:

 > Your arguments are heated but not coherent.  Denigrating a significant
 > class of users with the term "lobby" does not make them less important
 > or less worthy of respect.  You can't refute a practical argument
 > by targeting an imaginary prejudice.
 > 
 > It is not pertinent what fraction of all computer users use non-ASCII
 > characters daily.  That doesn't tell us what fraction of Emacs users
 > use non-ASCII characters -- but that is not pertinent either.  You
 > implicitly presume that they are in favor of this change, but that is
 > not true.  Some Emacs users that use non-ASCII characters daily _in
 > Emacs_ will be inconvenienced by this change.
 > 
 > The situation is simple: using curly quotes in doc strings would be a
 > substantial inconvenience for many users _for no practical benefit_.
 > 
 > If it provided a practical benefit for other users, we would need to
 > consider how many gain how much (and what roles they have in Emacs
 > development), versus how many lose how much.  But since it doesn't,
 > that question does not arise.

I see nothing there that is inconsistent with the description
"dismissing all possible benefits" (unless you want to quibble about
the difference between "practical" and "possible"), and it's quite
obvious that you oppose the change.  IIRC it transpired that you were
not even using an Emacs in which the change had been applied at the
time you posted that message.  So what was misrepresented by my words
as you quoted them?

By the way, your apparent belief that the discussion of the number of
non-ASCII users was a description of benefits of the change is a
complete misreading.  That was an argument that the cost of learning
input methods is zero, and the cost of using them nearly zero, for a
large number of computer users, though not all.

Also, it is false that I presume that anybody is in favor of this
change who hasn't posted in support of it.  That's my main point, that
none of us know what the majority of developer-users will think of
this change until they try it and tell us.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: acm@muc.de,
>     eggert@cs.ucla.edu,
>     rms@gnu.org,
>     emacs-devel@gnu.org
> Date: Tue, 23 Jun 2015 12:16:46 +0900
> 
> > Try the Windows console some day.
> 
> I have, and Japanese has worked fine for two decades AFAICR.  This
> includes various quotation marks, which have been accessible from
> MSIME since Windows 9x at the latest.  That might be specific to the
> Japanese localization though, since you can't do anything without
> input methods in Japan.

I'm not talking about input methods here, I'm talking about display.

In any case, localized versions of Windows (or any OS) are a thing of the
past; with Unicode we should be able to have all this regardless of the
locale, right?  But we don't.  Not every locale is .UTF-8, and on
Windows there's no UTF-8 locale at all.

It's amazing how Windows, which was the first mass OS that adopted Unicode,
still doesn't have any decent support for it on the console.  Displaying
Unicode in console programs needs to use undocumented halfheartedly
supported features, tricky programming using wchar_t I/O, and eventually
you bump into the basic limitation that there are no bundled console fonts
that support any large subset of the BMP, let alone the other planes.

> So please don't tell me that I'm dismissing your problems while
> quoting my denial that I was dismissing them and the explanation of
> what I *was* doing.

Then please don't tell me my words were out of context while explaining why
they are.

>  > > But guess what?  AFAICT, the rest of the software world doesn't have
>  > > these problems.  People are typing scores of odd characters in email
>  > > to me all the time.  And not just Japanese, but good ol' boys and
>  > > girls from the U S of A.  How do they manage that, I wonder?
>  > 
>  > My guess would be that the mail composing program inserts Unicode
>  > quotes when the user types ASCII quotes.
> 
> It's not just quotation marks.  It's foreign words spelled correctly,
> ellipses, the occasional math symbol (the lemniscate (infinity) is
> popular among the more flaky of my correspondents), emoticons, and
> various other symbols (dice, playing card suit symbols, enclosed
> characters such as circled numerals).  There are too many of them to
> excuse with "smart quotes"; users are using "input methods" of some
> sort for these characters that don't exist on their keyboards.

Not IME.  When I type "1st", "2nd", etc., one particularly popular mailer,
which will remain unnamed, automatically makes the "st" and "nd" parts
rendered as superscripts.  Ellipses appear if I type "...", em-dashes
appear if I type "--", and "→" if I type "-->".  If I type "naive", it gets
converted to "naïve".  Emoticons appear automatically if I type their
ASCII-art equivalents.  Likewise with "(c)", "(tm)", "(r)", and "(e)" (I'll
let you guess what the last one gets me).  There are special
auto-conversions for math symbols, too numerous to mention.  Plus, I've
counted more than a dozen of other such automatic conversions that I can
turn on or off.  And -- no less important -- each auto-converted text gets
a small widget shown near it, which allows to undo that particular
conversion with a single mouse click.

Given all that, why would one need an input method, except when typing in
some complex script?

We in Emacs have yet a way to go until we get there.  Are there any
motivated volunteers reading this who'd like to provide something like
that in Emacs text modes?

> Let's try loosening up and find out, no?

Yes, let's.  But let's not forget about the inconveniences this will bring
to some, either.  Let's not dismiss them.  Let's provide graceful fallbacks
for them.  As we are doing now.  Let's not be too religious about these
changes.

> You don't bother to mention that you're aware of benefits, or what
> they are, when you allege that I'm biased against admitting there are
> costs.  As far as I can tell from the words you post, you are indeed
> opposed to the changes, describing only costs at every turn.  I can't
> know what you're thinking but not posting, sorry.

You can see what I'm coding, though.  Messages I post here are only part of
what I do for Emacs; the rest is in the repository.  Who do you think just
spent several days getting emoticons to display correctly out of the box,
or on making the curved quotes display as reasonable fallbacks on a Windows
console?

>  > I just think that we shouldn't dismiss so easily the issues these
>  > changes bring with them.  IOW, we should see this issue in its
>  > delicate balance.
> 
> You're telling *me*?

Not just you, everyone who reads this list.  This ain't private mail, and
I'm not talking to you alone.

> It's certainly true that I've engaged in hyperbole.

That's my sole point: don't.  Hyperboles don't help a bit here.

> I'm sure Paul thought he was doing the right thing at the time.  The only
> way for him to know what you think is the right way is for you to review.
> You didn't

Yes, I did, see

  http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-05/msg00350.html
  http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-05/msg00438.html
  http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-06/msg00012.html

and a lot of others.

IOW, your mental model of what I think and do in these matters seems to be
flawed.

> Of course it's possible to go too far.  What's the appropriate
> balance?  I don't know.

My point is that we should make these steps carefully, one step at a time,
and provide fallbacks and ways to switch this off where possible on every
step.  Which is what we've been doing.  You seem to be advocating to charge
ahead and never look back, and that's not TRT, IMO.  That's all I wanted to
point out: there shouldn't be any fervor in this development, just
cautious, slow movement ahead.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I'm not talking about input methods here, I'm talking about
 > display.

OK, I see.  Again, Japanese kind of spoils you, since the JIS
standards incorporate a bunch of Latin characters as well as a few
non-Latin scripts (Greek and Cyrillic at least).  But I can't seem to
find curly quotes, so maybe I misremembered the display repertoire.

 > In any case, localized versions of Windows (or any OS) are a thing
 > of the past;

Not in Japan.  Windows is very much localized (for example, in
recognizing Japanese charsets in IE), although most of the components
are properly internationalized.

 > > It's not just quotation marks.  It's foreign words spelled correctly,
 > > ellipses, the occasional math symbol (the lemniscate (infinity) is
 > > popular among the more flaky of my correspondents), emoticons, and
 > > various other symbols (dice, playing card suit symbols, enclosed
 > > characters such as circled numerals).  There are too many of them to
 > > excuse with "smart quotes"; users are using "input methods" of some
 > > sort for these characters that don't exist on their keyboards.
 > 
 > Not IME.  When I type "1st", "2nd", etc., one particularly popular mailer,
 > which will remain unnamed, automatically makes the "st" and "nd" parts
 > rendered as superscripts.  Ellipses appear if I type "...", em-dashes
 > appear if I type "--", and "→" if I type "-->".  If I type "naive", it gets
 > converted to "naïve".  Emoticons appear automatically if I type their
 > ASCII-art equivalents.  Likewise with "(c)", "(tm)", "(r)", and "(e)" (I'll
 > let you guess what the last one gets me).  There are special
 > auto-conversions for math symbols, too numerous to mention.  Plus, I've
 > counted more than a dozen of other such automatic conversions that I can
 > turn on or off.  And -- no less important -- each auto-converted text gets
 > a small widget shown near it, which allows to undo that particular
 > conversion with a single mouse click.
 > 
 > Given all that, why would one need an input method, except when typing in
 > some complex script?

That *is* an input method, since it's not just remapping keystrokes to
different characters, but rather using sequences of keystrokes to
represent single characters.  That's precisely the way one inputs
Japanese, including the little widget.  OTOH, C-x 8 is just
paleolithic, it's the non-programmer's equivalent to C-q <octal>.

 > We in Emacs have yet a way to go until we get there.  Are there any
 > motivated volunteers reading this who'd like to provide something like
 > that in Emacs text modes?

I don't think we're really that far off.  Quail is pretty close.  I
think we need four things: (1) tables for these characters, (2) a way
to combine tables so that we can overlay quote or sub/superscript
translation on other translation tables and perhaps overlay AltGr-
style "native keyboard" methods (ie, non-Quail), (3) a way to
"decompose" a character back into a sequence of characters, and (4)
perhaps somewhat less "highlighting" of partial translations (eg, in
Latin input, Quail underlines a character in the process of being
composed -- I find that annoying).

I suppose we could also provide a little widget for (3), on displays
that can handle it.

 > > Let's try loosening up and find out, no?
 > 
 > Yes, let's.  But let's not forget about the inconveniences this will bring
 > to some, either.  Let's not dismiss them.  Let's provide graceful fallbacks
 > for them.  As we are doing now.  Let's not be too religious about these
 > changes.

 > >  > I just think that we shouldn't dismiss so easily the issues these
 > >  > changes bring with them.  IOW, we should see this issue in its
 > >  > delicate balance.
 > > 
 > > You're telling *me*?
 > 
 > Not just you, everyone who reads this list.  This ain't private mail, and
 > I'm not talking to you alone.

Point taken.  Nevertheless, in English your phrasing, taken in its
original context, strongly implies that *I* dismissed and *I'm*
unaware of the delicate balance.  I take exception to those
implications, and ask that you take the kind of care with them that
you request of me with respect to hyperbole.

 > > It's certainly true that I've engaged in hyperbole.
 > 
 > That's my sole point: don't.  Hyperboles don't help a bit here.

I'll try, but really, in adult conversation in the three languages I
speak well enough to recognize exaggeration, it is extremely frequent.
And I'm hardly the only offender on this list.  Like it or not,
readers are going to have to deal with it.

 > IOW, your mental model of what I think and do in these matters seems to be
 > flawed.

You're absolutely right.  I stand corrected, and I apologize.

 > My point is that we should make these steps carefully, one step at
 > a time, and provide fallbacks and ways to switch this off where
 > possible on every step.

Once again, your phrasing excludes me.  This is *our* point.

 > Which is what we've been doing.  You seem to be advocating to charge
 > ahead and never look back,

Not at all.  Surely you have noticed how often I've used the word
"experiment"!  It was I who risked insulting Paul by asking for an
explicit statement that the change would be backed out on request.  I
could go on.  If you haven't noticed that, maybe you should try
harder.

Sure, I've been emphatic, but mostly in response to self-described
reactionaries and other *very* conservative posters, who don't even
want to see experiments.

It's also true that I think that we should not display different
characters from what's in the buffer in programming modes (eg
displaying left single quotation mark when the buffer contains a grave
accent), and therefore I'm in favor of experimenting with curly quotes
in Lisp sources.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: acm@muc.de,
>     eggert@cs.ucla.edu,
>     rms@gnu.org,
>     emacs-devel@gnu.org
> Date: Wed, 24 Jun 2015 04:57:57 +0900
> 
>  > My point is that we should make these steps carefully, one step at
>  > a time, and provide fallbacks and ways to switch this off where
>  > possible on every step.
> 
> Once again, your phrasing excludes me.  This is *our* point.
> 
>  > Which is what we've been doing.  You seem to be advocating to charge
>  > ahead and never look back,
> 
> Not at all.  Surely you have noticed how often I've used the word
> "experiment"!  It was I who risked insulting Paul by asking for an
> explicit statement that the change would be backed out on request.  I
> could go on.  If you haven't noticed that, maybe you should try
> harder.

Then I guess we are in violent agreement (again).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But with respect to the coding conventions Oleh cited, it is true that
  > the use of ‘foo-bar’ instead of `foo-bar' is at least encouraged for
  > lisp docstrings (and comments) although both will be displayed like the
  > former with describe-*, right?

This change in the source format is a bad idea.  It is a pain in the
neck, providing no practical benefit.

This is a doc string convention, like \\[...], not a user interface.
It needs to be convenient, not pretty.

Inserting ‘ and ’ is as very inconvenient.  Hunting for how to insert
them is even more so, as it is not documented.

Because I have an instance of ‘ in the buffer now, I was able to use
C-u C-x = on it, which told me I could insert it with C-x 8 RET 2018
RET.

I will not remember that hex code.  I will be able to look it up again
next time -- if I have an instance in the buffer.  Of course, given
one the buffer, I will not use C-x 8, I will copy it with the kill
ring.  In other words, this character is so inconvenient to insert
that I will use workarounds rather than try.

In practice, to insert curly quotes, I will have to find an instance
to copy.

The Emacs Manual in Info provides no help on inputting them.  The word
"curly" does not appear in the manual, except in regard to another
character.  In the chapter on non-ASCII characters, the only advice it
gives is the general reference to C-x 8 -- nothing about how to use
C-x 8 to get curly quotes in particular.

There was talk some months ago about adding C-x 8 shortcuts for curly
quotes, but C-x 8 C-h does not show them.  I found nothing in etc/NEWS
either.  I am running from source fetched on May 8.

If those shortcuts are implemented, they will require 3 or 4
characters, one of them a control character.  That's better than 8
characters, but still a pain.

We must not make any convention involving those characters
in source code.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 1959 bytes --]

Richard Stallman <rms@gnu.org> writes:

> This change in the source format is a bad idea.  It is a pain in the
> neck, providing no practical benefit.
>
> This is a doc string convention, like \\[...], not a user interface.
> It needs to be convenient, not pretty.
>
> Inserting ‘ and ’ is as very inconvenient.  Hunting for how to insert
> them is even more so, as it is not documented.
>
> Because I have an instance of ‘ in the buffer now, I was able to use
> C-u C-x = on it, which told me I could insert it with C-x 8 RET 2018
> RET.
>
> I will not remember that hex code.  I will be able to look it up again
> next time -- if I have an instance in the buffer.  Of course, given
> one the buffer, I will not use C-x 8, I will copy it with the kill
> ring.  In other words, this character is so inconvenient to insert
> that I will use workarounds rather than try.
>
> In practice, to insert curly quotes, I will have to find an instance
> to copy.
>
> The Emacs Manual in Info provides no help on inputting them.  The word
> "curly" does not appear in the manual, except in regard to another
> character.  In the chapter on non-ASCII characters, the only advice it
> gives is the general reference to C-x 8 -- nothing about how to use
> C-x 8 to get curly quotes in particular.
>
> There was talk some months ago about adding C-x 8 shortcuts for curly
> quotes, but C-x 8 C-h does not show them.  I found nothing in etc/NEWS
> either.  I am running from source fetched on May 8.
>
> If those shortcuts are implemented, they will require 3 or 4
> characters, one of them a control character.  That's better than 8
> characters, but still a pain.
>
> We must not make any convention involving those characters
> in source code.

I agree, but then why do we have electric-quote-mode for?  When enabled,
it will insert curly quotes in docstrings in place of `'.
-- 
Nicolas Petton
http://nicolas-petton.fr

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> From: Nicolas Petton
> Richard Stallman <rms@gnu.org> writes:
>
> > This change in the source format is a bad idea...
> > We must not make any convention involving those characters
> > in source code.
> 
> I agree, but then why do we have electric-quote-mode for?  When
> enabled, it will insert curly quotes in docstrings in place of `'.

We do *not* have `electric-quote-mode' (yet).  It was added to
development code after the last release, as part of an individual
crusade to turn `...' into ‘...’.

As S. Turnbull has carefully pointed out, this ‘...’ stuff is just
a test, to see how much you love it, and it should be reverted if
you do not love it (modulo a knowing nod from our dear leader, of
course).

As another example, starting next month all Emacs text will appear
mirrored (derorrim).  This is a test to see if you prefer such a
presentation - it is regarded by some as a convenient (and modern!)
brain-eye strengthening aid.

Again, this enhancement too can be reverted, given enough clamour
and a d-l nod.  As Mr Turnbull has also reminded us, changes to
Emacs development code have *no* momentum or inertia.

Just because a boatload of changes have radically transformed
the development version of Emacs over a period of months or
years, that means nothing.  All is easily reversible, provided
you gather enough petition signatures and present a notarized
and watermarked note from your mother.

Such code changes are simply tests, to see if you are paying
attention.  This approach is, like, waaaaaay more dynamic and
21st-century than discussing potential changes with arguments
pro & con and offering patches for people to try on their own.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Marcin Borkowski]

On 2015-06-17, at 18:03, Drew Adams <drew.adams@oracle.com> wrote:

> As another example, starting next month all Emacs text will appear
> mirrored (derorrim).  This is a test to see if you prefer such a
> presentation - it is regarded by some as a convenient (and modern!)
> brain-eye strengthening aid.

Surely this must be part of a global conspiracy.  I'm not sure, however,
whether this one comes from Jews or Arabs...  ;-)

Personally, given that Emacs users are heavily into efficiency, and that
the community will undoubtfully be divided on this one, I would suggest
a compromise.  It is called "Boustrophedon" (see
https://en.wikipedia.org/wiki/Boustrophedon), and it reduces the effort
of moving your eyes from the right end of the previous line to the left
end of the next one.  With this clear efficiency booster, we will
finally be able to beat Vim users, eh?  Just in case someone would like
to test it immediately, here is a quick-and-dirty implementation:

--8<---------------cut here---------------start------------->8---
(defun boustrophedonize-buffer ()
  "Convert Emacs buffer into boustrophedon so that reading is
faster and more pleasant."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (forward-line)
    (while (not (eobp))
      (let* ((beg (line-beginning-position))
	     (end (line-end-position))
	     (line (buffer-substring beg end)))
	(delete-region beg end)
	(insert (reverse line))
	(forward-line 2)))))
--8<---------------cut here---------------end--------------->8---


On a more serious note, I have to say that even I am surprised that
I agree (mostly) with Emanuel Berg.  Even though my language does not
fit into ASCII - we have 9 letters outside the (modern) Latin alphabet -
I'm in favor of using ASCII alone - at least unless there is
a *convenient* method of typing all the "funny" symbols, reliable in all
relevant contexts (so also in isearch, for instance).

On the other hand, it would be great if we had an "ascii-folding"
option, making (some reasonable subset of) Unicode "equivalent" to
ASCII, so that we could easily search for e.g. the Polish word ‘żółw’
(meaning "turtle") by typing `zolw'.  (I have to say that lack of this
is one of my main gripes with A****n's K****e e-book reader - this
renders the "search" option unusable for non-English texts...)

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Date: Wed, 17 Jun 2015 20:30:14 +0200
> 
> Surely this must be part of a global conspiracy.  I'm not sure, however,
> whether this one comes from Jews or Arabs...  ;-)

More likely from bicycle-riders.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Date: Wed, 17 Jun 2015 20:30:14 +0200

>> Surely this must be part of a global conspiracy.  I'm not sure, however,
>> whether this one comes from Jews or Arabs...  ;-)

> More likely from bicycle-riders.

The sight of fat middle aged men in spandex will make programmers want
to poke their eyes out and will no longer care about what characters are
used in the emacs lisp code.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stefan Monnier]

> The sight of fat middle aged men in spandex will make programmers want

Eh!  Stop the slander: I don't wear spandex!


        Stefan

ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Stephen J. Turnbull]

Marcin Borkowski writes:

 > On the other hand, it would be great if we had an "ascii-folding"
 > option, making (some reasonable subset of) Unicode "equivalent" to
 > ASCII,

I believe Emacs already implements NFD normalization.  All you need
after that is to skip compose characters when searching.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Thu, 18 Jun 2015 13:52:49 +0900
> Cc: emacs-devel@gnu.org
> 
> Marcin Borkowski writes:
> 
>  > On the other hand, it would be great if we had an "ascii-folding"
>  > option, making (some reasonable subset of) Unicode "equivalent" to
>  > ASCII,
> 
> I believe Emacs already implements NFD normalization.

Yes, see ucs-normalize-NFD-region and friends.

> All you need after that is to skip compose characters when
> searching.

No, it's much more complex than that.  For starters, normalization
won't convert u+2018 etc. to their ASCII counterparts.  The Unicode
Standard doesn't consider those even compatibility-equivalent.  And
for matching just the base characters (which is what I presume is
meant here by "ascii-folding"), we'd need to handle correctly any
number of combinations of pre-composed and decomposed character
sequences in both the search string and the text we search, and
implement that on the fly, since the buffer text obviously cannot be
transformed for these purposes.

So yes, this feature is something that's sorely needed, but volunteers
need to know that the task is not too easy (or else it would have been
done long ago).  Interested individuals can start by studying the
following references:

  . Sections 5.18 "Case Mappings" and 5.19 "Mapping Compatibility
    Variants" of the Unicode Standard

  . UTN#5 "Canonical Equivalence in Applications"
    (http://www.unicode.org/notes/tn5/)

  . UTR#15 "Unicode Normalization Forms"
    (http://unicode.org/reports/tr15/)

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > No, it's much more complex than that.  For starters, normalization
 > won't convert u+2018 etc. to their ASCII counterparts.  The Unicode
 > Standard doesn't consider those even compatibility-equivalent.

True, but the OP asked for a "reasonable subset".  Given the context,
sure, we'd have to go beyond what NFD (or NFKD) gives, but that could
be done over time, starting with a few quotation characters (which can
probably be assembled by selecting on Unicode name).

 > And for matching just the base characters (which is what I presume
 > is meant here by "ascii-folding"), we'd need to handle correctly
 > any number of combinations of pre-composed and decomposed character
 > sequences in both the search string and the text we search, and
 > implement that on the fly, since the buffer text obviously cannot
 > be transformed for these purposes.

That's not at all obvious, for two reasons.  (1) If the applications
producing and consuming the buffer text claim Unicode conformance, we
sure can.  (2) Nobody said we have to do the transformation in place.

 > Interested individuals can start by studying the following
 > references:

I don't think that's the place to start.  The whole idea is heuristic.
Sure, at some point we'd want to improve accuracy by applying those
TRs, but anyone who wants to do this can start with just the
heuristic.

I'm not offering to do this myself, so your advice is better than
mine.  But it *could* be done this way.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 18 Jun 2015 16:48:58 +0900
> 
>  > Interested individuals can start by studying the following
>  > references:
> 
> I don't think that's the place to start.  The whole idea is heuristic.
> Sure, at some point we'd want to improve accuracy by applying those
> TRs, but anyone who wants to do this can start with just the
> heuristic.

Which heuristic?  What use cases will it support?

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

2015-06-18 6:27 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Date: Thu, 18 Jun 2015 13:52:49 +0900
>> Cc: emacs-devel@gnu.org
>>
>> Marcin Borkowski writes:
>>
>>  > On the other hand, it would be great if we had an "ascii-folding"
>>  > option, making (some reasonable subset of) Unicode "equivalent" to
>>  > ASCII,

If we're bringing this up again, I kindly suggest that people have a
look a previous thread here called "Character group folding in
searches". Here's the link:
https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00362.html

It offers 3 implementations. I think the second
(group-folding-with-case-table-lisp.patch) was quite adequade.
The only reason it wasn't applied is that Eli and Stefan wanted
something that also matched non-spacing marks, or composite
characters, whereas case tables can only map single characters to
single characters.

The first implementation (group-folding-with-regexp-lisp.patch) didn't
have this caveat because it didn't use this case-tables. The only
caveat it had is that it only works for non-regexp searches.

Attached are the two patches I mentioned here.

[-- Attachment #2: group-folding-with-case-table-lisp.patch --]
[-- Type: text/x-patch, Size: 2538 bytes --]

From 8f4be27dca714b168414171bde3eeee9fefc44e9 Mon Sep 17 00:00:00 2001
From: Artur Malabarba <address@hidden>
Date: Tue, 27 Jan 2015 14:08:01 -0200
Subject: [PATCH] (isearch-search-fun-default): Implement group folding in
 isearch.

(isearch-fold-groups group-fold-table): New variables.

When `isearch-fold-groups' is non-nil `group-fold-table' is used as the
case table.
---
 lisp/isearch.el | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/lisp/isearch.el b/lisp/isearch.el
index 99ca73f..7d568dd 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -272,6 +272,38 @@ Default value, nil, means edit the string instead."
   :version "23.1"
   :group 'isearch)
 
+(defcustom isearch-fold-groups t
+  "Whether regular isearch should do group folding.
+This means some characters will match entire groups of charactes,
+such as \" matching ”, for instance."
+  :type 'boolean
+  :group 'isearch
+  :version "25.1")
+
+(defvar group-fold-table
+  (eval-when-compile
+    (let ((table (make-char-table 'case-table))
+          (eq (make-char-table 'equiv)))
+      (require 'subr-x)
+      ;; Build the group table.
+      (dotimes (i (length eq))
+        (when-let ((d (get-char-code-property i 'decomposition))
+                   (k (car-safe d)))
+          (unless (eq i k)
+            (aset eq i (if (characterp k) k (cadr d))))))
+      ;; Put it in the right place.
+      (set-char-table-extra-slot table 1 eq)
+      table))
+  "Used for folding characters of the same group during search.")
+
+(defmacro with-group-folding (&rest body)
+  "Execute BODY with character-group folding turned on.
+This sets `group-fold-table' as the case-table during the
+execution of BODY."
+  `(let ((case-fold-search t))
+     (with-case-table group-fold-table
+       ,@body)))
+
 (defcustom isearch-lazy-highlight t
   "Controls the lazy-highlighting during incremental search.
 When non-nil, all text in the buffer matching the current search
@@ -2568,6 +2600,12 @@ Can be changed via `isearch-search-fun-function' for special needs."
 (defun isearch-search-fun-default ()
   "Return default functions to use for the search."
   (cond
+   (isearch-fold-groups
+    (lambda (&rest args)
+      (let* ((isearch-fold-groups nil)
+             (function (isearch-search-fun-default)))
+        (with-group-folding
+         (apply function args)))))
    (isearch-word
     (lambda (string &optional bound noerror count)
       ;; Use lax versions to not fail at the end of the word while
-- 
2.2.2


[-- Attachment #3: group-folding-with-regexp-lisp.patch --]
[-- Type: text/x-patch, Size: 3220 bytes --]

From f67ae7ed53e6a90cf4f97ac1bba9498b5d58e6dc Mon Sep 17 00:00:00 2001
From: Artur Malabarba <address@hidden>
Date: Tue, 27 Jan 2015 14:08:01 -0200
Subject: [PATCH] (isearch-search-fun-default): Implement group folding in
 isearch.

Use `isearch-fold-groups', `isearch-groups-alist', and
`isearch--replace-groups-in-string'.
---
 lisp/isearch.el | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/lisp/isearch.el b/lisp/isearch.el
index 99ca73f..accfb72 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -58,6 +58,7 @@
 ;;; Code:
 
 (eval-when-compile (require 'cl-lib))
+(eval-when-compile (require 'subr-x))
 \f
 ;; Some additional options and constants.
 
@@ -272,6 +273,23 @@ Default value, nil, means edit the string instead."
   :version "23.1"
   :group 'isearch)
 
+(defcustom isearch-fold-groups t
+  "Whether regular isearch should do group folding.
+This means some characters will match entire groups of charactes,
+such as \" matching ”, for instance."
+  :type 'boolean
+  :group 'isearch
+  :version "25.1")
+
+(defvar isearch-groups-alist
+  ;; FIXME: Add all the latin accented letters like Ã.
+  '((?\" . "[\"＂“””„⹂〞‟‟❞❝❠“„〝〟🙷🙶🙸❮❯«»‹›󠀢]")
+    (?' . "[`'❟❛❜‘’’‚‛‛‚‘󠀢❮❯«»‹›]")
+    ;; `isearch-fold-groups' doesn't interact with
+    ;; `isearch-lax-whitespace' yet.  So we need to add this here.
+    (?\s . "[ 	\r\n]+"))
+  "Alist of groups to use when `isearch-fold-groups' is non-nil.")
+
 (defcustom isearch-lazy-highlight t
   "Controls the lazy-highlighting during incremental search.
 When non-nil, all text in the buffer matching the current search
@@ -2565,6 +2583,18 @@ search for the first occurrence of STRING or its translation.")
 Can be changed via `isearch-search-fun-function' for special needs."
   (funcall (or isearch-search-fun-function 'isearch-search-fun-default)))
 
+(defun isearch--replace-groups-in-string (string)
+  "Return a group-folded regexp version of STRING.
+Any character that has an entry in `isearch-groups-alist' is
+replaced with the cdr of that entry (which should be a regexp).
+Other characters are `regexp-quote'd."
+  (apply #'concat
+    (mapcar (lambda (c)
+              (if-let ((entry (assq c isearch-groups-alist)))
+                  (cdr entry)
+                (regexp-quote (string c))))
+      string)))
+
 (defun isearch-search-fun-default ()
   "Return default functions to use for the search."
   (cond
@@ -2591,6 +2621,13 @@ Can be changed via `isearch-search-fun-function' for special needs."
       're-search-backward-lax-whitespace))
    (isearch-regexp
     (if isearch-forward 're-search-forward 're-search-backward))
+   ;; `isearch-regexp' is essentially a superset of
+   ;; `isearch-fold-groups'.  So fold-groups comes after it.
+   (isearch-fold-groups
+    (lambda (string &optional bound noerror count)
+      (funcall (if isearch-forward #'re-search-forward #'re-search-backward)
+        (isearch--replace-groups-in-string string)
+        bound noerror count)))
    ((and isearch-lax-whitespace search-whitespace-regexp)
     (if isearch-forward
 	'search-forward-lax-whitespace
-- 
2.2.2

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Thu, 18 Jun 2015 09:16:16 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, emacs-devel <emacs-devel@gnu.org>, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>
> 
> If we're bringing this up again, I kindly suggest that people have a
> look a previous thread here called "Character group folding in
> searches". Here's the link:
> https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00362.html
> 
> It offers 3 implementations. I think the second
> (group-folding-with-case-table-lisp.patch) was quite adequade.
> The only reason it wasn't applied is that Eli and Stefan wanted
> something that also matched non-spacing marks, or composite
> characters, whereas case tables can only map single characters to
> single characters.
> 
> The first implementation (group-folding-with-regexp-lisp.patch) didn't
> have this caveat because it didn't use this case-tables. The only
> caveat it had is that it only works for non-regexp searches.
> 
> Attached are the two patches I mentioned here.

Perhaps we should install the first one, it at least solves part of
the problem.  But I think it has a subtle bug: it assumes that
get-char-code-property returns value of the form '(SOMETHING CH1 ...)',
where CH1 is necessarily the character we want.  However, that is
false at least in some cases, for example:

  (get-char-code-property ?🄝 'decomposition) => (compat 40 78 41)

So I think perhaps we should subject the return value of
get-char-code-property to a few more tests, and either reject such
values, or maybe find the first character that matches [:alnum:], and
use that.

Btw, can the other patch be installed _in_addition_ to the first one?
IOW, they are orthogonal, or could be that, right?

Thanks.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

> or maybe find the first character that matches [:alnum:], and
> use that.

Sounds reasonable.

> Btw, can the other patch be installed _in_addition_ to the first one?
> IOW, they are orthogonal, or could be that, right?

Yes, they should be orthogonal. One patch turns regular input strings into
regexps, the other patch uses character folding tables which have always
worked fine with regexps.

[-- Attachment #2: Type: text/html, Size: 494 bytes --]

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Thu, 18 Jun 2015 10:50:01 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, stephen@xemacs.org
> 
> > Btw, can the other patch be installed _in_addition_ to the first one?
> > IOW, they are orthogonal, or could be that, right?
> 
> Yes, they should be orthogonal. One patch turns regular input strings into
> regexps, the other patch uses character folding tables which have always worked
> fine with regexps. 

So maybe we should have both, as separate features?

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

2015-06-18 11:13 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Thu, 18 Jun 2015 10:50:01 +0100
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, stephen@xemacs.org
>>
>> > Btw, can the other patch be installed _in_addition_ to the first one?
>> > IOW, they are orthogonal, or could be that, right?
>>
>> Yes, they should be orthogonal. One patch turns regular input strings into
>> regexps, the other patch uses character folding tables which have always worked
>> fine with regexps.
>
> So maybe we should have both, as separate features?

You mean use case-tables for single-character folding everywhere and
then use regexps only for multi-character folding in isearch?

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

By the way, here's a patch of how the regexp version can apply to both
isearch and query-replace while doing pretty much everything that's
been requested.
It fixes the issues raised by Eli, and (unlike the previous version)
is now fully implemented.
Now, "a" will match any character whose decomposition has "a" as the
first letter character (not necessarily the first character), so it
will match ⒜, for instance. "a" also matches the decomposition itself,
so it will match both the letter "á" and the two-character combo "á".
Note that the unicode characters still only match themselves, because
that's also how case-folding works (if you search uppercase you only
match uppercase).

I believe that is all we were asking for, correct?

Shall I merge? (It adds about 5 seconds of compile time in my laptop)

[-- Attachment #2: 0001-lisp-isearch.el-Fold-many-unicode-characters-to-ASCI.patch --]
[-- Type: text/x-patch, Size: 6098 bytes --]

From 1d2d66f88d7c86705095d93bbd3ba78955a91f36 Mon Sep 17 00:00:00 2001
From: Artur Malabarba <address@hidden>
Date: Tue, 27 Jan 2015 14:08:01 -0200
Subject: [PATCH] * lisp/isearch.el: Fold many unicode characters to ASCII

(isearch-character-fold-search, isearch--character-fold-extras)
(isearch--character-fold-table): New variable.
(isearch--character-folded-regexp): New function.
(isearch-search-fun-default): Use them.

* lisp/replace.el (replace-character-fold): New variable.
(replace-search): Use it.
---
 lisp/isearch.el | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lisp/replace.el |  9 +++++++
 2 files changed, 84 insertions(+)

diff --git a/lisp/isearch.el b/lisp/isearch.el
index d1b92bd..eb0f965 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -272,6 +272,74 @@ Default value, nil, means edit the string instead."
   :version "23.1"
   :group 'isearch)
 
+(defvar isearch-character-fold-search t
+  "Whether regular isearch should fold similar characters.
+This means some characters will match entire groups of charactes,
+such as \" matching all variants of double quotes, for instance.")
+
+(defconst isearch--character-fold-extras
+  '((?\" "＂" "“" "”" "”" "„" "⹂" "〞" "‟" "‟" "❞" "❝" "❠" "“" "„" "〝" "〟" "🙷" "🙶" "🙸" "«" "»")
+    (?' "`" "❟" "❛" "❜" "‘" "’" "‚" "‛" "‚" "󠀢" "❮" "❯" "‹" "›")
+    (?` "❛" "‘" "‛" "󠀢" "❮" "‹")
+    ;; `isearch-character-fold-search' doesn't interact with
+    ;; `isearch-lax-whitespace' yet.  So we need to add this here.
+    (?\s "	" "\r" "\n"))
+  "Extra entries to add to `isearch--character-fold-table'.
+Used to specify character folding not covered by unicode
+decomposition.  Each car is a character and each cdr is a list of
+strings that it should match (itself excluded).")
+
+(defvar isearch--character-fold-table
+  (eval-when-compile (funcall (byte-compile (lambda ()
+   (require 'subr-x)
+   (let ((equiv (make-char-table 'character-fold-table)))
+     ;; Compile a list of all complex characters that each simple
+     ;; character should match.
+     (dotimes (i (length equiv))
+       (let ((dd (get-char-code-property i 'decomposition))
+             d k found)
+         ;; Skip trivial cases (?a decomposes to (?a)).
+         (unless (and (eq i (car dd)))
+           ;; Discard a possible formatting tag.
+           (when (symbolp (car-safe dd))
+             (setq dd (cdr dd)))
+           ;; Is k a number or letter, per unicode standard?
+           (setq d dd)
+           (while (and d (not found))
+             (setq k (pop d))
+             (setq found (and (characterp k)
+                              (memq (get-char-code-property k 'general-category)
+                                    '(Lu Ll Lt Lm Lo Nd Nl No)))))
+           ;; If there's no number or letter on the
+           ;; decomposition, find the first character in it.
+           (setq d dd)
+           (while (and d (not found))
+             (setq k (pop d))
+             (setq found (characterp k)))
+           ;; Add i to the list of characters that k can
+           ;; represent. Also add its decomposition, so we can
+           ;; match multi-char representations like (format "a%c" 769)
+           (when (and found (not (eq i k)))
+             (aset equiv k (cons (apply #'string dd)
+                                 (cons (string i)
+                                       (aref equiv k))))))))
+     (dotimes (i (length equiv))
+       (when-let ((chars (append (cdr (assq i isearch--character-fold-extras))
+                                 (aref equiv i))))
+         (aset equiv i (regexp-opt (cons (string i) chars)))))
+     equiv)))))
+  "Used for folding characters of the same group during search.")
+
+(defun isearch--character-folded-regexp (string)
+  "Return a regexp matching anything that character-folds into STRING.
+That is, any character in STRING that has an entry in
+`isearch--character-fold-table' is replaced with that entry (which is a
+regexp).  Other characters are `regexp-quote'd."
+  (apply #'concat
+    (mapcar (lambda (c) (or (aref isearch--character-fold-table c)
+                       (regexp-quote (string c))))
+            string)))
+
 (defcustom isearch-lazy-highlight t
   "Controls the lazy-highlighting during incremental search.
 When non-nil, all text in the buffer matching the current search
@@ -2607,6 +2675,13 @@ Can be changed via `isearch-search-fun-function' for special needs."
       're-search-backward-lax-whitespace))
    (isearch-regexp
     (if isearch-forward 're-search-forward 're-search-backward))
+   ;; `isearch-regexp' is essentially a superset of
+   ;; `isearch-fold-groups'.  So fold-groups comes after it.
+   (isearch-character-fold-search
+    (lambda (string &optional bound noerror count)
+      (funcall (if isearch-forward #'re-search-forward #'re-search-backward)
+        (isearch--character-folded-regexp string)
+        bound noerror count)))
    ((and isearch-lax-whitespace search-whitespace-regexp)
     (if isearch-forward
 	'search-forward-lax-whitespace
diff --git a/lisp/replace.el b/lisp/replace.el
index 1bf1343..96bbd61 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -33,6 +33,14 @@
   :type 'boolean
   :group 'matching)
 
+(defcustom replace-character-fold t
+  "Non-nil means `query-replace' should do character folding in matches.
+This means, for instance, that ' will match a large variety of
+unicode quotes."
+  :type 'boolean
+  :group 'matching
+  :version "25.1")
+
 (defcustom replace-lax-whitespace nil
   "Non-nil means `query-replace' matches a sequence of whitespace chars.
 When you enter a space or spaces in the strings to be replaced,
@@ -2003,6 +2011,7 @@ It is called with three arguments, as if it were
   ;; used after `recursive-edit' might override them.
   (let* ((isearch-regexp regexp-flag)
 	 (isearch-word delimited-flag)
+         (isearch-character-fold-search replace-character-fold)
 	 (isearch-lax-whitespace
 	  replace-lax-whitespace)
 	 (isearch-regexp-lax-whitespace
-- 
2.4.4

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

> Shall I merge? (It adds about 5 seconds of compile time in my laptop)

Inlining the patch attached above (sorry, force of habit).

From: Artur Malabarba <address@hidden>
Date: Tue, 27 Jan 2015 14:08:01 -0200
Subject: [PATCH] * lisp/isearch.el: Fold many unicode characters to ASCII

(isearch-character-fold-search, isearch--character-fold-extras)
(isearch--character-fold-table): New variable.
(isearch--character-folded-regexp): New function.
(isearch-search-fun-default): Use them.

* lisp/replace.el (replace-character-fold): New variable.
(replace-search): Use it.
---
 lisp/isearch.el | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lisp/replace.el |  9 +++++++
 2 files changed, 84 insertions(+)

diff --git a/lisp/isearch.el b/lisp/isearch.el
index d1b92bd..eb0f965 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -272,6 +272,74 @@ Default value, nil, means edit the string instead."
   :version "23.1"
   :group 'isearch)

+(defvar isearch-character-fold-search t
+  "Whether regular isearch should fold similar characters.
+This means some characters will match entire groups of charactes,
+such as \" matching all variants of double quotes, for instance.")
+
+(defconst isearch--character-fold-extras
+  '((?\" "＂" "“" "”" "”" "„" "⹂" "〞" "‟" "‟" "❞" "❝" "❠" "“" "„" "〝"
"〟" "🙷" "🙶" "🙸" "«" "»")
+    (?' "`" "❟" "❛" "❜" "‘" "’" "‚" "‛" "‚" "󠀢" "❮" "❯" "‹" "›")
+    (?` "❛" "‘" "‛" "󠀢" "❮" "‹")
+    ;; `isearch-character-fold-search' doesn't interact with
+    ;; `isearch-lax-whitespace' yet.  So we need to add this here.
+    (?\s "    " "\r" "\n"))
+  "Extra entries to add to `isearch--character-fold-table'.
+Used to specify character folding not covered by unicode
+decomposition.  Each car is a character and each cdr is a list of
+strings that it should match (itself excluded).")
+
+(defvar isearch--character-fold-table
+  (eval-when-compile (funcall (byte-compile (lambda ()
+   (require 'subr-x)
+   (let ((equiv (make-char-table 'character-fold-table)))
+     ;; Compile a list of all complex characters that each simple
+     ;; character should match.
+     (dotimes (i (length equiv))
+       (let ((dd (get-char-code-property i 'decomposition))
+             d k found)
+         ;; Skip trivial cases (?a decomposes to (?a)).
+         (unless (and (eq i (car dd)))
+           ;; Discard a possible formatting tag.
+           (when (symbolp (car-safe dd))
+             (setq dd (cdr dd)))
+           ;; Is k a number or letter, per unicode standard?
+           (setq d dd)
+           (while (and d (not found))
+             (setq k (pop d))
+             (setq found (and (characterp k)
+                              (memq (get-char-code-property k
'general-category)
+                                    '(Lu Ll Lt Lm Lo Nd Nl No)))))
+           ;; If there's no number or letter on the
+           ;; decomposition, find the first character in it.
+           (setq d dd)
+           (while (and d (not found))
+             (setq k (pop d))
+             (setq found (characterp k)))
+           ;; Add i to the list of characters that k can
+           ;; represent. Also add its decomposition, so we can
+           ;; match multi-char representations like (format "a%c" 769)
+           (when (and found (not (eq i k)))
+             (aset equiv k (cons (apply #'string dd)
+                                 (cons (string i)
+                                       (aref equiv k))))))))
+     (dotimes (i (length equiv))
+       (when-let ((chars (append (cdr (assq i isearch--character-fold-extras))
+                                 (aref equiv i))))
+         (aset equiv i (regexp-opt (cons (string i) chars)))))
+     equiv)))))
+  "Used for folding characters of the same group during search.")
+
+(defun isearch--character-folded-regexp (string)
+  "Return a regexp matching anything that character-folds into STRING.
+That is, any character in STRING that has an entry in
+`isearch--character-fold-table' is replaced with that entry (which is a
+regexp).  Other characters are `regexp-quote'd."
+  (apply #'concat
+    (mapcar (lambda (c) (or (aref isearch--character-fold-table c)
+                       (regexp-quote (string c))))
+            string)))
+
 (defcustom isearch-lazy-highlight t
   "Controls the lazy-highlighting during incremental search.
 When non-nil, all text in the buffer matching the current search
@@ -2607,6 +2675,13 @@ Can be changed via
`isearch-search-fun-function' for special needs."
       're-search-backward-lax-whitespace))
    (isearch-regexp
     (if isearch-forward 're-search-forward 're-search-backward))
+   ;; `isearch-regexp' is essentially a superset of
+   ;; `isearch-fold-groups'.  So fold-groups comes after it.
+   (isearch-character-fold-search
+    (lambda (string &optional bound noerror count)
+      (funcall (if isearch-forward #'re-search-forward #'re-search-backward)
+        (isearch--character-folded-regexp string)
+        bound noerror count)))
    ((and isearch-lax-whitespace search-whitespace-regexp)
     (if isearch-forward
     'search-forward-lax-whitespace
diff --git a/lisp/replace.el b/lisp/replace.el
index 1bf1343..96bbd61 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -33,6 +33,14 @@
   :type 'boolean
   :group 'matching)

+(defcustom replace-character-fold t
+  "Non-nil means `query-replace' should do character folding in matches.
+This means, for instance, that ' will match a large variety of
+unicode quotes."
+  :type 'boolean
+  :group 'matching
+  :version "25.1")
+
 (defcustom replace-lax-whitespace nil
   "Non-nil means `query-replace' matches a sequence of whitespace chars.
 When you enter a space or spaces in the strings to be replaced,
@@ -2003,6 +2011,7 @@ It is called with three arguments, as if it were
   ;; used after `recursive-edit' might override them.
   (let* ((isearch-regexp regexp-flag)
      (isearch-word delimited-flag)
+         (isearch-character-fold-search replace-character-fold)
      (isearch-lax-whitespace
       replace-lax-whitespace)
      (isearch-regexp-lax-whitespace
-- 
2.4.4

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

> Now, "a" will match any character whose decomposition has "a" as the
> first letter character (not necessarily the first character), so it
> will match ⒜, for instance. "a" also matches the decomposition itself,
> so it will match both the letter "á" and the two-character combo "á".
> Note that the unicode characters still only match themselves, because
> that's also how case-folding works (if you search uppercase you only
> match uppercase).

Have you found a way to combine char-folding with case-folding?
(in https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00407.html
another hard problem was char-folding with a combining character.)

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

> Have you found a way to combine char-folding with case-folding?

Yes. This particular patch uses regexps, so it works independently of
case-fold-search. That is, if case-fold-search is on, then you will get
case folding in addition to this.

> (in https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00407.html
> another hard problem was char-folding with a combining character.)

That works too. "a" matches both "á" (the letter) and "á" (a letter with a
combining character).

[-- Attachment #2: Type: text/html, Size: 686 bytes --]

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

>> Have you found a way to combine char-folding with case-folding?
>
> Yes. This particular patch uses regexps, so it works independently of
> case-fold-search. That is, if case-fold-search is on, then you will get
> case folding in addition to this.
>
>> (in https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00407.html
>> another hard problem was char-folding with a combining character.)
>
> That works too. "a" matches both "á" (the letter) and "á" (a letter with a
> combining character).

It would be easy also to allow "á" match "a" (by normalizing the
original search string like "A" matches "a" in case insensitive mode),
but this could be added later.

Regarding the usage of the regexp, please consider a simpler way
to use your character-folding regexp that would be just:

(define-key isearch-mode-map "\M-sd" 'isearch-toggle-character-fold)
(defun isearch-toggle-character-fold ()
  (interactive)
  (setq isearch-word (unless (eq isearch-word 'isearch--character-folded-regexp)
		       'isearch--character-folded-regexp))
  (if isearch-word (setq isearch-regexp nil))
  (setq isearch-success t isearch-adjusted t)
  (isearch-update))
(put 'isearch--character-folded-regexp 'isearch-message-prefix "char-fold ")

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

2015-06-23 23:49 GMT+01:00 Juri Linkov >>> Have you found a way to
combine char-folding with case-folding?
>>
>> Yes. This particular patch uses regexps, so it works independently of
>> case-fold-search. That is, if case-fold-search is on, then you will get
>> case folding in addition to this.
>>
>>> (in https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00407.html
>>> another hard problem was char-folding with a combining character.)
>>
>> That works too. "a" matches both "á" (the letter) and "á" (a letter with a
>> combining character).

> It would be easy also to allow "á" match "a" (by normalizing the
> original search string

It's easy for things like “ⓐ” or “á”, which are single characters,
normalizing a combination of multiple characters (like “á”) can get a
little complicated. But then, I don't think we want to do either.

> like "A" matches "a" in case insensitive mode),

It doesn't for me. If I type C-s A, it only matches the capital “A”.
Whereas if I type C-s a it matches both “a” and “A”. The same happens
during query-replace and other commands. That's a feature, not a flaw.
If the user explicitly typed out “á”, they probably don't want to
match “a” (or they would have just typed “a” in the first place).

> Regarding the usage of the regexp, please consider a simpler way
> to use your character-folding regexp that would be just:
>
> (define-key isearch-mode-map "\M-sd" 'isearch-toggle-character-fold)
> (defun isearch-toggle-character-fold ()
>   (interactive)
>   (setq isearch-word (unless (eq isearch-word 'isearch--character-folded-regexp)
>                        'isearch--character-folded-regexp))
>   (if isearch-word (setq isearch-regexp nil))
>   (setq isearch-success t isearch-adjusted t)
>   (isearch-update))
> (put 'isearch--character-folded-regexp 'isearch-message-prefix "char-fold ")

Thanks, I was meaning to look into how to make it toggleable. Will
that snippet make it ON by default? If not, how could I do that?

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Wed, 24 Jun 2015 09:03:48 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, "Stephen J. Turnbull" <stephen@xemacs.org>, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>
> 
> > like "A" matches "a" in case insensitive mode),
> 
> It doesn't for me. If I type C-s A, it only matches the capital “A”.
> Whereas if I type C-s a it matches both “a” and “A”. The same happens
> during query-replace and other commands. That's a feature, not a flaw.

Did you try invoking search non-interactively?  I think the feature is
only in effect in interactive search.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

2015-06-24 15:47 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Wed, 24 Jun 2015 09:03:48 +0100
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>, "Stephen J. Turnbull" <stephen@xemacs.org>,
>>       Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>
>>
>> > like "A" matches "a" in case insensitive mode),
>>
>> It doesn't for me. If I type C-s A, it only matches the capital “A”.
>> Whereas if I type C-s a it matches both “a” and “A”. The same happens
>> during query-replace and other commands. That's a feature, not a flaw.
>
> Did you try invoking search non-interactively?  I think the feature is
> only in effect in interactive search.

Indeed, you're right.
Well, we can definitely add two-way folding if desired.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Stefan Monnier]

>> > like "A" matches "a" in case insensitive mode),
>> It doesn't for me. If I type C-s A, it only matches the capital “A”.
>> Whereas if I type C-s a it matches both “a” and “A”. The same happens
>> during query-replace and other commands. That's a feature, not a flaw.
> Did you try invoking search non-interactively?  I think the feature is
> only in effect in interactive search.

Indeed.  It's a feature of isearch to decide automatically whether to
use case-fold-search or not based on the presence of upper case
characters (by default: this can be overridden at run time with M-x c).


        Stefan

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

> It's easy for things like “ⓐ” or “á”, which are single characters,
> normalizing a combination of multiple characters (like “á”) can get a
> little complicated. But then, I don't think we want to do either.

I'm testing your changes right now and noticed the following problem:

0. emacs -Q
1. C-h H (‘view-hello-file’)
2. C-s i (search for ‘i’)

matches such substrings as “in”, “iv”, ‘ii’ …

Maybe we should add explicit exceptions for such cases?

>> (define-key isearch-mode-map "\M-sd" 'isearch-toggle-character-fold)
>> (defun isearch-toggle-character-fold ()
>>   (interactive)
>>   (setq isearch-word (unless (eq isearch-word 'isearch--character-folded-regexp)
>>                        'isearch--character-folded-regexp))
>>   (if isearch-word (setq isearch-regexp nil))
>>   (setq isearch-success t isearch-adjusted t)
>>   (isearch-update))
>> (put 'isearch--character-folded-regexp 'isearch-message-prefix "char-fold ")
>
> Thanks, I was meaning to look into how to make it toggleable. Will
> that snippet make it ON by default? If not, how could I do that?

It doesn't enable by default, so it could be combined with the same logic
like in ‘isearch-toggle-case-fold’.  In any case we have to find a good
mnemonic key binding to toggle the new character-folding mode.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

> I'm testing your changes right now and noticed the following problem:
>
> 0. emacs -Q
> 1. C-h H (‘view-hello-file’)
> 2. C-s i (search for ‘i’)
>
> matches such substrings as “in”, “iv”, ‘ii’ …
>
> Maybe we should add explicit exceptions for such cases?

I meant like you added ‘isearch--character-fold-extras’ also
to add something like ‘isearch--character-fold-excludes’.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

2015-06-24 22:44 GMT+01:00 Juri Linkov <juri@linkov.net>:
>> It's easy for things like “ⓐ” or “á”, which are single characters,
>> normalizing a combination of multiple characters (like “á”) can get a
>> little complicated. But then, I don't think we want to do either.
>
> I'm testing your changes right now and noticed the following problem:
>
> 0. emacs -Q
> 1. C-h H (‘view-hello-file’)
> 2. C-s i (search for ‘i’)
>
> matches such substrings as “in”, “iv”, ‘ii’ …
>
> Maybe we should add explicit exceptions for such cases?

Yes, I saw that too. I've added a clause that will generally prevent
one letter from mathing multiple letters (it can still match multiple
characters, as long as only one of them is a letter).

> It doesn't enable by default, so it could be combined with the same logic
> like in ‘isearch-toggle-case-fold’.  In any case we have to find a good
> mnemonic key binding to toggle the new character-folding mode.

I think I'll set it to M-s f. Both c and h are already taken (not sure
why, but `h' feels mnemonic here) and f is the next best thing I
think. Or we could use a quote M-s '

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

>> It doesn't enable by default, so it could be combined with the same logic
>> like in ‘isearch-toggle-case-fold’.  In any case we have to find a good
>> mnemonic key binding to toggle the new character-folding mode.
>
> I think I'll set it to M-s f. Both c and h are already taken (not sure
> why, but `h' feels mnemonic here) and f is the next best thing I
> think. Or we could use a quote M-s '

‘M-s f’ is already used by dired-isearch-filenames,
whereas ‘M-s '’ has good mnemonics indeed.

So the most important remaining problem is how to combine char-folding
with ‘isearch-lax-whitespace’ and ‘search-whitespace-regexp’.
It seems this will require duplicating some code between these two.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

> So the most important remaining problem is how to combine char-folding
> with ‘isearch-lax-whitespace’ and ‘search-whitespace-regexp’.
> It seems this will require duplicating some code between these two.

I took a shot at this on the last commit. `character-fold-to-regexp'
now takes a second argument (lax), when this is non-nil it appends a +
to the regexp generated from whitespace characters.
It doesn't take `search-whitespace-regexp' into consideration, but I
don't know if that would even make sense given that a character-folded
space is even more general (matches more characters) than what's
specified by `search-whitespace-regexp'.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

Isearch has a couple of toggles like case-fold, isearch-invisible and now
char-fold.

After updating, I noticed that the prompt for the regular `C-s` is now
"Char-fold Isearch: ".

Should that "Char-fold" prefix be there? If so, we should also display what
other isearch options are in effect (case-fold, etc).

Suggestion:
- The prefix be kept the same as before ("Isearch: ") for brevity.
- The moment a user starts Isearch (or when user calls a command during
Isearch using some binding), the effective isearch properties are flashed
in the echo area. That "flashing" would look just as the case-sensitivity
state is flashed on hitting `M-c` during isearch. An example of such
flashed message could be

    [Case-fold: on Char-fold: on  Search-invisible: off]

- If we don't want to flash that message, the message can be displayed and
be allowed to stay there till user starts/resumes typing in the minibuffer.

Thoughts?

On Wed, Jun 24, 2015 at 9:56 PM Artur Malabarba <bruce.connor.am@gmail.com>
wrote:

> > So the most important remaining problem is how to combine char-folding
> > with ‘isearch-lax-whitespace’ and ‘search-whitespace-regexp’.
> > It seems this will require duplicating some code between these two.
>
> I took a shot at this on the last commit. `character-fold-to-regexp'
> now takes a second argument (lax), when this is non-nil it appends a +
> to the regexp generated from whitespace characters.
> It doesn't take `search-whitespace-regexp' into consideration, but I
> don't know if that would even make sense given that a character-folded
> space is even more general (matches more characters) than what's
> specified by `search-whitespace-regexp'.
>
>

[-- Attachment #2: Type: text/html, Size: 2751 bytes --]

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

> Isearch has a couple of toggles like case-fold, isearch-invisible and now
> char-fold.
>
> After updating, I noticed that the prompt for the regular `C-s` is now
> "Char-fold Isearch: ".
>
> Should that "Char-fold" prefix be there? If so, we should also display what
> other isearch options are in effect (case-fold, etc).
>
> Suggestion:
> - The prefix be kept the same as before ("Isearch: ") for brevity.
> - The moment a user starts Isearch (or when user calls a command during
> Isearch using some binding), the effective isearch properties are flashed
> in the echo area. That "flashing" would look just as the case-sensitivity
> state is flashed on hitting `M-c` during isearch. An example of such
> flashed message could be
>
>     [Case-fold: on Char-fold: on  Search-invisible: off]
>
> - If we don't want to flash that message, the message can be displayed and
> be allowed to stay there till user starts/resumes typing in the minibuffer.
>
> Thoughts?

This will be fixed by bug#12988.  I'll send the latest patch there.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

> > - If we don't want to flash that message, the message can be displayed
> and
> > be allowed to stay there till user starts/resumes typing in the
> minibuffer.
> >
> > Thoughts?
>
> This will be fixed by bug#12988.  I'll send the latest patch there.
>

I don't know how you're planning on solving doing this,  but I was going to
suggest that isearch could just take over the mode-line while in effect,
that would it give it a lot of space to verbosely display the searching
options (and maybe even list the key binds for each) and each option could
be a button that toggles itself.

[-- Attachment #2: Type: text/html, Size: 839 bytes --]

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 2052 bytes --]

@Juri
I read that that bug report is more than 2 years old. Did you recently
submit a patch? Or were you referring to a patch from Drew in that bug
report?

@ Artur
Using the modeline is a very neat idea!

There is one package (anzu) I know of that makes partial use of the mode
line to display the search pattern counts.

We will need a solution that doesn't break these other packages. Or should
the info be simply added to `minor-mode-alist` with "button click
properties"? (I forgot the exact property name).

abo-abo creates a pseudo minibuffer on top of the actual minibuffer to
display the hints in his hydra.el package. That could also work here?

My earlier suggestion was simply based in the current code. I saw that a
`sleep-for` was used to flash the isearch setting change momentarily. I
wondered if we just keep that isearch setting change display static and
then remove that at the next search character input by the user.

Let's say that the user is search for "abc" and he has already typed "ab".

> Isearch: ab|

Now the user decides to toggle one of the isearch toggles, let's say
case-fold toggle and hits `M-c`. Then the minibuffer can display this

> Isearch: ab| [case-sensitive]

That should stay like that till the user continues typing

> Isearch: abc|

But, showing the isearch settings persistently in a separate area is a
better idea.

On Jun 26, 2015 5:32 AM, "Artur Malabarba" <bruce.connor.am@gmail.com>
wrote:

>
> > - If we don't want to flash that message, the message can be displayed
>> and
>> > be allowed to stay there till user starts/resumes typing in the
>> minibuffer.
>> >
>> > Thoughts?
>>
>> This will be fixed by bug#12988.  I'll send the latest patch there.
>>
>
> I don't know how you're planning on solving doing this,  but I was going
> to suggest that isearch could just take over the mode-line while in effect,
> that would it give it a lot of space to verbosely display the searching
> options (and maybe even list the key binds for each) and each option could
> be a button that toggles itself.
>

[-- Attachment #2: Type: text/html, Size: 2818 bytes --]

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> Using the modeline is a very neat idea!
>
> There is one package (anzu) I know of that makes partial use of the
> mode line to display the search pattern counts.

And for years isearch+.el has used the mode-line lighter to indicate
case-sensitivity.  And it uses faces on the prompt (and on the lighter,
for wraparound) to indicate word, regexp-vs-literal, and multi-buffer
search, as well as search wraparound.  (See my previous message.)

> We will need a solution that doesn't break these other packages.
> Or should the info be simply added to `minor-mode-alist` with
> "button click properties"? (I forgot the exact property name).
>
> abo-abo creates a pseudo minibuffer on top of the actual minibuffer
> to display the hints in his hydra.el package. That could also work
> here?
>
> My earlier suggestion was simply based in the current code. I saw
> that a `sleep-for` was used to flash the isearch setting change
> momentarily. I wondered if we just keep that isearch setting change
> display static and then remove that at the next search character
> input by the user.
>
> Let's say that the user is search for "abc" and he has already
> typed "ab".                      Isearch: ab|
> Now the user decides to toggle one of the isearch toggles, let's
> say case-fold toggle and hits `M-c`. Then the minibuffer can
> display this                     Isearch: ab| [case-sensitive]
> That should stay like that till the user continues typing 
>                                  Isearch: abc| 
> But, showing the isearch settings persistently in a separate area
> is a better idea.

My suggestion:

Isearch should *not* take over the mode-line.  Certainly not by
default.  At the same time that Isearch is used, users typically
need to see lots of other stuff in the mode-line as well.

Any other persistent (i.e., always visible while Isearching)
display of search info that is more verbose than a lighter
should be optional, turned off by default, and perhaps
toggleable by users during Isearch.

In Isearch+ I use different faces in the *prompt* to indicate
search wraparound, word, regexp-vs-literal, and multi-buffer
search.  This reminder is sufficient, I think, and it does not
sacrifice any space.

Wrt the current issue being discussed: Character folding could,
like case folding, be indicated in the lighter.  Something like
this, for example:

 Isearch - case sensitive
 ISEARCH - case folding
 Isearcĥ - char folding
 ISEARCĤ - char folding & case folding

However, as has been discussed in several threads in the past,
here and in the bug list, there are several char-folding
possibilities that we might want to offer.

There are any number of reasonable, default character equivalence
classes that you can think of.  And users themselves can come up
with any number of other such equivalence classes, some of which
are specific to a particular context.

Then bring multiple scripts and languages into the picture...
The possibilities are virtually limitless.

One person's common or useful char-folding use case is another's
YAGNI, and vice versa.

Succinct visual representation of the possible foldings is
clearly untenable in the general case.  That way lies folly.

It might be possible to instead offer a general visual indication
facility, which users themselves can develop or customize.

But I do not think it makes a lot of sense for Emacs to add any
such indication by default (and certainly not by hard-coding).

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

> However, as has been discussed in several threads in the past,
> here and in the bug list, there are several char-folding
> possibilities that we might want to offer.
>
> There are any number of reasonable, default character equivalence
> classes that you can think of.  And users themselves can come up
> with any number of other such equivalence classes, some of which
> are specific to a particular context.

True, and which equivalence classes to offer is a whole new topic all
by itself. But the implementation is there now, all it takes it to
offer more tables for it.

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> > However, as has been discussed in several threads in the past,
> > here and in the bug list, there are several char-folding
> > possibilities that we might want to offer.
> >
> > There are any number of reasonable, default character equivalence
> > classes that you can think of.  And users themselves can come up
> > with any number of other such equivalence classes, some of which
> > are specific to a particular context.
> 
> True, and which equivalence classes to offer is a whole new topic
> all by itself.

It's not a separate topic if we are now discussing whether to
indicate char folding in the mode-line etc.

> But the implementation is there now, all it takes it to offer
> more tables for it.

Irrelevant to the discussion of indicating char-folding in the
mode-line.

The point is that if multiple (and many!) kinds of char folding
are available, indicating any one of them - let alone any
combination of them - in the mode-line (as a whole, let alone in
the lighter!) is problematic.  And it would not be very helpful
to have only an indicator that said whether or not there was
currently *some* char folding.

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> > However, as has been discussed in several threads in the past,
> > here and in the bug list, there are several char-folding
> > possibilities that we might want to offer.
> >
> > There are any number of reasonable, default character equivalence
> > classes that you can think of.  And users themselves can come up
> > with any number of other such equivalence classes, some of which
> > are specific to a particular context.
> 
> True, and which equivalence classes to offer is a whole new topic
> all by itself. But the implementation is there now, all it takes
> it to offer more tables for it.

OK, you keep advertising that.  But it is irrelevant to the
discussion of whether to indicate char folding in the mode-line.
How we do char folding doesn't matter in this context.

What is relevant is that multiple kinds of char folding mean that
it makes little sense to try to indicate them in the mode-line.

IOW, we should not even think about going down that road, unless
someone has a brilliant idea of how to handle a plethora of char
foldings.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

> It's not a separate topic if we are now discussing whether to
> indicate char folding in the mode-line etc.

Whether char-folding is on or off (and how to indicate that) is
orthogonal to which char-folding table the user uses.
That is, unless you or someone else plans on implementing the
possibility of combining tables and toggling them individually (which
we don't support yet).

>> True, and which equivalence classes to offer is a whole new topic
>> all by itself. But the implementation is there now, all it takes
>> it to offer more tables for it.
>
> OK, you keep advertising that.

Ahn... do you realise that you replied twice to the same email?

> But it is irrelevant to the
> discussion of whether to indicate char folding in the mode-line.
> How we do char folding doesn't matter in this context.
> What is relevant is that multiple kinds of char folding mean that
> it makes little sense to try to indicate them in the mode-line.
>
> IOW, we should not even think about going down that road,

So you don't want to implement a simple visual indicator, because this
indicator might become harder to implement in the future when we
*might* have multiple tables that can be independently applied? (which
no one has offered to do yet, btw)

> unless
> someone has a brilliant idea of how to handle a plethora of char
> foldings.

Well, if we ever even reach that situation:
  - If no char-folding tables are in use, the indicator will be off.
  - If more than zero char-folding tables are in use, the indicator will be on.
I wouldn't call it brilliant (and please let's not go off on a tangent
discussing the (de)merits of this idea), I'm just saying we don't need
to block this feature now because of the future possibility of
multiple independent char-folding tables.

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> > It's not a separate topic if we are now discussing whether to
> > indicate char folding in the mode-line etc.
> 
> Whether char-folding is on or off (and how to indicate that) is
> orthogonal to which char-folding table the user uses.

Not if you assume, as I do, that it can be on for one kind of
char folding and off for another.  And there can be many kinds.

You see char folding as the feature to support with UI indication.
I see char foldings (one of which is case folding, BTW).  And I'm
not sure how they would best be indicated in the UI.

> That is, unless you or someone else plans on implementing the
> possibility of combining tables and toggling them individually
> (which we don't support yet).

Let's not assume anything about the possible implementations of
char-folding possibilities at this point.  (And no, I won't be
implementing them.)

 "(which we don't support yet)" -

We don't support *any* character folding yet, in any Emacs
release (except for case folding).

The point is to not assume there will only be one kind of char
folding, so that all we need is a binary mode-line indication
that it is turned on.

More than one kind of char folding is, yes, what we should aim
for, IMO, and what I thought we were aiming for.  Multiple
predefined equivalence sets, perhaps, and certainly the ability
for users to define their own equivalence classes.

So yes, I am assuming that is the goal.  And in that case it
makes little sense to propose mode-line indication now for char
folding, I think.  It would be either unworkable (too many
possibilities) or pretty useless (an all-or-none indicator
that says only that *some* folding is currently in effect).

Better would be to think, now, about how we might let users
know which foldings are in effect - whether via mode-line, an
additional line, the prompt, mode-line single-char symbols
that provide more info when hovered over or clicked, a key to
show a help buffer, or anything else.

> Ahn... do you realise that you replied twice to the same email?

No; sorry.  I received two copies from you (with different
recipient lists) and I didn't realize it.  I was doing something
else at the same time, and I apparently edited two different
replies and then sent them both.  Mille excuses.

> > But it is irrelevant to the
> > discussion of whether to indicate char folding in the mode-line.
> > How we do char folding doesn't matter in this context.
> > What is relevant is that multiple kinds of char folding mean that
> > it makes little sense to try to indicate them in the mode-line.
> >
> > IOW, we should not even think about going down that road,
> 
> So you don't want to implement a simple visual indicator, because
> this indicator might become harder to implement in the future when we
> *might* have multiple tables that can be independently applied?
> (which no one has offered to do yet, btw)

You talk in terms of an implementation (tables).  I'm talking
about a user feature: multiple char foldings, however it might
be implemented.

And yes, I think that we should, now, assume that we will, one way
or another, offer multiple char-folding possibilities.

How to indicate the foldings that are currently in effect should be
how we think about this, now.  We should not opt for a UI that we
are pretty sure now will not work for the feature we are aiming for. 

> > unless someone has a brilliant idea of how to handle a plethora
> > of char foldings.
> 
> Well, if we ever even reach that situation:
>   - If no char-folding tables are in use, the indicator will be off.
>   - If more than zero char-folding tables are in use, the indicator
>     will be on.

That's not very helpful, IMO.  And we *should* "ever even reach
that situation".  We've been talking about it for a while now.
Various implementation possibilities have been suggested, and
you have now added one as a start.

You know, *case* folding is itself one kind of char folding.
Why should we indicate it separately, if, as you seem to think,
one size fits all?

The answer is that it is important to be able to tell, for any
given folding, whether it is on or off.  We can do that, I'm sure;
we just need to think about it a bit more instead of jumping off
the dock into the first dingy we see go by.

> I wouldn't call it brilliant (and please let's not go off on a
> tangent discussing the (de)merits of this idea),

It's not a tangent.  All-or-nothing should not be how we indicate
char folding to users, IMO.

> I'm just saying we don't need to block this feature now

What feature?  Mode-line indication of char folding?  Why not
block that for now?  Why not discuss how to properly indicate
foldings instead?

It's not like it's urgent to indicate char folding in the mode
line.  We don't even indicate case folding in the mode-line!

I proposed such case indication years ago (and provided the
code).  Indicating whether search is currently case-sensitive
is as important as indicating other char foldings, no?

If we have not found it urgent to do that (and case folding
has been a feature for decades) then what's the rush to do it
for a not-yet-released, partial implementation of char folding?

Why throw something out there now that is inappropriate for
the char folding that we've been discussing and aiming toward?

> because of the future possibility of multiple independent
> char-folding tables.

Multiple char foldings, not necessarily tables (plumbing).

And yes, that is the feature to work on and discuss at this
point, IMO.  Not how to indicate the state to users.

You made, I assume, a good first step in that direction.
That's great.  I'm very thankful that we will soon have some
additional char folding beyond case folding - believe me.

Nothing prevents us from now coming up with a good way to
indicate which char foldings are currently in effect.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

2015-06-26 21:34 GMT+01:00 Drew Adams <drew.adams@oracle.com>:
> [...]

Just to be clear, when I talk about the possibility of this being
futurely implemented, I'm referring to the "independent" part, not the
"multiple" part.
I agree multiple char folding is where this should go, I just don't
agree it is important for these multiple options to be  visually
distinguishable (or independently toggleable) during isearch.

May be you can convince me otherwise. But I think that most users will
never customize the equivalence class used. From those that do
customize it, most will never toggle it during interactive searches.
From those that do toggle it, most of them will prefer an
"all-or-nothing" toggle/indicator.

And then there are the few of the few of the few who wish they could
see which equiv-classes are on or off, or wish they could toggle them
independently during isearch. Which is great! But we shouldn't come up
with a complex interface to satisfy this need if it's just gonna
confuse a lot of other people.

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> Just to be clear, when I talk about the possibility of this being
> futurely implemented, I'm referring to the "independent" part, not
> the "multiple" part.
>
> I agree multiple char folding is where this should go,

Great.  Then lets assume that and discuss the UI framework
that should handle it.

> I just don't agree it is important for these multiple options
> to be  visually distinguishable (or independently toggleable)
> during isearch.

Then you should feel the same about indicating case folding.
I don't.  And you should feel the same about distinguishing
case folding from what your implementation of char folding
provides.

In that case, you should want only one indicator, which shows
that search is currently folding *something*, and let users
guess what that something might be.

I think that, *especially* when there are multiple possible
char foldings, seeing which ones are currently in effect is
important/useful.

> May be you can convince me otherwise. But I think that most
> users will never customize the equivalence class used.

Most users never toggle case sensitivity, perhaps.  So what?
I still think it is a good idea to visually indicate (e.g., in
the lighter) whether search is currently case-sensitive.

Anyway, it's not only or especially about user creation of
custom equivalence classes.  (But yes, I think that that too
is important.)

I'm hoping for some predefined sets of common equivalences.

Things like diacritical marks, numerals of different kinds,
smallcaps/caps, smallcaps/lowercase, circled/uncircled,
parenthesized/unparenthesized, arrows & brackets & pointers
& such, quotes & guillmets, fullwidth-latin/latin, non-ascii,
math-symbol groups...

And classes that might make sense for different languages,
just as case insensitivity is useful for latin languages.

Who knows?  I can guess that there will be at least *two*
besides the current one (case-insensitivity).  And given that
possibility, users will want to distinguish them easily.

I'm pretty sure of that.  Just as I'm guessing that you probably
do want to distinguish case folding from your new char folding,
with a visual indication for each.

> From those that do customize it, most will never toggle it
> during interactive searches.

Why do you think that?  Will you never toggle abstracting from
accents/diacriticals?  Do you never toggle case sensitivity?
Why would a user-defined equivalence class be any different in
this regard?

> From those that do toggle it, most of them will prefer an
> "all-or-nothing" toggle/indicator.

Why?  By that logic, you would prefer only one now, for both
case sensitivity and your new (other) char folding.  Don't you
want to see *both* whether search is case sensitive and whether
it is abstracting from diacriticals or whatever?  Why not?

> And then there are the few of the few of the few

There are a lot of users out there, and more and more of them
use Unicode chars that you and I might never use.  Starting
with languages that you and I might never search.

> who wish they could see which equiv-classes are on or off,
> or wish they could toggle them independently during isearch.
> Which is great! But we shouldn't come up with a complex
> interface to satisfy this need if it's just gonna
> confuse a lot of other people.

We should come up with a flexible UI that is as simple as
possible while supporting the assumption of more than two
char foldings (with your addition we will already have two).

We don't need to predefine toggle commands for things that
are not there now.  No one is suggesting that we do that.

But we should make it easy for users to adapt what we do
provide for use with other character-equivalence classes -
both to add toggle commands and to add/modify a visual
indicator for their needs.

We should set the pattern now, based on assuming not just
two equivalence classes (including case sensitivity), but
3 or more.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

>> From those that do toggle it, most of them will prefer an
>> "all-or-nothing" toggle/indicator.
>
> Why? By that logic, you would prefer only one now, for both
> case sensitivity and your new (other) char folding.
>
>> I just don't agree it is important for these multiple options
>> to be visually distinguishable (or independently toggleable)
>> during isearch.
>
> Then you should feel the same about indicating case folding.

I don't. We can afford to distinguish 2 or 3 different types of
char-folding, We just can't afford to (and have little reason to) make an
arbitrary number of arbitrary equiv classes distinguishable during search.

case folding is significant enough to warrant being one of these 2 or 3
(not to mention it's trivial to do internally). If there's another
equivalence class that we think outshines the rest we can distinguish it
too.

> I think that, *especially* when there are multiple possible
> char foldings, seeing which ones are currently in effect is
> important/useful.

I disagree, because I think the vast majority of people will always use the
same set foldings. For this majority, it's better to only show whether it's
ON or OFF, because that's simpler to read and to discern the meaning.

>> And then there are the few of the few of the few
>
> There are a lot of users out there, and more and more of them
> use Unicode chars that you and I might never use.

Indeed. "The few" I'm referring to is not "users who use unicode", it's
"users who customize charfolding, and toggle it mid-isearch, and want to
toggle specific equiv classes mid-isearch".

>> Which is great! But we shouldn't come up with a complex
>> interface to satisfy this need if it's just gonna
>> confuse a lot of other people.
>
> We should come up with a flexible UI that is as simple as
> possible while supporting the assumption of more than two
> char foldings (with your addition we will already have two).

No, we *can* try come up with a utopic UI, that doesn't mean we should. And
it does't mean we should deny willing patches while we throw essays around
to come up with that UI. Specially when it's a simple thing that's trivial
to revert with git.

[-- Attachment #2: Type: text/html, Size: 2639 bytes --]

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Fri, 26 Jun 2015 15:54:58 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> > May be you can convince me otherwise. But I think that most
> > users will never customize the equivalence class used.
> 
> Most users never toggle case sensitivity, perhaps.  So what?

Please drop this futile argument.  Your dispute has no basis and no
evidence based on rel-life usage experience.  It has no hope of being
resolved in constructive way.

We have just installed that feature, and it's still being worked on to
eliminate some rough edges and unintended consequences.  Let it be
refined, then let it be used by users (including you, Drew) for some
time, and then let's collect user experience and complaints and see if
something needs to be fixed.  _Then_ we will have experience and real
data points to discuss constructively and efficiently.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

2015-06-26 13:13 GMT+01:00 Kaushal
> There is one package (anzu) I know of that makes partial use of the mode
> line to display the search pattern counts.
>
> We will need a solution that doesn't break these other packages. Or should
> the info be simply added to `minor-mode-alist` with "button click
> properties"? (I forgot the exact property name).

Yeah, it can go in the mode-line lighter. I just meant that, given how
temporary isearch is, the "status indicators" can afford to be a lot
more clear and verbose than a single character. These characters make
sense if you know what they mean, but mean nothing to a newcomer.

> abo-abo creates a pseudo minibuffer on top of the actual minibuffer to
> display the hints in his hydra.el package. That could also work here?

Hm. I think so. I don't know whether multiline echo-area is supported
everywhere, but if it is I would really like to see that. The only
thing to keep in mind is that it has to be fixed. That is, I don't
want a second line poping up and down while I search or toggle stuff,
it has to be permanent so as to not be distracting.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Juri Linkov]

> @Juri
> I read that that bug report is more than 2 years old. Did you recently
> submit a patch? Or were you referring to a patch from Drew in that bug
> report?

I meant a patch from bug#12078 and implementing the suggestion from bug#12988
that will allow Isearch to always start with the default short prompt ‘I-search’
and any changes from the default state will add more indicators to the prompt.
For example, if the default search is case-insensitive, then ‘C-s’ will display
“I-search”, and ‘M-s c’ will turn the prompt into “Case-sensitive I-search”.

Your suggestion of displaying flashing initially also makes sense.
This flashing could be similar to how e.g. “[case sensitive]” is flashed
momentarily on toggling with ‘M-s c’, but also on starting Isearch.

Whether to display verbose indicators (like currently are used, e.g.
“Regexp I-search”) or 1-letter cryptic ones (e.g. “I-search(r/c/')”
like in C-mode with its indicators such as “C/lahw”) is a matter of taste
and could be customizable.

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> I don't know how you're planning on solving doing this,  but I was
> going to suggest that isearch could just take over the mode-line
> while in effect, that would it give it a lot of space to verbosely
> display the searching options (and maybe even list the key binds
> for each) and each option could be a button that toggles itself.

FWIW - I have long done this in Isearch+.

But without "tak[ing] over the mode line" and without using "a lot
of space to verbosely display the searching options" or "list[ing]
the key binds for each" or adding buttons.

What isearch+.el does is update only the `Isearch' lighter to
show case sensitivity and search wraparound, as well as word,
regexp-vs-literal, and multi-buffer search.

Case sensitivity is shown just by the text of the lighter:

 Isearch - case-sensitive
 ISEARCH - case-insensitive (case folding)

Wraparound is also shown in the lighter, by using a different face.

The other states are each indicated by a different face on the
relevant part of the prompt, as they are orthogonal to case
sensitivity.

For example, a multi-buffer regexp search uses prompt

 Regexp multi I-search:

with `Regexp' in face `isearchp-regexp' and `multi' in face
`isearchp-multi'.  When wrapped it uses prompt

 Wrapped regexp multi I-search:

with faces as above plus `Wrapped' in face `isearchp-wrapped'.

(http://www.emacswiki.org/emacs/IsearchPlus)

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Mon, 22 Jun 2015 22:02:11 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Stephen Turnbull <stephen@xemacs.org>
> 
> By the way, here's a patch of how the regexp version can apply to both
> isearch and query-replace while doing pretty much everything that's
> been requested.
> It fixes the issues raised by Eli, and (unlike the previous version)
> is now fully implemented.
> Now, "a" will match any character whose decomposition has "a" as the
> first letter character (not necessarily the first character), so it
> will match ⒜, for instance. "a" also matches the decomposition itself,
> so it will match both the letter "á" and the two-character combo "á".
> Note that the unicode characters still only match themselves, because
> that's also how case-folding works (if you search uppercase you only
> match uppercase).
> 
> I believe that is all we were asking for, correct?

Yes.  But I'd still want to know what would it take to make this
applicable to other types of search.

> Shall I merge?

Fine with me, but please wait until we clarify the issue with
non-incremental search.

Thanks.

> (It adds about 5 seconds of compile time in my laptop)

Doesn't sound too bad to me.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

> Yes.  But I'd still want to know what would it take to make this
> applicable to other types of search.

Essentially, this patch does two things.
1. It provides a function that takes a string S and returns a regexp
that matches anything that folds into S. This is why it only works on
non-regexp searching, because it takes a regular string and turns it
into a regexp. But other than that, it's generic (it's not specific to
isearch).
2. It also uses this function to implement character-fold matching in
`isearch' and in `query-replace'. In order to use this
character-folding with other searches, these need to call the
conversion function too.

For instance, to get this character folding behavior with
`search-forward', instead of writing
    (search-forward STRING)
I would instead write (conditional on a variable being non-nil)
    (search-forward-regexp (isearch--character-folded-regexp STRING))

But I can't do that directly inside search-forward, because that's a c
function. So it needs to be done in the functions that call
search-forward. I was aware of isearch and query-replace, but there's
no reason it won't work on others.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Tue, 23 Jun 2015 18:20:41 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Stephen Turnbull <stephen@xemacs.org>
> 
> For instance, to get this character folding behavior with
> `search-forward', instead of writing
>     (search-forward STRING)
> I would instead write (conditional on a variable being non-nil)
>     (search-forward-regexp (isearch--character-folded-regexp STRING))
> 
> But I can't do that directly inside search-forward, because that's a c
> function. So it needs to be done in the functions that call
> search-forward. I was aware of isearch and query-replace, but there's
> no reason it won't work on others.

I think it would be good to have that in all the search commands that
search for fixed strings.  But that could be added as separate
commits.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Mon, 22 Jun 2015 21:49:35 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Stephen Turnbull <stephen@xemacs.org>
> 
> 2015-06-18 11:13 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
> >> Date: Thu, 18 Jun 2015 10:50:01 +0100
> >> From: Artur Malabarba <bruce.connor.am@gmail.com>
> >> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, stephen@xemacs.org
> >>
> >> > Btw, can the other patch be installed _in_addition_ to the first one?
> >> > IOW, they are orthogonal, or could be that, right?
> >>
> >> Yes, they should be orthogonal. One patch turns regular input strings into
> >> regexps, the other patch uses character folding tables which have always worked
> >> fine with regexps.
> >
> > So maybe we should have both, as separate features?
> 
> You mean use case-tables for single-character folding everywhere and
> then use regexps only for multi-character folding in isearch?

Maybe I'm confused (or forgot the details): aren't we talking about
search (not necessarily Isearch)?  So what do you mean by
"everywhere"?

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

> Maybe I'm confused (or forgot the details): aren't we talking about
> search (not necessarily Isearch)?

Yes, non-regexp search.

> So what do you mean by
> "everywhere"?

Regexps searching too.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Tue, 23 Jun 2015 18:19:46 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Stephen Turnbull <stephen@xemacs.org>
> 
> > Maybe I'm confused (or forgot the details): aren't we talking about
> > search (not necessarily Isearch)?
> 
> Yes, non-regexp search.
> 
> > So what do you mean by
> > "everywhere"?
> 
> Regexps searching too.

Then I guess yes, it would be nice to have both.

Thanks.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Ulrich Mueller]

>>>>> On Wed, 17 Jun 2015, Marcin Borkowski wrote:

> On the other hand, it would be great if we had an "ascii-folding"
> option, making (some reasonable subset of) Unicode "equivalent" to
> ASCII, so that we could easily search for e.g. the Polish word ‘żółw’
> (meaning "turtle") by typing `zolw'.  (I have to say that lack of this
> is one of my main gripes with A****n's K****e e-book reader - this
> renders the "search" option unusable for non-English texts...)

I have the following code in my .emacs which does exactly that:

;; Ignore accent and umlaut marks when searching.
;; Works for Emacs 19.30 and later.
(let ((eqv-list '("aAàÀáÁâÂãÃäÄåÅ"
		  "cCçÇ"
		  "eEèÈéÉêÊëË"
		  "iIìÌíÍîÎïÏ"
		  "nNñÑ"
		  "oOòÒóÓôÔõÕöÖøØ"
		  "uUùÙúÚûÛüÜ"
		  "yYýÝÿ"))
      (table (standard-case-table))
      canon)
  (setq canon (copy-sequence table))
  (mapcar (lambda (s)
	    (mapcar (lambda (c) (aset canon c (aref s 0))) s))
	  eqv-list)
  (set-char-table-extra-slot table 1 canon)
  (set-char-table-extra-slot table 2 nil)
  (set-standard-case-table table))

Maybe it could be used as the basis for a minor mode? Downside is that
the above will significantly slow down search in multibyte buffers.
This is because equivalent characters have a different number of bytes
in UTF-8, therefore the Boyer-Moore algorithm cannot be used any more.

Ulrich

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Thu, 18 Jun 2015 07:27:37 +0200
> From: Ulrich Mueller <ulm@gentoo.org>
> Cc: eggert@cs.ucla.edu, rms@gnu.org, Nicolas Petton <nicolas@petton.fr>,
> 	emacs-devel@gnu.org, Tassilo Horn <tsdh@gnu.org>, acm@muc.de,
> 	stephen@xemacs.org
> 
> ;; Ignore accent and umlaut marks when searching.
> ;; Works for Emacs 19.30 and later.
> (let ((eqv-list '("aAàÀáÁâÂãÃäÄåÅ"
> 		  "cCçÇ"
> 		  "eEèÈéÉêÊëË"
> 		  "iIìÌíÍîÎïÏ"
> 		  "nNñÑ"
> 		  "oOòÒóÓôÔõÕöÖøØ"
> 		  "uUùÙúÚûÛüÜ"
> 		  "yYýÝÿ"))
>       (table (standard-case-table))
>       canon)
>   (setq canon (copy-sequence table))
>   (mapcar (lambda (s)
> 	    (mapcar (lambda (c) (aset canon c (aref s 0))) s))
> 	  eqv-list)
>   (set-char-table-extra-slot table 1 canon)
>   (set-char-table-extra-slot table 2 nil)
>   (set-standard-case-table table))

This means you cannot search for, say, å, even if you want to find
only it and not the other "equivalents", right?  That's not how Emacs
works now wrt letter-case.  At the very least, this folding of
diacriticals should offer the same flexibility, i.e. if the user types
'a', she should be able to find all the variants, but if she types 'å',
should find only that character.

Also, this doesn't handle decomposed characters, as in 'å'.  So this
is not really Unicode-compliant, it's a half-measure of sorts.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Ulrich Mueller]

>>>>> On Thu, 18 Jun 2015, Eli Zaretskii wrote:

>> ;; Ignore accent and umlaut marks when searching.
>> ;; Works for Emacs 19.30 and later.
>> (let ((eqv-list '("aAàÀáÁâÂãÃäÄåÅ"
>> 		  "cCçÇ"
>> 		  "eEèÈéÉêÊëË"
>> 		  "iIìÌíÍîÎïÏ"
>> 		  "nNñÑ"
>> 		  "oOòÒóÓôÔõÕöÖøØ"
>> 		  "uUùÙúÚûÛüÜ"
>> 		  "yYýÝÿ"))
>>       (table (standard-case-table))
>>       canon)
>>   (setq canon (copy-sequence table))
>>   (mapcar (lambda (s)
>> 	    (mapcar (lambda (c) (aset canon c (aref s 0))) s))
>> 	  eqv-list)
>>   (set-char-table-extra-slot table 1 canon)
>>   (set-char-table-extra-slot table 2 nil)
>>   (set-standard-case-table table))

> This means you cannot search for, say, å, even if you want to find
> only it and not the other "equivalents", right?

Yes, the idea was to consider them as equivalent, so searching for any
element of the set would match any other.

> That's not how Emacs works now wrt letter-case. At the very least,
> this folding of diacriticals should offer the same flexibility, i.e.
> if the user types 'a', she should be able to find all the variants,
> but if she types 'å', should find only that character.

Good idea.

> Also, this doesn't handle decomposed characters, as in 'å'.  So this
> is not really Unicode-compliant, it's a half-measure of sorts.

The above code snippet predates Unicode Emacs, so you cannot expect it
to handle NFC and NFD and other intricacies of Unicode normalisation.
(Also I've never seen anything else than the NFC forms, e.g., for
German umlauts, in the texts that I usually edit.)

BTW, also isearch-forward doesn't match å when searching for å, and
vice versa. So by your above argument, search in Emacs isn't Unicode
compliant anyway. (But not sure if it should be, because I think that
this would break Boyer-Moore.)

Ulrich

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Thu, 18 Jun 2015 09:08:06 +0200
> Cc: mbork@mbork.pl, eggert@cs.ucla.edu, rms@gnu.org, nicolas@petton.fr,
>         emacs-devel@gnu.org, tsdh@gnu.org, acm@muc.de, stephen@xemacs.org
> From: Ulrich Mueller <ulm@gentoo.org>
> 
> >>>>> On Thu, 18 Jun 2015, Eli Zaretskii wrote:
> 
> >> ;; Ignore accent and umlaut marks when searching.
> >> ;; Works for Emacs 19.30 and later.
> >> (let ((eqv-list '("aAàÀáÁâÂãÃäÄåÅ"
> >> 		  "cCçÇ"
> >> 		  "eEèÈéÉêÊëË"
> >> 		  "iIìÌíÍîÎïÏ"
> >> 		  "nNñÑ"
> >> 		  "oOòÒóÓôÔõÕöÖøØ"
> >> 		  "uUùÙúÚûÛüÜ"
> >> 		  "yYýÝÿ"))
> >>       (table (standard-case-table))
> >>       canon)
> >>   (setq canon (copy-sequence table))
> >>   (mapcar (lambda (s)
> >> 	    (mapcar (lambda (c) (aset canon c (aref s 0))) s))
> >> 	  eqv-list)
> >>   (set-char-table-extra-slot table 1 canon)
> >>   (set-char-table-extra-slot table 2 nil)
> >>   (set-standard-case-table table))

Btw, the above doesn't work at all for me in Emacs 25: searching for
'a' doesn't find the variants with diacriticals.  Maybe I didn't use
it correctly -- is something else required beyond evaluating the
expression and making sure I-search does a case-insensitive search?

> > Also, this doesn't handle decomposed characters, as in 'å'.  So this
> > is not really Unicode-compliant, it's a half-measure of sorts.
> 
> The above code snippet predates Unicode Emacs, so you cannot expect it
> to handle NFC and NFD and other intricacies of Unicode normalisation.
> (Also I've never seen anything else than the NFC forms, e.g., for
> German umlauts, in the texts that I usually edit.)

Mac OS X's HFS filesystem holds file names in NFD, AFAIK.

And diacriticals are only the tip of the iceberg.  E.g., when you
search for 'n', won't you want to find 'ⁿ' and '🄝' as well, at least
sometimes, and likewise with '²' and '⒉' and '🄃' when looking for '2'?
These require support for compatibility decompositions, not just for
canonical decompositions as in the case of diacriticals.

> BTW, also isearch-forward doesn't match å when searching for å, and
> vice versa. So by your above argument, search in Emacs isn't Unicode
> compliant anyway.

Of course, Emacs isn't Unicode-compliant -- this is why I said this
feature is sorely needed, and that your proposal is a half-measure.

> (But not sure if it should be, because I think that this would break
> Boyer-Moore.)

It's already broken for multibyte characters anyway.  And yes,
handling equivalence in searching complicates the algorithm even more,
but that's a necessary payment for the extended functionality.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Ulrich Mueller]

>>>>> On Thu, 18 Jun 2015, Eli Zaretskii wrote:

>> >> ;; Ignore accent and umlaut marks when searching.
>> >> ;; Works for Emacs 19.30 and later.
>> >> (let ((eqv-list '("aAàÀáÁâÂãÃäÄåÅ"
>> >> 		  "cCçÇ"
>> >> 		  "eEèÈéÉêÊëË"
>> >> 		  "iIìÌíÍîÎïÏ"
>> >> 		  "nNñÑ"
>> >> 		  "oOòÒóÓôÔõÕöÖøØ"
>> >> 		  "uUùÙúÚûÛüÜ"
>> >> 		  "yYýÝÿ"))
>> >>       (table (standard-case-table))
>> >>       canon)
>> >>   (setq canon (copy-sequence table))
>> >>   (mapcar (lambda (s)
>> >> 	    (mapcar (lambda (c) (aset canon c (aref s 0))) s))
>> >> 	  eqv-list)
>> >>   (set-char-table-extra-slot table 1 canon)
>> >>   (set-char-table-extra-slot table 2 nil)
>> >>   (set-standard-case-table table))

> Btw, the above doesn't work at all for me in Emacs 25: searching for
> 'a' doesn't find the variants with diacriticals. Maybe I didn't use
> it correctly -- is something else required beyond evaluating the
> expression and making sure I-search does a case-insensitive search?

It should still work with Emacs from git master of today. But I guess
an additional (set-case-table (standard-case-table)) is needed when
evaluating it in an existing buffer.

Ulrich

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Also, this doesn't handle decomposed characters, as in 'å'.  So this
> is not really Unicode-compliant, it's a half-measure of sorts.

(I'm not sure it is relevant for this discussion, but to the actual
users of "å", "å" isn't an "a-with-an-accent" but a proper letter in the
alphabet with its own position in the alphabet.  When sorting, it would
in some countries using it, be expected to have "aa" sorted
equivalently, but never "a" (å is at or near the end of the alphabets
using it, "a" is at the start).  In a search engine hits with "aa"
instead of "å" would be expected, but in emacs search I would only
expect to hit "å" and "Å")

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 18 Jun 2015 19:23:24 +0200
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > Also, this doesn't handle decomposed characters, as in 'å'.  So this
> > is not really Unicode-compliant, it's a half-measure of sorts.
> 
> (I'm not sure it is relevant for this discussion, but to the actual
> users of "å", "å" isn't an "a-with-an-accent" but a proper letter in the
> alphabet with its own position in the alphabet.

That's true, but not really relevant.  What I was saying was that this
letter can be found in text either as a single codepoint, E5, or as a
sequence of 2 codepoints, 61 30A.  Unicode mandates that in certain
contexts, like search, users may want to treat both of these as
equivalent, and match them.

> When sorting

Sorting is a related, but different issue, because it also requires an
order relation between characters, whereas search only needs the
equal/not equal relation.

> In a search engine hits with "aa" instead of "å" would be expected,
> but in emacs search I would only expect to hit "å" and "Å")

You might, but others might have other expectations.  They should be
able to have them.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

>> In a search engine hits with "aa" instead of "å" would be expected,
>> but in emacs search I would only expect to hit "å" and "Å")

> You might, but others might have other expectations.  They should be
> able to have them.

I'm pretty sure all native users of "å" would never wish a search to hit
"a" (of course... being able to search for "å" in the first place, is
progress...). 

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Marcin Borkowski]

On 2015-06-18, at 20:13, Steinar Bang <sb@dod.no> wrote:

>>>>>> Eli Zaretskii <eliz@gnu.org>:
>
>>> In a search engine hits with "aa" instead of "å" would be expected,
>>> but in emacs search I would only expect to hit "å" and "Å")
>
>> You might, but others might have other expectations.  They should be
>> able to have them.
>
> I'm pretty sure all native users of "å" would never wish a search to hit
> "a" (of course... being able to search for "å" in the first place, is
> progress...). 

However, many non-native users might wish for that.  Use case:
a bibliography for a scientific article, written in Emacs + AUCTeX +
LaTeX, containing *lots* of completely foreign names, with the author
having *completely* no idea how the native users treat these letters -
he only want to look for that name containing an "å", and doesn't even
know which input method/code point/whatever can be used to actually type
this letter.

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Steinar Bang]

>>>>> Marcin Borkowski <mbork@mbork.pl>:
> he only want to look for that name containing an "å", and doesn't even
> know which input method/code point/whatever can be used to actually
> type this letter.

What...?  You don't have "å" on your keyboard...? :-)


- Steinår

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Marcin Borkowski <mbork@mbork.pl>:
>> he only want to look for that name containing an "å", and doesn't even
>> know which input method/code point/whatever can be used to actually
>> type this letter.

> What...?  You don't have "å" on your keyboard...? :-)

"Whåt"... I meånt...

> - Steinår

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Thu, 18 Jun 2015 20:13:13 +0200
> 
> I'm pretty sure all native users of "å" would never wish a search to hit
> "a"

You were talking about "aa" previously, not "a".

I think hitting "aa" when searching for "å" is as reasonable in
certain cultures as hitting "SS" when searching for "ß" in some
others, or as hitting "fi" when searching for "fi" (and vice versa).
Users don't _always_ want that, but sometimes they might.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

>> From: Steinar Bang <sb@dod.no>
>> Date: Thu, 18 Jun 2015 20:13:13 +0200

>> I'm pretty sure all native users of "å" would never wish a search to hit
>> "a"

> You were talking about "aa" previously, not "a".

Well, both actually.  One as the equivalent (or actually: old
orthography for the same sound), and one that looks similar but is an
entirely different sound. I wasn't sure which one you meant.

> I think hitting "aa" when searching for "å" is as reasonable in
> certain cultures as hitting "SS" when searching for "ß" in some
> others, or as hitting "fi" when searching for "fi" (and vice versa).
> Users don't _always_ want that, but sometimes they might.

It could be reasonable (if a bit unexpected...) for a Dane or a
Norwegian, but not necessarily reasonable for a Swede.

My personal preference would be to make "å" match "å" (and maybe "Å" for
case insensitive search).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Steinar Bang writes:

 > My personal preference would be to make "å" match "å" (and maybe "Å" for
 > case insensitive search).

In general that would be true, and the normal case.  However, here's a
case where you might want it to match "a", too.  You are pretty sure
you saw a quote of a Norwegian text or a Norwegian name in an
otherwise English document, and want to find it.  But you forgot the
spelling was not accurate: the word you're sure is in the quote
contains "å", but was spelled with "a" (by a non-Norwegian typist)
instead.

Rarely of use, I agree.

I think the more normal case (and the original motivation for this,
also pointed out by Marcin) would be an English-speaker who would find
it difficult to type "å", and so searches for the Norwegian text using
the word spelling with "a" (she doesn't know about "aa").

Steve

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Stephen J. Turnbull]

Drew Adams writes:

 > Such code changes are simply tests, to see if you are paying
 > attention.  This approach is, like, waaaaaay more dynamic and
 > 21st-century than discussing potential changes with arguments
 > pro & con and offering patches for people to try on their own.

Uh, no.  Beware who you mock: As far as I can tell on this particular
issue I'm in fact *exactly* aligned with Stefan Monnier.  I can afford
to choose sides, he can't, and that's 100% of the difference in tone
between our posts (aside from the fact that I'm an arrogant
a-----e[1], and he isn't, but most people can filter that part out).

As I've already implied, I think a lot of the changes made to Emacs
over the years were ill-considered, and conservatives like you and
Alan do a great service by asking that new ones be evaluated more
carefully.  I think the same about many of the changes that have been
vetoed, as well; they were not given the effort in evaluation they
deserved.

My experience is that some issues cannot be accurately judged in
advance, and so gendeken experiments and user polling are very
inaccurate.  My experience also shows that the people who try patches
on their own generally feel strongly one way or the other already, and
so rarely change their position yea or any, though they sometimes
modify it in one direction or the other.  Experiments with defaults
(or changing behavior, as in this case) have their cost too, but
sometimes the additional accuracy is worth the cost.  I believe
movement toward exploiting more of the Unicode repertoire for syntax
is one such case.

I also believe that this experiment will show that this movement is
feasible and potentially beneficial, but unlike the conservatives I
don't ask anyone to take my word for it in advance.

Footnotes: 
[1]  I take full responsibility for those words, and do not put them
in anyone else's mouth.  I'm not proud of my condition, but I know who
I am, and couldn't deny it believably after recent posts anyway.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Robert Pluim]

Richard Stallman <rms@gnu.org> writes:

> Inserting ‘ and ’ is as very inconvenient.  Hunting for how to insert
> them is even more so, as it is not documented.
>
> Because I have an instance of ‘ in the buffer now, I was able to use
> C-u C-x = on it, which told me I could insert it with C-x 8 RET 2018
> RET.
>
> I will not remember that hex code.  I will be able to look it up again
> next time -- if I have an instance in the buffer.  Of course, given
> one the buffer, I will not use C-x 8, I will copy it with the kill
> ring.  In other words, this character is so inconvenient to insert
> that I will use workarounds rather than try.

Based on what's in iso-transl.el, C-x 8 [ will insert a single quote
(and similarly C-x 8 { double quotes).  Personally I prefer A-[ and
A-{

I agree it would be very useful if C-u C-x = mentioned those bindings
instead of or in addition to the hex code version.

Robert

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Based on what's in iso-transl.el, C-x 8 [ will insert a single quote
  > (and similarly C-x 8 { double quotes).

It does not work here.

  >   Personally I prefer A-[ and
  > A-{

Are you using an Alt key separate from your Meta key?  I don't have those.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Robert Pluim]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Based on what's in iso-transl.el, C-x 8 [ will insert a single quote
>   > (and similarly C-x 8 { double quotes).
>
> It does not work here.
>

Paul added those bindings on May 10th, I believe you mentioned your
sources were from May 8th.

>   >   Personally I prefer A-[ and
>   > A-{
>
> Are you using an Alt key separate from your Meta key?  I don't have those.

Yes. It's very convenient for me since I never used my right-Alt key
as Meta anyway.

Regards

Robert

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> I am running from source fetched on May 8.

May 8 predates the changes involving curved quotes, so the comments in your 
email don't apply to the current Emacs master.  The current Emacs and Elisp 
manuals say how to enter curved quotes and etc/NEWS briefly notes the changes. 
It no longer takes 8 keystrokes (or even 4) to enter a curved quote -- I suggest 
using the new Electric Quote minor mode, where it typically takes just 1 
keystroke.  Other shorthands are also available.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Thu, 18 Jun 2015 00:05:48 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: acm@muc.de, stephen@xemacs.org, emacs-devel@gnu.org
> 
> May 8 predates the changes involving curved quotes, so the comments in your 
> email don't apply to the current Emacs master.  The current Emacs and Elisp 
> manuals say how to enter curved quotes and etc/NEWS briefly notes the changes. 
> It no longer takes 8 keystrokes (or even 4) to enter a curved quote -- I suggest 
> using the new Electric Quote minor mode, where it typically takes just 1 
> keystroke.  Other shorthands are also available.

I suggest to mention the Electric Quote mode (and perhaps other
related features for typing those characters) in NEWS, and have that
referenced right where we describe in NEWS their use in doc strings.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It no longer takes 8 keystrokes (or even 4) to enter a curved quote -- I suggest 
  > using the new Electric Quote minor mode, where it typically takes just 1 
  > keystroke.

With that mode, the inconvenience will be more or less limited to
those who don't know about it.  Will it be enabled by default in Emacs
Lisp mode?

Even if we eliminate the practical inconvenience, it is still added
complexity, and that has a cost, both in maintenance and in use.  What
practical advantage justifies this complexity?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> Will it be enabled by default in Emacs Lisp mode?

I'd prefer that, yes.  I suggested doing this for Emacs developers first, in the 
master branch.  Eli thought this too eager, so I did not install it.  Perhaps we 
should revisit this at some point.  In the meantime I've put the following into 
my ~/.emacs startup file:

(if (fboundp 'electric-quote-mode)
     (electric-quote-mode))

and this does enable electric quoting in Emacs Lisp mode.


> What practical advantage justifies this complexity?

There are practical advantages to a quoting style that can't easily be confused 
with ordinary Lisp code involving ' and `.  We've already done this in the Emacs 
manuals' Info files, and the idea is to extend this to *Help* buffers (already 
done in the master branch) and to diagnostics (not yet done).  This will provide 
a more consistent interface within Emacs.

Also, quoting with single quotes is easier for most non-experts to read. 
Emacs's traditional style of quoting with grave accent and apostrophe is 
idiosyncratic and is offputting for many new users; it used to work well in 
displays long ago and it was reasonably common in GNU applications, but that's 
no longer true and now it's one more thing to learn when using Emacs.  I have 
some interest and experience in this, as my assistants and I introduce Emacs to 
an average of about one new user per day.  It would be better for Emacs to use a 
quoting style similar to what people are accustomed to, so that they are not 
distracted by it.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

There are two different issues here: a UI issue and a programming issue.
I'm afraid you are mixing them up.

  > Also, quoting with single quotes is easier for most non-experts to read. 

Most non-experts don't read the Emacs source code.  Are you talking
about reading messages and doc strings?

  > Emacs's traditional style of quoting with grave accent and apostrophe is 
  > idiosyncratic and is offputting for many new users;

To new users reading the Emacs source code?  Or is this about what
they see in messages and doc strings?

  >   I have 
  > some interest and experience in this, as my assistants and I introduce Emacs to 
  > an average of about one new user per day.

When you introduce Emacs to new users, do you teach them to edit Emacs
Lisp code?  I have nothin against it, but it seems unlikely that they
all advance so far in one day.


There are two different issues here: a UI issue and a programming issue.
The UI issue is about what to display in messages doc strings.
The programming issue is about what to write in the source code.

I'm talking about the source code issue, but the reasons you give
seem to apply to the UI issue.

We've already settled the UI issue, with a feature that will display
the quotes in whatever form the user likes, defaulting based on what
the terminal can display.  That should please everyone.

This makes the source code issue totally independent of the UI issue.

What practical benefit comes from writing curly quotes in doc strings
in Lisp source code?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> When you introduce Emacs to new users, do you teach them to edit Emacs
> Lisp code?

Not in the first day, but we have people use Emacs over a period of ten weeks, 
as part of a software construction course.  A primary benefit of Emacs over 
editors like vim (which is what we've also used in the past) is its 
programmability.  I want students to be able to read and understand the source 
code of the tools they're using.

> I'm talking about the source code issue, but the reasons you give
> seem to apply to the UI issue.

Although they apply primarily to the UI, they also apply to source code. 
Previously a docstring couldn't unambiguously quote Lisp code containing ' or `. 
  Now it can.  Previously one could cut from a *Help* buffer and paste into a 
docstring; this remains true only because curved quotes now work in a docstring 
and so can be safely copied from a *Help* buffer to a docstring.

Also, nonexperts read Lisp code at times, and we're better off simplifying the 
process of reading the code and becoming expert.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You have presented practical arguments, but they are extremely minor
ones; they can't justify much.

  > Not in the first day, but we have people use Emacs over a period of ten weeks, 
  > as part of a software construction course.  A primary benefit of Emacs over 
  > editors like vim (which is what we've also used in the past) is its 
  > programmability.  I want students to be able to read and understand the source 
  > code of the tools they're using.

To understand Emacs Lisp (beyond the rudiments) one must learn an awful lot.
A convention such as `...' is insignificant by comparison.  Thus, the benefit
of eliminating `...' would be little.

As long as both `...' and curly quotes are being used, instead of a tiny
simplification it would be a tiny extra complication.

  > Although they apply primarily to the UI, they also apply to source code. 
  > Previously a docstring couldn't unambiguously quote Lisp code containing ' or `. 
  >   Now it can.

When we mention a complicated expression in a doc string, we typically
put it on separate lines, for readability.  Then we don't need any
sort of quotation marks around it.  Thus, this issue is not a real
problem in practice.

  >  Previously one could cut from a *Help* buffer and paste into a
  > docstring; this remains true only because curved quotes now work
  > in a docstring

I recommend copying from the source code of the function.
(One might want to copy more than just the doc string.)

  > Also, nonexperts read Lisp code at times, and we're better off simplifying the 
  > process of reading the code and becoming expert.

What I said about the first point applies here too: to guess the
meaning of `...' is insignificant by comparison with the rest of what
you have to guess, and as long as both forms are used, this means
a tiny complication rather than a tiny simplification.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> as long as both forms are used, this means
> a tiny complication rather than a tiny simplification

The intent is to move to the simpler form, which uses curved quotes uniformly. 
That will be simpler than what's in master now, and simpler than what's in Emacs 
24.5 where curved quotes are in documentation but grave accents are in *Help* 
buffers and diagnostics.  So the idea is to support the traditional form 
indefinitely to ease transition, but to recommend the simpler form.

> To understand Emacs Lisp (beyond the rudiments) one must learn an awful lot.

Comments and strings and doc strings are part of the rudiments.  The course I 
mentioned assumes six months' programming experience.  Even people with less 
experience than that should be able to read and write doc strings.

> When we mention a complicated expression in a doc string, we typically
> put it on separate lines, for readability.

This works if the definition of “complicated” includes any expression containing 
' or `, which requires separate lines that would not be otherwise needed.  And 
there is a reasonable likelihood of confusing docstrings lacking separate lines 
anyway.  For example, until I recently fixed it the docstring for 
‘recursive-edit’ contained this:

   To get out of the recursive edit, a command can throw to `exit' -- for
   instance `(throw 'exit nil)'.

Is one supposed to quasiquote the throw?  An expert should know the answer right 
away, but a non-expert is likely to be confused at first.  Curved-quote 
docstrings lessen this sort of confusion.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The intent is to move to the simpler form, which uses curved quotes uniformly. 

To call it "simpler" is an exaggeration, for such a tiny simplification.

But it may cause substantial problems for some people.

For instance, how will this affect developers whose terminals won't
display curly quotes, in editing the source code?
How does Emacs display curly quotes on such terminals?

  > > To understand Emacs Lisp (beyond the rudiments) one must learn an awful lot.

  > Comments and strings and doc strings are part of the rudiments.

That's more false than true.

The general ideas of comments and strings are part of the rudiments.
The general ideas of doc strings, not quite so much.

The full conventions for doc strings are definitely not
part of the rudiments.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 22 Jun 2015 09:55:27 -0400
> Cc: acm@muc.de, stephen@xemacs.org, emacs-devel@gnu.org, tsdh@gnu.org
> 
> For instance, how will this affect developers whose terminals won't
> display curly quotes, in editing the source code?
> How does Emacs display curly quotes on such terminals?

In those cases, we use a display table to display them as their ASCII
equivalents.  (The fact that the terminal doesn't support the curved
quotes is determined at Emacs startup.)

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > For instance, how will this affect developers whose terminals won't
  > > display curly quotes, in editing the source code?
  > > How does Emacs display curly quotes on such terminals?

  > In those cases, we use a display table to display them as their ASCII
  > equivalents.

Are the ASCII equivalents of the two curly braces ` and '?

If so, the code will look natural, and most editing will work fine.
However, searches for those characters will have unpredictable results.

As that user writes changes, he will insert real ` and ' rather than
curly braces.  But maybe the new electric quote mode will convert them.

If so, I don't see any problem off hand, except search.
That is not a grave problem, but it is a problem.

Given how little benefit this will give us, why make the change?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Nikolai Weibull]

On Tue, Jun 23, 2015 at 2:37 PM, Richard Stallman <rms@gnu.org> wrote:

>   > > For instance, how will this affect developers whose terminals won't
>   > > display curly quotes, in editing the source code?
>   > > How does Emacs display curly quotes on such terminals?
>
>   > In those cases, we use a display table to display them as their ASCII
>   > equivalents.

> Are the ASCII equivalents of the two curly braces ` and '?

I assume so, even though the true “equivalents” are ‘'’ and ‘'’.
ASCII doesn’t have a unique left and right single (or double)
quotation mark, instead employing a straight “bastardization” of one.

> That is not a grave problem, but it is a problem.

Great pun, Richard :-)

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, acm@muc.de, stephen@xemacs.org,
> 	emacs-devel@gnu.org, tsdh@gnu.org
> Date: Tue, 23 Jun 2015 08:37:54 -0400
> 
>   > > For instance, how will this affect developers whose terminals won't
>   > > display curly quotes, in editing the source code?
>   > > How does Emacs display curly quotes on such terminals?
> 
>   > In those cases, we use a display table to display them as their ASCII
>   > equivalents.
> 
> Are the ASCII equivalents of the two curly braces ` and '?

No, they are ' and ', in line with the recommendations of the GNU
Coding Standards.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Are the ASCII equivalents of the two curly braces ` and '?

  > No, they are ' and ', in line with the recommendations of the GNU
  > Coding Standards.

Does this mean that both curly braces, in Emacs Lisp doc strings,
will appear as ' on a terminal which can't display curly braces?

That's a total screw.  Developers using such a display
will not be able to distinguish the two.

If they display as ` and ', editing will be feasible once
you understand the convention.  If they both display as ',
it makes no sense to try this even as an experiment.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: acm@muc.de, stephen@xemacs.org, eggert@cs.ucla.edu, tsdh@gnu.org,
> 	emacs-devel@gnu.org
> Date: Tue, 23 Jun 2015 18:21:08 -0400
> 
>   > > Are the ASCII equivalents of the two curly braces ` and '?
> 
>   > No, they are ' and ', in line with the recommendations of the GNU
>   > Coding Standards.
> 
> Does this mean that both curly braces, in Emacs Lisp doc strings,
> will appear as ' on a terminal which can't display curly braces?

Yes.

> That's a total screw.  Developers using such a display
> will not be able to distinguish the two.

"C-x =" can tell the difference.  But yes, it's not very convenient.
However, standards.texi was recommending to quote 'like this' for
quite some time now, so I think that ship sailed long ago.

> If they display as ` and ', editing will be feasible once
> you understand the convention.  If they both display as ',
> it makes no sense to try this even as an experiment.

You can customize the display table if you want.  This is Emacs, after
all.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Does this mean that both curly braces, in Emacs Lisp doc strings,
  > > will appear as ' on a terminal which can't display curly braces?

  > Yes.

  > > That's a total screw.  Developers using such a display
  > > will not be able to distinguish the two.

  > "C-x =" can tell the difference.  But yes, it's not very convenient.
  > However, standards.texi was recommending to quote 'like this' for
  > quite some time now,

That's a different issue.

If you put two different curly quotes in the buffer, and they look
like ` and ', you will understand what's going on once you know
that convention.  But if they both look like ', it is a screw
for editing.

For this reason, we should not put any curly quotes in doc strings
at present.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: acm@muc.de, stephen@xemacs.org, eggert@cs.ucla.edu, tsdh@gnu.org,
> 	emacs-devel@gnu.org
> Date: Wed, 24 Jun 2015 10:56:49 -0400
> 
>   > However, standards.texi was recommending to quote 'like this' for
>   > quite some time now,
> 
> That's a different issue.

Sorry, I don't understand: how is that a different issue?

I meant this part of standards.texi:

  5.10 Quote Characters
  =====================

  In the C locale, the output of GNU programs should stick to plain ASCII
  for quotation characters in messages to users: preferably 0x22 (`"') or
  0x27 (`'') for both opening and closing quotes.  Although GNU programs
  traditionally used 0x60 (``') for opening and 0x27 (`'') for closing
  quotes, nowadays quotes ``like this'' are typically rendered
  asymmetrically, so quoting `"like this"' or `'like this'' typically
  looks better.

It clearly says that quoting 'like this' is better than `like this',
doesn't it?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Eli Zaretskii wrote:
> I meant this part of standards.texi:

I relied on that part of standards.texi as well, when I discussed this display 
issue with you a few weeks ago.  However, this discussion is leading me to see 
that I was mistaken.  The standards.texi document is talking about diagnostics, 
whereas this issue is about display, and they're different things.  If one is 
editing a random text file that contains curved quotes, it's not a diagnostic, 
it's data.

So I think RMS's suggestion is the right one, and that we should change Emacs to 
display ` (not ') for undisplayable left single quotation mark.  I'll prepare a 
patch along those lines assuming there's no objection.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So I think RMS's suggestion is the right one, and that we should change Emacs to 
  > display ` (not ') for undisplayable left single quotation mark.  I'll prepare a 
  > patch along those lines assuming there's no objection.

Thanks.  That will fix a significant problem.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

Richard Stallman wrote:
>    > So I think RMS's suggestion is the right one, and that we should change Emacs to
>    > display ` (not ') for undisplayable left single quotation mark.  I'll prepare a
>    > patch along those lines assuming there's no objection.
>
> Thanks.  That will fix a significant problem.

You're welcome.  I installed the attached patch.


[-- Attachment #2: 0001-Translate-undisplayable-to.txt --]
[-- Type: text/plain, Size: 4028 bytes --]

From 08bb4e9393611607190c495bc34ad1738b8e0da1 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 25 Jun 2015 07:21:20 -0700
Subject: [PATCH] =?UTF-8?q?Translate=20undisplayable=20=E2=80=98=20to=20`?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* doc/lispref/help.texi (Keys in Documentation):
* lisp/international/mule-cmds.el (set-locale-environment):
* lisp/term/w32console.el (terminal-init-w32console):
* src/doc.c (Fsubstitute_command_keys, Vhelp_quote_translation):
If ‘ is not displayable, transliterate it to `, not to '.  See:
http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00542.html
---
 doc/lispref/help.texi           | 2 +-
 lisp/international/mule-cmds.el | 4 ++--
 lisp/term/w32console.el         | 2 +-
 src/doc.c                       | 6 +++---
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/doc/lispref/help.texi b/doc/lispref/help.texi
index 44a680c..fde985d 100644
--- a/doc/lispref/help.texi
+++ b/doc/lispref/help.texi
@@ -357,7 +357,7 @@ quotes.  If the value is @code{?'} (apostrophe), the style is @t{'like
 this'} with apostrophes.  If the value is @code{?`} (grave accent),
 the style is @t{`like this'} with grave accent and apostrophe.  The
 default value @code{nil} means to use curved single quotes if
-displayable and apostrophes otherwise.
+displayable, and grave accent and apostrophe otherwise.
 @end defvar
 
 @defun substitute-command-keys string
diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 16c1003..248c89c 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -2731,9 +2731,9 @@ See also `locale-charset-language-names', `locale-language-names',
 	(set-terminal-coding-system 'utf-8)
 	(set-keyboard-coding-system 'utf-8)))
 
-    ;; If curved quotes don't work, display straight ASCII approximations.
+    ;; If curved quotes don't work, display ASCII approximations.
     (unless frame
-      (dolist (char-repl '((?‘ . [?\']) (?’ . [?\']) (?“ . [?\"]) (?” . [?\"])))
+      (dolist (char-repl '((?‘ . [?\`]) (?’ . [?\']) (?“ . [?\"]) (?” . [?\"])))
         (when (not (char-displayable-p (car char-repl)))
           (or standard-display-table
               (setq standard-display-table (make-display-table)))
diff --git a/lisp/term/w32console.el b/lisp/term/w32console.el
index 29ab2f1..2df1378 100644
--- a/lisp/term/w32console.el
+++ b/lisp/term/w32console.el
@@ -69,7 +69,7 @@
           ;; Since we changed the terminal encoding, we need to repeat
           ;; the test for Unicode quotes being displayable.
           (dolist (char-repl
-                   '((?‘ . [?\']) (?’ . [?\']) (?“ . [?\"]) (?” . [?\"])))
+                   '((?‘ . [?\`]) (?’ . [?\']) (?“ . [?\"]) (?” . [?\"])))
             (when (not (char-displayable-p (car char-repl)))
               (or standard-display-table
                   (setq standard-display-table (make-display-table)))
diff --git a/src/doc.c b/src/doc.c
index d2d3c8d..655b911 100644
--- a/src/doc.c
+++ b/src/doc.c
@@ -761,8 +761,8 @@ Otherwise, return a new string.  */)
       Lisp_Object dv = DISP_CHAR_VECTOR (XCHAR_TABLE (Vstandard_display_table),
 					 LEFT_SINGLE_QUOTATION_MARK);
       if (VECTORP (dv) && ASIZE (dv) == 1
-	  && EQ (AREF (dv, 0), make_number ('\'')))
-	quote_translation = apostrophe;
+	  && EQ (AREF (dv, 0), make_number ('`')))
+	quote_translation = grave_accent;
     }
 
   multibyte = STRING_MULTIBYTE (string);
@@ -1040,7 +1040,7 @@ Quote \\=‘like this\\=’ if the value is ?\\=‘ (left single quotation mark)
 Quote 'like this' if the value is ?' (apostrophe).
 Quote \\=`like this' if the value is ?\\=` (grave accent).
 The default value is nil, which means quote with left single quotation mark
-if displayable, and with apostrophe otherwise.  */);
+if displayable, and with grave accent otherwise.  */);
   Vhelp_quote_translation = Qnil;
 
   defsubr (&Sdocumentation);
-- 
2.1.0

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Oleh Krehel]

Richard Stallman <rms@gnu.org> writes:

>   > > Are the ASCII equivalents of the two curly braces ` and '?
>
>   > No, they are ' and ', in line with the recommendations of the GNU
>   > Coding Standards.
>
> Does this mean that both curly braces, in Emacs Lisp doc strings,
> will appear as ' on a terminal which can't display curly braces?

Not only there, but everywhere for people like me who will use the
display table to modify "‘" to display as "'", because I completely
disagree with the change and think that the new quotes are ugly and hard
to input.

> That's a total screw.  Developers using such a display
> will not be able to distinguish the two.

I think the biggest screw hasn't yet been mentioned. If the change is to
go through, I think the lesser evil would be to completely forbid the
old style of quoting. Which of course won't happen, even if the change
is to go through.

And then we are left with a greater evil: two styles of quotes forever
intertwined. There's no way I'm switching the quoting style in any of my
(3rd party) packages, unless the lesser evil path leads to them being
forbidden.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Oops.  I meant to write "quotes", not "braces".  It should be:

  > > Are the ASCII equivalents of the two curly quotes ` and '?

  > No, they are ' and ', in line with the recommendations of the GNU
  > Coding Standards.

Does this mean that both curly quotes, in Emacs Lisp doc strings,
will appear as ' on a terminal which can't display curly quotes?

That's a total screw.  Developers using such a display
will not be able to distinguish the two.

If they display as ` and ', editing will be feasible once
you understand the convention.  If they both display as ',
it makes no sense to try this even as an experiment.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Richard Stallman wrote:
> The full conventions for doc strings are definitely not
> part of the rudiments.

True, but quoting is pretty elementary.  Most doc strings use it, I daresay. 
For example, the doc strings for ‘car’, ‘cdr’, and ‘length’ all use quoting. 
These are the sorts of doc strings that new users are likely to see.

I agree with you, by the way, that the improvement due to using curved quotes is 
not a large one.  Emacs's use of an obsolescent quoting style is merely a 
constant minor annoyance, and it's obviously something that we can work around 
when we teach Emacs to students -- we can even give them a little history lesson 
as we go.  It's just one minor way that Emacs appears old-fashioned.  Obviously 
there are other, bigger ways, but I'm working on this one primarily because I 
finally got annoyed enough with it enough to invest the time to fix it, and 
because it's a minor task that I can tackle a bit at a time.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Wolfgang Jenkner]

On Tue, Jun 23 2015, Paul Eggert wrote:

> Richard Stallman wrote:
>> The full conventions for doc strings are definitely not
>> part of the rudiments.
>
> True, but quoting is pretty elementary.  Most doc strings use it,
> I daresay. For example, the doc strings for ‘car’, ‘cdr’, and ‘length’
> all use quoting. These are the sorts of doc strings that new users are
> likely to see.
>
> I agree with you, by the way, that the improvement due to using curved
> quotes is not a large one.  Emacs's use of an obsolescent quoting
> style is merely a constant minor annoyance, and it's obviously
> something that we can work around when we teach Emacs to students --
> we can even give them a little history lesson as we go.

You could equally well choose to present doc strings in emacs as an
example of how to decouple mark-up from presentation instead, though.

I mean, those quotes (any kind of quotes) in *Help* or *info* are a bit
odd anyway given that documentation presented in web browsers usually
manages to do fine without them.

How do you explain to your students why they are needed (the quotes, not
the students)?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Wolfgang Jenkner wrote:
> How do you explain to your students why they are needed (the quotes, not
> the students)?

The quotes themselves are fairly basic, but the details of what they're all used 
for are relatively advanced, and I wouldn't expect lower-division students to 
know all that.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Emacs's use of an obsolescent quoting style is merely a constant
  > minor annoyance, and it's obviously something that we can work
  > around when we teach Emacs to students -- we can even give them a
  > little history lesson as we go.

If the change causes no problems at all, it could be worth while
to fix such a minor problem.

But it will cause problems, with searching -- unless we implement a
fix for that.  People are discussing a different but related search
feature; perhaps the search problem for disguised curly quotes will be
eliminated.  But until it is, we should not use curly quotes in doc
strings.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Juri Linkov]

> In the meantime I've put the following into my ~/.emacs startup file:
>
> (if (fboundp 'electric-quote-mode)
>     (electric-quote-mode))

Or another way to do the same: (add-to-list 'insert-pair-alist '(?\' ?\‘ ?\’))
e.g. together with (define-key esc-map "'"  'insert-pair)

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Mon, 15 Jun 2015 14:22:37 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> My view is that the curly quotes should not replace 0x60 and 0x27 in our
> sources, but that the option to display them as curly quotes should be
> made available to those that want it.
> 
> As far as I am aware, there has been no poll to gather and analyse the
> views of Emacs developers on these changes, much less one for Emacs
> users.  This is a Bad Thing.
> 
> What do people think?

I think this ship has sailed long ago.  E.g., GCC has been spewing
UTF-8 encoded quotes at me for several years now, and many Info
manuals started using Unicode characters, that show as gibberish on my
text terminal, many moons ago.

Like you, I don't like these characters very much.  The make my life
slightly harder.  But I like fighting quixotic wars even less.
Instead of fighting this wave, we should adapt and move on.

And there are ways of adapting: convenient input methods for typing
the characters, replacements for displaying them on terminals that
cannot show the originals, run Grep from within Emacs if it's
inconvenient to type these characters from a terminal and observe the
results there, etc.  We already have all that, and perhaps will need
some more, which we will discover as we move on.  The next release of
Texinfo, just around the corner, will have a feature where the Info
reader will display these characters in a legible form, even if the
terminal is not UTF-8 capable.  So it's not a catastrophe, just
something new to adapt to, and make sure, like we already do, that the
degradation on less capable devices is graceful.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

On 06/15/2015 09:52 AM, Eli Zaretskii wrote:
>> Date: Mon, 15 Jun 2015 14:22:37 +0000
>> From: Alan Mackenzie <acm@muc.de>
>>
>> My view is that the curly quotes should not replace 0x60 and 0x27 in our
>> sources, but that the option to display them as curly quotes should be
>> made available to those that want it.
>>
>> As far as I am aware, there has been no poll to gather and analyse the
>> views of Emacs developers on these changes, much less one for Emacs
>> users.  This is a Bad Thing.
>>
>> What do people think?
> 
> I think this ship has sailed long ago.  E.g., GCC has been spewing
> UTF-8 encoded quotes at me for several years now, and many Info
> manuals started using Unicode characters, that show as gibberish on my
> text terminal, many moons ago.
> 
> Like you, I don't like these characters very much.  The make my life
> slightly harder.  But I like fighting quixotic wars even less.
> Instead of fighting this wave, we should adapt and move on.
> 
> And there are ways of adapting: convenient input methods for typing
> the characters, replacements for displaying them on terminals that
> cannot show the originals

Can we make " match fancy quotes in isearch?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

Daniel Colascione <dancol@dancol.org> writes:

>> And there are ways of adapting: convenient input methods for typing
>> the characters, replacements for displaying them on terminals that
>> cannot show the originals
>
> Can we make " match fancy quotes in isearch?

And ditto for ` and ' which I guess should also match ‘ and ’,
respectively.  That's the downside of having two alternatives for
marking symbols in comments and docstrings.

And of course one wants that convenience also with search-forward,
query-replace, grep, rgrep, occur, swiper, ag (3rd party), swiper (3rd
party), ...

Bye,
Tassilo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 212 bytes --]

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Alan Mackenzie]

Hello, Tassilo.

On Wed, Jun 17, 2015 at 10:15:48AM +0200, Tassilo Horn wrote:
> Daniel Colascione <dancol@dancol.org> writes:

> >> And there are ways of adapting: convenient input methods for typing
> >> the characters, replacements for displaying them on terminals that
> >> cannot show the originals

> > Can we make " match fancy quotes in isearch?

> And ditto for ` and ' which I guess should also match ‘ and ’,
> respectively.  That's the downside of having two alternatives for
> marking symbols in comments and docstrings.

Indeed.  With such a change to isearch, Emacs takes one more step from
being a sharp tool to being a blunt instrument.

> And of course one wants that convenience also with search-forward,
> query-replace, grep, rgrep, occur, swiper, ag (3rd party), swiper (3rd
> party), ...

Don't forget that Emacs is used for things other than developing Elisp
programs.  Such an identification of disparate characters is not going
to be universally welcome, and will cause confusion.

> Bye,
> Tassilo

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Yuri Khan]

On Wed, Jun 17, 2015 at 2:40 PM, Alan Mackenzie <acm@muc.de> wrote:
>> > Can we make " match fancy quotes in isearch?
>
> Indeed.  With such a change to isearch, Emacs takes one more step from
> being a sharp tool to being a blunt instrument.
>
> Don't forget that Emacs is used for things other than developing Elisp
> programs.  Such an identification of disparate characters is not going
> to be universally welcome, and will cause confusion.

Emacs already has a facility for ignoring distinctions between
disparate characters. It’s called case-insensitive search, and is
governed by a customizable global option and a toggleable local option
in isearch.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Tassilo Horn]

Yuri Khan <yuri.v.khan@gmail.com> writes:

>> Indeed.  With such a change to isearch, Emacs takes one more step
>> from being a sharp tool to being a blunt instrument.
>>
>> Don't forget that Emacs is used for things other than developing Elisp
>> programs.  Such an identification of disparate characters is not going
>> to be universally welcome, and will cause confusion.
>
> Emacs already has a facility for ignoring distinctions between
> disparate characters.  It’s called case-insensitive search, and is
> governed by a customizable global option and a toggleable local option
> in isearch.

So probably we should have some `quote-fold-search' with the same
toggles.  Then modes need to specify somehow which replacements are
valid, e.g., elisp-mode would define something like ((?` ?` ?‘) (?' ?'
?’)) meaning that ` matches itself and also an opening quote, and '
matches itself and also a closing quote.

Bye,
Tassilo

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Artur Malabarba]

2015-06-17 12:20 GMT+01:00 Tassilo Horn <tsdh@gnu.org>:
> Yuri Khan <yuri.v.khan@gmail.com> writes:
>
>>> Indeed.  With such a change to isearch, Emacs takes one more step
>>> from being a sharp tool to being a blunt instrument.
>>>
>>> Don't forget that Emacs is used for things other than developing Elisp
>>> programs.  Such an identification of disparate characters is not going
>>> to be universally welcome, and will cause confusion.
>>
>> Emacs already has a facility for ignoring distinctions between
>> disparate characters.  It’s called case-insensitive search, and is
>> governed by a customizable global option and a toggleable local option
>> in isearch.
>
> So probably we should have some `quote-fold-search' with the same
> toggles.  Then modes need to specify somehow which replacements are
> valid, e.g., elisp-mode would define something like ((?` ?` ?‘) (?' ?'
> ?’)) meaning that ` matches itself and also an opening quote, and '
> matches itself and also a closing quote.

Folding characters based on something different than case has been
discussed here at least 3 times since I joined this party (and I'm
pretty new here).
It's been coming up since `info' started round quoting stuff (which
was well before the stuff being discussed here).

There is a bug report on it (I think it's from last year, but might be
sooner) where I offered a patch for that, but it was rejected because
it would only apply to isearch and people prefered a more general
thing (like case-folding).
Some time after that I started a thread that had 3 patches proposing 3
different ways to do that (though they weren't finalized). IIRC that
one only got the attention of Eli and Stefan, and the improvements
that were suggested there were beyond my coding time/skills.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Nathan]

[-- Attachment #1: Type: text/plain, Size: 3830 bytes --]

Hello gentle emacs-devel members,

I am a boring person with a bog standard US keyboard, who uses emacs
extremely regularly, and I would prefer that the character set used be
7-bit ASCII, for the single and solitary reason that I would prefer to type
out characters rather than hunt them down on the Internet or some font
program to copy & paste the character into my emacs. The vast majority of
the programs I write and the files I work with are 7-bit ASCII. This is,
indeed, typical of the organizations I have worked with. Non-ASCII
rendering was confined to .po files and other i18n situations.

It is perhaps unfortunate and certainly an accident of history, but my
keyboard is limited, and is as limited in its expression as my ancient IBM
PS/2 keyboard.

It would satisfy me if I could have a variable *render-in-7-bit* that could
be set to T, and I could continue on my placid and slightly antique way,
with an aesthetic renderer taking over when *render-in-7-bit* is set to NIL.


On Mon, Jun 15, 2015 at 7:22 AM, Alan Mackenzie <acm@muc.de> wrote:

>
> Hello, Emacs.
>
> First, some definitions.
>
> A @dfn{working character} is a character you use in everyday life - you
> can type it easily on your keyboard for self-insert-command, and it is
> displayed clearly and unambiguously on your screen.  As Emacs developers,
> our working characters are basically ASCII, though each of us has his own
> characters (e.g. ä, ö, ü, ß for me) which count as "private" working
> characters.
>
> By contrast, a @dfn{display character}, or @dfn{non-working character} is
> a character which isn't a working character.  To insert it into a buffer,
> you need to type its numeric code, or use (and remember) a C-x 8 ...
> binding, or some other clumsy workaround.  It might or might not get
> properly displayed on your screen.
>
> Traditionally, Emacs sources have been written exclusively in working
> characters (with the essential exception of some characters which stand
> for themselves, such as some non-ASCII punctuation symbols used in
> `sentence-end').  This makes editing our source files optimally
> efficient.
>
> Recently, there has been a move, primarily by Paul, to introduce
> non-working characters (specifically, LEFT/RIGHT SINGLE QUOTATION MARKs,
> or (less shoutily) "curly quotes") into our sources and *Help* buffers,
> not just marginally, but routinely into all our doc strings and error
> strings.
>
> Paul and I have had an extensive discussion on many of the issues this
> raises, in the thread for bug #20707.  I am against these changes for
> several reasons: basically, they will make our sources more difficult to
> read and edit.  I am not even sure of the motivation for the changes,
> though I think it is mainly that some people find the appearance of 0x60
> and 0x27 as quote marks unaesthetic in some display environments.
>
> My main objection is there is no option to turn this new "feature" off.
> Some users are going to dislike these changes, possible dislike them a
> lot, but we are going to be forcing the curly quotes on them.  This is
> going to create much dissatisfaction.  For comparison, the fringe was a
> new feature in Emacs-21 which had no way of being disabled.  Many users
> hated it, and were up in arms about it until this was fixed in Emacs-22.
>
> My view is that the curly quotes should not replace 0x60 and 0x27 in our
> sources, but that the option to display them as curly quotes should be
> made available to those that want it.
>
> As far as I am aware, there has been no poll to gather and analyse the
> views of Emacs developers on these changes, much less one for Emacs
> users.  This is a Bad Thing.
>
> What do people think?
>
> --
> Alan Mackenzie (Nuremberg, Germany).
>
>

[-- Attachment #2: Type: text/html, Size: 4364 bytes --]

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

> We have just installed that feature, and it's still being worked on
> to eliminate some rough edges and unintended consequences.  Let it be
> refined, then let it be used by users (including you, Drew) for some
> time, and then let's collect user experience and complaints and see
> if something needs to be fixed.  _Then_ we will have experience and
> real data points to discuss constructively and efficiently.

I'm guessing that the feature you are talking about is the char
folding that Bruce/Artur installed, and *not* the suggestion by
Kaushal to "take over the mode line" and use "a lot of space to
verbosely display the searching options".

If so then we are in violent agreement and I think you did not
understand me.  My comments were directed to the suggested
mode-line feature, not to the char-folding feature.

My message was to not go down the road of changing the mode
line that way now without thinking about what might be
appropriate when there are more folding possibilities.

My point was precisely to let people use the new char folding
for a while, *without* also immediately jumping into a UI
change of the mode line to indicate whether it is on or off.

That's all.  I am in favor of the char-folding advancement,
and of its refinement over time, based on user experience etc.,
exactly as you describe.  I was quite clear that I am "very
thankful that we will soon have some additional char folding
beyond case folding."  This is a step forward.

I agree that we should "collect user experience and complaints
and see if something needs to be fixed."  And not just wrt
fixing something but wrt improvement suggestions, including
any suggestions about how to let users know what the current
search state is.  Time will tell what is most helpful for that.

My point was only that we should not prematurely opt now for a
new, mode-line UI means of conveying the status - something we
don't even do currently for case folding.

Wait and see what might work best for this - that was my
message.  As you put it, "_Then_ we will have experience and
real data points to discuss constructively and efficiently."

We could discuss it now, but we should not just throw some
take-over-the-mode-line UI change out there now, which might
end up tying our hands (yes, momentum) and not make sense if
and when more folding possibilities become available.  And
even discussion about the UI possibilities is best carried
out when we've had more experience with the new feature.

A priori, I am in favor of our eventually adding some way of
indicating the current search state, including perhaps using
the mode-line.  My objection is to us prematurely opting for
a way that might not fit well with the existence of multiple
foldings.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Eli Zaretskii]

> Date: Sat, 27 Jun 2015 08:34:46 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: bruce.connor.am@gmail.com, emacs-devel@gnu.org
> 
> > We have just installed that feature, and it's still being worked on
> > to eliminate some rough edges and unintended consequences.  Let it be
> > refined, then let it be used by users (including you, Drew) for some
> > time, and then let's collect user experience and complaints and see
> > if something needs to be fixed.  _Then_ we will have experience and
> > real data points to discuss constructively and efficiently.
> 
> I'm guessing that the feature you are talking about is the char
> folding that Bruce/Artur installed, and *not* the suggestion by
> Kaushal to "take over the mode line" and use "a lot of space to
> verbosely display the searching options".

Yes.

> If so then we are in violent agreement and I think you did not
> understand me.  My comments were directed to the suggested
> mode-line feature, not to the char-folding feature.

Sorry for my misunderstanding.

For the record, I'd like us to keep our application-level hands off
the mode line.  It is already too cramped with useful stuff that
shouldn't be removed.  The fact that applications can redefine the
mode line by changing mode-line-format should not IMO be interpreted
as an invitation to put arbitrary stuff there.

> My message was to not go down the road of changing the mode
> line that way now without thinking about what might be
> appropriate when there are more folding possibilities.

Agreed.

Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]

> > I'm guessing that the feature you are talking about is the char
> > folding that Bruce/Artur installed, and *not* the suggestion by
> > Kaushal to "take over the mode line" and use "a lot of space to
> > verbosely display the searching options".

Just to be fair, it was I (Artur) who suggested taking over the mode line
(and I've changed my mind against that now too). Kaushal's suggestion was
more discrete.

[-- Attachment #2: Type: text/html, Size: 516 bytes --]

RE: ASCII-folded search [was: Re: Upcoming loss of usability ...]

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

My bad. Apologies to Kaushal.

 

I must say that the use of HTML mail in a mailing list of mostly plain-text messages is (IMHO) a pity and a pain.

 

In my mail client, at least, Kaushal's reply to your message shows that quoted bit from your message at the top level, which makes it indistinguishable from the text Kaushal wrote as a reply.

 

But I now see the message displayed correctly at lists.gnu.org:

http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00593.html

 

 

Is there a special reason why you use HTML mail on this mailing list?

 

[FWIW, I use HTML mail in my work and leisure, along with everyone else I communicate with, but I don't use it on these mailing lists. I thought that doing so was discouraged. In any case, I, for one find it a PITA to reply to a mix and to try to reply to an HTML msg coherently using plain text.]

 

From: Artur Malabarba [mailto:bruce.connor.am@gmail.com] 
Sent: Saturday, June 27, 2015 9:47 AM
To: Eli Zaretskii
Cc: emacs-devel; Drew Adams
Subject: Re: ASCII-folded search [was: Re: Upcoming loss of usability ...]

 

> > I'm guessing that the feature you are talking about is the char
> > folding that Bruce/Artur installed, and *not* the suggestion by
> > Kaushal to "take over the mode line" and use "a lot of space to
> > verbosely display the searching options".

Just to be fair, it was I (Artur) who suggested taking over the mode line (and I've changed my mind against that now too). Kaushal's suggestion was more discrete. 

[-- Attachment #2: Type: text/html, Size: 6143 bytes --]

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> > What we should look for is both (1) a mirror pair of chars
> > that is not confusable with other uses of the same and
> > (2) a pair of chars that is simple to insert, search, etc. -
> > in general, simple to use in a rudimentary, plain-text
> > (perhaps even ASCII) context.
> 
> If you want mirror pairs, you should select only those pairs that
> are mentioned in the BidiBrackets.txt file that is part of the Unicode
> Character Database (UCD).  You can see them in Emacs by looking at
> characters that have non-nil 'paired-bracket' property.
> 
> If you select any other pairs, they will look wrong in R2L text.

Point taken.

But I'm not sure that what you say is true of "any other pairs",
where pair is regarded as two chars that we might consider using
as code/key phrase bookends.

I'm thinking of the pair ` and ', for instance.  Not exact mirror
images by any means, but workable.  The point is that there might
well be such pairs that are not problematic for Bidi - chars that
we humans might consider as pairable enough for our purposes.

Anyway, I'm not looking for a different pair from `...'.  If
someone finds one that satisfies the criteria, I'm open to it, of
course.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Mon, 29 Jun 2015 07:57:15 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: tsdh@gnu.org, dgutov@yandex.ru, eggert@cs.ucla.edu, rms@gnu.org,
>         emacs-devel@gnu.org, monnier@iro.umontreal.ca, acm@muc.de,
>         stephen@xemacs.org
> 
> But I'm not sure that what you say is true of "any other pairs",
> where pair is regarded as two chars that we might consider using
> as code/key phrase bookends.
> 
> I'm thinking of the pair ` and ', for instance.  Not exact mirror
> images by any means, but workable.

No, they aren't.  Here's how this will look in a right-to-left
paragraph:

`אבג'

(If this doesn't show in your mailer flushed all the way to the right
edge of the window, save this as a text file and visit that file in
Emacs.)

> The point is that there might well be such pairs that are not
> problematic for Bidi

I don't think so.  For them to not be problematic, they need to be
"mirrored" at display time -- and that only happens with characters in
BidiBrackets.txt.

> Anyway, I'm not looking for a different pair from `...'.  If
> someone finds one that satisfies the criteria, I'm open to it, of
> course.

Any pair in BidiBrackets.txt will do.  The question is: do we want to
use any of them?  I don't have an answer to that.

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> > But I'm not sure that what you say is true of "any other pairs",
> > where pair is regarded as two chars that we might consider using
> > as code/key phrase bookends.
> >
> > I'm thinking of the pair ` and ', for instance.  Not exact mirror
> > images by any means, but workable.
> 
> No, they aren't.  Here's how this will look in a right-to-left
> paragraph:
> 
> `אבג'
> 
> (If this doesn't show in your mailer flushed all the way to the
> right edge of the window, save this as a text file and visit that
> file in Emacs.)
> 
> > The point is that there might well be such pairs that are not
> > problematic for Bidi
> 
> I don't think so.  For them to not be problematic, they need to be
> "mirrored" at display time -- and that only happens with characters
> in BidiBrackets.txt.
> 
> > Anyway, I'm not looking for a different pair from `...'.  If
> > someone finds one that satisfies the criteria, I'm open to it, of
> > course.
> 
> Any pair in BidiBrackets.txt will do.  The question is: do we want
> to use any of them?  I don't have an answer to that.

Good to know.  So I guess you are saying that we already have a Bidi
problem wrt `...'.  And I guess your last question asks whether we
care about what happens to code enveloping by a char pair when RtL
is involved.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Mon, 29 Jun 2015 09:14:33 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: tsdh@gnu.org, dgutov@yandex.ru, eggert@cs.ucla.edu, rms@gnu.org,
>         emacs-devel@gnu.org, monnier@iro.umontreal.ca, acm@muc.de,
>         stephen@xemacs.org
> 
> > Any pair in BidiBrackets.txt will do.  The question is: do we want
> > to use any of them?  I don't have an answer to that.
> 
> Good to know.  So I guess you are saying that we already have a Bidi
> problem wrt `...'.

Yes.  But until now there was no requirement for the quote pair to be
a mirror pair.

> And I guess your last question asks whether we care about what
> happens to code enveloping by a char pair when RtL is involved.

I guess I'm asking what is really meant by "mirror pairs" in this
context.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Why are we concerned with Bidi for quotes in doc strings?
Doc strings should be written in English.

Of course, you CAN put in them whatever language you like; we don't
try to stop you.  But if you write them in some other language, you're
going against our conventions -- so there is no reason why other
conventions need to cater to that.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: drew.adams@oracle.com, eggert@cs.ucla.edu, tsdh@gnu.org,
> 	emacs-devel@gnu.org, monnier@iro.umontreal.ca,
> 	dgutov@yandex.ru, acm@muc.de, stephen@xemacs.org
> Date: Mon, 29 Jun 2015 17:52:45 -0400
> 
> Why are we concerned with Bidi for quotes in doc strings?
> Doc strings should be written in English.

Wouldn't we want to use the same quotes in tutorials, including
TUITORIAL.he?

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Why are we concerned with Bidi for quotes in doc strings?
  > > Doc strings should be written in English.

  > Wouldn't we want to use the same quotes in tutorials, including
  > TUITORIAL.he?

The tutorials are a totally different issue, in practical terms.  The
options available are different.

Tutorials are much a more flexible field.  There are just a few
tutorial files, all centrally maintained.  They don't have to follow
the conventions of Lisp code; we can make different conventions for
them.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: drew.adams@oracle.com, eggert@cs.ucla.edu, tsdh@gnu.org,
> 	emacs-devel@gnu.org, monnier@iro.umontreal.ca,
> 	dgutov@yandex.ru, acm@muc.de, stephen@xemacs.org
> Date: Tue, 30 Jun 2015 12:53:11 -0400
> 
>   > Wouldn't we want to use the same quotes in tutorials, including
>   > TUITORIAL.he?
> 
> The tutorials are a totally different issue, in practical terms.  The
> options available are different.

We currently use the same style of quoting there.  Quoting `like this'
in plain-text documents has been the GNU style for decades.

> Tutorials are much a more flexible field.  There are just a few
> tutorial files, all centrally maintained.  They don't have to follow
> the conventions of Lisp code; we can make different conventions for
> them.

Then my comment is limited to such documents, at least for now.
(There has been talk several times in the past about localizing
documentation in Emacs, but until there's a motivated volunteer
working on that, we can disregard that.)

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Eli Zaretskii wrote:
> Quoting `like this'
> in plain-text documents has been the GNU style for decades.

It certainly was, but other key GNU projects have switched to quoting 'like 
this' in ASCII English text files, and Emacs should do something similar. 
Several of Emacs's plain-text ASCII documents already do it that way, and at 
some point we should convert the remainder so that Emacs is consistent with 
itself and with other GNU projects.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Tue, 30 Jun 2015 15:20:49 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: tsdh@gnu.org, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
> 	dgutov@yandex.ru, acm@muc.de, stephen@xemacs.org, drew.adams@oracle.com
> 
> Eli Zaretskii wrote:
> > Quoting `like this'
> > in plain-text documents has been the GNU style for decades.
> 
> It certainly was, but other key GNU projects have switched to quoting 'like 
> this' in ASCII English text files, and Emacs should do something similar. 

Which is why I envisioned that soon enough we will want to use in the
tutorials the same quoting style as in *Help* buffers.  After all,
"C-h t" is a help command.  In fact, a few tutorials already do that.

That's why I mentioned the mirroring issue, which might become
relevant.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Paul Eggert]

Eli Zaretskii wrote:
> Which is why I envisioned that soon enough we will want to use in the
> tutorials the same quoting style as in *Help*  buffers.  After all,
> "C-h t" is a help command.  In fact, a few tutorials already do that.

The issue doesn't arise now with ‘C-h t’ in an English locale, since 
etc/tutorials/TUTORIAL quotes "like this" which is fine for an ASCII-only text 
file.  If this hypothetical issue ever happens I suppose we'll need to deal with 
it, but it's lower priority than the real quoting problems Emacs currently has.

Other languages can quote at will in tutorials using language-specific styles, 
e.g., TUTORIAL.de quotes »like this«.  This is fine too of course.

Re: Upcoming loss of usability of Emacs source files and Emacs.

[Eli Zaretskii]

> Date: Wed, 01 Jul 2015 12:33:14 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: rms@gnu.org, tsdh@gnu.org, emacs-devel@gnu.org, 
>  monnier@iro.umontreal.ca, dgutov@yandex.ru, acm@muc.de, 
>  stephen@xemacs.org, drew.adams@oracle.com
> 
> Eli Zaretskii wrote:
> > Which is why I envisioned that soon enough we will want to use in the
> > tutorials the same quoting style as in *Help*  buffers.  After all,
> > "C-h t" is a help command.  In fact, a few tutorials already do that.
> 
> The issue doesn't arise now with ‘C-h t’ in an English locale, since 
> etc/tutorials/TUTORIAL quotes "like this" which is fine for an ASCII-only text 
> file.

As long as we don't decide to switch to “like this”, yes.

RE: Upcoming loss of usability of Emacs source files and Emacs.

[Drew Adams]

> I guess I'm asking what is really meant by "mirror pairs" in this
> context.

Well, if it's any help (since I guess I introduced "mirror pairs"
to the conversation), all I meant was a pair of chars that "we"
think could be appropriate to set off code etc. char sequences from
the surrounding text.

My intention in using that term was just to contrast the use of two
different chars from use of the same char twice: '...', `...`, etc.

Personally, I don't think it's necessarily important that the two
different chars be visually exact mirror images in any sense.

(Certain quotation-mark pairs are anyway not left-right mirror
images.)