bug#24510: 25.1; Info: searching for ` does not find what looks like `

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> > Again, this is emacs -Q.
> 
> That doesn't matter, since Emacs doesn't control the fonts
> installed on the user's system, much less how they look.

IIUC, Emacs tries to influence the default font, even if it
cannot completely control what it will be.  And the font in
question is one of the most common (on MS Windows).

> Like I said: if some
> user cannot distinguish between these characters, I see no other
> solution for the problem except font customizations, since only the
> user on her system can DTRT with the fonts that are available.

Are you saying that Emacs has no role in guiding the default
font toward being Courier New on MS Windows - that there is no
way to guide it toward being another font?  Emacs has no influence
at all on which font is used by default?

In that case, indeed, we have a usability problem, Houston,
since this is a very common fixed-width font.

> > The chars look quite different with my own setup - this is
> > not a problem for me, personally.
> 
> Then what exactly is the nature of the problem you raise?

emacs -Q on MS Windows, as stated.  It's not about me (or
my setup).  I don't use emacs -Q, except to look into bugs.

> Is it a hypothetical issue that _might_ happen on someone's
> machine, even though none of us, including yourself, have
> actually witnessed it?

It is a real problem, which I witness systematically with
emacs -Q.  It is, I expect, a real problem for many, even
most, users on MS Windows.

Why do you insist on saying things like "none of us,
including yourself, have actually witnessed it?"

What is the point of such a preposterous claim?  Clearly
I _have_ witnessed the problem and reported it, including
showing a screenshot.  If you refuse to see that the
emperor has no clothes, that's not my fault.  I and my
little camera are witness.

> > > It might, if you turn on character-folding in Isearch.
> > > Unfortunately, that is off by default.
> >
> > I did mention that this is what a user sees by DEFAULT, no?
> 
> Weren't you one of those who lobbied for turning the character-folding
> off by default?  If it were not for that lobbying, your search would
> have succeeded.

No.  And there was no such "lobbying", by anyone.  Can you point
to any post by me, in the long and deep discussion about this on
emacs-devel ("On language-dependent defaults for character-folding")
where I did that?
 
And why would it be important if I had?  Are you trying to
mete out punishment, saying that it is my fault that Emacs
now presents, by default, a visual confusion of single-quoting?
You didn't get your way, but that's not my fault.

FYI, this is what I _actually_ said, in reply to your insistence
that the feature be on by default for 25.1, after the consensus
seemed to have been reached to leave it off by default (there
was one dissenter -you- and several, like me, for whom the
default behavior was much less important than other questions):

d>  I don't see anyone proposing to throw out the bathwater,
d>  much less the baby with it.
d>  
d>  Eli, you say here, quite often, that you think discussions
d>  about what the default behavior of a feature should be are
d>  typically fruitless, if not sterile.  But it seems clear
d>  that you care quite a lot about this default behavior.
d>  
d>  I'd say let it go.  There will be Emacs 25.2 and beyond.
d>  And users will try this new feature and give their feedback,
d>  which I expect will be overwhelmingly positive - and
d>  informative for further discussions here.
d>  
d>  Based on user feedback and further discussion and analysis
d>  here (this is not going away), Emacs Dev will improve and
d>  elaborate this feature.  We will have better ideas about how
d>  to handle all of the things that are currently not so clear.
d>  There is plenty of time to decide again whether this or that
d>  should be turned on by default.
d>  
d>  What seems clear to me for Emacs 25.1 is that the feature
d>  should be included AND that it should be simple to both (1)
d>  customize the default behavior for a given user (i.e., what
d>  behavior search starts with, a la `case-fold-search') and
d>  (2) toggle the behavior on the fly, during Isearch.
d>  
d>  Given (1) and (2), users can do what they like, and we can
d>  learn later from them what behaviors might best be adopted
d>  for defaulting.

And this is John's response to that:

 d> What seems clear to me for Emacs 25.1 is that the feature
 d> should be included AND that it should be simple to both (1)
 d> customize the default behavior for a given user (i.e., what
 d> behavior search starts with, a la `case-fold-search') and
 d> (2) toggle the behavior on the fly, during Isearch.
j>
j> I think Drew has summarized perfectly what I would like to
j> see happen. In addition, I'd add one more item: Once 25.1 is
j> released, I (or another) will write a blog article publicizing
j> this feature and touting its benefits, in order to encourage
j> people to try it out and discover how useful it can be.
j> 
j> However, making it a default in 25.1 is something I am simply
j> not comfortable doing, giving the diversity of opinion on this
j> list, plus my own misgivings about so new (and nuanced) a
j> feature. Yes, the visual equality of á and á is a powerful
j> argument, but as Drew said, there will be well-advertised ways
j> to both enable this feature, and to toggle it while searching.
j> Users will not lose any capacity by our decision, they will
j> simply not experience it as a default out of the box.
j> 
j> And so, my decision is that this feature will be off by default
j> in the 25.1 release, with the genuine hope that it can be made
j> solid enough to become a default in a future release. It
j> needn't even wait until 26.1, if we receive enough positive feedback.
j> 
j> My thanks to everyone for the extensive and conscientious debate,
j> and to Eli for sticking to his guns.  I am hopeful we will reach
j> general consensus over time, and that this feature will come to
j> be recognized as a compelling aspect of the Emacs feature set.
j> Until that day, please forgive me my reservations; I'm just not
j> there yet in wanting this to become a default behavior.

"Sticking to his guns" was a "nice" way to put it.
And he seems to still be brandishing them.

> > > Emacs doesn't provide any fonts.  Your system does.  On my system,
> > > these two characters look different, even with my old weary eyes.
> >
> > They look slightly different, yes.  Barely noticeable when you
> > see them side by side (see screenshot).
> 
> The look quite different to me (my system uses the same font),

You _know_ they are different.  Please look again at the screenshot
I sent, and imagine that it does not include the window showing the
two right next to each other, so you see what a user sees when s?he
searches for `.

It is not at all obvious, I would argue, that the ` in the Isearch
search string is a different character from the left quotation mark
in the *info* buffer.  And things are even less obvious if the user
cannot immediately see an occurrence of the latter character - e.g.,
when starting a search across nodes.

> so I guess we will have to agree to disagree about that.

Indeed we shall.

> I'm not dismissing the problem.  I'm saying that I don't see a
> solution except through user customizations.  Searching for similarly
> looking characters is not specific to Info, it can happen elsewhere.
> Users who bump into that frequently should customize their fonts
> and/or turn on character-folding in search commands.

That's not a solution for the default behavior, which is what
this bug is about.  If that is what you propose to solve the
problem here then you are indeed dismissing the problem.

Two possible solutions have been suggested in this thread, so far:

1) Emacs can try to avoid using Courier New as the default font.
2. Isearch can fold "single-quote" chars (i.e., chars that could
   be confused in this regard), by default.

Please note, BTW, that I have _not_ opposed #2, just as I did
_not_ "lobby for turning the character-folding off by default"
during the general emacs-devel discussion.

I think #1 is generally preferable.  Other solution proposals
are welcome.  What's not helpful is dismissing the problem or
casting blame for it.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Date: Fri, 23 Sep 2016 09:02:32 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 24510@debbugs.gnu.org
> 
> > > Again, this is emacs -Q.
> > 
> > That doesn't matter, since Emacs doesn't control the fonts
> > installed on the user's system, much less how they look.
> 
> IIUC, Emacs tries to influence the default font, even if it
> cannot completely control what it will be.  And the font in
> question is one of the most common (on MS Windows).

It is by far the most common and the best, which is why Emacs uses it
by default.

> > Like I said: if some
> > user cannot distinguish between these characters, I see no other
> > solution for the problem except font customizations, since only the
> > user on her system can DTRT with the fonts that are available.
> 
> Are you saying that Emacs has no role in guiding the default
> font toward being Courier New on MS Windows - that there is no
> way to guide it toward being another font?  Emacs has no influence
> at all on which font is used by default?

No, I'm saying Emacs has no control on how different characters in
different fonts look on different platforms.  We use the best fonts
that are commonly available, but that's only the best on average, it
doesn't guarantee everyone will be happy.  That's where customizations
come into play.

> emacs -Q on MS Windows, as stated.  It's not about me (or
> my setup).  I don't use emacs -Q, except to look into bugs.

Like I said: a non-existent problem.  One more opportunity for you to
raise the old problem and request that Cartage be destroyed.

> Why do you insist on saying things like "none of us,
> including yourself, have actually witnessed it?"

Because you yourself just said that the characters look different to
you.

> > Weren't you one of those who lobbied for turning the character-folding
> > off by default?  If it were not for that lobbying, your search would
> > have succeeded.
> 
> No.  And there was no such "lobbying", by anyone.  Can you point
> to any post by me, in the long and deep discussion about this on
> emacs-devel ("On language-dependent defaults for character-folding")
> where I did that?

You just did it yourself, so I can rest my case.

> And why would it be important if I had?  Are you trying to
> mete out punishment, saying that it is my fault that Emacs
> now presents, by default, a visual confusion of single-quoting?

I'm saying that yours are double standards.

> > The look quite different to me (my system uses the same font),
> 
> You _know_ they are different.  Please look again at the screenshot
> I sent, and imagine that it does not include the window showing the
> two right next to each other, so you see what a user sees when s?he
> searches for `.

I did.  They still look different.

> > I'm not dismissing the problem.  I'm saying that I don't see a
> > solution except through user customizations.  Searching for similarly
> > looking characters is not specific to Info, it can happen elsewhere.
> > Users who bump into that frequently should customize their fonts
> > and/or turn on character-folding in search commands.
> 
> That's not a solution for the default behavior, which is what
> this bug is about.  If that is what you propose to solve the
> problem here then you are indeed dismissing the problem.

Customizations exist in Emacs for a reason.  Saying that every problem
must be solved in "emacs -Q" is absurd, especially for minor problems
such as this one, with 2 characters that look differently.

> Two possible solutions have been suggested in this thread, so far:
> 
> 1) Emacs can try to avoid using Courier New as the default font.

Not an option.  Other fonts have much less coverage, and most of them
are much uglier.  You may wish looking at them before you suggest
this.

> 2. Isearch can fold "single-quote" chars (i.e., chars that could
>    be confused in this regard), by default.

I'm okay with that, but my opinions on that were overruled.

> Please note, BTW, that I have _not_ opposed #2, just as I did
> _not_ "lobby for turning the character-folding off by default"
> during the general emacs-devel discussion.

Sure, you did.  You just prefer to deny it now.

> What's not helpful is dismissing the problem or casting blame for
> it.

I agree.  Do you realize that this is what you are doing?

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 5717 bytes --]

On 2016-09-23 13:48, Eli Zaretskii wrote:
>> > 1) Emacs can try to avoid using Courier New as the default font.
>
> Not an option.  Other fonts have much less coverage, and most of them
> are much uglier.  You may wish looking at them before you suggest
> this.

Hi Eli,

I have not used MS Windows much in a while, but one thing I missed after I switched to GNU/Linux was Microsoft's excellent Consolas font. It was first included with Windows XP, if I remember correctly.  Do you think Emacs could try this one instead of Courier New by default?  I think both Visual Studio and Atom use it by default on Windows.

Here's how the two fonts compare in terms of coverage; I hope it's helpful!

    # Only in Courier New:
        ˉְֱֲֳִֵֶַָֹֻּֽ־ֿ׀ׁׂ׃אבגדהוזחטיךכלםמןנסעףפץצקרשתװױײ׳״،؛؟ءآأؤإئابةتثجحخدذرزسشصضطظع
        غـفقكلمنهوىيًٌٍَُِّْ٠١٢٣٤٥٦٧٨٩٪٫٭ٰٱٴپچژڤکگیەۤ۴۵۶′
        ﬠשׁשׂשּׁשּׂאַאָאּבּגּדּהּוּזּטּיּךּכּלּמּנּסּףּפּצּקּרּשּתּוֹבֿכֿפֿﭏﭑﭗﭘﭙﭫﭬﭭﭻﭼﭽﮋﮏﮐﮑ
        ﮓﮔﮕﯽﯾﯿﰈﰉﰎﰒﰱﰲﰿﱀﱁﱂﱃﱄﱎﱏﱘﱙﱞﱟﱠﱡﱢﱪﱭﱮﱯﱰﱳﱴﱵﲎﲏﲑﲔﲜﲝﲞﲟﲠﲡﲢﲣﲤﲥﲦﲨﲪﲬﲰﳉﳊﳋﳌﳍﳎﳏﳐﳑﳒﳓﳔﳕﳘﳚﳛﳜﳝﴰﴼﴽ﴾﴿ﶈﷲﺀ
        ﺂﺄﺆﺈﺊﺋﺌﺎﺐﺑﺒﺔﺖﺗﺘﺚﺛﺜﺞﺟﺠﺢﺣﺤﺦﺧﺨﺪﺬﺮﺰﺲﺳﺴﺶﺷﺸﺺﺻﺼﺾﺿﻀﻂﻃﻄﻆﻇﻈﻊﻋﻌﻎﻏﻐﻒﻓﻔﻖﻗﻘﻚﻛﻜﻞﻟﻠﻢﻣﻤﻦﻧﻨﻪﻫﻬﻮﻰﻲﻵ
        ﻶﻷﻸﻹﻺﻻﻼ

    # Only in Consolas:
        ƀƁƂƃƄƅƆƇƈƉƊƋƌƍƎƐƑƓƔƕƖƗƘƙƚƛƜƝƞƟƢƣƤƥƦƧƨƩƪƫƬƭƮƱƲƳƴƵƶƷƸƹƺƻƼƽƾƿǀǁǂǃDŽDždžLJLjljNJNjnjǝǞǟǠǡǢǣǤ
        ǥǦǧǨǩǪǫǬǭǮǯǰDZDzdzǴǵǶǷǸǹȀȁȂȃȄȅȆȇȈȉȊȋȌȍȎȏȐȑȒȓȔȕȖȗȘșȚțȜȝȞȟȠȡȢȣȤȥȦȧȨȩȪȫȬȭȮȯȰȱȲȳȴȵȶȷȸȹȺ
        ȻȼȽȾȿɀɁɂɃɄɅɆɇɈɉɊɋɌɍɎɏɐɑɒɓɔɕɖɗɘɚɛɜɝɞɟɠɡɢɣɤɥɦɧɨɩɪɫɬɭɮɯɰɱɲɳɴɵɶɷɸɹɺɻɼɽɾɿʀʁʂʃʄʅʆʇʈʉʊʋ
        ʌʍʎʏʐʑʒʓʔʕʖʗʘʙʚʛʜʝʞʟʠʡʢʣʤʥʦʧʨʩʪʫʬʭʮʯʰʱʲʳʴʵʶʷʸʹʺʻʼʽʾʿˀˁ˂˃˄˅ˈˊˋˌˍˎˏːˑ˒˓˔˕˖˗˞˟ˠˡˢˣˤ
        ˥˦˧˨˩˪˫ˬ˭ˮ˯˰˱˲˳˴˵˶˷˸˹˺˻˼˽˾˿̴̵̶̷̸̡̢̧̨̛̖̗̘̙̜̝̞̟̠̤̥̦̩̪̫̬̭̮̯̰̱̲̳̹̂̄̅̆̇̈̊̋̌̍̎̏̐̑̒̓̔̕̚
        ̺̻̼͇͈͉͍͎̽̾̿̀́͂̓̈́͆͊͋͌ͅ͏͓͔͕͖͙͚͐͑͒͗͛ͣͤͥͦͧͨͩͪͫͬͭͮͯ͘͜͟͢͝͞͠͡ʹ͵ͺͻͼͽ;ϐϑϒϓϔϕϖϗϘϙϚϛϜϝϞϟϠϡϢ
        ϣϤϥϦϧϨϩϪϫϬϭϮϯϰϱϲϳϴϵ϶ϷϸϹϺϻϼϽϾϿЀЍѐѝѠѡѢѣѤѥѦѧѨѩѪѫѬѭѮѯѰѱѲѳѴѵѶѷѸѹѺѻѼѽѾѿҀҁ҂҃҄҅҆҈҉ҊҋҌҍҎҏ
        ҔҕҘҙҞҟҠҡҤҥҦҧҨҩҪҫҬҭҴҵҶҷҼҽҾҿӀӁӂӃӄӅӆӇӈӉӊӋӌӍӎӏӐӑӒӓӔӕӖӗӘәӚӛӜӝӞӟӠӡӢӣӤӥӦӧӪӫӬӭӮӯӰӱӲӳӴӵӶӷ
        ӸӹӺӻӼӽӾӿԀԁԂԃԄԅԆԇԈԉԊԋԌԍԎԏԐԑԒԓ฿ᴀᴁᴂᴃᴄᴅᴆᴇᴈᴉᴊᴋᴌᴍᴎᴏᴐᴑᴒᴓᴔᴕᴖᴗᴘᴙᴚᴛᴜᴝᴞᴟᴠᴡᴢᴣᴤᴥᴦᴧᴨᴩᴪᴫᴬᴭᴮᴯᴰᴱᴲ
        ᴳᴴᴵᴶᴷᴸᴹᴺᴻᴼᴽᴾᴿᵀᵁᵂᵃᵄᵅᵆᵇᵈᵉᵊᵋᵌᵍᵎᵏᵐᵑᵒᵓᵔᵕᵖᵗᵘᵙᵚᵛᵜᵝᵞᵟᵠᵡᵢᵣᵤᵥᵦᵧᵨᵩᵪᵫᵬᵭᵮᵯᵰᵱᵲᵳᵴᵵᵶᵷᵸᵹᵺᵻᵼᵽᵾᵿᶀᶁᶂ
        ᶃᶄᶅᶆᶇᶈᶉᶊᶋᶌᶍᶎᶏᶐᶑᶒᶓᶔᶕᶖᶗᶘᶙᶚᶛᶜᶝᶞᶟᶠᶡᶢᶣᶤᶥᶦᶧᶨᶩᶪᶫᶬᶭᶮᶯᶰᶱᶲᶳᶴᶵᶶᶷᶸᶹᶺᶻᶼᶽᶾᶿ᷂᷊᷿᷀᷁᷃᷄᷅᷆᷇᷈᷉᷾ḀḁḂḃḄḅ
        ḆḇḈḉḊḋḌḍḎḏḐḑḒḓḔḕḖḗḘḙḚḛḜḝḞḟḠḡḢḣḤḥḦḧḨḩḪḫḬḭḮḯḰḱḲḳḴḵḶḷḸḹḺḻḼḽḾḿṀṁṂṃṄṅṆṇṈṉṊṋṌṍṎṏṐṑṒṓṔṕ
        ṖṗṘṙṚṛṜṝṞṟṠṡṢṣṤṥṦṧṨṩṪṫṬṭṮṯṰṱṲṳṴṵṶṷṸṹṺṻṼṽṾṿẆẇẈẉẊẋẌẍẎẏẐẑẒẓẔẕẖẗẘẙẚẛẞἀἁἂἃἄἅἆἇἈἉἊἋἌἍἎ
        ἏἐἑἒἓἔἕἘἙἚἛἜἝἠἡἢἣἤἥἦἧἨἩἪἫἬἭἮἯἰἱἲἳἴἵἶἷἸἹἺἻἼἽἾἿὀὁὂὃὄὅὈὉὊὋὌὍὐὑὒὓὔὕὖὗὙὛὝὟὠὡὢὣὤὥὦὧὨὩὪ
        ὫὬὭὮὯὰάὲέὴήὶίὸόὺύὼώᾀᾁᾂᾃᾄᾅᾆᾇᾈᾉᾊᾋᾌᾍᾎᾏᾐᾑᾒᾓᾔᾕᾖᾗᾘᾙᾚᾛᾜᾝᾞᾟᾠᾡᾢᾣᾤᾥᾦᾧᾨᾩᾪᾫᾬᾭᾮᾯᾰᾱᾲᾳᾴᾶᾷᾸᾹᾺΆᾼ᾽
        ᾿῀῁ῂῃῄῆῇῈΈῊΉῌ῍῎῏ῐῑῒΐῖῗῘῙῚΊ῝῞῟ῠῡῢΰῤῥῦῧῨῩῪΎῬ῭΅`ῲῳῴῶῷῸΌῺΏῼ´῾           ​‒‖‟ ‴‽⁃⁞ ⁰ⁱ
        ⁴⁵⁶⁷⁸⁹⁺⁻⁼⁽⁾₀₁₂₃₄₅₆₇₈₉₊₋₌₍₎ₐₑₒₓₔ₠₡₢₥₦₨₩₭₮₯₰₱₲₳₴₵₸₹₺⃝℗Ⅎ⅍ⅎ⅕⅖⅗⅘⅙⅚Ↄↄ∕①②③④⑤⑥⑦⑧⑨⑩⑪⑫⑬⑭⑮⑯
        ⑰⑱⑲⑳⓪⓫⓬⓭⓮⓯⓰⓱⓲⓳⓴⓿▴▸▾◂◌✶❶❷❸❹❺❻❼❽❾❿ⱠⱡⱢⱣⱤⱥⱦⱧⱨⱩⱪⱫⱬⱴⱵⱶⱷ⸗ꜗꜘꜙꜚ꜠꜡fifl︠︡︢︣

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Fri, 23 Sep 2016 16:07:51 -0400
> 
> >> > 1) Emacs can try to avoid using Courier New as the default font.
> >
> > Not an option.  Other fonts have much less coverage, and most of them
> > are much uglier.  You may wish looking at them before you suggest
> > this.
> 
> Hi Eli,
> 
> I have not used MS Windows much in a while, but one thing I missed after I switched to GNU/Linux was Microsoft's excellent Consolas font. It was first included with Windows XP, if I remember correctly.  Do you think Emacs could try this one instead of Courier New by default?  I think both Visual Studio and Atom use it by default on Windows.
> 
> Here's how the two fonts compare in terms of coverage; I hope it's helpful!

I hope, based on the coverage you presented, it should be clear that
Consolas is worse than Courier New.  Non-support of Arabic script is a
non-starter these days.  On my system, Consolas supports 16 Unicode
blocks and 598 characters, while Courier New supports 29 blocks and
1258 characters, almost twice as many.  Lucida Console suggested by
Drew supports 22 blocks and 663 characters, which is also too few.
(On Windows 8.1, the respective numbers are 38/2220, 41/3025, and
21/644 -- as you see, the numbers for Consolas are significantly
larger, but the growth is mainly due to symbols and Latin extensions,
it is still pretty much a Europe-centric font).

In terms of coverage, the best fonts available on Windows are Arial MS
Unicode, Cambria, and Times New Roman, but they are proportional
fonts, whereas we need a monospaced font.  (Cambria is also
unavailable before Vista.)

Symbola is my favorite as far as coverage goes (71 blocks and 7873
characters), but it is not installed by default on Windows systems.

So I don't think we have a candidate font on Windows that is better
than the current default.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1970 bytes --]

On 2016-09-24 03:28, Eli Zaretskii wrote:
> I hope, based on the coverage you presented, it should be clear that 
> Consolas is worse than Courier New.  Non-support of Arabic script is
> a non-starter these days.  On my system, Consolas supports 16
> Unicode blocks and 598 characters, while Courier New supports 29
> blocks and 1258 characters, almost twice as many.

> […] So I don't think we have a candidate font on Windows that is
> better than the current default.

Hi Eli,

I'm not sure it makes sense to equate "better" and "worse" with more or less coverage.   Is Consolas worse than Courier New in terms of number of characters supported?  Certainly.  But then why is Emacs not defaulting to MingLiu or NSimSun?  They are both monospace, both available in recent releases of Microsoft Windows, and they both cover multiple East-Asian scripts, (Courier New doesn't).  On my system, MingLiu supports 28955 glyphs; over 23 times as many as Courier New.

Do we have evidence that users of Emacs on Windows write significant amounts of code in Arabic, to the point that we would want to use Courier New as the default, instead of as a fallback?  If so, do we have evidence that more code is written in Emacs in Arabic than in Chinese and Japanese?  And finally, do we have evidence that users of scripts that Consolas does not support prefer having Courier New as the default, rather than Consolas with a fallback to Courier New, or MingLiu?

I'm not too familiar with editors on Windows.  Do many of them default to Courier New?  AFAICT Netbeans apparently does, but Atom doesn't (it uses Consolas), Visual Studio doesn't (it uses Consolas), Sublime Text doesn't (it uses Consolas), Eclipse doesn't (it changed from Courier New to Consolas in 2011), and Notepad++ doesn't (it changed from Courier New to Source Code Pro in 2015).  What makes things that are acceptable for so many other editors non-starters for us?

Cheers,
Clément. 




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Cc: 24510@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 24 Sep 2016 12:30:07 -0400
> 
> I'm not sure it makes sense to equate "better" and "worse" with more or less coverage.   Is Consolas worse than Courier New in terms of number of characters supported?  Certainly.  But then why is Emacs not defaulting to MingLiu or NSimSun?  They are both monospace, both available in recent releases of Microsoft Windows, and they both cover multiple East-Asian scripts, (Courier New doesn't).  On my system, MingLiu supports 28955 glyphs; over 23 times as many as Courier New.

Not just the number of characters matters: the number of Unicode
blocks also matters, maybe even more.  Each block is some script, so
supporting less blocks means less scripts supported by the default
font.  Emacs will have then look for a different font, which makes
less pleasant display, creates text alignment problems, etc.

> Do we have evidence that users of Emacs on Windows write significant amounts of code in Arabic, to the point that we would want to use Courier New as the default, instead of as a fallback?  If so, do we have evidence that more code is written in Emacs in Arabic than in Chinese and Japanese?  And finally, do we have evidence that users of scripts that Consolas does not support prefer having Courier New as the default, rather than Consolas with a fallback to Courier New, or MingLiu?

I'm not sure I understand where these questions go.  We don't have
evidence either way, so the issues you raise cannot help us make the
decision.

I personally consider Consolas worse than Courier New, because
Consolas's coverage is clearly biased towards European scripts.  Also,
Emacs has used Courier New for a couple of decades, so changing that
would need a good reason, not some vague doubts that we cannot resolve
based on data.

> I'm not too familiar with editors on Windows.  Do many of them default to Courier New?  AFAICT Netbeans apparently does, but Atom doesn't (it uses Consolas), Visual Studio doesn't (it uses Consolas), Sublime Text doesn't (it uses Consolas), Eclipse doesn't (it changed from Courier New to Consolas in 2011), and Notepad++ doesn't (it changed from Courier New to Source Code Pro in 2015).  What makes things that are acceptable for so many other editors non-starters for us?

One reason is that Emacs has a wider range of different applications,
where being able to support as many languages and scripts as possible
is more important than in Studio or even Vim.  E.g., at least some of
the editors you mention are never used as email/news clients or Web
browsers, where the ability to support as many scripts as possible is
important.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 3758 bytes --]

On 2016-09-24 13:08, Eli Zaretskii wrote:
>> Cc: 24510@debbugs.gnu.org From: Clément Pit--Claudel
>> <clement.pit@gmail.com> Date: Sat, 24 Sep 2016 12:30:07 -0400
>> 
>> I'm not sure it makes sense to equate "better" and "worse" with
>> more or less coverage.   Is Consolas worse than Courier New in
>> terms of number of characters supported?  Certainly.  But then why
>> is Emacs not defaulting to MingLiu or NSimSun?  They are both
>> monospace, both available in recent releases of Microsoft Windows,
>> and they both cover multiple East-Asian scripts, (Courier New
>> doesn't).  On my system, MingLiu supports 28955 glyphs; over 23
>> times as many as Courier New.
> 
> Not just the number of characters matters: the number of Unicode 
> blocks also matters, maybe even more.  Each block is some script, so 
> supporting less blocks means less scripts supported by the default 
> font.  Emacs will have then look for a different font, which makes 
> less pleasant display, creates text alignment problems, etc.

Thanks for the explanation. But MingLiu supports 34 blocks, while Courier New supports 23.  In particular, MingLiu has support for traditional Chinese and Japanese Hiragana and Katakana, which Courier New lacks.

>> Do we have evidence that users of Emacs on Windows write
>> significant amounts of code in Arabic, to the point that we would
>> want to use Courier New as the default, instead of as a fallback?
>> If so, do we have evidence that more code is written in Emacs in
>> Arabic than in Chinese and Japanese?  And finally, do we have
>> evidence that users of scripts that Consolas does not support
>> prefer having Courier New as the default, rather than Consolas with
>> a fallback to Courier New, or MingLiu?
> 
> I'm not sure I understand where these questions go.  We don't have 
> evidence either way, so the issues you raise cannot help us make the 
> decision.

The question was whether we had evidence in one way, and you answered it; thanks!
I don't know about either way, though: we do have evidence that many other editors don't do the same thing as we do.

> I personally consider Consolas worse than Courier New, because 
> Consolas's coverage is clearly biased towards European scripts.

True.  But Courier is biased too.  In fact, virtually all programming fonts are biased in that way, maybe as an unfortunate artifact of most programming languages themselves being biased towards ASCII.  In selecting the default Emacs font, we're trying to ensure that users have an agreeable experience: if most of what they look at in a monospace font is covered by ASCII, then it's not clear to me that it makes sense to select a monospace font based on coverage only.

>> I'm not too familiar with editors on Windows.  Do many of them
>> default to Courier New?  AFAICT Netbeans apparently does, but Atom
>> doesn't (it uses Consolas), Visual Studio doesn't (it uses
>> Consolas), Sublime Text doesn't (it uses Consolas), Eclipse doesn't
>> (it changed from Courier New to Consolas in 2011), and Notepad++
>> doesn't (it changed from Courier New to Source Code Pro in 2015).
>> What makes things that are acceptable for so many other editors
>> non-starters for us?
> 
> One reason is that Emacs has a wider range of different
> applications, where being able to support as many languages and
> scripts as possible is more important than in Studio or even Vim.
> E.g., at least some of the editors you mention are never used as
> email/news clients or Web browsers, where the ability to support as
> many scripts as possible is important.

Do many of these applications require a monospace font?  Eww now defaults to a proportional face, doesn't it?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Cc: 24510@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 24 Sep 2016 13:24:50 -0400
> 
> > Not just the number of characters matters: the number of Unicode 
> > blocks also matters, maybe even more.  Each block is some script, so 
> > supporting less blocks means less scripts supported by the default 
> > font.  Emacs will have then look for a different font, which makes 
> > less pleasant display, creates text alignment problems, etc.
> 
> Thanks for the explanation. But MingLiu supports 34 blocks, while Courier New supports 23.  In particular, MingLiu has support for traditional Chinese and Japanese Hiragana and Katakana, which Courier New lacks.

Out of 34 blocks MingLiu supports, one third is CJK blocks.  The other
blocks support Latin and Greek scripts, and that's about all.

We don't use Courier New for CJK anyway, as it doesn't cover those
well.  But neither does Consolas, and I find it hard to believe you
are saying that we should switch from Courier New to MingLiu as the
default font: that'd be absurd.

> > I personally consider Consolas worse than Courier New, because 
> > Consolas's coverage is clearly biased towards European scripts.
> 
> True.  But Courier is biased too.

It's less biased, though.

> In fact, virtually all programming fonts are biased in that way, maybe as an unfortunate artifact of most programming languages themselves being biased towards ASCII.  In selecting the default Emacs font, we're trying to ensure that users have an agreeable experience: if most of what they look at in a monospace font is covered by ASCII, then it's not clear to me that it makes sense to select a monospace font based on coverage only.

Emacs supports non-programming applications as well, not just
programming modes.  In fact, the trigger for this discussion was Info,
a non-programming mode very close to Text mode.

> > One reason is that Emacs has a wider range of different
> > applications, where being able to support as many languages and
> > scripts as possible is more important than in Studio or even Vim.
> > E.g., at least some of the editors you mention are never used as
> > email/news clients or Web browsers, where the ability to support as
> > many scripts as possible is important.
> 
> Do many of these applications require a monospace font?

We use a monospaced font for the default face because it suits well
both programming and non-programming modes, and because aligning text
is much easier with such a font.  Text alignment is important in modes
that present summaries in tabular form, like modes that show listing
of buffers, email summaries, Dired, Proced, etc.

> Eww now defaults to a proportional face, doesn't it?

By default, yes; but there's "M-x eww-toggle-fonts".

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1529 bytes --]

On 2016-09-24 13:57, Eli Zaretskii wrote:
> I find it hard to believe you
> are saying that we should switch from Courier New to MingLiu as the
> default font: that'd be absurd.

It doesn't feel more more absurd than using Courier New in the first place, frankly.  But sorry for diverting the thread.  I don't use Emacs on Windows, so I don't really care about good defaults there.

> Emacs supports non-programming applications as well, not just
> programming modes.  In fact, the trigger for this discussion was Info,
> a non-programming mode very close to Text mode.

Why does info use a monospace font?

>>> One reason is that Emacs has a wider range of different
>>> applications, where being able to support as many languages and
>>> scripts as possible is more important than in Studio or even Vim.
>>> E.g., at least some of the editors you mention are never used as
>>> email/news clients or Web browsers, where the ability to support as
>>> many scripts as possible is important.
>>
>> Do many of these applications require a monospace font?
> 
> We use a monospaced font for the default face because it suits well
> both programming and non-programming modes, and because aligning text
> is much easier with such a font.  Text alignment is important in modes
> that present summaries in tabular form, like modes that show listing
> of buffers, email summaries, Dired, Proced, etc.

Doesn't we the special 'space' display property make it possible to properly align variable-width fonts?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Cc: 24510@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 24 Sep 2016 14:14:52 -0400
> 
> On 2016-09-24 13:57, Eli Zaretskii wrote:
> > I find it hard to believe you
> > are saying that we should switch from Courier New to MingLiu as the
> > default font: that'd be absurd.
> 
> It doesn't feel more more absurd than using Courier New in the first place, frankly.

It does to me.

> > Emacs supports non-programming applications as well, not just
> > programming modes.  In fact, the trigger for this discussion was Info,
> > a non-programming mode very close to Text mode.
> 
> Why does info use a monospace font?

Because Texinfo produces filled text on the assumption that it will be
displayed with monospaced font.  The Info format was invented for
text-mode readers, and it uses hard newlines where the line should be
broken.  This doesn't work well with variable-pitch fonts.

> > We use a monospaced font for the default face because it suits well
> > both programming and non-programming modes, and because aligning text
> > is much easier with such a font.  Text alignment is important in modes
> > that present summaries in tabular form, like modes that show listing
> > of buffers, email summaries, Dired, Proced, etc.
> 
> Doesn't we the special 'space' display property make it possible to properly align variable-width fonts?

Of course, it does.  But most Lisp applications don't bother doing
that, even though the facility is available since Emacs 21.  Counting
columns is so much easier.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 795 bytes --]

On 2016-09-24 14:34, Eli Zaretskii wrote:
>>> > > Emacs supports non-programming applications as well, not just
>>> > > programming modes.  In fact, the trigger for this discussion was Info,
>>> > > a non-programming mode very close to Text mode.
>> > 
>> > Why does info use a monospace font?
> Because Texinfo produces filled text on the assumption that it will be
> displayed with monospaced font.  The Info format was invented for
> text-mode readers, and it uses hard newlines where the line should be
> broken.  This doesn't work well with variable-pitch fonts.

Ok.  In that case, maybe we should just change the default font of the Info reader.  Do we know of any Info manuals written in Arabic or Hebrew?  If not, is there a point to using Courier New there?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Cc: 24510@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 24 Sep 2016 14:40:54 -0400
> 
> In that case, maybe we should just change the default font of the Info reader.

On MS-Windows only?  Doesn't sound like important enough to me.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> I'm not too familiar with editors on Windows.  Do many of them
> default to Courier New?

I too am not very familiar with other editors on MS Windows.

I can mention that TextPad (the only other editor I sometimes
use on Windows) seems to default to Courier New.  (But it also
handles multiple languages differently, I believe.)

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> > Why do you insist on saying things like "none of us,
> > including yourself, have actually witnessed it?"
> 
> Because you yourself just said that the characters look
> different to you.

No.  I said clearly that they can easily be mistaken for each
other, even though it is true that they are not _exactly_ the
same visually:

  See attached screenshot.  No, they are not EXACTLY the same.  But
  a user who is used to searching for ` (from the backquote key) in
  such a context can easily think that it IS the same character.
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  S?he can easily either think (1) that the text s?he (thought s?he
  was) searching for is not present (e.g., if s?he does not see it)
  or (2) that Isearch is not working properly for some reason.

And I said clearly, in the message before that one:

  They look slightly different, yes.  Barely noticeable when
            ^^^^^^^^                  ^^^^^^^^^^^^^^^^^
  you see them side by side (see screenshot).

Barely noticeable EVEN when you see them side by side.
That was my judgment then, and I'm sticking to it now.

  But otherwise not noticeable enough that someone used to Emacs
                ^^^^^^^^^^^^^^^^^^^^^
  < 25.1 would expect that the ` s?he types (as s?he always has,
  for this) does not result in the same character s?he sees as a
  name left delimiter.

And I fell upon this bug by accident, when I tried to do
what the bug recipe describes, with emacs -Q: search for
"`list" in a topic where that text occurs, but with a left
single quotation mark instead of `.

It's a real report of a real experience, with emacs -Q.

You keep trying to show that there is no problem by
referring to the fact that I myself usually use Emacs with
a different font.  But this is not about me.  It's about
the default behavior of Emacs.

I included the screenshot so that you and others can
easily see for yourselves how much the two chars resemble
each other.

OK, so you disagree that they do.  To you it is crystal
clear that they are not the same, and you cannot imagine
that anyone could see things differently.  For you, I'm
making this all up just to bother you, perhaps?

Am I the only one who sees these characters as similar
in appearance, and easy to mistake, one for the other?
Dunno.  But no one need take my word for the appearance,
at least.  The screenshot is there for anyone to check
and come to their own conclusion.

> > > Weren't you one of those who lobbied for turning the character-folding
> > > off by default?  If it were not for that lobbying, your search would
> > > have succeeded.
> >
> > No.  And there was no such "lobbying", by anyone.  Can you point
> > to any post by me, in the long and deep discussion about this on
> > emacs-devel ("On language-dependent defaults for character-folding")
> > where I did that?
> 
> You just did it yourself, so I can rest my case.

Is this really the way you try to reason, and discuss something?

You apparently cannot show that I lobbied for turning char
folding off by default, so you resort to claiming that I
have argued that just now?  Where?

Are you perhaps watching too much Trump these days?

In fact, I stated clearly in this thread that turning it on,
at least for quotation, could be one of the possible fixes
for this bug.

My position on the default has been, from the beginning
(in case you are really interested, though it is _irrelevant_
for this bug report), is that the default behavior is not
very important.

And that what is more important is that users (1) be able
to easily customize the behavior, (2) be able to toggle it
during isearch, and (3) be told about it in clear terms
(especially if the default behavior represents a change),
in both the doc and the NEWS.

THAT was my point of view during the default discussion.
And it has not changed.

However, now that I have come across this usability bug,
I think the bug should be fixed.  But the default behavior
is still less important that what I argued for.

IF you choose to fix the bug by changing the default
behavior for folding quotes, that will be better, I think,
than not fixing it.

I'm hoping that there might be a better fix available.
I'm hoping that someone will come up with something good.
But you've rejected the only other one I've come up with:
change the default font, to avoid the similarity of appearance.

> > And why would it be important if I had?  Are you trying to
> > mete out punishment, saying that it is my fault that Emacs
> > now presents, by default, a visual confusion of single-quoting?
> 
> I'm saying that yours are double standards.

Show us, please.  What _are_ you talking about?

Better yet, please address the bug, and not just attack the
messenger.

> > > The look quite different to me (my system uses the same font),
> >
> > You _know_ they are different.  Please look again at the screenshot
> > I sent, and imagine that it does not include the window showing the
> > two right next to each other, so you see what a user sees when s?he
> > searches for `.
> 
> I did.  They still look different.

OK.  Clearly you have good eyes.

But even with your good eyes you cannot see that someone (else)
could easily mistake one for the other?   Especially since s?he
has long been able to search for quoted names using `?  You can't
see a problem here for others, even if you yourself have no such
problem?

> > > I'm not dismissing the problem.  I'm saying that I don't see a
> > > solution except through user customizations.  Searching for similarly
> > > looking characters is not specific to Info, it can happen elsewhere.
> > > Users who bump into that frequently should customize their fonts
> > > and/or turn on character-folding in search commands.
> >
> > That's not a solution for the default behavior, which is what
> > this bug is about.  If that is what you propose to solve the
> > problem here then you are indeed dismissing the problem.
> 
> Customizations exist in Emacs for a reason.  Saying that every problem
> must be solved in "emacs -Q" is absurd, especially for minor problems
> such as this one, with 2 characters that look differently.

Another straw-man argument.  Did someone argue that every
problem must be solved in "emacs -Q"?  No, of course not.

> > Two possible solutions have been suggested in this thread, so far:
> >
> > 1) Emacs can try to avoid using Courier New as the default font.
> 
> Not an option.  Other fonts have much less coverage, and most of them
> are much uglier.  You may wish looking at them before you suggest
> this.

I'm no expert on fonts.  Are you speaking authoritatively,
about ALL other fonts, when you say "other fonts"?

I can't argue about coverage (I don't use RTL or many
non-English characters, personally).  And I won't try to
argue about what is ugly or not.  For English, I can offer
the font I use as one that I find useful for MS Windows,
FWIW: Lucida Console.  I don't claim that it could or
should be adopted by Emacs.

Do we have a prioritized list of the important qualities
of a default font for Emacs?  How important is "coverage",
and just what do we mean by it?  How important is visual
distinction/confusion of characters (including letters vs
numbers)?

Again, the default font is not the most important thing.
It need not be perfect.  But it can help or hinder
understanding of Emacs, including by searching for
information.

> > 2. Isearch can fold "single-quote" chars (i.e., chars that could
> >    be confused in this regard), by default.
> 
> I'm okay with that, but my opinions on that were overruled.
> 
> > Please note, BTW, that I have _not_ opposed #2, just as I did
> > _not_ "lobby for turning the character-folding off by default"
> > during the general emacs-devel discussion.
> 
> Sure, you did.  You just prefer to deny it now.

See above.  Including my citing #2, from the beginning,
as one possible fix for this bug.  And note your lack
of finding any post by me where I lobbied to turn off
folding by default.

And I spent time looking through the thread, to see
what people, including you and me, actually said there.
Please put some evidence where your claims are.

Or is it perhaps that you were alone in being _adamant_
about turning it on by default, so that even those who
were on the fence about the default behavior have now
been put on your enemies list and must be thoroughly
castigated?

> > What's not helpful is dismissing the problem or casting
> > blame for it.
> 
> I agree.  Do you realize that this is what you are doing?

Uh, how so, Eli?  Have I blamed you for this bug?  No way.

You, on the other hand _have_:

* dismissed the problem, saying that there can be
  no difficulty distinguishing the characters.

* blamed me for the default search behavior of not
  character-folding.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Date: Fri, 23 Sep 2016 14:47:12 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 24510@debbugs.gnu.org
> 
> I'm no expert on fonts.  Are you speaking authoritatively,
> about ALL other fonts, when you say "other fonts"?

At some point, I looked at all the fonts installed on a typical
Windows machine.  Whether that makes me an "authority" is not for me
to say.

> I can't argue about coverage (I don't use RTL or many
> non-English characters, personally).  And I won't try to
> argue about what is ugly or not.  For English, I can offer
> the font I use as one that I find useful for MS Windows,
> FWIW: Lucida Console.  I don't claim that it could or
> should be adopted by Emacs.

See my other message: its coverage is too small.

> Do we have a prioritized list of the important qualities
> of a default font for Emacs?

It should be a monospaced font, and it should cover as many Unicode
blocks for as many scripts as possible.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Byung-Hee HWANG (황병희, 黃炳熙)]

Eli Zaretskii <eliz@gnu.org> 께서 쓰시길,
 《記事 全文 <83eg49da94.fsf@gnu.org> 에서》:

>> [...]
> It should be a monospaced font, and it should cover as many Unicode
> blocks for as many scripts as possible.

Eli, i like these your mind, very beautiful, thanks!

-- 
^고맙습니다 _地平天成_ 감사합니다_^))//

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

emacs -Q

C-h r g Dired
C-s `list

finds nothing, missing the occurrence of ‘list-directory’ that is in the
same node.

What looks exactly (AFAICT) like the backquote you typed at the prompt
is NOT the same as the character that precedes "list-directory" in the
buffer.  And it is not matched to that character by Isearch.  That
character is LEFT SINGLE QUOTATION MARK.

What you see is definitely NOT what you get.  And you wonder...

Emacs provides a default font where these two characters are visually
indistinguishable.  And then it distinguishes them by default when you
try to search.

This is extremely user-UNfriendly.  The default font for Emacs should be
changed.  Or else search should, by default, find the character used
when you type a backquote char.  Or else the character used, by default,
should be backquote, not LEFT SINGLE QUOTATION MARK.  Or some other
reasonable solution should be found, to remedy this annoyance.

The DEFAULT behavior of Emacs should not be so confusing,
user-unfriendly, cumbersome, and, well, unusable.

It is a common operation to search through our manuals for names that
are within "single-quote" delimiters.  In the past, these delimiters
were backquote and apostrophe: `...', and keys that insert those chars
are nearly ubiquitous on western keyboards.  Not so, the "single-quote"
delimiters used now.

Hard to believe this annoyance has gotten as far as it has (release
25.1).  Did no one try searching our manuals using Emacs -Q?


In GNU Emacs 25.1.1 (x86_64-w64-mingw32)
 of 2016-09-17 built on LAPHROAIG
Windowing system distributor 'Microsoft Corp.', version 6.1.7601
Configured using:
 'configure --without-dbus --without-compress-install CFLAGS=-static'

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Date: Thu, 22 Sep 2016 10:04:27 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> emacs -Q
> 
> C-h r g Dired
> C-s `list
> 
> finds nothing, missing the occurrence of ‘list-directory’ that is in the
> same node.
> 
> What looks exactly (AFAICT) like the backquote you typed at the prompt
> is NOT the same as the character that precedes "list-directory" in the
> buffer.

It doesn't look exactly like it here.  In fact, it looks quite
different.  Although it could be that on your system Emacs uses
different fonts than those on mine.

But if 2 characters from 2 different fonts look the same on your
system, I think you should customize the Info faces to countermand
that.  I don't see any other solution.

> And it is not matched to that character by Isearch.

It might, if you turn on character-folding in Isearch.  Unfortunately,
that is off by default.

> Emacs provides a default font where these two characters are visually
> indistinguishable.

Emacs doesn't provide any fonts.  Your system does.  On my system,
these two characters look different, even with my old weary eyes.

> Hard to believe this annoyance has gotten as far as it has (release
> 25.1).  Did no one try searching our manuals using Emacs -Q?

We did.  It works for me.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 3207 bytes --]

> > emacs -Q
> > C-h r g Dired
> > C-s `list
> > finds nothing, missing the occurrence of ‘list-directory’ that
> > is in the same node.
> >
> > What looks exactly (AFAICT) like the backquote you typed at the
> > prompt is NOT the same as the character that precedes
> > "list-directory" in the buffer.
> 
> It doesn't look exactly like it here.  In fact, it looks quite
> different.  Although it could be that on your system Emacs uses
> different fonts than those on mine.

See attached screenshot.  No, they are not EXACTLY the same.  But
a user who is used to searching for ` (from the backquote key) in
such a context can easily think that it IS the same character.

S?he can easily either think (1) that the text s?he (thought s?he
was) searching for is not present (e.g., if s?he does not see it)
or (2) that Isearch is not working properly for some reason.

> But if 2 characters from 2 different fonts look the same on your
> system, I think you should customize the Info faces to countermand
> that.  I don't see any other solution.

Again, this is emacs -Q.  The chars look quite different with my
own setup - this is not a problem for me, personally.

This is a problem for Emacs.  This is using the default font
(on my platform, which is nevertheless a common one: MS Windows).

This is the font used by emacs -Q:
-outline-Courier New-normal-normal-normal-mono-17-*-*-*-c-*-iso8859-1

That's a very common MS Windows fixed-width font.  And it has,
in the past, been a reasonable default choice.  Doesn't seem
so reasonable anymore.

> > And it is not matched to that character by Isearch.
> 
> It might, if you turn on character-folding in Isearch.
> Unfortunately, that is off by default.

I did mention that this is what a user sees by DEFAULT, no?

And I did propose that one fix could be for search to find
this character - by default - when you hit the backquote key:

  The default font for Emacs should be changed.  Or else search
  should, by default, find the character used when you type a
  backquote char.  Or else the character used, by default,
  should be backquote, not LEFT SINGLE QUOTATION MARK.  Or some
  other reasonable solution should be found, to remedy this
  annoyance.

> > Emacs provides a default font where these two characters are
> > visually indistinguishable.
> 
> Emacs doesn't provide any fonts.  Your system does.  On my system,
> these two characters look different, even with my old weary eyes.

They look slightly different, yes.  Barely noticeable when you
see them side by side (see screenshot).

But otherwise not noticeable enough that someone used to Emacs
< 25.1 would expect that the ` s?he types (as s?he always has,
for this) does not result in the same character s?he sees as a
name left delimiter.

If you do not want to accept that this is an obvious gotcha, fine.
Perhaps you will hear from other users about this; perhaps not.

> > Hard to believe this annoyance has gotten as far as it has (release
> > 25.1).  Did no one try searching our manuals using Emacs -Q?
> 
> We did.  It works for me.

If it works for you then that's all we can hope for, I guess.

[-- Attachment #2: throw-Emacs-25-backquote-search.png --]
[-- Type: image/png, Size: 49487 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 869 bytes --]

On 2016-09-22 15:58, Drew Adams wrote:
>>> And it is not matched to that character by Isearch.
>> > 
>> > It might, if you turn on character-folding in Isearch.
>> > Unfortunately, that is off by default.
>
> I did mention that this is what a user sees by DEFAULT, no?
> 
> And I did propose that one fix could be for search to find
> this character - by default - when you hit the backquote key:

The fix that you propose was already discussed, albeit as part of a larger change (the one that Eli pointed to, namely enabling character folding by default).  And Eli was, IIRC, in favor of it.

Do you think it would be useful to have a specific hack for just ` vs ‘?  I'd find it very confusing.

Maybe we can come up with a more limited version of character folding, and enable that by default?  This should solve this problem, right?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> >>> And it is not matched to that character by Isearch.
> >> >
> >> > It might, if you turn on character-folding in Isearch.
> >> > Unfortunately, that is off by default.
> >
> > I did mention that this is what a user sees by DEFAULT, no?
> >
> > And I did propose that one fix could be for search to find
> > this character - by default - when you hit the backquote key:
> 
> The fix that you propose was already discussed, albeit as part of a larger
> change (the one that Eli pointed to, namely enabling character folding by
> default).  And Eli was, IIRC, in favor of it.
> 
> Do you think it would be useful to have a specific hack for just ` vs ‘?
> I'd find it very confusing.
> 
> Maybe we can come up with a more limited version of character folding, and
> enable that by default?  This should solve this problem, right?

I've brought the problem to your attention.  A solution would be
good to have.  It should not be enough that the response is (1)
"It works for me" and (2) users need to customize their fonts,
to fix things on their end.  That's not the right approach, IMO.

From a user point of view, and ignoring whatever good or bad
reasons might be behind the changes made, (1) Info now uses curly
quotes, by default, to delimit names, and (2) Isearch does not,
by default, fold ` and ' so that they match those new delimiters.
This combination is not good, for users.

If the delimiters must be changed to curly quotes, AND if they
are not noticeably different (i.e., quite obviously different)
from the characters that users have used to search for delimited
names for decades (since the beginning of Emacs), then Isearch
needs to fold them by default, I think.

Another solution, which I also mentioned, is to try to ensure
that in the default font, for at least common platforms (and MS
Windows is one of those, I think), the delimiter chars are quite
obviously different from ` and '.

If you do not want to enable char-folding by default in 25.1,
or at least quote-folding, then I think the default font needs
to be changed, to make this newly imposed character discrepency
blatantly obvious to users.

And the doc and NEWS need to prominently point out to users that
names in Info are now delimited using characters for which they
(probably) do not have simple keyboard keys.  That message should
also tell them how to insert the delimiter chars in search strings
(toggle char-folding or otherwise insert them).

Do you see another solution?  Pop up a message whenever they
insert a backquote followed by a letter into the search string,
warning them that a backquote does not match a name delimiter?
Warn them to customize their font, if they want to avoid confusion?
I don't think so.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Eli Zaretskii]

> Date: Thu, 22 Sep 2016 12:58:49 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 24510@debbugs.gnu.org
> 
> > But if 2 characters from 2 different fonts look the same on your
> > system, I think you should customize the Info faces to countermand
> > that.  I don't see any other solution.
> 
> Again, this is emacs -Q.

That doesn't matter, since Emacs doesn't control the fonts installed
on the user's system, much less how they look.  Like I said: if some
user cannot distinguish between these characters, I see no other
solution for the problem except font customizations, since only the
user on her system can DTRT with the fonts that are available.

> The chars look quite different with my own setup - this is not a
> problem for me, personally.

Then what exactly is the nature of the problem you raise?  Is it a
hypothetical issue that _might_ happen on someone's machine, even
though none of us, including yourself, have actually witnessed it?

> > It might, if you turn on character-folding in Isearch.
> > Unfortunately, that is off by default.
> 
> I did mention that this is what a user sees by DEFAULT, no?

Weren't you one of those who lobbied for turning the character-folding
off by default?  If it were not for that lobbying, your search would
have succeeded.

> > Emacs doesn't provide any fonts.  Your system does.  On my system,
> > these two characters look different, even with my old weary eyes.
> 
> They look slightly different, yes.  Barely noticeable when you
> see them side by side (see screenshot).

The look quite different to me (my system uses the same font), so I
guess we will have to agree to disagree about that.

> If you do not want to accept that this is an obvious gotcha, fine.
> Perhaps you will hear from other users about this; perhaps not.
> 
> > > Hard to believe this annoyance has gotten as far as it has (release
> > > 25.1).  Did no one try searching our manuals using Emacs -Q?
> > 
> > We did.  It works for me.
> 
> If it works for you then that's all we can hope for, I guess.

I'm not dismissing the problem.  I'm saying that I don't see a
solution except through user customizations.  Searching for similarly
looking characters is not specific to Info, it can happen elsewhere.
Users who bump into that frequently should customize their fonts
and/or turn on character-folding in search commands.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Clément Pit--Claudel]

[-- Attachment #1.1.1: Type: text/plain, Size: 510 bytes --]

On 2016-09-22 13:04, Drew Adams wrote:
> What looks exactly (AFAICT) like the backquote you typed at the prompt
> is NOT the same as the character that precedes "list-directory" in the
> buffer.  And it is not matched to that character by Isearch.  That
> character is LEFT SINGLE QUOTATION MARK.
> 
> What you see is definitely NOT what you get.  And you wonder...

Hi Drew,

I wonder which fonts you're using.  I've attached a screenshot of what it looks for me in emacs -Q.

Cheers,
Clément.

[-- Attachment #1.1.2: Screenshot from 2016-09-22 16-00-43.png --]
[-- Type: image/png, Size: 78241 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Noam Postavsky]

found 24510 24.5
severity 24510 minor
quit

On Thu, Sep 22, 2016 at 1:04 PM, Drew Adams <drew.adams@oracle.com> wrote:
> Hard to believe this annoyance has gotten as far as it has (release
> 25.1).

I see the same in the 24.5 release.

>  Did no one try searching our manuals using Emacs -Q?

Nobody reads the manual.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> I see the same in the 24.5 release.

Yes; same here.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Drew Adams <drew.adams@oracle.com> writes:

> emacs -Q
>
> C-h r g Dired
> C-s `list
>
> finds nothing, missing the occurrence of ‘list-directory’ that is in the
> same node.
>
> What looks exactly (AFAICT) like the backquote you typed at the prompt
> is NOT the same as the character that precedes "list-directory" in the
> buffer.  And it is not matched to that character by Isearch.  That
> character is LEFT SINGLE QUOTATION MARK.

It would be possible to fold certain character groups into equivalence
classes for isearch in these modes (I'm thinking info-mode and
help-mode) so that ` and ‘ are treated equivalently -- that might help
users and shouldn't lead to regressions?

Perhaps Juri has some comments; added to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> that might help users and shouldn't lead to regressions?

(We already have a regression in the behavior,
 introduced by the substitution of curly quotes
 for the traditional `...'.)

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

>> C-h r g Dired
>> C-s `list
>>
>> finds nothing, missing the occurrence of ‘list-directory’ that is in the
>> same node.
>>
>> What looks exactly (AFAICT) like the backquote you typed at the prompt
>> is NOT the same as the character that precedes "list-directory" in the
>> buffer.  And it is not matched to that character by Isearch.  That
>> character is LEFT SINGLE QUOTATION MARK.
>
> It would be possible to fold certain character groups into equivalence
> classes for isearch in these modes (I'm thinking info-mode and
> help-mode) so that ` and ‘ are treated equivalently -- that might help
> users and shouldn't lead to regressions?

It's possible to enable char-fold search only in Info mode,
but probably this should be left for user customization,
because this problem is not Info-specific: most likely the user
might want ` and ‘ to be treated equivalently pretty much everywhere.
But still if the user want this only in Info, it's very easy to type
C-s M-s ' `list.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> It's possible to enable char-fold search only in Info mode,
> but probably this should be left for user customization,
> because this problem is not Info-specific: most likely the user
> might want ` and ‘ to be treated equivalently pretty much everywhere.

My thorough research shows that 99.7% of Emacs users don't see a ‘
outside of *Help* and *info*, so doing something special there seems
useful.

> But still if the user want this only in Info, it's very easy to type
> C-s M-s ' `list.

That does more than just fold `' with ‘’, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> My thorough research shows that 99.7% of Emacs users don't see a ‘
> outside of *Help* and *info*, so doing something special there seems
> useful.

Possibly, yes.  Optionally (maybe opt-out, not opt-in).

> > But still if the user want this only in Info, it's 
> > very easy to type C-s M-s ' `list.
> 
> That does more than just fold `' with ‘’, though.

That too is a good point.  Maybe a command that folds
only some (definable) set of confusables?

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

>> It's possible to enable char-fold search only in Info mode,
>> but probably this should be left for user customization,
>> because this problem is not Info-specific: most likely the user
>> might want ` and ‘ to be treated equivalently pretty much everywhere.
>
> My thorough research shows that 99.7% of Emacs users don't see a ‘
> outside of *Help* and *info*, so doing something special there seems
> useful.
>
>> But still if the user want this only in Info, it's very easy to type
>> C-s M-s ' `list.
>
> That does more than just fold `' with ‘’, though.

So only folding `' with ‘’ is required?  Then it's easy to do:

diff --git a/lisp/info.el b/lisp/info.el
index 8ca6c54979..656a3b0b11 100644
--- a/lisp/info.el
+++ b/lisp/info.el
@@ -41,7 +41,9 @@
 
 ;;; Code:
 
-(eval-when-compile (require 'cl-lib))
+(eval-when-compile
+  (require 'cl-lib)
+  (require 'subr-x))
 
 (defgroup info nil
   "Info subsystem."
@@ -2147,7 +2149,10 @@ Info-isearch-search
 		   (funcall isearch-regexp-function string lax)
 		 (word-search-regexp string lax))))
 	    (isearch-regexp string)
-	    (t (regexp-quote string)))
+	    (t (thread-last
+		 (regexp-quote string)
+		 (replace-regexp-in-string "`" "[`‘]")
+		 (replace-regexp-in-string "'" "['’]"))))
 	   bound noerror count
 	   (unless isearch-forward 'backward)))
 	(point))

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> So only folding `' with ‘’ is required?

Yup, I think so.

> Then it's easy to do:

[...]

> @@ -2147,7 +2149,10 @@ Info-isearch-search
>  		   (funcall isearch-regexp-function string lax)
>  		 (word-search-regexp string lax))))
>  	    (isearch-regexp string)
> -	    (t (regexp-quote string)))
> +	    (t (thread-last
> +		 (regexp-quote string)
> +		 (replace-regexp-in-string "`" "[`‘]")
> +		 (replace-regexp-in-string "'" "['’]"))))
>  	   bound noerror count
>  	   (unless isearch-forward 'backward)))
>  	(point))

Nice.  But we should also have this in *Help* (and possibly other
buffers where we do the `' -> ‘’ dance, but I can't think of any others
at the moment), so would it possible to have this like a little minor
mode?  Or would that be difficult?  (I've barely looked at the char-fold
machinery, so I have no idea.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> we should also have this in *Help* (and possibly other
> buffers where we do the `' -> ‘’ dance, 

The real, right solution is not to do that
convoluted dance.  Not by default, anyway.

The automatic refrain will be "that ship has
sailed", alas.  Same old song.

But that ship/mistake could be recalled.

A car company recalls vehicles that were
poorly designed and thus create problems.

We should be brave and honest enough to
recognize the damage we're causing and
make amends.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

> Nice.  But we should also have this in *Help* (and possibly other
> buffers where we do the `' -> ‘’ dance, but I can't think of any others
> at the moment), so would it possible to have this like a little minor
> mode?  Or would that be difficult?  (I've barely looked at the char-fold
> machinery, so I have no idea.)

I have a patch that shows the contents of help buffers in Info mode.
Then searching for ‘’ in help text will be handled by Info-isearch-search.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> I have a patch that shows the contents of help buffers in Info mode.
> Then searching for ‘’ in help text will be handled by Info-isearch-search.

Hm...  I think I'd like to keep help buffers in help mode, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

>> I have a patch that shows the contents of help buffers in Info mode.
>> Then searching for ‘’ in help text will be handled by Info-isearch-search.
>
> Hm...  I think I'd like to keep help buffers in help mode, though.

It's possible to do this in help mode, but what keys should toggle
char-folding of quotes while using Isearch in help buffers?

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> It's possible to do this in help mode, but what keys should toggle
> char-folding of quotes while using Isearch in help buffers?

I may be misunderstanding the question -- should the normal keys toggle
that?  I was thinking that the default in those buffers would be to
fold the quoting characters, but that would be totally outside the
normal folding machinery, so that wouldn't be affected at all.

But I don't know whether that's possible (or easy).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

>> It's possible to do this in help mode, but what keys should toggle
>> char-folding of quotes while using Isearch in help buffers?
>
> I may be misunderstanding the question -- should the normal keys toggle
> that?  I was thinking that the default in those buffers would be to
> fold the quoting characters, but that would be totally outside the
> normal folding machinery, so that wouldn't be affected at all.

I thought that by default Isearch always should be literal,
so the user could search only for plain ', and not to match
its fancy variants.  Currently, M-s ' switches to mode
that folds all possible characters.  But how to switch to new mode
that folds only quoting characters?

> But I don't know whether that's possible (or easy).

This should not be too hard to do, but I don't understand
what UI it could have.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> I thought that by default Isearch always should be literal,
> so the user could search only for plain ', and not to match
> its fancy variants.

No, I want the default Isearch to fold `' with the fancy variants in
these modes.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> > I thought that by default Isearch always should be literal,
> > so the user could search only for plain ', and not to match
> > its fancy variants.
> 
> No, I want the default Isearch to fold `' with the 
> fancy variants in these modes.  :-)

Do what you want for your own use.  Please read
the bug report for what this reporter reported as
the problem and suggested solutions.

Juri is right, here.  For Emacs default behavior,
Isearch should do what it does generally.

But there should be a simple Isearch binding (at
least in places such as Info) that toggles
folding of _these particular_ confusables
(misguidedly introduced by foisting curly quotes
on Info).  This toggling should be separate from
toggling the more general char-folding.

[And see what I suggested about maybe providing
feedback when users can't find ` matches etc.
There's unfortunately no easy, complete solution
to this problem, I'm afraid.]

Users should also be able to specify whether,
for them, such folding is on by default.

IOW, make it easy for users to control this. But
don't change Isearch behavior here by default.

Yes, unfortunately, that won't help uninitiated
users deal with these confusables as well as
would folding them by default.  But it's saner.

And it would be better if the default font, at
least for Info and Help, were one that did not
confuse these quotes so easily visually.  The
default, at least on MS Windows, is especially
pernicious in this regard.

This is just another annoyance/headache caused
by Emacs adopting curly quoting everywhere.

Emacs has tied itself in knots to confuse users
and make their life more difficult in Info and
Help, all in the name of becoming more "modern"
by changing the brilliant, simple text feature
that was `...' to plain old curly quotes.  Bad
dog.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

>> I thought that by default Isearch always should be literal,
>> so the user could search only for plain ', and not to match
>> its fancy variants.
>
> No, I want the default Isearch to fold `' with the fancy variants in
> these modes.  :-)

So the users will have no way to search for ' literally?
Or maybe better to add a new Isearch mode that char-folds only `'
and enable it only in Help buffers?

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

>> No, I want the default Isearch to fold `' with the fancy variants in
>> these modes.  :-)
>
> So the users will have no way to search for ' literally?

Yes, I don't think anybody would want to.  But like I said, I'd prefer
if it was possible to have this as a minor mode so that people could
switch it off (even if they probably won't want to).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> >> No, I want the default Isearch to fold `' with the fancy variants in
> >> these modes.  :-)
> >
> > So the users will have no way to search for ' literally?
> 
> Yes, I don't think anybody would want to.

That's 100% wrong.  Especially in the Emacs
and Elisp manuals.  Same thing for `.  Same
thing for _any_ char, no matter how common.

> But like I said, I'd prefer
> if it was possible to have this as a minor mode so that people could
> switch it off (even if they probably won't want to).

We don't need a minor mode for this.  This is
about Isearch.  We need a toggle key for it.

On the other hand, we could consider whether
the same choice/decision to fold should/could
affect both Isearch and other things, such as
query-replace.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

>>> No, I want the default Isearch to fold `' with the fancy variants in
>>> these modes.  :-)
>>
>> So the users will have no way to search for ' literally?
>
> Yes, I don't think anybody would want to.  But like I said, I'd prefer
> if it was possible to have this as a minor mode so that people could
> switch it off (even if they probably won't want to).

If you are sure about this, here is the simplest way to implement it,
and the users can disable it easily in help-mode-hook:


[-- Attachment #2: help-mode-search-default.patch --]
[-- Type: text/x-diff, Size: 1574 bytes --]

diff --git a/lisp/help-mode.el b/lisp/help-mode.el
index 94bd591131..4ad8e36070 100644
--- a/lisp/help-mode.el
+++ b/lisp/help-mode.el
@@ -414,6 +414,13 @@ help-mode
   (setq-local tool-bar-map
               help-mode-tool-bar-map)
   (setq-local help-mode--current-data nil)
+  (unless search-default-mode
+    (setq-local search-default-mode
+                (lambda (string &optional _lax)
+                  (thread-last
+                    (regexp-quote string)
+                    (replace-regexp-in-string "`" "[`‘]")
+                    (replace-regexp-in-string "'" "['’]")))))
   (setq-local bookmark-make-record-function
               #'help-bookmark-make-record))
 
diff --git a/lisp/info.el b/lisp/info.el
index abfb77b055..7e0d4f646f 100644
--- a/lisp/info.el
+++ b/lisp/info.el
@@ -4487,6 +4487,14 @@ Info-mode
   (setq-local isearch-wrap-function #'Info-isearch-wrap)
   (setq-local isearch-push-state-function #'Info-isearch-push-state)
   (setq-local isearch-filter-predicate #'Info-isearch-filter)
+  (unless search-default-mode
+    (setq-local search-default-mode
+                (lambda (string &optional _lax)
+                  (thread-last
+                    (regexp-quote string)
+                    (replace-regexp-in-string "`" "[`‘]")
+                    (replace-regexp-in-string "'" "['’]")
+                    (replace-regexp-in-string "\"" "[\"“”]")))))
   (setq-local revert-buffer-function #'Info-revert-buffer-function)
   (setq-local font-lock-defaults '(Info-mode-font-lock-keywords t t))
   (Info-set-mode-line)

[-- Attachment #3: Type: text/plain, Size: 136 bytes --]


PS: ISTR, there was a discussion about supporting simultaneous replacements
instead of using thread-last.  Is this a work in progress?

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> >>> No, I want the default Isearch to fold `' with the fancy variants
> in
> >>> these modes.  :-)
> >>
> >> So the users will have no way to search for ' literally?
> >
> > Yes, I don't think anybody would want to.  But like I said, I'd
> > prefer if it was possible to have this as a minor mode so that
> > people could switch it off (even if they probably won't want to).
> 
> If you are sure about this, here is the simplest way to implement it,
> and the users can disable it easily in help-mode-hook:

Please don't do this.  Users shouldn't have to fiddle
with mode hooks to get or remove this folding.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I definitely wnat to be able to search literally for each kind
of quote character.

Folding of quotes might also be a useful feature, but don't
make it difficult to search for them literally.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> If you are sure about this, here is the simplest way to implement it,
> and the users can disable it easily in help-mode-hook:

[...]

> +  (unless search-default-mode
> +    (setq-local search-default-mode
> +                (lambda (string &optional _lax)
> +                  (thread-last
> +                    (regexp-quote string)
> +                    (replace-regexp-in-string "`" "[`‘]")
> +                    (replace-regexp-in-string "'" "['’]")))))

Thanks; I separated this out into a tiny minor mode, and switched that
on in help-mode and info-mode instead.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> Thanks; I separated this out into a tiny minor mode, 
> and switched that on in help-mode and info-mode instead.

Too bad.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Juri Linkov]

>> +  (unless search-default-mode
>> +    (setq-local search-default-mode
>> +                (lambda (string &optional _lax)
>> +                  (thread-last
>> +                    (regexp-quote string)
>> +                    (replace-regexp-in-string "`" "[`‘]")
>> +                    (replace-regexp-in-string "'" "['’]")))))
>
> Thanks; I separated this out into a tiny minor mode, and switched that
> on in help-mode and info-mode instead.

I noticed that Info also contains double quotation marks,
so these could be added as well:

                        (replace-regexp-in-string "\"" "[\"“”]")

Also this improves tests:

diff --git a/test/lisp/subr-tests.el b/test/lisp/subr-tests.el
index 6bcac2a5eb..8f3ee66e00 100644
--- a/test/lisp/subr-tests.el
+++ b/test/lisp/subr-tests.el
@@ -1065,6 +1065,9 @@ test-local-set-state
     (let ((state (buffer-local-set-state global 10
                                          local 20
                                          unexist 30)))
+      (should (= global 10))
+      (should (= local 20))
+      (should (= unexist 30))
       (buffer-local-restore-state state)
       (should (= global 1))
       (should (= local 2))

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

> I noticed that Info also contains double quotation marks,
> so these could be added as well:
>
>                         (replace-regexp-in-string "\"" "[\"“”]")
>
> Also this improves tests:

Makes sense; please go ahead and push.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> >> I thought that by default Isearch always should be literal,
> >> so the user could search only for plain ', and not to match
> >> its fancy variants.
> >
> > No, I want the default Isearch to fold `' with the fancy variants in
> > these modes.  :-)
> 
> So the users will have no way to search for ' literally?

No; see my other msgs in the thread about this.
Users _absolutely_ need the ability to search
for ` and ', as well as to search for ` and left
curly, and ' and right curly.

This is the problem.  We need to give them ways
to do everything, easily, and we need to provide
reasonable default behavior.  I agreed with you
that by default Isearch in Info and Help should
_not_ fold such confusables.  But users need to
know that they can easily do that (e.g. a key).

And we should I think come up with reasonable
ways to help with some feedback when Emacs can
guess that maybe the user meant to get folded
behavior but didn't, e.g., looked for ` and
didn't find any occurrences.  Such feedback
from guessing needs to be optional - it must
not be in-your-face and annoy.

> Or maybe better to add a new Isearch mode that char-folds only `'

I suggested having an Isearch _key_ (a toggle),
not a mode, for just that, as opposed to the
more drastic full char-folding.  And I suggested
having an option to configure the confusables
to fold.

> and enable it only in Help buffers?

No, I think that any such behavior should be
configurable, e.g. by an option that lists the
modes or buffer-name regexps, to define where
to apply it.
___

There are lots of possibilities for trying
to deal with all of this, and I don't have a
perfect solution (there isn't any, I
suspect).  It's worth brainstorming etc.
___

Even if we didn't have the catastrophe of the
imposition everywhere of curly quotes, we'd
have this problem (only __much__ less so) of
optionally folding confusables.  It merits
some reflection and discussion - better on
emacs-devel than here, I think.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> > My thorough research shows that 99.7% of Emacs users don't see a ‘
> > outside of *Help* and *info*, so doing something special there seems
> > useful.
> >
> >> But still if the user want this only in Info, it's very easy to type
> >> C-s M-s ' `list.
> >
> > That does more than just fold `' with ‘’, though.
> 
> So only folding `' with ‘’ is required?  Then it's easy to do:

Then how does a user actually search only for ‘
or only for ', etc.?

The ability to search with or without folding 
such confusables is needed.

We have that, with `M-s '' (thank you).

But the problem is that users won't necessarily
know that they need to use that to get past the
curly-vs-traditional confusion.  Searching for
` or ' won't find curly quotes by default.

"Confusable" means they might not even notice a
difference between `...' and curly quotes.  And
no, the answer to that Emacs-imposed gotcha is
not to tell users it's their fault for choosing
a font that doesn't make distinguishing them
easier.

If users don't notice a visible difference then
they're likely to get misled - confused.  We've
_done that to them gratuitously_, by imposing 
curly quoting by default - replacing unambiguous
notation with, well, a confusing mess.  Sorry,
but that's the case.

And if we change Info mode to fold these chars by
default then users still won't necessarily know
they're doing that, and so won't they need to, or
even perhaps won't know how to, search unfoldingly
(to coin an adverb).

It might be helpful to (1) document the folding
by default in the mode doc, (2) document a toggle
command for this folding (specifically), and
maybe even (3) show a message in some cases
during search that involves these chars, to let
users know that they are, or are not, folding,
and what that means.

But all such attempts to help users are really
only poor workarounds for the mess we've thrown
at them.

Finally, as I said in an earlier msg, this gotcha
is not limited to Info and Help.

  It's about searching for something, and not
  finding it because you're typing a different
  char - it's about (~semi)confusables, I guess.

The willy-nilly replacement of `...' by curly
quotes has not finished to mess things up.  We're
still quite a ways from cleaning up after it and
helping users navigate it (and work around it).

It was so much simpler - transparent for everyone
- with just `...'.  No ambiguity.  Very Emacsy.
Alas, we've tossed out that baby, with an eye
that mistakenly saw clear bathwater as dirty.

bug#24510: 25.1; Info: searching for ` does not find what looks like `

[Drew Adams]

> It's possible to enable char-fold search only in Info mode,
> but probably this should be left for user customization,
> because this problem is not Info-specific: most likely the user
> might want ` and ‘ to be treated equivalently pretty much everywhere.
> But still if the user want this only in Info, it's very easy to type
> C-s M-s ' `list.

Yes, `C-s M-s '' is great.  But it requires having
some idea that you want/need to fold chars.

The problem (gotcha) is that a user can easily not
recognize (know) that the char s?he's looking for
is not just ` (backquote).  As I said in the bug
report:

  S?he can easily either think (1) that the text
  s?he (thought s?he was) searching for is not
  present (e.g., if s?he does not see it) or
  (2) that Isearch is not working properly for
  some reason.

We should help users with this problem.  Somehow.

And yes, the problem is more general than just
` (backquote), and it's more general than Info.

It's about searching for something, and not finding
it because you're typing a different char - it's
about (~semi)confusables, I guess.
___

Maybe (?) - dunno whether this would be reasonable -
a minor mode or option whereby search failure, when
the search string involves confusables (where that
set of chars is customizable or bindable in a given
context/mode), lets you know that you might want to
try char-folding.  (At least for literal searches.)