Quotes do not display correctly

Quotes do not display correctly

[Nick Helm]

[-- Attachment #1: Type: text/plain, Size: 1677 bytes --]

I use Emacs to write and edit long-form prose, mainly in org-mode,
markdown-mode and other text-mode derivatives. While doing
so, I frequently use different types of quotation characters at the
same time.

However, since Emacs 25.x, I have lost the ability to work with some
quote chars – in particular quote pairs 0x2018 and 0x2019, and 0x201C
and 0x201D. These are Unicode chars LEFT/RIGHT SINGLE QUOTATION MARK
and LEFT/RIGHT DOUBLE QUOTATION MARK, or what some people call
typographic or curly quotes.

For some reason, Emacs now shows these quote marks in my org-mode
and markdown-mode buffers (and, as far as I can tell, everywhere else in
Emacs, including fundamental mode) as either a grave accent (0x60) and
single quote (0x27) pair, or a double quote (0x22) pair (see screen
shot). I say shows, because when I do C-u C-x = on these chars, the
decomposition tells me they are, in fact, the quote chars I expect, Emacs
just chooses to display them as something else.

The quotes also appear to have a display table and face applied, but I
don’t understand the significance of this information, or if it’s a clue to
solving my problem.

I read in NEWS about a new variable, text-quoting-style, which
manipulates how quote are displayed, but none of its documented options
seem to have an effect on the appearance of quote marks in my writing
buffers, or anywhere else for that matter. Perhaps this is unrelated.

I would like Emacs to simply display the actual quote chars in my
buffers without applying any kind of styling.

Any help would be greatly appreciated!

Thanks,
Nick

[cid:DB688C1D-D533-467A-8A1B-8F27263C0E24]



[-- Attachment #2: quote problem.png --]
[-- Type: image/png, Size: 292839 bytes --]

Re: Quotes do not display correctly

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

Hi Nick,

FWIW, the curly quotes show up fine for me. I am on the latest build of
emacs-25 branch. I have kept the value of text-quoting-style at its default
value.

Is it possible that your fonts don't support the curly quotes? I haven't
used Menlo, so I cannot comment on that for sure. Try switching to other
fonts like DejaVu Sans Mono or Source Code Pro and see if the issue still
persists.

Before that, also check if the issue persists in an emacs -Q session.

[image: pasted1]

>
>

[-- Attachment #2: pasted1 --]
[-- Type: image/png, Size: 160487 bytes --]

RE: Quotes do not display correctly

[Drew Adams]

`M-x report-emacs-bug' might be appropriate here.

Try to repro the problem starting Emacs without an
init file (`emacs -Q').

Re: Quotes do not display correctly

[Eli Zaretskii]

> From: Nick Helm <nick@tenpoint.co.nz>
> Date: Thu, 7 Apr 2016 09:20:47 +0000
> 
> However, since Emacs 25.x, I have lost the ability to work with some
> quote chars – in particular quote pairs 0x2018 and 0x2019, and 0x201C
> and 0x201D. These are Unicode chars LEFT/RIGHT SINGLE QUOTATION MARK
> and LEFT/RIGHT DOUBLE QUOTATION MARK, or what some people call
> typographic or curly quotes.
> 
> For some reason, Emacs now shows these quote marks in my org-mode
> and markdown-mode buffers (and, as far as I can tell, everywhere else in
> Emacs, including fundamental mode) as either a grave accent (0x60) and
> single quote (0x27) pair, or a double quote (0x22) pair (see screen
> shot). I say shows, because when I do C-u C-x = on these chars, the
> decomposition tells me they are, in fact, the quote chars I expect, Emacs
> just chooses to display them as something else.
> 
> The quotes also appear to have a display table and face applied, but I
> don’t understand the significance of this information, or if it’s a clue to
> solving my problem.

It _is_ a clue.  But to understand what goes wrong, more information
is needed, because Emacs was supposed to set up the display table only
if your display is unable to show the Unicode quote characters.

So please make a bug report, and include there the details about how
you started Emacs.  Also, please make sure you try reproducing the
problem with the latest emacs-25 branch of the Emacs repository, or at
least with the latest pretest of Emacs 25, because it's possible that
this problem was already solved.

Re: Quotes do not display correctly

[Nick Helm]

> On 8/04/2016, at 3:22 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> The quotes also appear to have a display table and face applied, but I
>> don’t understand the significance of this information, or if it’s a clue to
>> solving my problem.
> 
> It _is_ a clue.  But to understand what goes wrong, more information
> is needed, because Emacs was supposed to set up the display table only
> if your display is unable to show the Unicode quote characters.
> 
> So please make a bug report, and include there the details about how
> you started Emacs.  Also, please make sure you try reproducing the
> problem with the latest emacs-25 branch of the Emacs repository, or at
> least with the latest pretest of Emacs 25, because it's possible that
> this problem was already solved.

Thanks for the suggestions everyone,

Reproducing this from Emacs -Q has been difficult, but I think I've
narrowed it down. It's quite specific.

I'm on OS X and I launch Emacs using an Applescript script contained
within an Automator bundle. At startup, it essentially does this 
(with -Q added):

  do shell script "/Applications/Emacs.app/Contents/MacOS/Emacs -Q
                   --daemon " 
  do shell script "/Applications/Emacs.app/Contents/MacOS/bin/
                   emacsclient -n -c "

The problem only occurs when I launch this way - that is, Emacs does not
exhibit the quoting problem when I launch the /Applications/Emacs.app
bundle directly via double click, nor when started on the command line
using Emacs -Q --daemon and emacsclient -n -c, not even when starting 
the script from the command line with oascript.

Rather than being a bug, is it possible Emacs isn't receiving the
correct startup environment and is simply doing its best to display
extended chars on what it thinks is a terminal with limited display
capabilities? On the other hand, I've used this launch method for years
and this issue only surfaced recently. It's also interesting that other
Unicode chars display without any problems. I tested on 25.0.92.1 and
25.1.50.1, which both have the issue, and 24.5.1, which does not.

A few more clues. Where quotes do not show correctly, I can see a
different value for standard-display-table. Replacing this with a value
from an Emacs that does show them correctly fixes the problem. I did a
bit of reading and this led me to a new function named
startup--setup-quote-display in startup.el, which seems to manipulate
standard-display-table to let Emacs display Unicode quote char
equivalents in terminals that only support ASCII chars. It looks like
this function uses internal-char-font and char-displayable-p to
determine what changes to make, but I can't figure out how these
functions determine the capabilities of the terminal/display in the
first place.

Ok, more reading and it looks like it's determined by the default coding
system and language environment that Emacs picks up from the system
during startup. I have $LANG set in my environment, but for some reason
the script isn't seeing it. Changing it to something like this:

  do shell script "export LANG=en_NZ.UTF-8; /Applications/Emacs.app/
                   Contents/MacOS/Emacs -Q --daemon "

seems to fix the problem, at least standard-display-table remains
unchanged, and the quote chars display correctly in my buffers (and the
rest of Emacs it seems). text-quoting-style now seems to have its
intended effect as well.

I'm not sure if this is the correct solution, but it seems work.

Re: Quotes do not display correctly

[Bob Proulx]

Nick Helm wrote:
> I'm on OS X and I launch Emacs using an Applescript script contained
> within an Automator bundle. At startup, it essentially does this 
>...

> Ok, more reading and it looks like it's determined by the default coding
> system and language environment that Emacs picks up from the system
> during startup. I have $LANG set in my environment, but for some reason
> the script isn't seeing it.

How are you setting LANG?  In a ~/.bashrc file?  I don't know anything
about Apple OS X but in regular X11 settings in ~/.bashrc files only
affect interactive shells and not other programs that are not
interactive bash shells.  In order to set the environment in X Windows
for example one way would be in a .xsessionrc file which is
specifically loaded by the window system.  I would guess it is similar
with other systems.

> Changing it to something like this:
> 
>   do shell script "export LANG=en_NZ.UTF-8; /Applications/Emacs.app/
>                    Contents/MacOS/Emacs -Q --daemon "

That line break and indent seems unintentional.  Mailer problem?  I
assume it was meant to be this.  Which I am going to comment upon.

> do shell script "export LANG=en_NZ.UTF-8; /Applications/Emacs.app/Contents/MacOS/Emacs -Q --daemon "

> seems to fix the problem, at least standard-display-table remains
> unchanged, and the quote chars display correctly in my buffers (and the
> rest of Emacs it seems). text-quoting-style now seems to have its
> intended effect as well.
> 
> I'm not sure if this is the correct solution, but it seems work.

I wanted to comment upon the way you set the variable.  That works,
obviously.  But it isn't very idiomatic.  It is possible that the
inclusion of a ';' shell metacharacter in the string will require a
"/bin/sh -c" to be invoked in order to interpret the script.  Often
programs optimize the /bin/sh call out if there are no shell
meta-characters.  But the above requires it.

Better would be to use the "env" command to run something in a
modified environment.  It is one of the POSIX standard commands and
very portably found on systems.  This avoids the need for shell
meta-characters, may allow the system to optimize the shell out, and
allows very fine control of the environment.  With that it would be:

  env LANG=en_NZ.UTF-8 /Applications/Emacs.app/Contents/MacOS/Emacs -Q --daemon

Having said all of the above I am not sure this is the best solution.
This is simply the first environment variable you have set somewhere
(~/.bashrc ?) and you may run into other problems.  It might be better
to figure out how to get your environment set somewhere, such as in
the X11 ~/.xsessionrc file case, rather than play the wack-a-mole game
hitting individual environment variables one by one as you run into
problems not having them.

Good Luck and Happy Hacking,
Bob

Re: Quotes do not display correctly

[Nick Helm]

> How are you setting LANG?  In a ~/.bashrc file?  I don't know anything
> about Apple OS X but in regular X11 settings in ~/.bashrc files only
> affect interactive shells and not other programs that are not
> interactive bash shells.  

I have LANG set in my ~/.bash_profile. As I understand it, OSX runs a new
login shell each for each Terminal.app instance, so I have never bothered
to create a ~/.bashrc file on Macs.

> In order to set the environment in X Windows for example one way would be in
> a .xsessionrc file which is specifically loaded by the window system. I
> would guess it is similar with other systems.

I think that's pretty much how it works on OSX too, as I understand it.
Although there seems to be several ways to do it, including setting variables
in /etc/launchd.conf (the global config file for OSX's service manager/process
launcher), or creating ~/Library/LaunchAgents/environment.plist, which seems
to be an XML script that launchd uses to configure the environment when
launching an app.

> That line break and indent seems unintentional.  Mailer problem?  I
> assume it was meant to be this.  Which I am going to comment upon.

Yes, a problem manually introduced by me. Sorry about that.

> I wanted to comment upon the way you set the variable.  That works,
> obviously.  But it isn't very idiomatic.  It is possible that the
> inclusion of a ';' shell metacharacter in the string will require a
> "/bin/sh -c" to be invoked in order to interpret the script.  Often
> programs optimize the /bin/sh call out if there are no shell
> meta-characters.  But the above requires it.
>  
> Better would be to use the "env" command to run something in a
> modified environment.  It is one of the POSIX standard commands and
> very portably found on systems.  This avoids the need for shell
> meta-characters, may allow the system to optimize the shell out, and
> allows very fine control of the environment.  With that it would be:
> 
>  env LANG=en_NZ.UTF-8 /Applications/Emacs.app/Contents/MacOS/Emacs -Q --daemon

Thank you, this is good advice. At the time, I was convinced my script was at
fault, so I was attempting to use export as a way to force an environment
change within each "do shell script" to test what was going on. I didn't
know I could use env in this way.

> Having said all of the above I am not sure this is the best solution.
> This is simply the first environment variable you have set somewhere
> (~/.bashrc ?) and you may run into other problems.  It might be better
> to figure out how to get your environment set somewhere, such as in
> the X11 ~/.xsessionrc file case, rather than play the wack-a-mole game
> hitting individual environment variables one by one as you run into
> problems not having them.

I agree, this not ideal, so I removed the export commands in my script and
tweaked my ~/Library/LaunchAgents/environemt.plist to include the 
following (XML omitted):

   launchctl setenv LANG en_NZ.UTF-8 
   launchctl setenv TERM xterm-256color
   launchctl setenv HOME /Users/nick 
   launchctl setenv PATH /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin 

This also fixes my quoting problem and I can interrogate LANG and the other
variables from within Emacs with `getenv' and get the values I expect. As a
bonus, it now looks like Emacs is getting the correct PATH, which, up until
now, I’ve had to fudge with this ancient tweak in my init.el:

   (setenv "PATH" (concat "/usr/local/bin:" (getenv "PATH"))) 
   (setq exec-path (append '("/usr/local/bin") exec-path))

Which suggests I've played whack-a-mole before, so thanks for the prompt to
sort this out.

Re: Quotes do not display correctly

[Pascal J. Bourguignon]

Nick Helm <nick@tenpoint.co.nz> writes:

>> How are you setting LANG?  In a ~/.bashrc file?  I don't know anything
>> about Apple OS X but in regular X11 settings in ~/.bashrc files only
>> affect interactive shells and not other programs that are not
>> interactive bash shells.  
>
> I have LANG set in my ~/.bash_profile.

Why?

[pjb@kuiper :0.0 ~]$ LANG=fr_FR date
Fri Apr 15 19:34:04 CEST 2016
[pjb@kuiper :0.0 ~]$ LC_ALL=fr_FR date
vendredi 15 avril 2016, 19:34:08 (UTC+0200)
[pjb@kuiper :0.0 ~]$



-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: Quotes do not display correctly

[Eli Zaretskii]

> From: Nick Helm <nick@tenpoint.co.nz>
> Date: Fri, 8 Apr 2016 02:20:49 +0000
> 
> I'm on OS X and I launch Emacs using an Applescript script contained
> within an Automator bundle. At startup, it essentially does this 
> (with -Q added):
> 
>   do shell script "/Applications/Emacs.app/Contents/MacOS/Emacs -Q
>                    --daemon " 
>   do shell script "/Applications/Emacs.app/Contents/MacOS/bin/
>                    emacsclient -n -c "
> 
> The problem only occurs when I launch this way - that is, Emacs does not
> exhibit the quoting problem when I launch the /Applications/Emacs.app
> bundle directly via double click, nor when started on the command line
> using Emacs -Q --daemon and emacsclient -n -c, not even when starting 
> the script from the command line with oascript.
> 
> Rather than being a bug, is it possible Emacs isn't receiving the
> correct startup environment and is simply doing its best to display
> extended chars on what it thinks is a terminal with limited display
> capabilities?

Most probably, yes.

> On the other hand, I've used this launch method for years
> and this issue only surfaced recently. It's also interesting that other
> Unicode chars display without any problems. I tested on 25.0.92.1 and
> 25.1.50.1, which both have the issue, and 24.5.1, which does not.

The feature of displaying ‘..’ as `..' on less-capable displays is new
in Emacs 25.  Since Emacs now produces such characters when it
displays doc strings, it was important to make sure these characters
are always shown as something legible.

> A few more clues. Where quotes do not show correctly, I can see a
> different value for standard-display-table. Replacing this with a value
> from an Emacs that does show them correctly fixes the problem.

You can work around the problem by changing the value of
standard-display-table, indeed.

> I did a bit of reading and this led me to a new function named
> startup--setup-quote-display in startup.el, which seems to
> manipulate standard-display-table to let Emacs display Unicode quote
> char equivalents in terminals that only support ASCII chars. It
> looks like this function uses internal-char-font and
> char-displayable-p to determine what changes to make, but I can't
> figure out how these functions determine the capabilities of the
> terminal/display in the first place.

In a GUI session, they look at the standard fonts; in a TTY session,
they look at the locale's encoding.  So Bob's advice to set LANG
correctly might be a better solution than messing with
standard-display-table.

> Ok, more reading and it looks like it's determined by the default coding
> system and language environment that Emacs picks up from the system
> during startup. I have $LANG set in my environment, but for some reason
> the script isn't seeing it. Changing it to something like this:
> 
>   do shell script "export LANG=en_NZ.UTF-8; /Applications/Emacs.app/
>                    Contents/MacOS/Emacs -Q --daemon "
> 
> seems to fix the problem, at least standard-display-table remains
> unchanged, and the quote chars display correctly in my buffers (and the
> rest of Emacs it seems). text-quoting-style now seems to have its
> intended effect as well.
> 
> I'm not sure if this is the correct solution, but it seems work.

Setting LANG to a correct value is the correct solution.

Re: Quotes do not display correctly

[Pascal J. Bourguignon]

Bob Proulx <bob@proulx.com> writes:

> Nick Helm wrote:
>> I'm on OS X and I launch Emacs using an Applescript script contained
>> within an Automator bundle. At startup, it essentially does this 
>>...
>
>> Ok, more reading and it looks like it's determined by the default coding
>> system and language environment that Emacs picks up from the system
>> during startup. I have $LANG set in my environment, but for some reason
>> the script isn't seeing it.
>
> How are you setting LANG?

LANG, and LC_*, and LC_ALL.

When LC_ALL is set, it's prioritary over the other environment
variables!

When one of the LC_* environment variables is set, they're prioritary
over LANG!

LANG is used only in last recourse.


cf. http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk