Character with codepoint #o223 is displayed as \223, do I have a font problem?

Character with codepoint #o223 is displayed as \223, do I have a font problem?

[N. Jackson]

I'm finding that the name "Oscar" (with an accent on the initial letter)
is displayed as

    Óscar

That first character (the accented A) is:

                 position: 258 of 309 (83%), column: 41
                character: Ã (displayed as Ã) (codepoint 195, #o303, #xc3)
        preferred charset: unicode (Unicode (ISO10646))
    code point in charset: 0xC3
                   script: latin
                   syntax: w 	which means: word
                 category: .:Base, L:Left-to-right (strong), j:Japanese, l:Latin, v:Viet
                 to input: type "C-x 8 RET c3" or "C-x 8 RET LATIN CAPITAL LETTER A WITH TILDE"
              buffer code: #xC3 #x83
                file code: #xC3 #x83 (encoded by coding system utf-8-unix)
                  display: by this font (glyph code)
        xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1 (#x85)

    Character code properties: customize what to show
      name: LATIN CAPITAL LETTER A WITH TILDE
      old-name: LATIN CAPITAL LETTER A TILDE
      general-category: Lu (Letter, Uppercase)
      decomposition: (65 771) ('A' '̃')

    There are text properties here:
      fontified            t

and the second character is:

                 position: 259 of 309 (83%), column: 42
                character: “ (displayed as “) (codepoint 147, #o223, #x93)
        preferred charset: unicode (Unicode (ISO10646))
    code point in charset: 0x93
                   syntax: w 	which means: word
                 category: l:Latin
                 to input: type "C-x 8 RET 93" or "C-x 8 RET SET TRANSMIT STATE"
              buffer code: #xC2 #x93
                file code: #xC2 #x93 (encoded by coding system utf-8-unix)
                  display: by this font (glyph code)
        xft:-PfEd-Unifont-normal-normal-normal-*-12-*-*-*-d-0-iso10646-1 (#x96)

    Character code properties: customize what to show
      old-name: SET TRANSMIT STATE
      general-category: Cc (Other, Control)
      decomposition: (147) ('“')

    There are text properties here:
      fontified            t

Does this mean that the character with codepoint #o223 is missing from
DejaVu Sans Mono (my default font), or that something else is wrong? (I
tried setting my default font to several other faces but didn't see any
change.)

I'm guessing it's some sort of composition problem, which takes me a
long way beyond anything I know anything about!

Thanks in advance for any pointers,
N.

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Mar 18, 2016 at 12:17:58PM -0300, N. Jackson wrote:
> I'm finding that the name "Oscar" (with an accent on the initial letter)
> is displayed as
> 
>     Óscar
> 
> That first character (the accented A) is:
> 
>                  position: 258 of 309 (83%), column: 41
>                 character: Ã (displayed as Ã) (codepoint 195, #o303, #xc3)
>         preferred charset: unicode (Unicode (ISO10646))
>     code point in charset: 0xC3
>                    script: latin
>                    syntax: w 	which means: word
>                  category: .:Base, L:Left-to-right (strong), j:Japanese, l:Latin, v:Viet
>                  to input: type "C-x 8 RET c3" or "C-x 8 RET LATIN CAPITAL LETTER A WITH TILDE"
>               buffer code: #xC3 #x83
>                 file code: #xC3 #x83 (encoded by coding system utf-8-unix)
>                   display: by this font (glyph code)
>         xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1 (#x85)
> 
>     Character code properties: customize what to show
>       name: LATIN CAPITAL LETTER A WITH TILDE
>       old-name: LATIN CAPITAL LETTER A TILDE
>       general-category: Lu (Letter, Uppercase)
>       decomposition: (65 771) ('A' '̃')
> 
>     There are text properties here:
>       fontified            t
> 
> and the second character is:
> 
>                  position: 259 of 309 (83%), column: 42
>                 character: “ (displayed as “) (codepoint 147, #o223, #x93)
>         preferred charset: unicode (Unicode (ISO10646))
>     code point in charset: 0x93
>                    syntax: w 	which means: word
>                  category: l:Latin
>                  to input: type "C-x 8 RET 93" or "C-x 8 RET SET TRANSMIT STATE"
>               buffer code: #xC2 #x93
>                 file code: #xC2 #x93 (encoded by coding system utf-8-unix)
>                   display: by this font (glyph code)
>         xft:-PfEd-Unifont-normal-normal-normal-*-12-*-*-*-d-0-iso10646-1 (#x96)
> 
>     Character code properties: customize what to show
>       old-name: SET TRANSMIT STATE
>       general-category: Cc (Other, Control)
>       decomposition: (147) ('“')
> 
>     There are text properties here:
>       fontified            t
> 
> Does this mean that the character with codepoint #o223 is missing from
> DejaVu Sans Mono (my default font), or that something else is wrong? (I
> tried setting my default font to several other faces but didn't see any
> change.)

No, that's not a font problem. That's an utf8 character "interpreted"
as an 8 bit character set (most probably iso-8859-1 aka latin 1 or some
near cousin).

In utf8, Ó (capital letter O with acute) is represented by the byte
sequence 0xc3 0x93, which is what you have above. But in iso-8859-1,
0xc3 is a capital A with tilde, the 93 is in some non-printable area
and tends to look funny.

> I'm guessing it's some sort of composition problem, which takes me a
> long way beyond anything I know anything about!

I guess that your Emacs is trying the wrong encoding. There is a heuristic
to decide on that (the files themselves don't "know" their encoding, so
Emacs has to guess).

How did you get at the text?

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlbsGJoACgkQBcgs9XrR2kZpygCfYX/Me0waTpwryhr1+0yHswk9
GXgAn1/x8L5y/Js0ASfFebcqou/C9lYS
=UkLP
-----END PGP SIGNATURE-----

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[N. Jackson]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

At 16:02 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:
> 
> On Fri, Mar 18, 2016 at 12:17:58PM -0300, N. Jackson wrote:
>> I'm finding that the name "Oscar" (with an accent on the initial letter)
>> is displayed as
>> 
>>     Óscar
>> 
> No, that's not a font problem. That's an utf8 character "interpreted"
> as an 8 bit character set (most probably iso-8859-1 aka latin 1 or
> some near cousin).
>
> In utf8, Ó (capital letter O with acute) is represented by the byte
> sequence 0xc3 0x93, which is what you have above. But in iso-8859-1,
> 0xc3 is a capital A with tilde, the 93 is in some non-printable area
> and tends to look funny.

At 17:47 +0200 on Friday 2016-03-18, Eli Zaretskii wrote:
>
> it means that Emacs decoded a UTF-8 encoded text as Latin-1. And then
> the result which produced 2 characters out of one, was re-encoded as
> UTF-8.

> composition is not involved. It's just that the UTF-8 encoding of Ó is
> a pair of bytes \303\223.

Ah, I see. Thank you both for the explanations and education. 

At 16:02 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:
> 
> How did you get at the text?

It was displayed in a message in Gnus. (It was an email message to the
Emacs devel mailing list (from Eli, coincidentally), but I was reading
it as news through Gmane, which unfortunately adds an extra layer of
complication.)

In the raw text of the message, the text is

    =C3=83=C2=93scar

and the message includes the following headers:

    Mime-Version: 1.0
    Content-Type: text/plain; charset=utf-8
    Content-Transfer-Encoding: QUOTED-PRINTABLE

FWIW, the raw text of the message is attached.

The message in question can also be viewed in the mailing list at
http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg01020.html
where the text displays (in my browser) as:

    Ãscar

I'm wondering if there's a way to get Gnus to handle such text
correctly, or if the problem is in the original message.

N.


[-- Attachment #2: Raw text of the problematic message. --]
[-- Type: text/plain, Size: 11155 bytes --]

Path: news.gmane.org!not-for-mail
From: Eli Zaretskii <eliz@gnu.org>
Newsgroups: gmane.emacs.devel
Subject: Re: Move to a cadence release model?
Date: Wed, 11 Nov 2015 17:37:32 +0200
Lines: 155
Approved: news@gmane.org
Message-ID: <837flolf7n.fsf@gnu.org>
References: <CAJnXXogfOJEO3fbiVnE-v3q_KAM71wO_YUw7NBehwTE4S1SMsQ@mail.gmail.com>
	<E1ZwDWb-00077I-Go@fencepost.gnu.org>
	<CAAF+z6FbrbEt5=Z3kr_zSmqfGPXjAYns3B+83+W1FvyH9v5z4w@mail.gmail.com>
	<m2io59tnj7.fsf@Vulcan.attlocal.net>
	<CAJnXXoi0_zAJqMUDN+gB8arRJEkbpkqn6y53HS9qm4Sw2C6V2Q@mail.gmail.com>
Reply-To: Eli Zaretskii <eliz@gnu.org>
NNTP-Posting-Host: plane.gmane.org
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: QUOTED-PRINTABLE
X-Trace: ger.gmane.org 1447256340 2787 80.91.229.3 (11 Nov 2015 15:39:00 GMT)
X-Complaints-To: usenet@ger.gmane.org
NNTP-Posting-Date: Wed, 11 Nov 2015 15:39:00 +0000 (UTC)
Cc: xfq.free@gmail.com, rms@gnu.org, emacs-devel@gnu.org
To: John Yates <john@yates-sheets.org>
Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Nov 11 16:38:50 2015
Return-path: <emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org>
Envelope-to: ged-emacs-devel@m.gmane.org
Original-Received: from lists.gnu.org ([208.118.235.17])
	by plane.gmane.org with esmtp (Exim 4.69)
	(envelope-from <emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org>)
	id 1ZwXU1-0001b5-9Q
	for ged-emacs-devel@m.gmane.org; Wed, 11 Nov 2015 16:38:49 +0100
Original-Received: from localhost ([::1]:41298 helo=lists.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org>)
	id 1ZwXU0-0000ne-Ge
	for ged-emacs-devel@m.gmane.org; Wed, 11 Nov 2015 10:38:48 -0500
Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:50180)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eliz@gnu.org>) id 1ZwXTc-0000fn-7D
	for emacs-devel@gnu.org; Wed, 11 Nov 2015 10:38:28 -0500
Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eliz@gnu.org>) id 1ZwXTY-000772-Tu
	for emacs-devel@gnu.org; Wed, 11 Nov 2015 10:38:24 -0500
Original-Received: from mtaout27.012.net.il ([80.179.55.183]:53489)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@gnu.org>)
	id 1ZwXTY-00076m-Fl; Wed, 11 Nov 2015 10:38:20 -0500
Original-Received: from conversion-daemon.mtaout27.012.net.il by mtaout27.012.net.il
	(HyperSendmail v2007.08) id <0NXN00500PQB5800@mtaout27.012.net.il>;
	Wed, 11 Nov 2015 17:32:45 +0200 (IST)
Original-Received: from HOME-C4E4A596F7 ([84.94.185.246]) by mtaout27.012.net.il
	(HyperSendmail v2007.08) with ESMTPA id
	<0NXN00MKCPUK7V70@mtaout27.012.net.il>;
	Wed, 11 Nov 2015 17:32:44 +0200 (IST)
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and
	truncated.
In-reply-to: <CAJnXXoi0_zAJqMUDN+gB8arRJEkbpkqn6y53HS9qm4Sw2C6V2Q@mail.gmail.com>
X-012-Sender: halo1@inter.net.il
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 80.179.55.183
X-BeenThere: emacs-devel@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "Emacs development discussions." <emacs-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/emacs-devel>,
	<mailto:emacs-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/emacs-devel>
List-Post: <mailto:emacs-devel@gnu.org>
List-Help: <mailto:emacs-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/emacs-devel>,
	<mailto:emacs-devel-request@gnu.org?subject=subscribe>
Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org
Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org
Xref: news.gmane.org gmane.emacs.devel:194070
Archived-At: <http://permalink.gmane.org/gmane.emacs.devel/194070>

> Date: Tue, 10 Nov 2015 20:55:59 -0500
> From: John Yates <john@yates-sheets.org>
>=20
> Re Eli's points: In my short time at Mathworks it is clear to me th=
at the model is very different from what he imagines.  At Mathworks a=
 culture of cadence-base release starts with an absolute awareness th=
at no one knows what will be in the next release.

I sincerely doubt that.  Someone in the company's management must
know.  Moreover, someone must actively _control_ that.  You cannot ru=
n
a successful business any other way.

> Concretely that translates into _never_ making such commitments to =
customers.

I never said it should be known to customers.  It should, however, be
known internally.  Otherwise, Mathworks is just another chaotic firm
which we have nothing to learn from (and can actually teach them some=
 ;-).

> I am told that what corresponds to "roadmap presentations" come dow=
n to describing areas that the company views as strategic and hence w=
here it is investing development resources.

There you go: we don't have any such roadmap in Emacs.  And I actuall=
y
very much doubt if we can practically have one.  At least all past
attempts to start something like that failed spectacularly.  Countles=
s
discussions about adding WYSIWYG word-processor like editing features=
,
developing an Emacs IDE, etc. -- these all are attempts to set just
the first milestone on that roadmap.  They all eat dust every single
time the issues are raised here.  Doesn't that say something about ou=
r
culture and preferences?

> The clear message is that shipping a high-quality product on a pred=
ictable schedule is more important than delaying the release or shipp=
ing something buggy.

Emacs doesn't "ship buggy", so that's not the issue here.

As for predictable release schedule, whether this is important should
be based on user surveys.  Did someone make such surveys for Emacs?  =
I
don't think so.  FWIW, it makes little sense to me to install a new
non-bugfix release of a major package such as Emacs without getting
significant new features, but I'm just one user, and most probably no=
t
the typical one.

In any case, if we want to focus on frequent high-quality releases, w=
e
should work on our QA.  That's the point =C3=83=C2=93scar was making:=
 our QA is
abysmally inadequate.

> rapid availability of bug fix releases containing _nothing_ but hig=
h priority bug fixes.

That is relatively much easier, and we already tried that on the
Emacs-24 branch: the last bugfix release was out the door in about 10
days since the first pretest (which we declared an RC).  But this is
only possible on the release branch, and only if we allow absolutely
nothing on that branch except relatively safe bug fixes.

> I have been thinking about the difference between Emacs' developmen=
t culture and that of project with a strong commitment to maintaining=
 a cadence.  My conclusion is that it comes down to the effort put in=
to keeping the master branch so healthy that a release branch could b=
e cut at nearly any moment.  Inevitably this comes with a strong emph=
asis on code review, gatekeeping and usually continuous testing.  The=
 Linux kernel is somewhat atypical in having its gatekeeping function=
  entrusted entirely to humans (Linus' lieutenants and their underlin=
gs).  (I am unfamiliar with Firefox.  I did read https://www.mozilla.=
org/en-US/about/governance/policies/commit/access-policy/ but failed =
to get a good sense of how things work there.)  All of the remaining =
projects (Chromium, LibreOffice and OpenStack) use gerrit to enforce =
a review processes (and to support continuous integration testing).  =
The important part of all of these review cultures is that work _must=
_ be presented in reviewable uni

So we essentially agree: switching to any cadence model requires deep
changes in how Emacs development works, which in turn requires a much
larger core development team and a very different organization of tha=
t
team.  E.g., a gatekeeper model won't work without a gatekeeper.  We
can try working towards such new models, if we believe we can achieve
them.  But until we get close to that goal, discussing such radically
different models will remain a pipe dream at best.

> Re current pattern of 6 month code freeze: This seems to be a manif=
estation of that fact that once a sufficient collection of new featur=
es have been accumulated we recognize in our heart of hearts that our=
 code is not ready for release.  At a minimum it has not necessarily =
been built on all of the platforms we claim to support, various featu=
re have received waivers (hence are incomplete) and lots of documenta=
tion remains to be written.  Moving more rapidly from cutting a relea=
se branch to asserting a release-ready tarball requires addressing th=
ose cultural patterns.

There is no 6-month code freeze.  There are long pretest periods for
major version releases.  Emacs 21.1, which included a complete rewrit=
e
of the display engine and many other significant changes, took 11
months since the first pretest till the release.  Emacs 22.1 took 7.5
months, Emacs 23.1 5 months, Emacs 24.1 8.5 months.  A large portion
of that time is spent updating the documentation to match the changes=
,
especially documenting the new features, and then proof-reading the
manuals.  (I'm guessing at Mathworks you don't have developers writin=
g
and proof-reading the documentation, you have special staff for that.
We don't have such luxury.)  Slashing those months-long pretest
periods means requesting that every user-visible change be accompanie=
d
by the corresponding documentation changes -- are we prepared to do
that?  Do we have enough people in place to review and comment on
those documentation changes if and when they do come?  Don't forget
that some of us cannot write good English documentation and/or don't
command written technical English enough to produce something decent.
And I didn't even start talking about how many of us _want_ to deal
with documentation even if their English is fine.

The other reason for the long pretests is indeed the need to test the
upcoming release on a wide array of different platforms and system
configurations.  We used to have a group of pretesters who represente=
d
the supported platforms, and who were instrumental in that process.
We no longer seem to have such a group, and the platforms we care
about changed anyway.  As result, our coverage of the possible
configurations is mostly random, and we can no longer be sure that a
long enough pretest period will shake out enough bugs.  Note that it
is not enough to just build on a platform, you must actually use Emac=
s
there to find bugs.  And since a typical user uses only a small
fraction of Emacs features, a single tester per platform/configuratio=
n
is not enough for thorough testing.  Some work in that area is needed=
,
e.g. to identify the features which might be platform-dependent
(example: file notifications), and then compile a list of platforms
that share the same implementation of a given feature.  Then we need =
a
few testers for each such group, and we need to inform them about eac=
h
pretest (there used to be a mailing list for that) and be sure to
collect their reports.

Lots of job opportunities there, and too few volunteers...

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Mar 18, 2016 at 04:26:06PM -0300, N. Jackson wrote:
> At 16:02 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:

[...]

> At 16:02 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:
> > 
> > How did you get at the text?
> 
> It was displayed in a message in Gnus. (It was an email message to the
> Emacs devel mailing list (from Eli, coincidentally), but I was reading
> it as news through Gmane, which unfortunately adds an extra layer of
> complication.)
> 
> In the raw text of the message, the text is
> 
>     =C3=83=C2=93scar

Hm. This looks as if the catastrophe already happened at the other end
(somewhere between the sender and you). This looks like *two* UTF-8
characters, a capital letter A with tilde (Ã) and codepoint 93 (this
is control character "SET TRANSMIT STATE" somewhere in the limbo between
0x80 and 0x9f).

This mght happen when something in the transfer chain interprets the
(utf8) Ó (capital O with acute) as Latin1 and something "converts" this
"Latin1" again to utf8. If that's the quoted-printable which arrives
at you it seems the problem is beyond your control. You can, of course,
"fudge" it back...

How do you "see" the non-ascii characters in *this* mail?

regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlbsX80ACgkQBcgs9XrR2kaXzwCfTcZn0wLD0fuswz1Zgf1yR0gm
KkQAn1daHS/3YTqwl2TYEphpNxnsuGbk
=Pagq
-----END PGP SIGNATURE-----

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[N. Jackson]

At 21:06 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:
>
> How do you "see" the non-ascii characters in *this* mail?

Just fine, I think. Ã displays as a capital 'A' with a tilde on top. Ó
displays as a capital 'O' with an acute accent.

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[Emanuel Berg]

nljlistbox2@gmail.com (N. Jackson) writes:

>> How do you "see" the non-ascii characters in
>> *this* mail?
>
> Just fine, I think. Ã displays as a capital 'A' with
> a tilde on top. Ó displays as a capital 'O' with an
> acute accent.

Those two chars are different in more ways than being
different chars.

Because in the Linux VTs - obviously the best place to
be - I only see the acute accent'd O - the other is
a diamond.

What does it look like to you in the above citation?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[Eli Zaretskii]

> From: nljlistbox2@gmail.com (N. Jackson)
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 18 Mar 2016 16:26:06 -0300
> 
> At 16:02 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:
> > 
> > How did you get at the text?
> 
> It was displayed in a message in Gnus. (It was an email message to the
> Emacs devel mailing list (from Eli, coincidentally), but I was reading
> it as news through Gmane, which unfortunately adds an extra layer of
> complication.)
> 
> In the raw text of the message, the text is
> 
>     =C3=83=C2=93scar
> 
> and the message includes the following headers:
> 
>     Mime-Version: 1.0
>     Content-Type: text/plain; charset=utf-8
>     Content-Transfer-Encoding: QUOTED-PRINTABLE
> 
> FWIW, the raw text of the message is attached.
> 
> The message in question can also be viewed in the mailing list at
> http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg01020.html
> where the text displays (in my browser) as:
> 
>     Ãscar
> 
> I'm wondering if there's a way to get Gnus to handle such text
> correctly, or if the problem is in the original message.

The problem was in the original message.  Gnus behaves correctly.

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Mar 18, 2016 at 10:26:46PM +0200, Eli Zaretskii wrote:
[...]

> The problem was in the original message.  Gnus behaves correctly.

Yes, my take too.

regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlbsYAgACgkQBcgs9XrR2kb+KACfW0pcB6kExzxuIQGyQ8YuQg/J
E5cAnjlSWXBukhtO9ea3YHN3sVkTZi9c
=Lluy
-----END PGP SIGNATURE-----

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[N. Jackson]

Hi Eli and tomás,

At 21:07 +0100 on Friday 2016-03-18, tomas@tuxteam.de wrote:
>
> On Fri, Mar 18, 2016 at 10:26:46PM +0200, Eli Zaretskii wrote:
> [...]
>> The problem was in the original message.  Gnus behaves correctly.
>
> Yes, my take too.

Thank you both.

I learned a bit and I don't have anything to fix, so that's all good! :)

N.

Re: Character with codepoint #o223 is displayed as \223, do I have a font problem?

[Eli Zaretskii]

> From: nljlistbox2@gmail.com (N. Jackson)
> Date: Fri, 18 Mar 2016 12:17:58 -0300
> 
> I'm finding that the name "Oscar" (with an accent on the initial letter)
> is displayed as
> 
>     Óscar
> 
> That first character (the accented A) is:
> 
>                  position: 258 of 309 (83%), column: 41
>                 character: Ã (displayed as Ã) (codepoint 195, #o303, #xc3)
>         preferred charset: unicode (Unicode (ISO10646))
>     code point in charset: 0xC3
>                    script: latin
>                    syntax: w 	which means: word
>                  category: .:Base, L:Left-to-right (strong), j:Japanese, l:Latin, v:Viet
>                  to input: type "C-x 8 RET c3" or "C-x 8 RET LATIN CAPITAL LETTER A WITH TILDE"
>               buffer code: #xC3 #x83
>                 file code: #xC3 #x83 (encoded by coding system utf-8-unix)
>                   display: by this font (glyph code)
>         xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-12-*-*-*-m-0-iso10646-1 (#x85)
> 
>     Character code properties: customize what to show
>       name: LATIN CAPITAL LETTER A WITH TILDE
>       old-name: LATIN CAPITAL LETTER A TILDE
>       general-category: Lu (Letter, Uppercase)
>       decomposition: (65 771) ('A' '̃')
> 
>     There are text properties here:
>       fontified            t
> 
> and the second character is:
> 
>                  position: 259 of 309 (83%), column: 42
>                 character: “ (displayed as “) (codepoint 147, #o223, #x93)
>         preferred charset: unicode (Unicode (ISO10646))
>     code point in charset: 0x93
>                    syntax: w 	which means: word
>                  category: l:Latin
>                  to input: type "C-x 8 RET 93" or "C-x 8 RET SET TRANSMIT STATE"
>               buffer code: #xC2 #x93
>                 file code: #xC2 #x93 (encoded by coding system utf-8-unix)
>                   display: by this font (glyph code)
>         xft:-PfEd-Unifont-normal-normal-normal-*-12-*-*-*-d-0-iso10646-1 (#x96)
> 
>     Character code properties: customize what to show
>       old-name: SET TRANSMIT STATE
>       general-category: Cc (Other, Control)
>       decomposition: (147) ('“')
> 
>     There are text properties here:
>       fontified            t
> 
> Does this mean that the character with codepoint #o223 is missing from
> DejaVu Sans Mono (my default font), or that something else is wrong? (I
> tried setting my default font to several other faces but didn't see any
> change.)

No, it means that Emacs decoded a UTF-8 encoded text as Latin-1.  And
then the result which produced 2 characters out of one, was re-encoded
as UTF-8.

> I'm guessing it's some sort of composition problem, which takes me a
> long way beyond anything I know anything about!

No, composition is not involved.  It's just that the UTF-8 encoding of
Ó is a pair of bytes \303\223.