bug#3501: 23.0.94; Use Unicode in Info (?)

bug#3501: 23.0.94; Use Unicode in Info (?)

[Drew Adams]

emacs -Q
 
C-h r
g acknowledgements
 
Can't we use Unicode or in some other way use accented characters in
people's names (when appropriate)?
 
For example: Tibor S<imko, Franc,ois Pinard, Jan Dja"rv, W/lodek Bzyl,
Sascha Lu"decke, Arne J/orgensen. I'm assuming that punctuation
characters such as " and / should really be composed with one of the
other chars here - dunno. Dunno about a name such as Ken'ichi - is the
' a separator here, or should it be combined with some another
character?
 
In GNU Emacs 23.0.94.1 (i386-mingw-nt5.1.2600)
 of 2009-05-24 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'
 

bug#3501: 23.0.94; Use Unicode in Info (?)

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Mon, 8 Jun 2009 14:58:22 -0700
> Cc: 
> Reply-To: Drew Adams <drew.adams@oracle.com>, 3501@emacsbugs.donarmstrong.com
> 
> emacs -Q
>  
> C-h r
> g acknowledgements
>  
> Can't we use Unicode or in some other way use accented characters in
> people's names (when appropriate)?
>  
> For example: Tibor S<imko, Franc,ois Pinard, Jan Dja"rv, W/lodek Bzyl,
> Sascha Lu"decke, Arne J/orgensen. I'm assuming that punctuation
> characters such as " and / should really be composed with one of the
> other chars here - dunno. Dunno about a name such as Ken'ichi - is the
> ' a separator here, or should it be combined with some another
> character?

Are you asking that Emacs Info reader displays "S<" as Ŝ and "/o" as
ø?  Or are you asking that the Info files themselves include non-ASCII
characters?

What goes into the Info files depends on the directives in the Texinfo
sources and on the command line arguments used when makeinfo was
invoked to produce the Info files.  You will see in the `doc'
directory that we already invoke makeinfo with --enable-encoding
switch in some cases.

bug#3501: 23.0.94; Use Unicode in Info (?)

[Drew Adams]

> > Can't we use Unicode or in some other way use accented characters in
> > people's names (when appropriate)?
> >  
> > For example: Tibor S<imko, Franc,ois Pinard, Jan Dja"rv, W/lodek
> > Bzyl, Sascha Lu"decke, Arne J/orgensen. I'm assuming that punctuation
> > characters such as " and / should really be composed with one of the
> > other chars here - dunno. Dunno about a name such as Ken'ichi - is the
> > ' a separator here, or should it be combined with some another
> > character?
> 
> Are you asking that Emacs Info reader displays "S<" as S and "/o" as
> ø?

Yes. (Dunno about the particular fancy chars that should result, but yes.)

> Or are you asking that the Info files themselves include non-ASCII
> characters?

Dunno whether I'm asking that also. If it's important for some reason to what is
perceived by users, then yes; otherwise, no.

I was thinking of display - what the user sees. But it should, if possible,
affect also searching (including regexp searching) and any other behavior the
user can notice. The user should experience only the fancy character, both
visually and every other way. What is in the actual file is not important here,
except in so far as it might affect perception.

> What goes into the Info files depends on the directives in the Texinfo
> sources and on the command line arguments used when makeinfo was
> invoked to produce the Info files.  You will see in the `doc'
> directory that we already invoke makeinfo with --enable-encoding
> switch in some cases.

Whatever is already done is not sufficient in this regard, as indicated in the
original report: I don't see composed characters; I see punctuation in the
middle of people's names. I see J/orgensen, not Jørgensen.

bug#3501: 23.0.94; Use Unicode in Info (?)

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Mon, 8 Jun 2009 20:23:49 -0700
> 
> Whatever is already done is not sufficient in this regard, as indicated in the
> original report: I don't see composed characters; I see punctuation in the
> middle of people's names. I see J/orgensen, not Jørgensen.

Nothing is ``already done'' in the current Emacs.  Info files, at
least originally, were pure ASCII files by design, because Info
readers could not cope with non-ASCII characters.  If we want to
display such characters as their non-ASCII equivalents, we need to do
this in Emacs.

Personally, I don't like the idea of Emacs converting Info files for
display.  At the very least, it destroys the formatting of the text,
and at worst makes the text all but unreadable.  I think if it's so
important to us to have the names in their native scripts, we should
simply use UTF-8 in the Texinfo sources and use --enable-encoding when
we produce the Info files.

bug#3501: 23.0.94; Use Unicode in Info (?)

[Drew Adams]

> > Whatever is already done is not sufficient in this regard, 
> > as indicated in the original report: I don't see composed
> > characters; I see punctuation in the
> > middle of people's names. I see J/orgensen, not Jørgensen.
> 
> Nothing is ``already done'' in the current Emacs.

I was describing the current state of the pretest the bug was reported for,
nothing more. In this pretest, I see J/orgensen, and it would be better to see
Jørgensen.

> Info files, at least originally, were pure ASCII files by design,
> because Info readers could not cope with non-ASCII characters.
> If we want to display such characters as their non-ASCII equivalents,
> we need to do this in Emacs.
> 
> Personally, I don't like the idea of Emacs converting Info files for
> display.  At the very least, it destroys the formatting of the text,
> and at worst makes the text all but unreadable.  I think if it's so
> important to us to have the names in their native scripts, we should
> simply use UTF-8 in the Texinfo sources and use --enable-encoding when
> we produce the Info files.

That is the solution then. IIUYC, if the Texinfo source file for node
Acknowledgements used Unicode and --enable-encoding were used to produce the
Info file, then the effect would be as I described for the user: s?he would
perceive only the "fancy" chars.

If that's correct, then that sounds like the way to go. I certainly agree that
if things can be done from the outset in Unicode, and if that has the proper
effect in Info for the user (which is the point), then there is no reason to
start with ASCII source and somehow twiddle the display to come up with the
effect of composed chars. That's not what I had in mind at all. 

I assumed that the solution was just to use Unicode chars in the first place (in
the source files), which is what I think you're saying also.

When you asked whether I wanted the files or the Info experience to have the
fancy chars I replied the latter, since that is the _aim_. But if the former as
a means implies the latter as the end effect, then obviously that is the way to
go.

bug#3501: 23.0.94; Use Unicode in Info (?)

[Alan Mackenzie]

Hi, Drew!

On Mon, Jun 08, 2009 at 08:23:49PM -0700, Drew Adams wrote:
> > > Can't we use Unicode or in some other way use accented characters in
> > > people's names (when appropriate)?

> I was thinking of display - what the user sees. But it should, if
> possible, affect also searching (including regexp searching) and any
> other behavior the user can notice. The user should experience only the
> fancy character, both visually and every other way.

Please no.  I absolutely do not want to "experience" fancy unicode characters
when reading info.  It's bad enough getting them in email and in usenet
postings from Xah Lee.  ;-)  ASCII can be displayed perfectly on any
screen or teletype or even punched card puncher that can display English
at all.  Unicode, by contrast, needs a fancy setup, even if lots of
computers already have such a setup.

Unless accented hackers (of whom we're not two) find the ASCII rendering
of their names offensive, but I haven't seen any evidence of this.

> What is in the actual file is not important here, except in so far as
> it might affect perception.

> > What goes into the Info files depends on the directives in the
> > Texinfo sources and on the command line arguments used when makeinfo
> > was invoked to produce the Info files.  You will see in the `doc'
> > directory that we already invoke makeinfo with --enable-encoding
> > switch in some cases.

> Whatever is already done is not sufficient in this regard, as indicated
> in the original report: I don't see composed characters; I see
> punctuation in the middle of people's names. I see J/orgensen, not
> Jørgensen.

Is that bad?  J/orgensen is more readable (IMHO) than JÀ«rgensen (or
whatever that letter's two bytes actually are).

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#3501: 23.0.94; Use Unicode in Info (?)

[Drew Adams]

> Please no.  I absolutely do not want to "experience" fancy 
> unicode characters when reading info.  It's bad enough
> getting them in email and in usenet postings from Xah Lee. ;-)

What'll you tell your new Unicode toaster, when your old toaster breaks? ;-)
Better give up on iPhone and other such new-fangled gadgets altogether.

> ASCII can be displayed perfectly on any
> screen or teletype or even punched card puncher that can 
> display English at all.  Unicode, by contrast, needs a fancy setup,
> even if lots of computers already have such a setup.

Wow! We should reduce Emacs display to the level of a card punch? Presumably a
hand-crank model - no reason to assume electrical support.

Isn't Emacs capable of somehow knowing whether the current display can show
non-ASCII chars? If not, we'll forever remain with horse-and-buggy, I guess.

I'm not against coddling your sturdy old card punch, but not at the price of
giving up the world beyond ASCII for the rest of, well, the world beyond ASCII
(does your punch _really_ speak ASCII, or does it speak EBCDIC or perhaps Univac
field-data chars?).

Would you by the same token remove the possibility of Emacs files to use Unicode
chars? Library buff-menu.el uses utf-8 encoding, for example, and displays a
U+2014 (em dash char) if available. It tests like this:

;; Use U+2014 (EM DASH) to underline if possible,
;; else use ASCII (i.e. U+002D, HYPHEN-MINUS).
(if (char-displayable-p ?\u2014) ?\u2014 ?-)

Presumably something similar (but preferably more general) can be done to ensure
that your card punch can swallow J/orgensen if it can't digest Jørgensen.

> Is that bad?  J/orgensen is more readable (IMHO) than JÀ«rgensen (or
> whatever that letter's two bytes actually are).

Jørgensen is more readable than J/orgensen, but J/orgensen is fine for a card
punch. ;-)

bug#3501: 23.0.94; Use Unicode in Info (?)

[Lennart Borgman]

Hi ÅoAle/an

On Tue, Jun 9, 2009 at 7:17 PM, Alan Mackenzie<acm@muc.de> wrote:
>
> Is that bad?  J/orgensen is more readable (IMHO) than JÀ«rgensen (or
> whatever that letter's two bytes actually are).

I think you are a bit wrong on this point ;-)

bug#3501: 23.0.94; Use Unicode in Info (?)

[Alan Mackenzie]

'Evening, Drew!

On Tue, Jun 09, 2009 at 10:43:01AM -0700, Drew Adams wrote:
> > Please no.  I absolutely do not want to "experience" fancy 
> > unicode characters when reading info.  It's bad enough
> > getting them in email and in usenet postings from Xah Lee. ;-)

> What'll you tell your new Unicode toaster, when your old toaster
> breaks? ;-) Better give up on iPhone and other such new-fangled gadgets
> altogether.

Exactly the same I told the old toaster: they both understand ASCII.

> > ASCII can be displayed perfectly on any screen or teletype or even
> > punched card puncher that can display English at all.  Unicode, by
> > contrast, needs a fancy setup, even if lots of computers already have
> > such a setup.

> Isn't Emacs capable of somehow knowing whether the current display can
> show non-ASCII chars? If not, we'll forever remain with
> horse-and-buggy, I guess.

Emacs is capable of anything, provided you put enough effort into telling
it.  Assuming you're running on a pure ASCII display, or one running an
ISO-8859 character set (as I do), how much effort must you put into
telling Emacs (and standalone Info) that you really, really don't want
random Unicode bytes cluttering up your screen? 

> I'm not against coddling your sturdy old card punch, but not at the
> price of giving up the world beyond ASCII for the rest of, well, the
> world beyond ASCII (does your punch _really_ speak ASCII, or does it
> speak EBCDIC or perhaps Univac field-data chars?).

> Would you by the same token remove the possibility of Emacs files to
> use Unicode chars?

Not at all.  Quite a lot of people want Unicode, but quite a lot don't.
We shouldn't force it upon them.  

> Library buff-menu.el uses utf-8 encoding, for example, and displays a
> U+2014 (em dash char) if available. It tests like this:

> ;; Use U+2014 (EM DASH) to underline if possible,
> ;; else use ASCII (i.e. U+002D, HYPHEN-MINUS).
> (if (char-displayable-p ?\u2014) ?\u2014 ?-)

> Presumably something similar (but preferably more general) can be done
> to ensure that your card punch can swallow J/orgensen if it can't
> digest Jørgensen.

Who's volunteering to do this?

> > Is that bad?  J/orgensen is more readable (IMHO) than JÀ«rgensen (or
> > whatever that letter's two bytes actually are).

> Jørgensen is more readable than J/orgensen, but J/orgensen is fine for
> a card punch. ;-)

Possibly.  But I have just the _tiniest_ suspicion that this is the thin
end of the wedge, the crack in the dyke, the floodgates wanting to burst
open.  How long before people start using Xah's fancy 3-byte quote marks,
with associated dangly bits hanging off them?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#3501: 23.0.94; Use Unicode in Info (?)

[Drew Adams]

> > Isn't Emacs capable of somehow knowing whether the current 
> > display can show non-ASCII chars?
> 
> Emacs is capable of anything, provided you put enough effort 
> into telling it.  Assuming you're running on a pure ASCII
> display, or one running an ISO-8859 character set (as I do),
> how much effort must you put into telling Emacs (and
> standalone Info) that you really, really don't want
> random Unicode bytes cluttering up your screen?

I meant Emacs, not the user. Can't Emacs itself tell whether to show Unicode or
not? See my example regarding buff-menu.el. Emacs already decides whether to try
to show you images and other display features.

> > I'm not against coddling your sturdy old card punch, but not at the
> > price of giving up the world beyond ASCII for the rest of, well, the
> > world beyond ASCII (does your punch _really_ speak ASCII, or does it
> > speak EBCDIC or perhaps Univac field-data chars?).
> 
> > Would you by the same token remove the possibility of Emacs files to
> > use Unicode chars?
> 
> Not at all.  Quite a lot of people want Unicode, but quite a 
> lot don't. We shouldn't force it upon them.

If their Emacs supports Unicode, they would see the people's real names. If
their Emacs does not support Unicode, they would see an ASCII-art version of the
real names, like now. Where's the forcing? If they want to stick with ASCII,
they'll see ASCII.

If users want only black & white, they get it. If they don't want a mouse, they
don't need a mouse. Likewise menus, highlighting, and all the rest. No problem.

I agree that one should not need to have Unicode support to be able to use Info.
It does not follow that Info must present itself, by default and to everyone,
using only the lowest common denominator for every possible user interaction.

We've finally added font-locking by default, and we've gotten rid of vestigial
limits due to 900-baud communication - by default, at least. And you can turn
off font-locking.

No one stops a user from wearing a hair shirt and sleeping on nails in a
Siberian cabin without windows or running water. But that doesn't mean that the
default should be to expect that most users have such preferences.

Unicode is today's ASCII, like it or not. And I notice that you said you use ISO
8859. Why is that? Did you get tired of using `$' to represent a pound or a
euro? For you, I guess, whatever you use is OK for the default - ISO 8859 would
be OK for Info, instead of ASCII? 

Why don't you limit yourself to Extended ASCII, which has a pound sign and lots
of other "fancy" stuff (http://www.asciitable.com/)? Why go all of the way to
the wild side, to ISO 8859? Aren't you afraid that that will open the flood
gates to using such fancy characters (including character graphics) all over the
place? Who can trust users to behave themselves with such means in their hands?

> > Jørgensen is more readable than J/orgensen, but J/orgensen 
> > is fine for a card punch. ;-)
> 
> Possibly.  But I have just the _tiniest_ suspicion that this 
> is the thin end of the wedge, the crack in the dyke, the floodgates 
> wanting to burst open.  How long before people start using Xah's
> fancy 3-byte quote marks, with associated dangly bits hanging off them?

Fear can be a powerful motivator. But not necessarily a wise counselor.

It's not because you _can_ make text proportional or italic or magenta or
23-point or blinking or boxed or gothic that you _will_ do so everywhere. 

Your reaction reminds me of conservatives who try to deny access to information
and new possibilities because of fear of the unknown. Keep women at home or
covered in black. No libraries, so we don't incite the unwashed masses with
ideas they won't be able to handle responsibly. No voting for the ignorant or
unpropertied. Close the box, quick!

(How do you spell etidull?)

bug#3501: 23.0.94; Use Unicode in Info (?)

[Alan Mackenzie]

Hi, there!

On Tue, Jun 09, 2009 at 12:56:24PM -0700, Drew Adams wrote:
> > > Isn't Emacs capable of somehow knowing whether the current display
> > > can show non-ASCII chars?

> > Emacs is capable of anything, provided you put enough effort into
> > telling it.  Assuming you're running on a pure ASCII display, or one
> > running an ISO-8859 character set (as I do), how much effort must you
> > put into telling Emacs (and standalone Info) that you really, really
> > don't want random Unicode bytes cluttering up your screen?

> I meant Emacs, not the user. Can't Emacs itself tell whether to show
> Unicode or not? See my example regarding buff-menu.el. Emacs already
> decides whether to try to show you images and other display features.

> > Quite a lot of people want Unicode, but quite a lot don't. We
> > shouldn't force it upon them.

> If their Emacs supports Unicode, they would see the people's real
> names. If their Emacs does not support Unicode, they would see an
> ASCII-art version of the real names, like now. Where's the forcing? If
> they want to stick with ASCII, they'll see ASCII.

If you can do this, then great.  I don't think it can be done, in
general, without tarting up honest text files (like Info) with
un-displayed annotations.  UTF-8 isn't compatible with 8-bit codes like
8859-n.

> I agree that one should not need to have Unicode support to be able to
> use Info.  It does not follow that Info must present itself, by default
> and to everyone, using only the lowest common denominator for every
> possible user interaction.

For manuals written in English, it's a sensible optimisation.

> We've finally added font-locking by default, and we've gotten rid of
> vestigial limits due to 900-baud communication - by default, at least.
> And you can turn off font-locking.

> No one stops a user from wearing a hair shirt and sleeping on nails in
> a Siberian cabin without windows or running water. But that doesn't
> mean that the default should be to expect that most users have such
> preferences.

> Unicode is today's ASCII, like it or not.

No.  It's quite different.  ASCII was designed for elegance and ease of
use.  Unicode is a ghastly cludge, hacked together by some committee.
It's lacking in technical merit.  It's bodge nature is demonstrated by
the fact that people "have expertise" in it.  How many people "have
expertise" in ASCII?

The only thing in its favour is that it's used by "everybody", but then
so is Microsoft Windows.

We went through variable length encodings 30 years ago - George (the
operating system) had 6 bit characters, except some of them were the
equivalent of the <shift> key, or the <shift-lock> key.  It was such a
relief to get a sensible 7- or 8-bit uniform character set.  It's nice to
be able to count the number of characters in a string without needing a
finite state machine.

> And I notice that you said you use ISO 8859.  Why is that?

It's efficient for both English and German, and also does French and
Spanish (for the few times I need to write French and Spanish words).

> Did you get tired of using `$' to represent a pound or a euro? For you,
> I guess, whatever you use is OK for the default - ISO 8859 would be OK
> for Info, instead of ASCII? 

For me, personally, yes, but it would be a bad choice for Emacs users as
a whole - a bit like Unicode, really.

> Why don't you limit yourself to Extended ASCII, which has a pound sign
> and lots of other "fancy" stuff (http://www.asciitable.com/)? Why go
> all of the way to the wild side, to ISO 8859? Aren't you afraid that
> that will open the flood gates to using such fancy characters
> (including character graphics) all over the place? Who can trust users
> to behave themselves with such means in their hands?

You're not getting sarcastic here, are you, by any chance?  ;-)

> > > Jørgensen is more readable than J/orgensen, but J/orgensen 
> > > is fine for a card punch. ;-)

> > Possibly.  But I have just the _tiniest_ suspicion that this is the
> > thin end of the wedge, the crack in the dyke, the floodgates wanting
> > to burst open.  How long before people start using Xah's fancy 3-byte
> > quote marks, with associated dangly bits hanging off them?

> Fear can be a powerful motivator. But not necessarily a wise counselor.

> It's not because you _can_ make text proportional or italic or magenta
> or 23-point or blinking or boxed or gothic that you _will_ do so
> everywhere. 

I won't.  Many do, though.

> Your reaction reminds me of conservatives who try to deny access to
> information and new possibilities because of fear of the unknown.

That's unfair.  If I see somebody peddling something "new", I don't just
accept that it's automatically better.  Often, it's not.  Think of new
drugs which work no better than old ones.  That perl isn't better than
lisp, or MS Visual Studio better than Emacs, for example.  Unicode, with
its snakepit of variations, and coding schemas, and what have you,
certainly has its uses.  But just who was it that decided that Unicode is
the universal panacea?  I don't remember anybody asking me.  It was
stealthily foisted on us by somebody or other with interests different
from ours.  For the vast swathes of computer users, who use one single
language with possibly a bit of English on the side, something like
ISO-8859-n is technically far superior.

> Keep women at home or covered in black. No libraries, so we don't
> incite the unwashed masses with ideas they won't be able to handle
> responsibly. No voting for the ignorant or unpropertied. Close the box,
> quick!

> (How do you spell etidull?)

I don't.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#3501: 23.0.94; Use Unicode in Info (?)

[Lennart Borgman]

On Tue, Jun 9, 2009 at 11:10 PM, Alan Mackenzie<acm@muc.de> wrote:
>
> from ours.  For the vast swathes of computer users, who use one single
> language with possibly a bit of English on the side, something like
> ISO-8859-n is technically far superior.

Hi Alan,

China is huge. So is India.

The technical vision changed because of this and Internet.

bug#3501: 23.0.94; Use Unicode in Info (?)

[Jason Rumney]

Alan Mackenzie wrote:

> Emacs is capable of anything, provided you put enough effort into telling
> it.  Assuming you're running on a pure ASCII display, or one running an
> ISO-8859 character set (as I do), how much effort must you put into
> telling Emacs (and standalone Info) that you really, really don't want
> random Unicode bytes cluttering up your screen? 
>   
(set-terminal-coding-system 'us-ascii)

bug#3501: 23.0.94; Use Unicode in Info (?)

[Stefan Monnier]

[ I can't believe we got into this insane thread. ]

> Can't we use Unicode or in some other way use accented characters in
> people's names (when appropriate)?

Yes, of course, we should now use utf-8 for that Texinfo file and
pass --enable-encoding to makeinfo.


        Stefan