wikipedia and UTF8 coding (emacs21)

wikipedia and UTF8 coding (emacs21)

[Uwe Brauer]

Hello


I would like to edit wikipedia articles, using (X)emacs as an external
editor. The perl prg which I use for that purpose allows me to access
to the wiki file either 

    -  in iso8859-1 coding or 

    -  in UTF8.

The iso coding seem to be broken, I am using xemacs no mule so the
UTF8 option does not work, so I thought of using emacs21 for the
moment, however when opening the relevant file, the UTF8 is not
displayed correctly, I tried to play around with the language setting
but now the coding remains broken, that is I see symbols like Brüder
which should be displayed like Brüder but it is not.

Has anybody experience with wikipedia article using UTF8 coding.


Thanks


Uwe Brauer 

Re: wikipedia and UTF8 coding (emacs21)

[Peter Dyballa]

Am 03.11.2005 um 15:37 schrieb Uwe Brauer:

> Has anybody experience with wikipedia article using UTF8 coding.
>

I'm not experienced with this, but I am sure that you need at least two 
things:

	1. fonts (in X11 or such) with these glyphs
	2. GNU Emacs 23 from CVS

GNU Emacsen 21.4 and 22 (from CVS) fail to show a particular glyph when 
it's not in the font or fontset used. GNU Emacs 23 is able to get the 
missing glyph from other fonts -- usually at the wrong size.

But I doubt that you're on the right path! HTML uses Brüder for 
your example ...

--
Mit friedvollen Grüßen

   Pete

The human animal differs from the lesser primates in his passion for 
lists of "Ten Best".
		-- H. Allen Smith

Re: wikipedia and UTF8 coding (emacs21)

[Stefan Monnier]

> The iso coding seem to be broken, I am using xemacs no mule so the
> UTF8 option does not work, so I thought of using emacs21 for the
> moment, however when opening the relevant file, the UTF8 is not
> displayed correctly,

How is it displayed incorrectly?  Do you see some other chars than what you
expect, or do you see empty squares?  If you see empty squares (i.e. chars
for which Emacs can't find a font), place point on one of them and then do
C-u C-x = which will tell you what this char is.

> I tried to play around with the language setting but now the coding
> remains broken, that is I see symbols like Brüder which should be
> displayed like Brüder but it is not.

If UTF-8 is not auto-detected, try (prefer-coding-system 'utf-8).
Or if you don't want utf-8 to be your default and you know the the general
shape of the names of your wikipedia files, you can play with
file-coding-system-alist.


        Stefan

Re: wikipedia and UTF8 coding (emacs21)

[Uwe Brauer]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

   >> The iso coding seem to be broken, I am using xemacs no mule so the
   >> UTF8 option does not work, so I thought of using emacs21 for the
   >> moment, however when opening the relevant file, the UTF8 is not
   >> displayed correctly,

   Stefan> How is it displayed incorrectly?  Do you see some other
   Stefan> chars than what you expect, or do you see empty squares?
   Stefan> If you see empty squares (i.e. chars for which Emacs can't
   Stefan> find a font), place point on one of them and then do C-u
   Stefan> C-x = which will tell you what this char is.
I presume you don't a difference in 
Brüder and Brüder 
I see (using ascii-7bits)
Br~A1/4der and Br"uder

I did your C-u C-x = which tells me:


character: Ã (0303, 195, 0xc3)
    charset: eight-bit-graphic (8-bit graphic char (0xA0..0xFF))
 code point: 195
     syntax: whitespace
   category:
buffer code: 0xC3
  file code: 0xC3 (encoded by coding system raw-text-unix)
       font:
       -Adobe-Courier-Bold-R-Normal--18-180-75-75-M-110-ISO8859-1

So indeed it looks like a font problem since emacs uses the iso8859-1
font.


I looked in the font list shift down mouse 1 (in the options->mule)
but there the coding is not specified. I am on debian, does anybody
know whether I have to install a special package for UTF8 fonts?

Thanks

Uwe Brauer 

Re: wikipedia and UTF8 coding (emacs21)

[Peter Dyballa]

Am 03.11.2005 um 19:45 schrieb Uwe Brauer:

> I am on debian, does anybody know whether I have to install a special 
> package for UTF8 fonts?

It has good Unicode encoded TrueType fonts.

Try xfontsel or xlsfonts to determine the fonts' encodings! (xlsfonts 
-fn '*-iso10646-1')

--
Mit friedvollen Grüßen

   Pete

Der Unterschied zwischen Theorie und Praxis ist in
Praxis meist größer als in der Theorie

Re: wikipedia and UTF8 coding (emacs21)

[Uwe Brauer]

>>>>> "Peter" == Peter Dyballa <Peter_Dyballa@Web.DE> writes:

   Peter> Am 03.11.2005 um 19:45 schrieb Uwe Brauer:

   >> I am on debian, does anybody know whether I have to install a
   >> special package for UTF8 fonts?

   Peter> It has good Unicode encoded TrueType fonts.

   Peter> Try xfontsel or xlsfonts to determine the fonts' encodings! (xlsfonts 
   Peter> -fn '*-iso10646-1')


Thanks, so xfontsel does not find any fonts in this coding, however
xslfonts does:
for example 
-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1
etc

The point is why don't show these fonts show up in emacs-mule options?

And shall I start emacs the old way:


emacs -font "-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1"

Thanks

Uwe 

Re: wikipedia and UTF8 coding (emacs21)

[Peter Dyballa]

Am 03.11.2005 um 20:09 schrieb Uwe Brauer:

> And shall I start emacs the old way:
>
>
> emacs -font "-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1"
>

This is one option. You can change -0-0- to specify some size.

You can use X resources too (Emacs*font: 
-*-courier-medium-r-*-*-*-120-75-*-m-*-iso10646-* or Emacs.Font: 
fontset-10pt_courier).

And you can create fontsets:

;;;;;;;;; Monotype Courier New - Unicode encoded TrueType font, version 
2.82, 1318 glyphs
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
     (create-fontset-from-fontset-spec "-monotype-courier 
new-medium-r-*-*-9-*-*-*-*-*-fontset-09pt_courier" t 'noerror)
	(set-fontset-font "fontset-09pt_courier"	'latin-iso8859-1  '("courier 
new" . "iso8859-1"))
	(set-fontset-font "fontset-09pt_courier"	'latin-iso8859-2  '("courier 
new" . "iso8859-2"))
	(set-fontset-font "fontset-09pt_courier"	'latin-iso8859-3  '("courier 
new" . "iso8859-3"))
	(set-fontset-font "fontset-09pt_courier"	'latin-iso8859-4  '("courier 
new" . "iso8859-4"))
	(set-fontset-font "fontset-09pt_courier"    'cyrillic-iso8859-5  
'("courier new" . "iso8859-5"))
	(set-fontset-font "fontset-09pt_courier"      'arabic-iso8859-6  
'("courier new" . "iso8859-6"))
	(set-fontset-font "fontset-09pt_courier"	'greek-iso8859-7  '("courier 
new" . "iso8859-7"))
	(set-fontset-font "fontset-09pt_courier"      'hebrew-iso8859-8  
'("courier new" . "iso8859-8"))
	(set-fontset-font "fontset-09pt_courier"	'latin-iso8859-9  '("courier 
new" . "iso8859-9"))
	(set-fontset-font "fontset-09pt_courier"	'latin-iso8859-15 '("courier 
new" . "iso8859-15"))
	(set-fontset-font "fontset-09pt_courier" 'mule-unicode-0100-24ff 
'("courier new" . "iso10646-1"))
	(set-fontset-font "fontset-09pt_courier" 'mule-unicode-2500-33ff 
'("courier new" . "iso10646-1"))
	(set-fontset-font "fontset-09pt_courier" 'mule-unicode-e000-ffff 
'("courier new" . "iso10646-1"))
;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;; Adobe Courier - Unicode encoded TrueType font, OpenType font, 
version 1.020, 374 glyphs
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
     (create-fontset-from-fontset-spec 
"-adobe-courier-medium-r-*-*-9-*-*-*-*-*-fontset-09pt_adobe_courier" t 
'noerror)
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-1  
'("adobe-courier" . "iso8859-1"))
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-2  
'("adobe-courier" . "iso8859-2"))
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-3  
'("adobe-courier" . "iso8859-3"))
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-4  
'("adobe-courier" . "iso8859-4"))
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-9  
'("adobe-courier" . "iso8859-9"))
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-14 
'("adobe-courier" . "iso8859-14"))
	(set-fontset-font "fontset-09pt_adobe_courier"       'latin-iso8859-15 
'("adobe-courier" . "iso8859-15"))
	(set-fontset-font "fontset-09pt_adobe_courier" 'mule-unicode-0100-24ff 
'("adobe-courier" . "iso10646-1"))
	(set-fontset-font "fontset-09pt_adobe_courier" 'mule-unicode-2500-33ff 
'("adobe-courier" . "iso10646-1"))
	(set-fontset-font "fontset-09pt_adobe_courier" 'mule-unicode-e000-ffff 
'("adobe-courier" . "iso10646-1"))
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0370) (decode-char 'ucs #x03cf)) '("courier new" . "iso10646-1"))	; 
Greek
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x03d0) (decode-char 'ucs #x03ff)) '("lucida sans typewriter" . 
"iso10646-1"))	; Coptic
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0400) (decode-char 'ucs #x04ff)) '("lucida sans typewriter" . 
"iso10646-1"))	; Cyrillic
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0500) (decode-char 'ucs #x052f)) '("lucida sans typewriter" . 
"iso10646-1"))	; Cyrillic Suppll
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0530) (decode-char 'ucs #x058f)) '("aramian unicode" . 
"iso10646-1"))	; Armenian (sylfaen
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0590) (decode-char 'ucs #x05ff)) '("courier new" . "iso10646-1"))	; 
Hebrew
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0600) (decode-char 'ucs #x06ff)) '("lucida sans typewriter" . 
"iso10646-1"))	; Arabic
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0700) (decode-char 'ucs #x074f)) '("courier new" . "iso10646-1"))	; 
Syriac
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0780) (decode-char 'ucs #x07bf)) '("courier new" . "iso10646-1"))	; 
Thaana
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0900) (decode-char 'ucs #x097f)) '("courier new" . "iso10646-1"))	; 
Devanagari
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0980) (decode-char 'ucs #x09ff)) '("courier new" . "iso10646-1"))	; 
Bengali
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0a00) (decode-char 'ucs #x0a7f)) '("courier new" . "iso10646-1"))	; 
Gurmukhi
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0a80) (decode-char 'ucs #x0aff)) '("courier new" . "iso10646-1"))	; 
Gujarati
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0b00) (decode-char 'ucs #x0b7f)) '("courier new" . "iso10646-1"))	; 
Oriya
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0b80) (decode-char 'ucs #x0bff)) '("courier new" . "iso10646-1"))	; 
Tamil
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0c00) (decode-char 'ucs #x0c7f)) '("courier new" . "iso10646-1"))	; 
Telugu
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0c80) (decode-char 'ucs #x0cff)) '("courier new" . "iso10646-1"))	; 
Kannada
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0d00) (decode-char 'ucs #x0d7f)) '("courier new" . "iso10646-1"))	; 
Malayalam
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0d80) (decode-char 'ucs #x0dff)) '("courier new" . "iso10646-1"))	; 
Sinhala
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0e00) (decode-char 'ucs #x0e7f)) '("courier new" . "iso10646-1"))	; 
Thai
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0e80) (decode-char 'ucs #x0eff)) '("courier new" . "iso10646-1"))	; 
Lao
	(set-fontset-font "fontset-09pt_adobe_courier" (cons (decode-char 'ucs 
#x0f00) (decode-char 'ucs #x0fff)) '("courier new" . "iso10646-1"))	; 
Tibetan
;


--
Mit friedvollen Grüßen

   Pete

Claiming that the Macintosh is inferior to Windows because most people
use Windows, is like saying that all other restaurants serve food that 
is
inferior to McDonald's.

Re: wikipedia and UTF8 coding (emacs21)

[Uwe Brauer]

>>>>> "Peter" == Peter Dyballa <Peter_Dyballa@Web.DE> writes:

   Peter> Am 03.11.2005 um 20:09 schrieb Uwe Brauer:

   >> And shall I start emacs the old way:
   >> 
   >> 
   >> emacs -font "-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1"
   >> 

   Peter> This is one option. You can change -0-0- to specify some size.

   Peter> You can use X resources too (Emacs*font: 
   Peter> -*-courier-medium-r-*-*-*-120-75-*-m-*-iso10646-* or Emacs.Font: 
   Peter> fontset-10pt_courier).

   Peter> And you can create fontsets:
ok thanks for this so in summary it was a font problem nothing else.

Uwe 

Re: wikipedia and UTF8 coding (emacs21)

[Reiner Steib]

On Thu, Nov 03 2005, Peter Dyballa wrote:

> You can use X resources too (Emacs*font:
> -*-courier-medium-r-*-*-*-120-75-*-m-*-iso10646-* or Emacs.Font:
> fontset-10pt_courier).

Instead of iso10646-* you should use iso10646-1 (or iso8859-1) as
suggested in (info "(emacs)Font X").

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: wikipedia and UTF8 coding (emacs21)

[Stefan Monnier]

> The point is why don't show these fonts show up in emacs-mule options?

Emacs's font selection meachanism doesn't work by giving it a full
X font name.  This is because different fonts can be used for different
chars (in the same buffer).
So instead, you should only specify the font family, and size, while the
charset is automatically selected by Emacs (which will use the iso10646-1
font for the Unicode chars, ...).

> emacs -font "-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1"

Better not.


        Stefan

font problem persists (was: wikipedia and UTF8 coding (emacs21))

[Uwe Brauer]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:


   Stefan> So instead, you should only specify the font family, and
   Stefan> size, while the charset is automatically selected by Emacs
   Stefan> (which will use the iso10646-1 font for the Unicode chars,
   Stefan> ...).

   >> emacs -font  "-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1"

Ok then I tried

   >> emacs -font
   "-adobe-courier-bold-o-normal--0-0-100-100-m-0-*-*"

And the behaviour is the  same, that is 
 C-h h displays Hebrew and Greek correctly.
However when I open a file, which I recently have edited with Greek
and Hebrew symbols and saved as UTF-8, those symbols are displayed as
empty boxes.

Strange enough, when I chose an input mode, say Greek or Hebrew (no
BIDI support but anyway), the symbols I *then type* but only these are
correctly displayed, I save them as UTF-8 leave the buffer, reenter
the buffer and _again_ I just see just empty boxes.

Now I just downloaded and installed fonts from
http://www.m17n.org/emacs-bidi the m17nfonts. These fonts allow to display
even Hebrew with nikkud.
Now

When I start emacs like

alias e6   'emacs -font
"-m17n-mule-medium-r-normal--20-140-100-100-p-90-iso10646-1" -bg
grey86 \!* &'

(I know I should not) then everything is displayed nicely, however
these fonts a proportional fonts, I would prefer fixed fonts of the
courier type.


So what can I do?


Thanks

Uwe Brauer

Re: font problem persists (was: wikipedia and UTF8 coding (emacs21))

[Peter Dyballa]

Am 02.12.2005 um 18:58 schrieb Uwe Brauer:

> So what can I do?
>

Try to find Courier New! Or Lucida Sans Typewriter (in Java). Both have 
some Greek and Hebrew glyphs. Or Everson Mono Unicode, a bit more 
complete. (I can send you overviews)

--
Mit friedvollen Grüßen

   Pete

Es ist, glaub' ich, nichts so heiß, wie's gerne scheinen möchte.

Re: font problem persists

[Stefan Monnier]

Stefan> So instead, you should only specify the font family, and
Stefan> size, while the charset is automatically selected by Emacs
Stefan> (which will use the iso10646-1 font for the Unicode chars,
Stefan> ...).

>>> emacs -font  "-adobe-courier-bold-o-normal--0-0-100-100-m-0-iso10646-1"

> Ok then I tried

>>> emacs -font
>    "-adobe-courier-bold-o-normal--0-0-100-100-m-0-*-*"

> And the behaviour is the  same, that is 
>  C-h h displays Hebrew and Greek correctly.
> However when I open a file, which I recently have edited with Greek
> and Hebrew symbols and saved as UTF-8, those symbols are displayed as
> empty boxes.

The internal representation of non-ascii chars in MULE (later inherited by
Emacs and XEmacs) is based on iso-2022 which, contrary to Unicode which
didn't really exist yet at that time, does not unify "identical" characters.

So what you're seeing is that the Hebrew and Greek chars used in C-h
H are not the same chars as the ones used in utf-8 files (try C-u C-x =
on each of those chars to see which charset they are part of).

The Hebrew chars you see in C-h H are displayed using a Hebrew font.
The Greek chars you see in C-h H are displayed using a Greek font.
The Hebrew and Greek chars you don't see in your utf-8 file are displayed
with a Unicode font.

With Emacs-CVS ("soon" to become Emacs-22), you can set
utf-fragment-on-decoding, which should help for some of those chars
(probably the Greek chars, maybe not the Hebrew).


        Stefan

Re: font problem persists

[Uwe Brauer]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:


   Stefan> So what you're seeing is that the Hebrew and Greek chars
   Stefan> used in C-h H are not the same chars as the ones used in
   Stefan> utf-8 files (try C-u C-x = on each of those chars to see
   Stefan> which charset they are part of).

   Stefan> The Hebrew chars you see in C-h H are displayed using a
   Stefan> Hebrew font. The Greek chars you see in C-h H are displayed
   Stefan> using a Greek font. The Hebrew and Greek chars you don't
   Stefan> see in your utf-8 file are displayed with a Unicode font.

   Stefan> With Emacs-CVS ("soon" to become Emacs-22), you can set
   Stefan> utf-fragment-on-decoding, which should help for some of
   Stefan> those chars (probably the Greek chars, maybe not the
   Stefan> Hebrew).

Ok, what I don't understand is, if I use those UTF8 fonts of
http://www.m17n.org/emacs-bidi --the m17nfonts-- [1] then everything
works nicely. So are these fonts not REAL UTF8 fonts, as the courier
fonts I mentioned earlier?

Uwe 

[1] which seem to be UTF8 fonts,

Re: font problem persists

[Uwe Brauer]

>>>>> "Peter" == Peter Dyballa <Peter_Dyballa@Web.DE> writes:

   Peter> Am 02.12.2005 um 18:58 schrieb Uwe Brauer:

   >> So what can I do?
   >> 

   Peter> Try to find Courier New! Or Lucida Sans Typewriter (in
   Peter> Java). Both have some Greek and Hebrew glyphs. Or Everson
   Peter> Mono Unicode, a bit more complete. (I can send you
   Peter> overviews)

Please do, I desperately need these fonts.

Uwe 

Re: wikipedia and UTF8 coding (emacs21)

[Reiner Steib]

On Thu, Nov 03 2005, Uwe Brauer wrote:

> I presume you don't a difference in Brüder and Brüder I see (using
> ascii-7bits) Br~A1/4der and Br"uder

This is not a font issue.  If Emacs doesn't know that the file is
encoded as UTF-8 you'll see several iso-8859-* character instead of a
single character.  Stefan's suggestions (`prefer-coding-system',
`file-coding-system-alist') should help Emacs 21 to recognize it
correctly.  (AFAIK, Emacs 22 handles automatic UTF-8 recognition
better than Emacs 21.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: wikipedia and UTF8 coding (emacs21)

[Stefan Monnier]

> I presume you don't a difference in 
> Brüder and Brüder 
  ^^^^^^^
This shows that the utf-8 format was not treated as such and instead Emacs
read the file as if it were encoded in latin-1 or binary.

> I did your C-u C-x = which tells me:

> character: Ã (0303, 195, 0xc3)
>     charset: eight-bit-graphic (8-bit graphic char (0xA0..0xFF))
               ^^^^^^^^^^^^^^^^^

This means "unknown char, thus treated as a byte".

If you tell your Emacs to use utf-8 for this file (try C-x RET u utf-8 RET
before C-x C-f, or set file-coding-system-alist so it's done automatically
everytime), your char's charset should be either latin-8859-1 or
mule-unicode-nnnn.

All the above assumes you're not using Emacs in unibyte mode.  Unibyte mode
(which you sadly get if you have a lingering `special-display-european' in
your .emacs) can't handle UTF-8.


        Stefan

Re: wikipedia and UTF8 coding (emacs21)

[Reiner Steib]

On Tue, Nov 22 2005, Stefan Monnier wrote:

> All the above assumes you're not using Emacs in unibyte mode.  Unibyte mode
> (which you sadly get if you have a lingering `special-display-european' in
> your .emacs) can't handle UTF-8.

You meant `standard-display-european' here, I think.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: wikipedia and UTF8 coding (emacs21)

[Stefan Monnier]

>> All the above assumes you're not using Emacs in unibyte mode.  Unibyte mode
>> (which you sadly get if you have a lingering `special-display-european' in
>> your .emacs) can't handle UTF-8.
> You meant `standard-display-european' here, I think.

Indeed.  Braino,


        Stefan

Re: wikipedia and UTF8 coding (emacs21)

[Denis H. G.]

Uwe Brauer <oub@mat.ucm.es> writes:

> Hello
> 
> 
> I would like to edit wikipedia articles, using (X)emacs as an external
> editor. The perl prg which I use for that purpose allows me to access
> to the wiki file either 
> 
>     -  in iso8859-1 coding or 
> 
>     -  in UTF8.
> 
> The iso coding seem to be broken, I am using xemacs no mule so the
> UTF8 option does not work, so I thought of using emacs21 for the
> moment, however when opening the relevant file, the UTF8 is not
> displayed correctly, I tried to play around with the language setting
> but now the coding remains broken, that is I see symbols like Brüder
> which should be displayed like Brüder but it is not.
> 
> Has anybody experience with wikipedia article using UTF8 coding.
> 
> 
> Thanks
> 
> 
> Uwe Brauer 
For me, emacs21 + mule-ucs works fine in a UTF-8 locale. Certainly you need
proper X11 fonts to make it work.

Good luck.

Regards

-- 
Denis H. G.

Re: wikipedia and UTF8 coding (emacs21)

[Reiner Steib]

On Thu, Nov 03 2005, Peter Dyballa wrote:

> Am 03.11.2005 um 15:37 schrieb Uwe Brauer:
>
>> Has anybody experience with wikipedia article using UTF8 coding.
>>
> I'm not experienced with this, but I am sure that
      ^^^^^^^^^^^^^^^                ^^^^^^^^^

Convincing.  :-)

> you need at least two things:
>
> 	1. fonts (in X11 or such) with these glyphs

ACK

> 	2. GNU Emacs 23 from CVS

I'd guess that Uwe prevailingly needs Spanish and German non-ASCII
characters.  Those are sufficiently supported by already in Emacs 21
with (add-to-list 'file-coding-system-alist '("/wikipedia/" utf-8)) as
suggested by Stefan.  With this entry every file in .../wikipedia/ is
de- and encoded with UTF-8.

For example, I wrote the following in Emacs 21 and everything is
displayed correctly:

  ÄÖÜ äöü ß „Hallo Welt und Grüß Gott“
  éíúáó ÉÍÚÁÓ ñ¡¿ ¡Hola!
  “Hello world” ∑ α 

> GNU Emacsen 21.4 and 22 (from CVS) fail to show a particular glyph
> when it's not in the font or fontset used.

Maybe there's something wrong in your fontset config or your system.

My Emacs 21, correctly uses
-Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO8859-1 for “ü” and
-Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO10646-1 for “α”.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: wikipedia and UTF8 coding (emacs21)

[Uwe Brauer]

>>>>> "Reiner" == Reiner Steib <reinersteib+from-uce@imap.cc> writes:

   Reiner> On Thu, Nov 03 2005, Peter Dyballa wrote:
   >> Am 03.11.2005 um 15:37 schrieb Uwe Brauer:
   >> 
   >>> Has anybody experience with wikipedia article using UTF8
   >>> coding.
   >>> 
   >> I'm not experienced with this, but I am sure that
   Reiner>       ^^^^^^^^^^^^^^^                ^^^^^^^^^

   Reiner> Convincing.  :-)


Right, logic rationale and all that 


   >> you need at least two things:
   >> 
   >> 1. fonts (in X11 or such) with these glyphs

   Reiner> ACK

   >> 2. GNU Emacs 23 from CVS

   Reiner> I'd guess that Uwe prevailingly needs Spanish and German
   Reiner> non-ASCII characters.  Those are sufficiently supported by
   Reiner> already in Emacs 21 with (add-to-list
   Reiner> 'file-coding-system-alist '("/wikipedia/" utf-8)) as
   Reiner> suggested by Stefan.  With this entry every file in
   Reiner> .../wikipedia/ is de- and encoded with UTF-8.

Honestly I don't know enough of wiki(pedia) but I am oops sure/not
sure that the coding is more general. I tried to convert the document
using iconv to iso8859-1 and it failed. I have seen in those documents
Hebrew, Cyrillic and other codings, so that might be "true" utf-8.

Uwe 

   Reiner> For example, I wrote the following in Emacs 21 and
   Reiner> everything is displayed correctly:

   Reiner>   ÄÖÜ äöü ß „Hallo Welt
   und Grüß Gott“ 
   Reiner>   éíúáó ÉÍÚÁÓ ñ¡¿ ¡Hola!
   Reiner>   “Hello world” ∑ α 

   >> GNU Emacsen 21.4 and 22 (from CVS) fail to show a particular
   >> glyph when it's not in the font or fontset used.

I check that, but brute force 
 emacs -font
 "-adobe-courier-bold-r-normal--18-180-75-75-m-110-iso10646-1" -bg
 grey86 \!* & works so far, so that is fine for me, right now
   Reiner> Maybe there's something wrong in your fontset config or your system.

Could be, I will try thanks anyhow

   Reiner> My Emacs 21, correctly uses
   Reiner>
   -Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO8859-1 for
   “ü” and 
   Reiner>
   -Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO10646-1
   for “α”. 

PS right now I send this using xemacs no mule, hope the message does
get messed up if not, I might consider any sort of mule solution in
the future 

Uwe 

Re: wikipedia and UTF8 coding (emacs21)

[Stefan Monnier]

Reiner> I'd guess that Uwe prevailingly needs Spanish and German
Reiner> non-ASCII characters.  Those are sufficiently supported by
Reiner> already in Emacs 21 with (add-to-list
Reiner> 'file-coding-system-alist '("/wikipedia/" utf-8)) as
Reiner> suggested by Stefan.  With this entry every file in
Reiner> .../wikipedia/ is de- and encoded with UTF-8.

> Honestly I don't know enough of wiki(pedia) but I am oops sure/not
> sure that the coding is more general. I tried to convert the document
> using iconv to iso8859-1 and it failed. I have seen in those documents
> Hebrew, Cyrillic and other codings, so that might be "true" utf-8.

Emacs-21.4's handling of UTF-8 takes care of significantly more than just
latin-1.  What Reiner was saying is that you probably only really care about
the latin-1 chars (which will assuredly be displayed properly) and don't
mind if Emacs doesn't properly display some of the other characters, as long
as it does preserve them across load&save (it will display them as an empty
box if your unicode font doesn't have them).


        Stefan

using emacs 21 (was: Re: wikipedia and UTF8 coding (emacs21)

[Uwe Brauer]

Reiner Steib <reinersteib+from-uce@imap.cc> writes:

 > 
 >   ÄÖÜ äöü ß „Hallo Welt und Grüß Gott“
 >   éíúáó ÉÍÚÁÓ ñ¡¿ ¡Hola!
 >   “Hello world” ∑ α 
 > 

I am just using emacs 21, and I see almost all the symbols, save in
the last line which starts with hello world, I see 2 empty boxes..

Uwe 

Re: using emacs 21 (was: Re: wikipedia and UTF8 coding (emacs21)

[Peter Dyballa]

Am 06.11.2005 um 11:16 schrieb Uwe Brauer:

>>   ÄÖÜ äöü ß „Hallo Welt und Grüß Gott“
>>   éíúáó ÉÍÚÁÓ ñ¡¿ ¡Hola!
>>   “Hello world” ∑ α
>>

--
Mit friedvollen Grüßen

   Pete

Either this man is dead or my watch has stopped. - Groucho Marx

Re: using emacs 21 (was: Re: wikipedia and UTF8 coding (emacs21)

[Peter Dyballa]

Am 06.11.2005 um 11:16 schrieb Uwe Brauer:

>>
>>   ÄÖÜ äöü ß „Hallo Welt und Grüß Gott“
>>   éíúáó ÉÍÚÁÓ ñ¡¿ ¡Hola!
>>   “Hello world” ∑ α
>>
>
> I am just using emacs 21, and I see almost all the symbols, save in
> the last line which starts with hello world, I see 2 empty boxes..
>

Continuation of a font problem ...

The first box hides a mathematical symbol, N-ARY SUMMATION, at U+2211, 
the second box hides a GREEK SMALL LETTER ALPHA at U+03B1. Switch 
through all your fonts and see whether there is one which contains 
these glyphs too. I am using -b&h-lucida sans 
typewriter-medium-r-normal in iso10646-1 encoding. There is too an X 
font displayer, xfd.

--
Mit friedvollen Grüßen

   Pete

There's no place like ~
                           (UNIX Guru)

Encodings in message (was: wikipedia and UTF8 coding (emacs21))

[Marc Girod]

Hello,

I experience related but different problems.
In mails and news postings, which I process using Gnus, I tend to see
all the characters correctly (most of them anyway), but to send while
replying, what my correspondents see as garbage.

Indeed, looking at my messages with:

  (find-charset-region (point) (mark))

I often notice different charsets.

My question: how to force using only one (iso8859-1 -- forgetting the
few non supported characters)?
I currently know how to do it interactively, with query-replace, but
fail to do it programmatically.

I cannot use subst-char-in-region, since eight-bit graphic character
have not the same size as iso ones, and using search-forward and
replace-match, the syntax to pas the characters escapes me... I
believe this is precisely for UTF8 encoded chars, although... I have
no example at hand to check.

-- 
Marc Girod        P.O. Box 370        Voice:  +358-71 80 25581
Nokia BI          00045 NOKIA Group   Mobile: +358-50 38 78415
Kara RD B 480.2   Finland             Fax:    +358-71 80 64474

Re: Encodings in message

[Reiner Steib]

On Fri, Nov 04 2005, Marc Girod wrote:

> In mails and news postings, which I process using Gnus, I tend to see
> all the characters correctly (most of them anyway), but to send while
> replying, what my correspondents see as garbage.
>
> Indeed, looking at my messages with:
>
>   (find-charset-region (point) (mark))
>
> I often notice different charsets.
>
> My question: how to force using only one (iso8859-1 -- forgetting the
> few non supported characters)?

Instead of such a brute force destruction of information, you should
analyze the problem.  But "see as garbage" is not a sufficient
description to even guess what could be wrong.

If you use Gnus/5.9 Emacs/21.2 as your headers suggest, I'd strongly
recommend to upgrade at least to Emacs 21.4.  The Gnus version (also
labeled 5.9) of Emacs 21.4 provides `mm-coding-system-priorities':

,----[ <f1> v mm-coding-system-priorities RET ]
| mm-coding-system-priorities is a variable defined in `mm-util.el'.
| Its value is 
| (iso-8859-1 iso-8859-2 iso-8859-15 mule-utf-8)
| 
| Documentation:
| Preferred coding systems for encoding outgoing messages.
| 
| More than one suitable coding system may be found for some text.
| By default, the coding system with the highest priority is used
| to encode outgoing messages (see `sort-coding-systems').  If this
| variable is set, it overrides the default priority.
`----

Even better than using Emacs 21.4 + Gnus 5.9 would be to upgrade to
Gnus 5.10 available from www.gnus.org.

Unfortunately the latest stable release is still 5.10.6 although the
stable branch in CVS ("v5-10") contains lots of bug fixes to warrant a
release 5.10.7.  Maybe I should tickle Lars again.  :-)

Bye, Reiner.

Followup-To: gnu.emacs.gnus set, since this is Gnus specific.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/