regex encoding

regex encoding

[Chip Coldwell]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 790 bytes --]


Looking at lisp/textmodes/ispell.el, there are a number of regular
expressions such as this one taken from the "german8" entry of
ispell-dictionary-alist-4:

[a-zA-Z\304\326\334\344\366\337\374]

The last seven numeric character codes in the above are the iso-8859-1
(ISO Latin-1) encodings for letters used in German that are not
encoded by ASCII.  If your mail reader can handle UTF-8, the
characters represented by these iso-8859-1 codes look like this:

[a-zA-ZÄÖÜäößü]

My question is: are emacs regex character classes limited to the
iso-8859-1 character set, or is there some way to represent Unicode
(such as UTF-8) characters in a character class?

Thanks,

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: regex encoding

[Reiner Steib]

On Tue, Aug 01 2006, Chip Coldwell wrote:

> If your mail reader can handle UTF-8, the characters represented by
> these iso-8859-1 codes look like this:
>
> [a-zA-ZÄÖÜäößü]

| [a-zA-ZÄÖÜäößü]

If you intend to send UTF-8, you MUA should not declare it as
"Content-Type: TEXT/PLAIN; charset=ISO-8859-1".  ;-)

> My question is: are emacs regex character classes limited to the
> iso-8859-1 character set, or is there some way to represent Unicode
> (such as UTF-8) characters in a character class?

AFAIK, you can write the chars in UTF-8 if you specify the encoding of
the lisp file, cf. (info "(emacs)Specify Coding"):

--8<---------------cut here---------------start------------->8---
;; -*- coding: utf-8 -*-
(defun rs-test ()
  (interactive)
  (re-search-forward "[ÄÖÜäößü]"))
--8<---------------cut here---------------end--------------->8---

I don't know if there's a reason why isn't used in `ispell.el'.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: regex encoding

[Chip Coldwell]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2199 bytes --]

On Tue, 1 Aug 2006, Reiner Steib wrote:
>
> | [a-zA-ZÄÖÜäößü]
>
> If you intend to send UTF-8, you MUA should not declare it as
> "Content-Type: TEXT/PLAIN; charset=ISO-8859-1".  ;-)

Ooops.  I haven't tamed my MUA yet.  It's only been ten years.

>> My question is: are emacs regex character classes limited to the
>> iso-8859-1 character set, or is there some way to represent Unicode
>> (such as UTF-8) characters in a character class?
>
> AFAIK, you can write the chars in UTF-8 if you specify the encoding of
> the lisp file, cf. (info "(emacs)Specify Coding"):
>
> --8<---------------cut here---------------start------------->8---
> ;; -*- coding: utf-8 -*-
> (defun rs-test ()
>  (interactive)
>  (re-search-forward "[ÄÖÜäößü]"))
> --8<---------------cut here---------------end--------------->8---
>
> I don't know if there's a reason why isn't used in `ispell.el'.

The particular issue is that ispell is currently broken if your LANG
environment variable specifies UTF-8 encoding, your buffer is UTF-8
encoded and contains one of these non-ASCII characters, and you
specify the "deutsch8" dictionary.  ispell-word generates the error:

"Ispell and its process have different character maps"

What happens is that emacs transcodes the word to iso-8859-1 before
sending it to the aspell process, which most likely is respecting a
the LANG environment variable.  If you change the value of the
ispell-dictionary-alist for "deutch8" from

    ("deutsch8"
     "[a-zA-Z\304\326\334\344\366\337\374]"
     "[^a-zA-Z\304\326\334\344\366\337\374]"
     "[']" t ("-C" "-d" "deutsch") "~latin1" iso-8859-1)

to

    ("deutsch8"
     "[a-zA-Z\304\326\334\344\366\337\374]"
     "[^a-zA-Z\304\326\334\344\366\337\374]"
     "[']" t ("-C" "-d" "deutsch") "~latin1" utf-8)

the regex doesn't match words in the buffer properly.  Changing it to

    ("deutsch8"
     "[a-zA-Z\304\326\334\344\366\337\374]"
     "[^a-zA-Z\304\326\334\344\366\337\374]"
     "[']" t ("-C" "-d" "deutsch" "--encoding=iso8859-1") "~latin1" iso-8859-1)

does seem to work.

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: regex encoding

[Chip Coldwell]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1531 bytes --]

On Tue, 1 Aug 2006, Chip Coldwell wrote:

> On Tue, 1 Aug 2006, Reiner Steib wrote:
>> 
>> | [a-zA-ZÄÖÜäößü]
>> 
>> If you intend to send UTF-8, you MUA should not declare it as
>> "Content-Type: TEXT/PLAIN; charset=ISO-8859-1".  ;-)
>
> Ooops.  I haven't tamed my MUA yet.  It's only been ten years.
>
>>> My question is: are emacs regex character classes limited to the
>>> iso-8859-1 character set, or is there some way to represent Unicode
>>> (such as UTF-8) characters in a character class?
>> 
>> AFAIK, you can write the chars in UTF-8 if you specify the encoding of
>> the lisp file, cf. (info "(emacs)Specify Coding"):
>> 
>> --8<---------------cut here---------------start------------->8---
>> ;; -*- coding: utf-8 -*-
>> (defun rs-test ()
>>  (interactive)
>>  (re-search-forward "[ÄÖÜäößü]"))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> I don't know if there's a reason why isn't used in `ispell.el'.
>
> The particular issue is that ispell is currently broken if your LANG
> environment variable specifies UTF-8 encoding, your buffer is UTF-8
> encoded and contains one of these non-ASCII characters, and you
> specify the "deutsch8" dictionary.  ispell-word generates the error:
>
> "Ispell and its process have different character maps"

I should also mention that this is with aspell version 0.6; version
0.5 seems to do the right thing regardless.

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel