Subsets of Unicode and coding systems

Subsets of Unicode and coding systems

[Ulrich Mueller]

Hi,
Sometimes I want to check if a file saved in UTF-8 encoding contains
only characters from a certain subset of the Unicode character
repertoire, like the MES-* [1] or WGL4 [2] subsets. (For example, for
things published on the WWW one might rather avoid exotic characters
for better compatibility.)

My idea was now to define a coding system e.g. for MES-2 and set the
buffer-file-coding-system accordingly, so that Emacs would check it
upon saving the file.

Now I have the following questions:
1. Is using coding systems for this purpose a reasonable approach
   at all, or is there a better way to achieve this?
2. Where can I find documentation how to define a coding system?
   The Elisp Reference Manual isn't really helpful, it only says:
   "How to define a coding system is an arcane matter, and is not
   documented here."

Ulrich

[1] <http://www.evertype.com/standards/iso10646/pdf/cwa13873.pdf>
[2] <http://en.wikipedia.org/wiki/Windows_Glyph_List_4>

Re: Subsets of Unicode and coding systems

[Eli Zaretskii]

> Date: Tue, 6 Dec 2011 12:38:57 +0100
> From: Ulrich Mueller <ulm@gentoo.org>
> 
> Sometimes I want to check if a file saved in UTF-8 encoding contains
> only characters from a certain subset of the Unicode character
> repertoire, like the MES-* [1] or WGL4 [2] subsets. (For example, for
> things published on the WWW one might rather avoid exotic characters
> for better compatibility.)
> 
> My idea was now to define a coding system e.g. for MES-2 and set the
> buffer-file-coding-system accordingly, so that Emacs would check it
> upon saving the file.
> 
> Now I have the following questions:
> 1. Is using coding systems for this purpose a reasonable approach
>    at all, or is there a better way to achieve this?

I don't think defining a new coding-system is the best way to go about
this.  I would suggest to define a new category instead, see
"(elisp)Categories" (which includes a working example), and then make
a category table (which is just a special type of char-table) where
the relevant characters are marked with the appropriate categories.
Then you can simply scan the buffer with a regexp that uses "\\Cx"
where "x" is the letter you assign to your category.  (You will
probably need a separate letter for each of the MES-* and WGL4 sets.)

> 2. Where can I find documentation how to define a coding system?

You don't want to know that ;-)

Re: Subsets of Unicode and coding systems

[Werner LEMBERG]

>> 2. Where can I find documentation how to define a coding system?
> 
> You don't want to know that ;-)

Tsk, tsk, tsk.  This is at the border of heresy.


    Werner

Re: Subsets of Unicode and coding systems

[Kenichi Handa]

> Date: Tue, 6 Dec 2011 12:38:57 +0100
> From: Ulrich Mueller <ulm@gentoo.org>
> 
> Sometimes I want to check if a file saved in UTF-8 encoding contains
> only characters from a certain subset of the Unicode character
> repertoire, like the MES-* [1] or WGL4 [2] subsets. (For example, for
> things published on the WWW one might rather avoid exotic characters
> for better compatibility.)
> 
> My idea was now to define a coding system e.g. for MES-2 and set the
> buffer-file-coding-system accordingly, so that Emacs would check it
> upon saving the file.
> 
> Now I have the following questions:
> 1. Is using coding systems for this purpose a reasonable approach
>    at all, or is there a better way to achieve this?
> 2. Where can I find documentation how to define a coding system?

I agree with Eli, defining proper categories provides more
flexible usage, but FYI, something like this defines the
coding system utf-8-mes-2.

(let ((repertory '((0 . #x017f) #x018f #x0192 #x01b4 
		   (#x01de . #x01ef) (#x01fa . #x01ff)
		   ;;; add more here
		   ))
      list)
  (dolist (elm repertory)
    (if (consp elm)
	(let ((from (car elm))
	      (to (cdr elm)))
	  (while (<= from to)
	    (setq list (cons from (cons from list)))
	    (setq from (1+ from))))
      (setq list (cons elm (cons elm list)))))
  (define-charset 'unicode-mes-2 "MES-2 subset of Unicode"
    :code-space [0 255 0 255]
    :ascii-compatible-p t
    :map (apply 'vector (nreverse list)))
  (define-coding-system 'utf-8-mes-2 "UTF-8 for MES-2 subset"
    :mnemonic ?M
    :coding-type 'utf-8
    :charset-list '(unicode-mes-2)
    :mime-charset 'utf-8))

See the docstring of define-charset and define-coding-system
to understand the code.

---
Kenichi Handa
handa@m17n.org