bug#8308: 23.3; Use utf-8 for writing abbrev file

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

Is it OK to change the encoding for abbrev file to utf-8?

=== modified file 'lisp/abbrev.el'
--- a/lisp/abbrev.el	2011-03-21 05:49:12 +0000
+++ b/lisp/abbrev.el	2011-03-21 06:20:36 +0000
@@ -225,9 +225,9 @@
 		    abbrev-file-name)))
   (or (and file (> (length file) 0))
       (setq file abbrev-file-name))
-  (let ((coding-system-for-write 'emacs-mule))
+  (let ((coding-system-for-write 'utf-8))
     (with-temp-file file
-      (insert ";;-*-coding: emacs-mule;-*-\n")
+      (insert ";;-*-coding: utf-8;-*-\n")
       (dolist (table
                ;; We sort the table in order to ease the automatic
                ;; merging of different versions of the user's abbrevs


Leo

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Eli Zaretskii]

> From: Leo <sdl.web@gmail.com>
> Date: Mon, 21 Mar 2011 14:22:24 +0800
> Cc: 
> 
> Is it OK to change the encoding for abbrev file to utf-8?

What will that do to characters that are not unified into the range of
valid Unicode code points?

Can you tell what is the purpose of this change?

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

On 2011-03-21 17:00 +0800, Eli Zaretskii wrote:
>> From: Leo <sdl.web@gmail.com>
>> Date: Mon, 21 Mar 2011 14:22:24 +0800
>> Cc: 
>> 
>> Is it OK to change the encoding for abbrev file to utf-8?
>
> What will that do to characters that are not unified into the range of
> valid Unicode code points?

That's a valid concern. But

,----
| M -- emacs-mule
| 
| Emacs 21 internal format used in buffer and string.
| Type: emacs-mule (Emacs 21 internal encoding)
| EOL type: Automatic selection from:
| 	[emacs-mule-unix emacs-mule-dos emacs-mule-mac]
| This coding system can encode all emacs-mule charsets.
| 
| [back]
`----

,----[ (info "(elisp)Text Representations") ]
|    (1) This internal representation is based on one of the encodings
| defined by the Unicode Standard, called "UTF-8", for representing any
| Unicode codepoint, but Emacs extends UTF-8 to represent the additional
| codepoints it uses for raw 8-bit bytes and characters not unified with
| Unicode.
`----

Would you agree to use utf-8-emacs instead, which covers all characters.

>
> Can you tell what is the purpose of this change?

Make abbrev file editable to other editors.

Leo

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Eli Zaretskii]

> From: Leo <sdl.web@gmail.com>
> Date: Mon, 21 Mar 2011 18:01:17 +0800
> Cc: 
> 
> Would you agree to use utf-8-emacs instead, which covers all characters.

That's better, but the characters outside Unicode are still going to
do bad things to any software except Emacs.  AFAIK, emacs-mule is a
superset of iso-2022 in the same way as utf-8-emacs is a superset of
utf-8.

> > Can you tell what is the purpose of this change?
> 
> Make abbrev file editable to other editors.

If we are really keen on making the abbrev files editable to other
editors, we should make sure they are encoded in some encoding that
these other editors will understand.  That probably calls for using
utf-8 for everything that's covered by Unicode, and using other
appropriate encodings for characters outside Unicode.

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Andreas Röhler]

Am 21.03.2011 11:54, schrieb Eli Zaretskii:
>> From: Leo<sdl.web@gmail.com>
>> Date: Mon, 21 Mar 2011 18:01:17 +0800
>> Cc:
>>
>> Would you agree to use utf-8-emacs instead, which covers all characters.
>
> That's better, but the characters outside Unicode are still going to
> do bad things to any software except Emacs.  AFAIK, emacs-mule is a
> superset of iso-2022 in the same way as utf-8-emacs is a superset of
> utf-8.
>
>>> Can you tell what is the purpose of this change?
>>
>> Make abbrev file editable to other editors.
>
> If we are really keen on making the abbrev files editable to other
> editors, we should make sure they are encoded in some encoding that
> these other editors will understand.  That probably calls for using
> utf-8 for everything that's covered by Unicode, and using other
> appropriate encodings for characters outside Unicode.
>
>
>
>

Hi,

sounds interesting for me, as not just other editors are at stake AFAIU, 
but auto-generated abbrevs produced by programms.

These might be theme-specific, cover items of medicine, jura etc.
Could offer modes with preloaded abbrevs resp. to matter of writing.

Regards,

Andreas

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Stefan Monnier]

> Is it OK to change the encoding for abbrev file to utf-8?
> === modified file 'lisp/abbrev.el'
> --- a/lisp/abbrev.el	2011-03-21 05:49:12 +0000
> +++ b/lisp/abbrev.el	2011-03-21 06:20:36 +0000
> @@ -225,9 +225,9 @@
>  		    abbrev-file-name)))
>    (or (and file (> (length file) 0))
>        (setq file abbrev-file-name))
> -  (let ((coding-system-for-write 'emacs-mule))
> +  (let ((coding-system-for-write 'utf-8))
>      (with-temp-file file
> -      (insert ";;-*-coding: emacs-mule;-*-\n")
> +      (insert ";;-*-coding: utf-8;-*-\n")
>        (dolist (table
>                 ;; We sort the table in order to ease the automatic
>                 ;; merging of different versions of the user's abbrevs

Sounds good in general, but I'm wondering whether we should worry about
the presence of abbrevs which include bytes (aka eight-bit-chars).
Using `utf-8-emacs' should fix those issues, but would then bump into
the problem that such abbrev files wouldn't be compatible with Emacs-22.


        Stefan

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

On 2011-03-21 22:50 +0800, Stefan Monnier wrote:
> Sounds good in general, but I'm wondering whether we should worry about
> the presence of abbrevs which include bytes (aka eight-bit-chars).
> Using `utf-8-emacs' should fix those issues, but would then bump into
> the problem that such abbrev files wouldn't be compatible with Emacs-22.

I think we should just use utf-8-emacs. What do other people think?

Leo

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Eli Zaretskii]

> From: Leo <sdl.web@gmail.com>
> Date: Mon, 21 Mar 2011 23:37:41 +0800
> Cc: 
> 
> I think we should just use utf-8-emacs.

Why do you think so?

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

On 2011-03-22 02:45 +0800, Eli Zaretskii wrote:
>> I think we should just use utf-8-emacs.
>
> Why do you think so?

By the time 24.1 is released, it will be 1-2 years from now and there
will be two major stable releases that work with utf-8-emacs, which are
backward-compatible enough. But I don't know so I'll forget about this
bug and let the gurus figure it out.

Leo

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Stefan Monnier]

>>> I think we should just use utf-8-emacs.
>> Why do you think so?
> By the time 24.1 is released, it will be 1-2 years from now and there
> will be two major stable releases that work with utf-8-emacs, which are
> backward-compatible enough. But I don't know so I'll forget about this
> bug and let the gurus figure it out.

I think it might be OK to do it for Emacs-25, but since Emacs-22 can't
handle utf-8-emacs, I think it's a bit early to switch to it in
Emacs-24.  If utf-8 is sufficient, OTOH it's the best choice.  So maybe
we should check the buffer first to see if utf-8 is safe, and only fall
back to emacs-mule if utf-8 is not safe.


        Stefan

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

On 2011-03-22 10:48 +0800, Stefan Monnier wrote:
> I think it might be OK to do it for Emacs-25, but since Emacs-22 can't
> handle utf-8-emacs, I think it's a bit early to switch to it in
> Emacs-24.  If utf-8 is sufficient, OTOH it's the best choice.  So maybe
> we should check the buffer first to see if utf-8 is safe, and only fall
> back to emacs-mule if utf-8 is not safe.

I think default to utf-8 is good, which is sufficient for most people.
Any comments on the following patch? I don't know how to introduce a
char unencodable with utf-8 to the abbrevs. So it is only partially
tested.


=== modified file 'lisp/abbrev.el'
--- lisp/abbrev.el	2011-01-25 04:08:28 +0000
+++ lisp/abbrev.el	2011-03-22 03:30:52 +0000
@@ -225,21 +225,29 @@
 		    abbrev-file-name)))
   (or (and file (> (length file) 0))
       (setq file abbrev-file-name))
-  (let ((coding-system-for-write 'emacs-mule))
-    (with-temp-file file
-      (insert ";;-*-coding: emacs-mule;-*-\n")
+  (let ((coding-system-for-write 'utf-8))
+    (with-temp-buffer
       (dolist (table
-               ;; We sort the table in order to ease the automatic
-               ;; merging of different versions of the user's abbrevs
-               ;; file.  This is useful, for example, for when the
-               ;; user keeps their home directory in a revision
-               ;; control system, and is therefore keeping multiple
-               ;; slightly-differing copies loosely synchronized.
-               (sort (copy-sequence abbrev-table-name-list)
-                     (lambda (s1 s2)
-                       (string< (symbol-name s1)
-                                (symbol-name s2)))))
-	(insert-abbrev-table-description table nil)))))
+	       ;; We sort the table in order to ease the automatic
+	       ;; merging of different versions of the user's abbrevs
+	       ;; file.  This is useful, for example, for when the
+	       ;; user keeps their home directory in a revision
+	       ;; control system, and is therefore keeping multiple
+	       ;; slightly-differing copies loosely synchronized.
+	       (sort (copy-sequence abbrev-table-name-list)
+		     (lambda (s1 s2)
+		       (string< (symbol-name s1)
+				(symbol-name s2)))))
+	(insert-abbrev-table-description table nil))
+      (when (unencodable-char-position (point-min) (point-max) 'utf-8)
+	(setq coding-system-for-write
+	      (if (> emacs-major-version 24)
+		  'utf-8-emacs
+		;; For compatibility with Emacs 22
+		'emacs-mule)))
+      (goto-char (point-min))
+      (insert (format ";;-*-coding: %s;-*-\n" coding-system-for-write))
+      (write-region nil nil file nil 0))))
 \f
 (defun add-mode-abbrev (arg)
   "Define mode-specific abbrev for last word(s) before point.

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Stefan Monnier]

> I think default to utf-8 is good, which is sufficient for most people.
> Any comments on the following patch? I don't know how to introduce a
> char unencodable with utf-8 to the abbrevs. So it is only partially
> tested.

(unibyte-string 129) returns a string containing an unencodable char.
So you can test with it.
The patch looks good,


        Stefan

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

On 2011-03-22 13:24 +0800, Stefan Monnier wrote:
> (unibyte-string 129) returns a string containing an unencodable char.
> So you can test with it.

I still cannot get any byte into the abbrevs. For example,
(unibyte-string 129) returns byte \201 but when it is written to abbrev
file by write-abbrev-file, it is changed to \ 2 0 1, so utf-8 appear
sufficient even for bytes.

Leo

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Stefan Monnier]

>> (unibyte-string 129) returns a string containing an unencodable char.
>> So you can test with it.
> I still cannot get any byte into the abbrevs. For example,
> (unibyte-string 129) returns byte \201 but when it is written to abbrev
> file by write-abbrev-file, it is changed to \ 2 0 1, so utf-8 appear
> sufficient even for bytes.

Good.  In any case your unencodable-foo test would trigger if there were
eight-bit-chars in there, so it works correctly in this respect.
Please install your patch.


        Stefan

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Leo]

Version: 24.1.

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Andreas Röhler]

Am 21.03.2011 15:50, schrieb Stefan Monnier:
>> Is it OK to change the encoding for abbrev file to utf-8?
>> === modified file 'lisp/abbrev.el'
>> --- a/lisp/abbrev.el	2011-03-21 05:49:12 +0000
>> +++ b/lisp/abbrev.el	2011-03-21 06:20:36 +0000
>> @@ -225,9 +225,9 @@
>>   		    abbrev-file-name)))
>>     (or (and file (>  (length file) 0))
>>         (setq file abbrev-file-name))
>> -  (let ((coding-system-for-write 'emacs-mule))
>> +  (let ((coding-system-for-write 'utf-8))
>>       (with-temp-file file
>> -      (insert ";;-*-coding: emacs-mule;-*-\n")
>> +      (insert ";;-*-coding: utf-8;-*-\n")
>>         (dolist (table
>>                  ;; We sort the table in order to ease the automatic
>>                  ;; merging of different versions of the user's abbrevs
>
> Sounds good in general, but I'm wondering whether we should worry about
> the presence of abbrevs which include bytes (aka eight-bit-chars).
> Using `utf-8-emacs' should fix those issues, but would then bump into
> the problem that such abbrev files wouldn't be compatible with Emacs-22.
>
>
>          Stefan
>

Hi,

so maybe not hard-code it, rather have a variable?

Andreas

bug#8308: 23.3; Use utf-8 for writing abbrev file

[Eli Zaretskii]

> Date: Mon, 21 Mar 2011 19:24:16 +0100
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> Cc: 
> 
> so maybe not hard-code it, rather have a variable?

A constant encoding will never DTRT in all cases.