string-as-unibyte

string-as-unibyte

[Stefan Monnier]

Could you explain the need for the change below:

2005-07-16  YAMAMOTO Mitsuharu  <mituharu@math.s.chiba-u.ac.jp>

	* mac.c [TARGET_API_MAC_CARBON] (Fmac_code_convert_string):
	Use Fstring_as_unibyte instead of string_make_unibyte.

My experience is that string-as-unibyte is extremely rarely the right answer
to solve a problem.  If you described your motivation, I could add a comment
in the code making it clear why this is needed here (or else come up with
a better solution).


        Stefan

Re: string-as-unibyte

[YAMAMOTO Mitsuharu]

>>>>> On Mon, 18 Jul 2005 17:33:02 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:

> Could you explain the need for the change below:

> 2005-07-16 YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>

> 	* mac.c [TARGET_API_MAC_CARBON] (Fmac_code_convert_string):
> Use Fstring_as_unibyte instead of string_make_unibyte.

It is at the preparation stage of code conversion.  So I think the
following comment in decode_coding_string (coding.c) is also
applicable to this case.

  if (STRING_MULTIBYTE (str))
    {
      /* Decoding routines expect the source text to be unibyte.  */
      str = Fstring_as_unibyte (str);

> My experience is that string-as-unibyte is extremely rarely the
> right answer to solve a problem.  If you described your motivation,
> I could add a comment in the code making it clear why this is needed
> here (or else come up with a better solution).

I was trying to make a coding system that almost works as utf-8, but
additionally does "HFS+ composition" (canonical composition with some
exclusions) on decoding.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

;; For the Carbon port, Mac OS X 10.2 or later.
(make-coding-system
 'mac-hfs+
 0
 (coding-system-mnemonic 'utf-8)
 "Like utf-8, but additionally does Mac HFS+ composition on decoding."
 (coding-system-flags 'utf-8)
 (list (cons 'safe-charsets (coding-system-get 'utf-8 'safe-charsets))
       '(post-read-conversion . mac-hfs+-post-read-conversion)
       '(pre-write-conversion . mac-hfs+-pre-write-conversion)))

(defun mac-hfs+-post-read-conversion (length)
  (save-excursion
    (save-restriction
      (narrow-to-region (point) (+ (point) length))
      (let ((str (mac-code-convert-string (buffer-string)
					  'utf-8 'utf-8 'HFS+C)))
	(when str
	  (erase-buffer)
	  (insert (if enable-multibyte-characters
		      (string-as-multibyte str) str)))
	(setq length (decode-coding-region (point-min) (point-max) 'utf-8))
	;; We are inside a post-read-conversion function, so the
	;; original post-read-conversion for utf-8 is not
	;; automatically called.
	(goto-char (point-min))
	(funcall (or (coding-system-get 'utf-8 'post-read-conversion)
		     'identity)
		 length)))))

(defun mac-hfs+-pre-write-conversion (beg end)
  (funcall (or (coding-system-get 'utf-8 'pre-write-conversion) 'ignore)
	   beg (+ beg (encode-coding-region beg end 'utf-8))))

(setq default-file-name-coding-system 'mac-hfs+)

Re: string-as-unibyte

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 19 Jul 2005 07:41:33 +0900, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> said:

> I was trying to make a coding system that almost works as utf-8, but
> additionally does "HFS+ composition" (canonical composition with
> some exclusions) on decoding.

> 	(when str
> 	  (erase-buffer)
> 	  (insert (if enable-multibyte-characters
> 		      (string-as-multibyte str) str)))

Maybe this part should have been string-to-multibyte.

BTW, I noticed that backslashes in the docstring of
string-as-multibyte are stripped off as follows:

If you're not sure, whether to use `string-as-multibyte' or
`string-to-multibyte', use `string-to-multibyte'.  Beware:
   (aref (string-as-multibyte "201") 0) -> 129 (aka ?201)
   (aref (string-as-multibyte "300") 0) -> 192 (aka ?300)
   (aref (string-as-multibyte "300201") 0) -> 192 (aka ?300)
   (aref (string-as-multibyte "300201") 1) -> 129 (aka ?201)
but
   (aref (string-as-multibyte "201300") 0) -> 2240
   (aref (string-as-multibyte "201300") 1) -> <error>

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: string-as-unibyte

[Kenichi Handa]

In article <wl3bqbisf1.wl%mituharu@math.s.chiba-u.ac.jp>, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

> BTW, I noticed that backslashes in the docstring of
> string-as-multibyte are stripped off as follows:

> If you're not sure, whether to use `string-as-multibyte' or
> `string-to-multibyte', use `string-to-multibyte'.  Beware:
>    (aref (string-as-multibyte "201") 0) -> 129 (aka ?201)
[...]

Thank you for finding this bug.  I've just installed a fix.

---
Kenichi Handa
handa@m17n.org

Re: string-as-unibyte

[Kenichi Handa]

In article <wl4qarag9u.wl%mituharu@math.s.chiba-u.ac.jp>, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> writes:

>>>>>>  On Mon, 18 Jul 2005 17:33:02 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:
>>  Could you explain the need for the change below:

>>  2005-07-16 YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>

>>  	* mac.c [TARGET_API_MAC_CARBON] (Fmac_code_convert_string):
>>  Use Fstring_as_unibyte instead of string_make_unibyte.

> It is at the preparation stage of code conversion.  So I think the
> following comment in decode_coding_string (coding.c) is also
> applicable to this case.

>   if (STRING_MULTIBYTE (str))
>     {
>       /* Decoding routines expect the source text to be unibyte.  */
>       str = Fstring_as_unibyte (str);

If a multibyte string is given to mac-code-convert-string,
and the string is made mutlibyte by string-to-multibyte from
the raw-byte sequence (ex. inserting a file by raw-text in a
mutlibyte buffer and extracting a string by
buffer-substring), using Fstring_as_unibyte is correct.
Please note that we don't have Fstring_to_unibyte because it
should work the same way as Fstring_as_unibyte.

---
Kenichi Handa
handa@m17n.org

Re: string-as-unibyte

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 19 Jul 2005 11:56:37 +0900, Kenichi Handa <handa@m17n.org> said:

> If a multibyte string is given to mac-code-convert-string, and the
> string is made mutlibyte by string-to-multibyte from the raw-byte
> sequence (ex. inserting a file by raw-text in a mutlibyte buffer and
> extracting a string by buffer-substring), using Fstring_as_unibyte
> is correct.

That's the case for mac-code-convert-string.  Thanks for clarifying.

As for the `mac-hfs+' coding system shown in my previous mail, its
coding-system-type should have been 5 (raw-text), rather than 0
(emacs-mule) so that the leading bytes for eight-bit-control and
eight-bit-graphic may not be eaten.  And I had to do
(set-buffer-multibyte t) explicitly for the case of
decode-coding-string.  Maybe " *code-converting-work*" in
ctext-post-read-conversion should be " *code-conversion-work*" ?

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

(make-coding-system
 'mac-hfs+
 5
 (coding-system-mnemonic 'utf-8)
 "Like utf-8, but additionally does Mac HFS+ composition on decoding."
 (coding-system-flags 'utf-8)
 (list (cons 'safe-charsets (coding-system-get 'utf-8 'safe-charsets))
       '(post-read-conversion . mac-hfs+-post-read-conversion)
       '(pre-write-conversion . mac-hfs+-pre-write-conversion)))

(defun mac-hfs+-post-read-conversion (length)
  (save-excursion
    (save-restriction
      (narrow-to-region (point) (+ (point) length))
      (let ((in-workbuf (string= (buffer-name) " *code-conversion-work*"))
	    (str (mac-code-convert-string (buffer-string)
					  'utf-8 'utf-8 'HFS+C)))
	(when str
	  (erase-buffer)
	  (insert (if enable-multibyte-characters
		      (string-to-multibyte str) str)))
	(if in-workbuf
	    (set-buffer-multibyte t))
	(setq length (decode-coding-region (point-min) (point-max) 'utf-8))
	;; We are inside a post-read-conversion function, so the
	;; original post-read-conversion for utf-8 is not
	;; automatically called.
	(goto-char (point-min))
	(funcall (or (coding-system-get 'utf-8 'post-read-conversion)
		     'identity)
		 length)))))

(defun mac-hfs+-pre-write-conversion (beg end)
  (funcall (or (coding-system-get 'utf-8 'pre-write-conversion) 'ignore)
	   beg (+ beg (encode-coding-region beg end 'utf-8))))

(setq default-file-name-coding-system 'mac-hfs+)

Re: string-as-unibyte

[Stefan Monnier]

> If a multibyte string is given to mac-code-convert-string,
> and the string is made mutlibyte by string-to-multibyte from
> the raw-byte sequence (ex. inserting a file by raw-text in a
> mutlibyte buffer and extracting a string by
> buffer-substring), using Fstring_as_unibyte is correct.

Indeed, but sadly so.  In order to make it clear that the multibyte string
is expected to only contain single-byte chars (including eight-bit-* chars),
a comment is in order.

> Please note that we don't have Fstring_to_unibyte because it
> should work the same way as Fstring_as_unibyte.

Actually no.  string-to-unibyte should signal an error if it encounters
a non-ascii non-eight-bit-* char.


        Stefan

Re: string-as-unibyte

[Kenichi Handa]

In article <87wtnmg9iw.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  If a multibyte string is given to mac-code-convert-string,
>>  and the string is made mutlibyte by string-to-multibyte from
>>  the raw-byte sequence (ex. inserting a file by raw-text in a
>>  mutlibyte buffer and extracting a string by
>>  buffer-substring), using Fstring_as_unibyte is correct.

> Indeed, but sadly so.  In order to make it clear that the multibyte string
> is expected to only contain single-byte chars (including eight-bit-* chars),
> a comment is in order.

I agree.

>>  Please note that we don't have Fstring_to_unibyte because it
>>  should work the same way as Fstring_as_unibyte.

> Actually no.  string-to-unibyte should signal an error if it encounters
> a non-ascii non-eight-bit-* char.

Ah, hmmm, you are right.  I remember to implement it in
emacs-unicode.

---
Kenichi Handa
handa@m17n.org