[Unicode-2] `read' always returns multibyte symbol

[Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

Hi,

The following Lisp snippet emulates what Gnus does when reading
active data for the local.テスト newsgroup.  The buffer contains
data which have been retrieved from the nntp server.  Note that
the newsgroup name contains non-ASCII characters, which has been
encoded by utf-8 in the server.

--8<---------------cut here---------------start------------->8---
(let ((string (encode-coding-string "local.テスト" 'utf-8)))
  (with-temp-buffer
    (set-buffer-multibyte t)
    (insert (string-to-multibyte string))
    (goto-char (point-min))
    (multibyte-string-p (symbol-name (read (current-buffer))))))
--8<---------------cut here---------------end--------------->8---

While Emacs trunk returns nil for this, Emacs Unicode-2 returns t.

If it is not intentional, I hope `read' behaves just like it does
in Emacs trunk.  Otherwise, is there a way to make `read' return
a unibyte symbol (without slowing down)?

In the inside of Gnus, non-ASCII group names are all treated as
unibyte strings, that are the ones that the server has encoded
with certain coding systems.  Because of the present behavior of
`read' in Emacs Unicode-2, Gnus doesn't work with such newsgroups
perfectly.  You can find the actual code in gnus-start.el as
follows:

--8<---------------cut here---------------start------------->8---
;; Read an active file and place the results in `gnus-active-hashtb'.
(defun gnus-active-to-gnus-format (&optional method hashtb ignore-errors
					     real-active)
[...]
	      ;; group gets set to a symbol interned in the hash table
	      ;; (what a hack!!) - jwz
	      (setq group (let ((obarray hashtb)) (read cur)))
--8<---------------cut here---------------end--------------->8---

As you can see, it needs to work fast because there might be a
lot of newsgroups.  So, if possible, I don't want to modify it
into:

--8<---------------cut here---------------start------------->8---
 (setq group (intern (mm-string-as-unibyte (symbol-name (read cur))) hashtb))
--8<---------------cut here---------------end--------------->8---

Regards,

Re: [Unicode-2] `read' always returns multibyte symbol

[Kenichi Handa]

[-- Attachment #1: Type: text/plain, Size: 3483 bytes --]

In article <b4moddycwjv.fsf@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

> The following Lisp snippet emulates what Gnus does when reading
> active data for the local.テスト newsgroup.  The buffer contains
> data which have been retrieved from the nntp server.  Note that
> the newsgroup name contains non-ASCII characters, which has been
> encoded by utf-8 in the server.

> --8<---------------cut here---------------start------------->8---
> (let ((string (encode-coding-string "local.テスト" 'utf-8)))
>   (with-temp-buffer
>     (set-buffer-multibyte t)
>     (insert (string-to-multibyte string))
>     (goto-char (point-min))
>     (multibyte-string-p (symbol-name (read (current-buffer))))))
> --8<---------------cut here---------------end--------------->8---

> While Emacs trunk returns nil for this, Emacs Unicode-2 returns t.

That is because `read' decides the name is unibyte or
multibyte by whether the name is a valid multibyte sequence
or not.  In the trunk, utf-8 byte sequecne is not a valid
multibyte sequecne, but in emacs-unicode-2, it is valid.

> If it is not intentional, I hope `read' behaves just like it does
> in Emacs trunk.

The relevant code for `read' is very complicated and I want
to avoid touching it if there's another way.

In addition, I think it is the right thing that the above
code return t; i.e. any symbol created by reading a
multibyte buffer should have a multibyte string name.  The
bug to fix is that the following code also returns t in
emacs-unicode-2.

< --8<---------------cut here---------------start------------->8---
< (let ((string (encode-coding-string "local.テスト" 'utf-8)))
<   (with-temp-buffer
<     (set-buffer-multibyte nil)
<     (insert string)
<     (goto-char (point-min))
<     (multibyte-string-p (symbol-name (read (current-buffer))))))
< --8<---------------cut here---------------end--------------->8---

> Otherwise, is there a way to make `read' return a unibyte
> symbol (without slowing down)?

The replacement of the above code is simple as this:

(multibyte-string-p (intern (encode-coding-string "local.テスト" 'utf-8)))

But, hmmm, it seems that we can't use such a code in gnus...

> In the inside of Gnus, non-ASCII group names are all treated as
> unibyte strings, that are the ones that the server has encoded
> with certain coding systems.  Because of the present behavior of
> `read' in Emacs Unicode-2, Gnus doesn't work with such newsgroups
> perfectly.  You can find the actual code in gnus-start.el as
> follows:

> --8<---------------cut here---------------start------------->8---
> ;; Read an active file and place the results in `gnus-active-hashtb'.
> (defun gnus-active-to-gnus-format (&optional method hashtb ignore-errors
> 					     real-active)
> [...]
> 	      ;; group gets set to a symbol interned in the hash table
> 	      ;; (what a hack!!) - jwz
> 	      (setq group (let ((obarray hashtb)) (read cur)))
> --8<---------------cut here---------------end--------------->8---

How about this?

(setq group
       (let ((obarray hashtb) pos)
	 (skip-syntax-forward "^w_")
	 (setq pos (point))
	 (skip-syntax-forward "w_")
	 (intern (buffer-substring pos (point)))))

I think the overhead is just several more function calls.  The
actual task (searching for a range of symbol constituents,
make string from them, and intern it) is almost the same.

---
Kenichi Handa
handa@ni.aist.go.jp

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

> --8<---------------cut here---------------start------------->8---
> (let ((string (encode-coding-string "local.テスト" 'utf-8)))
>   (with-temp-buffer
>     (set-buffer-multibyte t)
>     (insert (string-to-multibyte string))
>     (goto-char (point-min))
>     (multibyte-string-p (symbol-name (read (current-buffer))))))
> --8<---------------cut here---------------end--------------->8---

I'm not sure what Emacs should do in such a case, but in the example
above, using a multibyte buffer is asking for trouble.
Can't Gnus use a unibyte buffer in its corresponding code?  That would
speed things up, save you the use of string-to-multibyte, and make it
crystal clear that the result should be unibyte.


        Stefan "trying hard not to say that the use of a multibyte
                buffer here is a plain bug ;-)"

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

> That is because `read' decides the name is unibyte or multibyte by
> whether the name is a valid multibyte sequence or not.

Yuck.

> The bug to fix is that the following code also returns t in
> emacs-unicode-2.

> < --8<---------------cut here---------------start------------->8---
> < (let ((string (encode-coding-string "local.テスト" 'utf-8)))
> <   (with-temp-buffer
> <     (set-buffer-multibyte nil)
> <     (insert string)
> <     (goto-char (point-min))
> <     (multibyte-string-p (symbol-name (read (current-buffer))))))
> < --8<---------------cut here---------------end--------------->8---

Yes, that's a clear bug.


        Stefan

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Kenichi Handa <handa@ni.aist.go.jp> wrote:

> In addition, I think it is the right thing that the above
> code return t; i.e. any symbol created by reading a
> multibyte buffer should have a multibyte string name.

I agree with that behavior.

> The bug to fix is that the following code also returns t in
> emacs-unicode-2.

> < --8<---------------cut here---------------start------------->8---
> < (let ((string (encode-coding-string "local.テスト" 'utf-8)))
> <   (with-temp-buffer
> <     (set-buffer-multibyte nil)
> <     (insert string)
> <     (goto-char (point-min))
> <     (multibyte-string-p (symbol-name (read (current-buffer))))))
> < --8<---------------cut here---------------end--------------->8---

Sure.  I'll try using a unibyte buffer to parse active data
(after the bug is fixed).

Regards,

Re: [Unicode-2] `read' always returns multibyte symbol

[Kenichi Handa]

In article <jwvoddykwt5.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > The bug to fix is that the following code also returns t in
> > emacs-unicode-2.

> > < --8<---------------cut here---------------start------------->8---
> > < (let ((string (encode-coding-string "local.テスト" 'utf-8)))
> > <   (with-temp-buffer
> > <     (set-buffer-multibyte nil)
> > <     (insert string)
> > <     (goto-char (point-min))
> > <     (multibyte-string-p (symbol-name (read (current-buffer))))))
> > < --8<---------------cut here---------------end--------------->8---

> Yes, that's a clear bug.

I've just installed a fix.

---
Kenichi Handa
handa@ni.aist.go.jp

[Unicode-2] `C-h f' error (was Re: `read' always returns multibyte symbol)

[Katsumi Yamaoka]

>>>>> Kenichi Handa <handa@ni.aist.go.jp> wrote:

> I've just installed a fix.

Thanks.

Isn't it a side effect of this change?  The `C-h f' command causes
the following error, though it can be solved by reloading "help".

Debugger entered--Lisp error: (setting-constant :validate)
  function-called-at-point()
  [...]
  call-interactively(describe-function)

It seems that the `with-syntax-table' macro, that
`function-called-at-point' uses, was not expanded properly when
dumping Emacs:

(disassemble 'function-called-at-point)
0	constant  syntax-table
1	call	  0
2	current-buffer 
3	varbind	  :validate
4	varbind	  setup-function
5	constant  (<byte code>...)
   0	   save-current-buffer 
   1	   varref    :validate
   2	   set-buffer 

Regards,

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

>>>>> Katsumi Yamaoka wrote:

> I'll try using a unibyte buffer to parse active data (after the bug
> is fixed).

Handa-san, thank you for the fix in Unicode-2.  I've also made a
change in the Gnus CVS trunk so that it may use a unibyte buffer.
Now it works not only with Emacs 23.0.60 but also with Emacs 22.1,
22.1.50, and 23.0.50.

BTW, I found another problem with Emacs 21 (Gnus still supports
Emacs 21, IIUC).  So, I'll go on looking into it further.

The diff between Gnus trunk and Unicode-2 is here:

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1379 bytes --]

*** gnus-start.el~	Sun Nov 11 21:51:22 2007
--- gnus-start.el	Wed Nov 14 11:32:28 2007
***************
*** 2106,2112 ****
  			    (if (equal method gnus-select-method)
  				(gnus-make-hashtable
  				 (count-lines (point-min) (point-max)))
! 			      (gnus-make-hashtable 4096)))))))
      ;; Delete unnecessary lines.
      (goto-char (point-min))
      (cond
--- 2106,2113 ----
  			    (if (equal method gnus-select-method)
  				(gnus-make-hashtable
  				 (count-lines (point-min) (point-max)))
! 			      (gnus-make-hashtable 4096))))))
! 	group max min)
      ;; Delete unnecessary lines.
      (goto-char (point-min))
      (cond
***************
*** 2141,2148 ****
  		      (insert prefix)
  		      (zerop (forward-line 1)))))))
      ;; Store the active file in a hash table.
!     (goto-char (point-min))
!     (let (group max min)
        (while (not (eobp))
  	(condition-case ()
  	    (progn
--- 2142,2153 ----
  		      (insert prefix)
  		      (zerop (forward-line 1)))))))
      ;; Store the active file in a hash table.
!     ;; Use a unibyte buffer in order to make `read' read non-ASCII
!     ;; group names (which have been encoded) as unibyte strings.
!     (mm-with-unibyte-buffer
!       (insert-buffer-substring cur)
!       (setq cur (current-buffer))
!       (goto-char (point-min))
        (while (not (eobp))
  	(condition-case ()
  	    (progn

[-- Attachment #3: Type: text/plain, Size: 9 bytes --]

Regards,

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: [Unicode-2] `C-h f' error (was Re: `read' always returns multibyte symbol)

[Kenichi Handa]

In article <b4mejete26t.fsf_-_@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> Kenichi Handa <handa@ni.aist.go.jp> wrote:
> > I've just installed a fix.

> Thanks.

> Isn't it a side effect of this change?  The `C-h f' command causes
> the following error, though it can be solved by reloading "help".

> Debugger entered--Lisp error: (setting-constant :validate)
>   function-called-at-point()
>   [...]
>   call-interactively(describe-function)

I can't reproduce that bug.  Perhaps because I did "make
bootstrap".  How about you?

> It seems that the `with-syntax-table' macro, that
> `function-called-at-point' uses, was not expanded properly when
> dumping Emacs:

> (disassemble 'function-called-at-point)
> 0	constant  syntax-table
> 1	call	  0
> 2	current-buffer 
> 3	varbind	  :validate
> 4	varbind	  setup-function
> 5	constant  (<byte code>...)
>    0	   save-current-buffer 
>    1	   varref    :validate
>    2	   set-buffer 

I got this in *Disassemble* buffer.

byte code for function-called-at-point:
  doc:  Return a function around point or else called by the list containing point. ...
  args: nil
0	constant  syntax-table
1	call	  0
2	current-buffer 
3	varbind	  required-features
4	varbind	  standard-display-european-internal
5	constant  (<byte code>...)
   0	   save-current-buffer 
   1	   varref    required-features
   2	   set-buffer 
   3	   discard   
   4	   constant  set-syntax-table
   5	   varref    standard-display-european-internal
   6	   call	     1
   7	   discard   
   8	   unbind    1
   9	   constant  set-syntax-table
   10	   return    

6	unwind-protect 
7	constant  set-syntax-table
8	varref	  emacs-lisp-mode-syntax-table
9	call	  1
10	discard	  
11	constant  nil
12	constant  <byte code>
   0	   save-excursion 
   1	   constant  zerop
   2	   constant  skip-syntax-backward
   3	   constant  "_w"
   4	   call	     1
   5	   call	     1
   6	   goto-if-nil 1
   9	   following-char 
   10	   char-syntax 
   11	   constant  119
   12	   eq	     
   13	   goto-if-not-nil 1
   16	   following-char 
   17	   char-syntax 
   18	   constant  95
   19	   eq	     
   20	   goto-if-not-nil 1
   23	   constant  forward-sexp
   24	   constant  -1
   25	   call	     1
   26	   discard   
   27:1	   constant  "'"
   28	   constant  nil
   29	   skip-chars-forward 
   30	   discard   
   31	   constant  read
   32	   current-buffer 
   33	   call	     1
   34	   dup	     
   35	   varbind   obj
   36	   symbolp   
   37	   goto-if-nil-else-pop 2
   40	   constant  fboundp
   41	   varref    obj
   42	   call	     1
   43	   goto-if-nil-else-pop 2
   46	   varref    obj
   47:2	   unbind    2
   48	   return    

13	constant  ((error))
14	condition-case 
15	goto-if-not-nil-else-pop 1
18	constant  nil
19	constant  <byte code>
   0	   save-excursion 
   1	   save-restriction 
   2	   point-min 
   3	   point     
   4	   constant  1000
   5	   diff	     
   6	   max	     
   7	   point-max 
   8	   narrow-to-region 
   9	   discard   
   10	   constant  backward-up-list
   11	   constant  1
   12	   call	     1
   13	   discard   
   14	   constant  1
   15	   forward-char 
   16	   discard   
   17	   constant  looking-at
   18	   constant  "[ 	]"
   19	   call	     1
   20	   goto-if-nil 1
   23	   constant  error
   24	   constant  "Probably not a Lisp function call"
   25	   call	     1
   26	   discard   
   27:1	   constant  read
   28	   current-buffer 
   29	   call	     1
   30	   dup	     
   31	   varbind   obj
   32	   symbolp   
   33	   goto-if-nil-else-pop 2
   36	   constant  fboundp
   37	   varref    obj
   38	   call	     1
   39	   goto-if-nil-else-pop 2
   42	   varref    obj
   43:2	   unbind    3
   44	   return    

20	constant  ((error))
21	condition-case 
22:1	unbind	  3
23	goto-if-not-nil-else-pop 6
26	constant  find-tag-default
27	call	  0
28	dup	  
29	varbind	  str
30	goto-if-nil-else-pop 2
33	constant  intern-soft
34	varref	  str
35	call	  1
36:2	dup	  
37	varbind	  sym
38	goto-if-nil 3
41	constant  fboundp
42	varref	  sym
43	call	  1
44	goto-if-nil 3
47	varref	  sym
48	goto	  5
51:3	constant  match-data
52	call	  0
53	varbind	  save-match-data-internal
54	constant  (<byte code>...)
   0	   constant  set-match-data
   1	   varref    save-match-data-internal
   2	   constant  evaporate
   3	   call	     2
   4	   return    

55	unwind-protect 
56	varref	  str
57	goto-if-nil-else-pop 4
60	constant  string-match
61	constant  "\\`\\W*\\(.*?\\)\\W*\\'"
62	varref	  str
63	call	  2
64	goto-if-nil-else-pop 4
67	constant  intern-soft
68	constant  match-string
69	constant  1
70	varref	  str
71	call	  2
72	call	  1
73	varset	  sym
74	constant  fboundp
75	varref	  sym
76	call	  1
77	goto-if-nil-else-pop 4
80	varref	  sym
81:4	unbind	  2
82:5	unbind	  2
83:6	return	  

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

> !     ;; Use a unibyte buffer in order to make `read' read non-ASCII
> !     ;; group names (which have been encoded) as unibyte strings.
> !     (mm-with-unibyte-buffer
> !       (insert-buffer-substring cur)

Why is `cur' a multibyte buffer?  Since it contains encoded strings, I'd
expect it would be better (more robust and convenient) to use a unibyte
buffer for it.


        Stefan

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Stefan Monnier wrote:

>> !     ;; Use a unibyte buffer in order to make `read' read non-ASCII
>> !     ;; group names (which have been encoded) as unibyte strings.
>> !     (mm-with-unibyte-buffer
>> !       (insert-buffer-substring cur)

> Why is `cur' a multibyte buffer?  Since it contains encoded strings, I'd
> expect it would be better (more robust and convenient) to use a unibyte
> buffer for it.

Good point.  The `cur' is `nntp-server-buffer' (" *nntpd*") or
`gnus-work-buffer' (" *gnus work*") as the case may be.  Gnus uses
those buffers for various purposes.  Although there looks no
situation where it is necessary to have multibyte data as far as
I can observe, Gnus explicitly sets them as multibyte buffers (see
`nnheader-init-server-buffer' and `gnus-set-work-buffer').  I
believe the reason they do so is to prevent from breaking data
when copying them to another multibyte buffer (IIUC, copying data
from a multibyte buffer to a unibyte buffer causes no problem).
So, I didn't modify those buffers' multibyteness.  If I introduced
a new unibyte work buffer (such as " *gnus binary work*"), it
required that `gnus-read-active-file-2' binds `nntp-server-buffer'
to it for example.  It is used by all the back ends but I'm not
sure it never causes a problem with them all.

Regards,

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

>>> !     ;; Use a unibyte buffer in order to make `read' read non-ASCII
>>> !     ;; group names (which have been encoded) as unibyte strings.
>>> !     (mm-with-unibyte-buffer
>>> !       (insert-buffer-substring cur)

>> Why is `cur' a multibyte buffer?  Since it contains encoded strings, I'd
>> expect it would be better (more robust and convenient) to use a unibyte
>> buffer for it.

> Good point.  The `cur' is `nntp-server-buffer' (" *nntpd*") or
> `gnus-work-buffer' (" *gnus work*") as the case may be.

Don't know about gnus-work-buffer, but nntp-server-buffer should only
ever contain unibyte data AFAICT, so it would be better to put it in
unibyte mode.

> Gnus uses those buffers for various purposes.  Although there looks no
> situation where it is necessary to have multibyte data as far as I can
> observe, Gnus explicitly sets them as multibyte buffers (see
> `nnheader-init-server-buffer' and `gnus-set-work-buffer').

> I believe the reason they do so is to prevent from breaking data when
> copying them to another multibyte buffer (IIUC, copying data from
> a multibyte buffer to a unibyte buffer causes no problem).

I'm not sure I understand: copying data from a multibyte buffer to
a unibyte buffer is exactly the case that can cause problems.


        Stefan

Re: [Unicode-2] `C-h f' error

[Katsumi Yamaoka]

>>>>> Kenichi Handa wrote:
> In article <b4mejete26t.fsf_-_@jpl.org>,
>	Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> Debugger entered--Lisp error: (setting-constant :validate)
>>   function-called-at-point()
>>   [...]
>>   call-interactively(describe-function)

> I can't reproduce that bug.  Perhaps because I did "make
> bootstrap".  How about you?

I always do "make bootstrap" with a copy of the source checked
out from CVS.  I did it again but it made no difference.  Hm.

Though I'm not sure it is useful, the function definition of
`function-called-at-point' that has been dumped in the Emacs
executable is here (it differs from the one in help.elc):

(fset 'function-called-at-point (read (base64-decode-string "\
I1tuaWwgIlwzMDYgcBgZXDMwN1wyMTZcMzEwCiFcMjEwXDMxMVwzMTJcMzEzXDIxN1wyMDYWAFwz
MTFcMzE0XDMxNVwyMTcrXDIwNlMAXDMxNiBcMjExG1wyMDUkAFwzMTcLIVwyMTEcXDIwMzMAXDMy
MAwhXDIwMzMADFwyMDJSAFwzMjEgHVwzMjJcMjE2C1wyMDVRAFwzMjNcMzI0C1wiXDIwNVEAXDMx
N1wzMjVcMzI2C1wiIRRcMzIwDCFcMjA1UQAMKipcMjA3IiBbOnZhbGlkYXRlIHNldHVwLWZ1bmN0
aW9uIGVtYWNzLWxpc3AtbW9kZS1zeW50YXgtdGFibGUgc3RyIHN5bSBzYXZlLW1hdGNoLWRhdGEt
aW50ZXJuYWwgc3ludGF4LXRhYmxlICgoYnl0ZS1jb2RlICJyCHFcMjEwXDMwMgkhXDIxMClcMzAy
XDIwNyIgWzp2YWxpZGF0ZSBzZXR1cC1mdW5jdGlvbiBzZXQtc3ludGF4LXRhYmxlXSAyKSkgc2V0
LXN5bnRheC10YWJsZSBuaWwgKGJ5dGUtY29kZSAiXDIxMlwzMDFcMzAyXDMwMyEhXDIwMxsAZ3pc
MzA0PVwyMDQbAGd6XDMwNT1cMjA0GwBcMzA2XDMwNyFcMjEwXDMxMFwzMTF3XDIxMFwzMTJwIVwy
MTEYOVwyMDUvAFwzMTMIIVwyMDUvAAgqXDIwNyIgW29iaiB6ZXJvcCBza2lwLXN5bnRheC1iYWNr
d2FyZCAiX3ciIDExOSA5NSBmb3J3YXJkLXNleHAgLTEgIiciIG5pbCByZWFkIGZib3VuZHBdIDQp
ICgoZXJyb3IpKSAoYnl0ZS1jb2RlICJcMjEyXDIxNGVgXDMwMVpdZH1cMjEwXDMwMlwzMDMhXDIx
MFwzMDN1XDIxMFwzMDRcMzA1IVwyMDMbAFwzMDZcMzA3IVwyMTBcMzEwcCFcMjExGDlcMjA1KwBc
MzExCCFcMjA1KwAIK1wyMDciIFtvYmogMTAwMCBiYWNrd2FyZC11cC1saXN0IDEgbG9va2luZy1h
dCAiWyAJXSIgZXJyb3IgIlByb2JhYmx5IG5vdCBhIExpc3AgZnVuY3Rpb24gY2FsbCIgcmVhZCBm
Ym91bmRwXSA0KSAoKGVycm9yKSkgZmluZC10YWctZGVmYXVsdCBpbnRlcm4tc29mdCBmYm91bmRw
IG1hdGNoLWRhdGEgKChieXRlLWNvZGUgIlwzMDEIXDMwMlwiXDIwNyIgW3NhdmUtbWF0Y2gtZGF0
YS1pbnRlcm5hbCBzZXQtbWF0Y2gtZGF0YSBldmFwb3JhdGVdIDMpKSBzdHJpbmctbWF0Y2ggIlxc
YFxcVypcXCguKj9cXClcXFcqXFwnIiBtYXRjaC1zdHJpbmcgMV0gNSA5MDkyNDRd")))

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Stefan Monnier wrote:

> Don't know about gnus-work-buffer, but nntp-server-buffer should only
> ever contain unibyte data AFAICT, so it would be better to put it in
> unibyte mode.

I think it's better, too.  However, there might be a code that
copies data from nntp-server-buffer to a multibyte buffer.  I'm
not capable to check all the Gnus code.

>> (IIUC, copying data from a multibyte buffer to a unibyte buffer
>> causes no problem).

> I'm not sure I understand: copying data from a multibyte buffer to
> a unibyte buffer is exactly the case that can cause problems.

I agree that's generally true.  But in Gnus' case, data in a
multibyte work buffer are the multibyte version of binary data.
I don't know proper words to explain it, sorry.  In other words,
they are the one which `string-to-multibyte' converted binary
data to.  For example:

(with-temp-buffer
  (set-buffer-multibyte t)
  (insert (string-to-multibyte (encode-coding-string "日本語" 'utf-8)))
  (let ((buffer (current-buffer)))
    (with-temp-buffer
      (set-buffer-multibyte nil)
      (insert-buffer-substring buffer)
      (decode-coding-string (buffer-string) 'utf-8))))
 => "日本語"

I'm not sure it works with any data, though.

Regards,

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

> I think it's better, too.  However, there might be a code that
> copies data from nntp-server-buffer to a multibyte buffer.  I'm
> not capable to check all the Gnus code.

I understand the desire to avoid changing code, but I think in the long
run it'll pay off.

>>> (IIUC, copying data from a multibyte buffer to a unibyte buffer
>>> causes no problem).

>> I'm not sure I understand: copying data from a multibyte buffer to
>> a unibyte buffer is exactly the case that can cause problems.

> I agree that's generally true.  But in Gnus' case, data in a
> multibyte work buffer are the multibyte version of binary data.
> I don't know proper words to explain it, sorry.  In other words,
> they are the one which `string-to-multibyte' converted binary
> data to.  For example:

> (with-temp-buffer
>   (set-buffer-multibyte t)
>   (insert (string-to-multibyte (encode-coding-string "日本語" 'utf-8)))
>   (let ((buffer (current-buffer)))
>     (with-temp-buffer
>       (set-buffer-multibyte nil)
>       (insert-buffer-substring buffer)
>       (decode-coding-string (buffer-string) 'utf-8))))
>  => "日本語"

> I'm not sure it works with any data, though.

I'm not sure what you're saying.  But IIUC the source buffer in your
example would be nntp-server-buffer, in which case turning it into
unibyte will not introduce any problem.  On the contrary, it'll make it
more obviously correct.


        Stefan

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Katsumi Yamaoka wrote:

> BTW, I found another problem with Emacs 21 (Gnus still supports
> Emacs 21, IIUC).  So, I'll go on looking into it further.

I realized a network process that is created by
`open-network-stream' in Emacs 21 breaks encoded non-ASCII group
names if the process buffer is in the multibyte mode even if the
process coding system is binary.  It behaves as if
`toggle-enable-multibyte-characters' modifies binary data when
turning on the multibyteness of a buffer.  So, I made changes in
nntp.el in the Gnus trunk so that it makes a process buffer
unibyte.  I also modified the nntp functions that copy data from
a unibyte buffer to a multibyte buffer.

Regards,

Re: [Unicode-2] `read' always returns multibyte symbol

[Kenichi Handa]

In article <b4m1war6ca5.fsf@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> Katsumi Yamaoka wrote:
> > BTW, I found another problem with Emacs 21 (Gnus still supports
> > Emacs 21, IIUC).  So, I'll go on looking into it further.

> I realized a network process that is created by
> `open-network-stream' in Emacs 21 breaks encoded non-ASCII group
> names if the process buffer is in the multibyte mode even if the
> process coding system is binary.  It behaves as if
> `toggle-enable-multibyte-characters' modifies binary data when
> turning on the multibyteness of a buffer.

If "modifies" means that 8-bit bytes are converted to
multibyte characters as what string-as-multibyte does, it's
an expected behaviour.

I long ago proposed a facility that turns on the
multibyteness of a buffer while converting 8-bit bytes to
multibyte characters as what string-to-multibyte does, but
not accepted.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Kenichi Handa wrote:
> In article <b4m1war6ca5.fsf@jpl.org>,
>	Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> I realized a network process that is created by
>> `open-network-stream' in Emacs 21 breaks encoded non-ASCII group
>> names if the process buffer is in the multibyte mode even if the
>> process coding system is binary.  It behaves as if
>> `toggle-enable-multibyte-characters' modifies binary data when
>> turning on the multibyteness of a buffer.

(The changes that I made in nntp.el has been archived in
 <URL:http://article.gmane.org/gmane.emacs.gnus.commits/5519>.)

> If "modifies" means that 8-bit bytes are converted to
> multibyte characters as what string-as-multibyte does, it's
> an expected behaviour.

What I observed was different.  The group name "テスト" is
encoded by utf-8 by the nntp server into:

"\343\203\206\343\202\271\343\203\210"

After it is transferred to Gnus, in the nntp process bufer it is
modified into:

"\343\203XY\343\203\210"

Where X is (make-char 'greek-iso8859-7 99)
  and Y is (make-char 'latin-iso8859-2 57).

Since Gnus treats a group name as a unibyte string, finally it
is made into:

"\343\203\343\271\343\203\210"

> I long ago proposed a facility that turns on the
> multibyteness of a buffer while converting 8-bit bytes to
> multibyte characters as what string-to-multibyte does, but
> not accepted.

But the modern Emacsen does do so, doesn't it?

Re: [Unicode-2] `read' always returns multibyte symbol

[Kenichi Handa]

[-- Attachment #1: Type: text/plain, Size: 1611 bytes --]

In article <b4moddv20sy.fsf@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

> > If "modifies" means that 8-bit bytes are converted to
> > multibyte characters as what string-as-multibyte does, it's
> > an expected behaviour.

> What I observed was different.  The group name "テスト" is
> encoded by utf-8 by the nntp server into:

> "\343\203\206\343\202\271\343\203\210"

> After it is transferred to Gnus, in the nntp process bufer it is
> modified into:

> "\343\203XY\343\203\210"

> Where X is (make-char 'greek-iso8859-7 99)
>   and Y is (make-char 'latin-iso8859-2 57).

That is exactly what string-as-multibyte does. \206\343 and
\202\271 are valid multibyte forms in the current Emacs,
thus are treated as multibyte characters.

> Since Gnus treats a group name as a unibyte string, finally it
> is made into:

> "\343\203\343\271\343\203\210"

It seems that gnus treats "\343\203XY\343\203\210" as
unibyte by converting it by string-make-unibyte.

Please try this:

(string-make-unibyte
 (string-as-multibyte "\343\203\206\343\202\271\343\203\210"))

You'll get the above result, ... yes, very weird.

On the other hand,

(string-as-unibyte
 (string-as-multibyte "\343\203\206\343\202\271\343\203\210"))
 =>  "\343\203\206\343\202\271\343\203\210"

> > I long ago proposed a facility that turns on the
> > multibyteness of a buffer while converting 8-bit bytes to
> > multibyte characters as what string-to-multibyte does, but
> > not accepted.

> But the modern Emacsen does do so, doesn't it?

No.

---
Kenichi Handa
handa@ni.aist.go.jp

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

> If "modifies" means that 8-bit bytes are converted to
> multibyte characters as what string-as-multibyte does, it's
> an expected behaviour.

99% of the uses of string-as-multibyte are bugs.


        Stefan

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Kenichi Handa <handa@ni.aist.go.jp> wrote:

> In article <b4moddv20sy.fsf@jpl.org>,
>	Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> What I observed was different.

> That is exactly what string-as-multibyte does. \206\343 and
> \202\271 are valid multibyte forms in the current Emacs,
> thus are treated as multibyte characters.

I understood why such readable characters appeared abruptly.

[...]

> Please try this:

> (string-make-unibyte
>  (string-as-multibyte "\343\203\206\343\202\271\343\203\210"))

> You'll get the above result, ... yes, very weird.

Oh, it made me surprised a bit.  But I often view such a scene
while playing with unibyte and multibyte things, and it always
confuses me.

> On the other hand,

> (string-as-unibyte
>  (string-as-multibyte "\343\203\206\343\202\271\343\203\210"))
>  =>  "\343\203\206\343\202\271\343\203\210"

>>> I long ago proposed a facility that turns on the
>>> multibyteness of a buffer while converting 8-bit bytes to
>>> multibyte characters as what string-to-multibyte does, but
>>> not accepted.

>> But the modern Emacsen does do so, doesn't it?

> No.

Oops.  I misunderstood that the reason why Emacs 22 and 23 don't
break 8-bit data while they are being fed into a multibyte buffer
from a network process of which the process coding system is
binary.  So, maybe the best ways for the present are still to
use a unibyte buffer for unibyte data and to use a multibyte
buffer for multibyte data.  And use a string, not a buffer, to
encode and decode data if the multibyteness of data will change,
like:

(insert (prog1
	    (decode-coding-string (buffer-string) 'coding)
	  (erase-buffer)
	  (set-buffer-multibyte t)))

Re: [Unicode-2] `read' always returns multibyte symbol

[Kenichi Handa]

In article <jwvd4ubttyx.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > If "modifies" means that 8-bit bytes are converted to
> > multibyte characters as what string-as-multibyte does, it's
> > an expected behaviour.

> 99% of the uses of string-as-multibyte are bugs.

Sure.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `read' always returns multibyte symbol

[Kenichi Handa]

In article <b4my7czxeze.fsf@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Oops.  I misunderstood that the reason why Emacs 22 and 23 don't
> break 8-bit data while they are being fed into a multibyte buffer
> from a network process of which the process coding system is
> binary.  So, maybe the best ways for the present are still to
> use a unibyte buffer for unibyte data and to use a multibyte
> buffer for multibyte data.  And use a string, not a buffer, to
> encode and decode data if the multibyteness of data will change,
> like:

> (insert (prog1
> 	    (decode-coding-string (buffer-string) 'coding)
> 	  (erase-buffer)
> 	  (set-buffer-multibyte t)))

The best is to decide buffer's multibyteness just after it
is created, and don't change the multibyteness later.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `read' always returns multibyte symbol

[Katsumi Yamaoka]

>>>>> Kenichi Handa wrote:
> In article <b4my7czxeze.fsf@jpl.org>,
>	Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> (insert (prog1
>> 	    (decode-coding-string (buffer-string) 'coding)
>> 	  (erase-buffer)
>> 	  (set-buffer-multibyte t)))

> The best is to decide buffer's multibyteness just after it
> is created, and don't change the multibyteness later.

I see.  In relation to this, I've been wanting to exterminate
the `mm-with-unibyte-current-buffer' macro that Gnus uses here
and there (if you have time, please look at how it is evil, in
mm-util.el).

Re: [Unicode-2] `read' always returns multibyte symbol

[Stefan Monnier]

> I see.  In relation to this, I've been wanting to exterminate
> the `mm-with-unibyte-current-buffer' macro that Gnus uses here
> and there (if you have time, please look at how it is evil, in
> mm-util.el).

Yes, I spotted it a while ago already (I'm using a few local hacks to
try and catch some multi/unibyte abuses so I tend to bump into bugs
a bit earlier than in normal use).

I think a mistake of Emacs's handling of encoding issues is that we use
"unibyte" and "multibyte" rather than "byteS" and chars".


        Stefan


PS: Here are some hunks from my local changes.


@@ -1034,16 +1068,18 @@
 (defmacro mm-with-unibyte-buffer (&rest forms)
   "Create a temporary buffer, and evaluate FORMS there like `progn'.
 Use unibyte mode for this."
-  `(let (default-enable-multibyte-characters)
-     (with-temp-buffer ,@forms)))
+  `(with-temp-buffer
+     (mm-disable-multibyte)
+     ,@forms))
 (put 'mm-with-unibyte-buffer 'lisp-indent-function 0)
 (put 'mm-with-unibyte-buffer 'edebug-form-spec '(body))
 
 (defmacro mm-with-multibyte-buffer (&rest forms)
   "Create a temporary buffer, and evaluate FORMS there like `progn'.
 Use multibyte mode for this."
-  `(let ((default-enable-multibyte-characters t))
-     (with-temp-buffer ,@forms)))
+  `(with-temp-buffer
+     (mm-enable-multibyte)
+     ,@forms))
 (put 'mm-with-multibyte-buffer 'lisp-indent-function 0)
 (put 'mm-with-multibyte-buffer 'edebug-form-spec '(body))
 
@@ -1058,24 +1094,29 @@
 harmful since it is likely to modify existing data in the buffer.
 For instance, it converts \"\\300\\255\" into \"\\255\" in
 Emacs 23 (unicode)."
-  (let ((multibyte (make-symbol "multibyte"))
-	(buffer (make-symbol "buffer")))
-    `(if mm-emacs-mule
-	 (let ((,multibyte enable-multibyte-characters)
-	       (,buffer (current-buffer)))
-	   (unwind-protect
-	       (let (default-enable-multibyte-characters)
-		 (set-buffer-multibyte nil)
-		 ,@forms)
-	     (set-buffer ,buffer)
-	     (set-buffer-multibyte ,multibyte)))
-       (let (default-enable-multibyte-characters)
-	 ,@forms))))
+  (message "Braindeadly defined macro: mm-with-unibyte-current-buffer")
+  ;; (let ((multibyte (make-symbol "multibyte"))
+  ;;       (buffer (make-symbol "buffer")))
+  ;;   `(if mm-emacs-mule
+  ;;        (let ((,multibyte enable-multibyte-characters)
+  ;;              (,buffer (current-buffer)))
+  ;;          (unwind-protect
+  ;;              (let (default-enable-multibyte-characters)
+  ;;       	 (set-buffer-multibyte nil)
+  ;;       	 ,@forms)
+  ;;            (set-buffer ,buffer)
+  ;;            (set-buffer-multibyte ,multibyte)))
+  ;;      (let (default-enable-multibyte-characters)
+  ;;        ,@forms)))
+  `(progn (assert (not enable-multibyte-characters))
+          ,@forms)
+  )
 (put 'mm-with-unibyte-current-buffer 'lisp-indent-function 0)
 (put 'mm-with-unibyte-current-buffer 'edebug-form-spec '(body))
 
 (defmacro mm-with-unibyte (&rest forms)
   "Eval the FORMS with the default value of `enable-multibyte-characters' nil."
+  (message "Braindead macro: mm-with-unibyte")
   `(let (default-enable-multibyte-characters)
      ,@forms))
 (put 'mm-with-unibyte 'lisp-indent-function 0)
@@ -1083,6 +1124,7 @@
 
 (defmacro mm-with-multibyte (&rest forms)
   "Eval the FORMS with the default value of `enable-multibyte-characters' t."
+  (message "Braindead macro: mm-with-multibyte")
   `(let ((default-enable-multibyte-characters t))
      ,@forms))
 (put 'mm-with-multibyte 'lisp-indent-function 0)

Re: [Unicode-2] `read' always returns multibyte symbol

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 15 Nov 2007 10:22:12 -0500
> Cc: Katsumi Yamaoka <yamaoka@jpl.org>, ding@gnus.org, emacs-devel@gnu.org
> 
> 99% of the uses of string-as-multibyte are bugs.

Should we emit a warning from the byte compiler about that?  (Sorry if
we already do: I didn't have time to look.)

Re: [Unicode-2] `C-h f' error

[Katsumi Yamaoka]

>>>>> Katsumi Yamaoka wrote:

>>> Debugger entered--Lisp error: (setting-constant :validate)
>>>   function-called-at-point()
>>>   [...]
>>>   call-interactively(describe-function)

I think I have reached to the real cause of this problem.  Though
it may happen only to me, I've tested it with two machines running
different OS (Fedora 8 and RHL 9).  The necessary conditions to
make it happen are:

  A function is dumped into the Emacs executable.
  It uses a macro in which uninterned symbols are used in `let'.

In that case, uninterned symbols seem to be replaced with the
interned ones when dumping into Emacs.  The way I reproduced it is:

1. Make the /tmp/test.el file (attached below) and byte compile it.
2. Modify the lisp/loadup.el file as follows:

--8<---------------cut here---------------start------------->8---
*** loadup.el~	Sun Nov 11 21:51:19 2007
--- loadup.el	Mon Nov 19 08:14:23 2007
***************
*** 85,88 ****
--- 85,89 ----
  (load "simple")
  
+ (load "/tmp/test")
  (load "help")
  
--8<---------------cut here---------------end--------------->8---

3. Dump Emacs in this way:
    $ cd src
    $ ./temacs -batch -l loadup dump

4. Run Emacs as:
    $ ./emacs -batch -Q -eval '(foo)'
   I got:
set-display-table-and-terminal-coding-system reset-language-environment English

5. Run Emacs as:
    $ ./emacs -batch -Q -l /tmp/test -eval '(foo)'
  I got:
foo bar baz

The test.el file is here:

--8<---------------cut here---------------start------------->8---
(defmacro foo-macro nil
  (let ((foo (make-symbol "foo"))
	(bar (make-symbol "bar"))
	(baz (make-symbol "baz")))
    `(message "%s %s %s" ',foo ',bar ',baz)))

(defun foo nil
  (foo-macro))
--8<---------------cut here---------------end--------------->8---

Regards,

Re: [Unicode-2] `C-h f' error

[CHENG Gao]

*On Mon, 19 Nov 2007 17:31:02 +0900
* Also sprach Katsumi Yamaoka <yamaoka@jpl.org>:

> I think I have reached to the real cause of this problem.  Though
> it may happen only to me, I've tested it with two machines running
> different OS (Fedora 8 and RHL 9).  The necessary conditions to
> make it happen are:
Yesterday I reported this problem. I just read you already found this.
This is to confirm this problem does not only happen to you.

-- 
Vivere est cogitare

Re: [Unicode-2] `C-h f' error

[Katsumi Yamaoka]

>>>>> CHENG Gao wrote:
> *On Mon, 19 Nov 2007 17:31:02 +0900
> * Also sprach Katsumi Yamaoka <yamaoka@jpl.org>:

>> I think I have reached to the real cause of this problem.  Though
>> it may happen only to me, I've tested it with two machines running
>> different OS (Fedora 8 and RHL 9).  The necessary conditions to
>> make it happen are:
> Yesterday I reported this problem. I just read you already found this.
> This is to confirm this problem does not only happen to you.

Thanks for the confirmation.  I tried bootstrapping Unicode-2
with lread.c before Handa-san changed.  It works normal.  For
the Lisp form

(aref (symbol-function (function function-called-at-point)) 2)

it returns

[buffer table emacs-lisp-mode-syntax-table str sym ...

while the latest Unicode-2 returns:

[:validate setup-function emacs-lisp-mode-syntax-table str sym ...

Regards,

Re: [Unicode-2] `C-h f' error

[Kenichi Handa]

In article <b4mbq9nlvf3.fsf@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Thanks for the confirmation.  I tried bootstrapping Unicode-2
> with lread.c before Handa-san changed.  It works normal.  For
> the Lisp form

> (aref (symbol-function (function function-called-at-point)) 2)

> it returns

> [buffer table emacs-lisp-mode-syntax-table str sym ...

> while the latest Unicode-2 returns:

> [:validate setup-function emacs-lisp-mode-syntax-table str sym ...

Thank you for investigating this problem.  But, as I don't
have a time to work on it at the moment, I fixed lread.c so
that it works as previously.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `C-h f' error

[Katsumi Yamaoka]

>>>>> Kenichi Handa wrote:

> Thank you for investigating this problem.  But, as I don't
> have a time to work on it at the moment, I fixed lread.c so
> that it works as previously.

No need to fix it in a hurry (at least for me), please take
your time.  Thanks.

Re: [Unicode-2] `C-h f' error

[Richard Stallman]

    Thank you for investigating this problem.  But, as I don't
    have a time to work on it at the moment, I fixed lread.c so
    that it works as previously.

What is the lread.c behavior that you changed?
What is the new behavior?

Re: [Unicode-2] `C-h f' error

[Kenichi Handa]

In article <E1Iv1mx-0003q8-0I@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     Thank you for investigating this problem.  But, as I don't
>     have a time to work on it at the moment, I fixed lread.c so
>     that it works as previously.

> What is the lread.c behavior that you changed?

Make the Lisp reader to generate a symbol of unibyte name
when it is read from a unibyte buffer.  Previously, the
multibyteness of a symbol name is determined by the byte
sequence (by using make_string).

> What is the new behavior?

The following phenomenon was reported.

Katsumi Yamaoka <yamaoka@jpl.org> writes:
> Isn't it a side effect of this change?  The `C-h f' command causes
> the following error, though it can be solved by reloading "help".

> Debugger entered--Lisp error: (setting-constant :validate)
>   function-called-at-point()
>   [...]
>   call-interactively(describe-function)

> It seems that the `with-syntax-table' macro, that
> `function-called-at-point' uses, was not expanded properly when
> dumping Emacs:

> (disassemble 'function-called-at-point)
> 0	constant  syntax-table
> 1	call	  0
> 2	current-buffer 
> 3	varbind	  :validate
> 4	varbind	  setup-function
> 5	constant  (<byte code>...)
>    0	   save-current-buffer 
>    1	   varref    :validate
>    2	   set-buffer 

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `C-h f' error

[Richard Stallman]

    Make the Lisp reader to generate a symbol of unibyte name
    when it is read from a unibyte buffer.  Previously, the
    multibyteness of a symbol name is determined by the byte
    sequence (by using make_string).

Please add a comment explaining the bad results that happened
when we tried the other way.  (Unless you already did so.)
It is very important to explain this IN the source file.

Re: [Unicode-2] `C-h f' error

[Johan Bockgård]

Kenichi Handa <handa@ni.aist.go.jp> writes:

> Thank you for investigating this problem.  But, as I don't
> have a time to work on it at the moment, I fixed lread.c so
> that it works as previously.

Your change replaced make_symbol with Fmake_symbol (and intern with
Fintern), and make_symbol does

    Fmake_symbol ((!NILP (Vpurify_flag)
                   ? make_pure_string (str, len, len, 0)
                   : make_string (str, len)));

In the make_symbol/Fmake_symbol pair of functions, the Vpurify_flag
check is in the former (so is not done after the change); but in the
intern/Fintern pair it is in the latter. Isn't this the problem?

-- 
Johan Bockgård

Re: [Unicode-2] `C-h f' error

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 813 bytes --]

In article <yoijve7tvvis.fsf@remote1.student.chalmers.se>, bojohan+news@dd.chalmers.se (Johan Bockg^[$(D+)^[(Brd) writes:

> Your change replaced make_symbol with Fmake_symbol (and intern with
> Fintern), and make_symbol does

>     Fmake_symbol ((!NILP (Vpurify_flag)
>                    ? make_pure_string (str, len, len, 0)
>                    : make_string (str, len)));

> In the make_symbol/Fmake_symbol pair of functions, the Vpurify_flag
> check is in the former (so is not done after the change); but in the
> intern/Fintern pair it is in the latter. Isn't this the problem?

Ah! Perhaps.  But, I don't understand the reason of calling
make_pure_string always with the last arg multibyte as 0.
Richard, don't you remember anything?  It seems that this
part was lastly modified by you about 10 years ago.

Re: [Unicode-2] `C-h f' error

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 988 bytes --]

In article <yoijve7tvvis.fsf@remote1.student.chalmers.se>, bojohan+news@dd.chalmers.se (Johan Bockg^[$(D+)^[(Brd) writes:

> Your change replaced make_symbol with Fmake_symbol (and intern with
> Fintern), and make_symbol does

>     Fmake_symbol ((!NILP (Vpurify_flag)
>                    ? make_pure_string (str, len, len, 0)
>                    : make_string (str, len)));

> In the make_symbol/Fmake_symbol pair of functions, the Vpurify_flag
> check is in the former (so is not done after the change); but in the
> intern/Fintern pair it is in the latter. Isn't this the problem?

Ah! Good point!  Perhaps you are right.

But, I don't understand the reason of calling
make_pure_string always with the last arg multibyte as 0.
It isn't consistent with the case of Vpurify_flag is nil
(letting make_string determine the multibyteness).

Richard, don't you remember anything?  It seems that this
part was lastly modified by you about 10 years ago.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `C-h f' error

[Richard Stallman]

    >     Fmake_symbol ((!NILP (Vpurify_flag)
    >                    ? make_pure_string (str, len, len, 0)
    >                    : make_string (str, len)));

    > In the make_symbol/Fmake_symbol pair of functions, the Vpurify_flag
    > check is in the former (so is not done after the change); but in the
    > intern/Fintern pair it is in the latter. Isn't this the problem?

    Ah! Perhaps.  But, I don't understand the reason of calling
    make_pure_string always with the last arg multibyte as 0.
    Richard, don't you remember anything?

Not any more.
Sorry.

Re: [Unicode-2] `C-h f' error

[Kenichi Handa]

In article <E1IywL8-0006cL-L4@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

> In the make_symbol/Fmake_symbol pair of functions, the Vpurify_flag
> check is in the former (so is not done after the change); but in the
> intern/Fintern pair it is in the latter. Isn't this the problem?

>     Ah! Perhaps.  But, I don't understand the reason of calling
>     make_pure_string always with the last arg multibyte as 0.
>     Richard, don't you remember anything?

> Not any more.
> Sorry.

Ok, I fixed the code by checking Vpurfiy_flag.  Please check
if the C-h f problem is fixed or not.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: [Unicode-2] `C-h f' error

[Katsumi Yamaoka]

>>>>> Kenichi Handa wrote:

> Ok, I fixed the code by checking Vpurfiy_flag.  Please check
> if the C-h f problem is fixed or not.

It works as expected.  Thank you.