gnus doesn't encode non-ascii attachment filenames

gnus doesn't encode non-ascii attachment filenames

[Zhang Wei]

If the attached file has a non-ascii(chinese actually) filename, gnus
failed to correctly encode the filename. But it do decode non-ascii
attachment filename composed by other MUA.

-- 
Zhang Wei or Brep
<brep@smth.org>

Re: gnus doesn't encode non-ascii attachment filenames

[Zhang Wei]

"Richard M. Stallman" <rms@gnu.org> writes:

> I suggest you provide a precise test case so people can reproduce the
> problem.

Sorry, false alarm. This bug is only with the emacs-unicode-2
branch. No problem with emacs 22.

The difference of internal character representation between emacs23
and emacs22, makes gnus can't handle non-ascii attachment filename
correctly.

To generate a rfc2231 encoded filename such as "%d6%d0%ce%c4",
function (following-char) returns "d6" for the first char of the
filename "d6d0cec4" in emacs22, while in emacs23 this function returns
"3fffd6".

This is an ugly hack to solve this problem:

*** rfc2231.el	08  Feb 2006 20:05:31 +0800	7.13
--- rfc2231.el	14  Feb 2006 09:12:16 +0800	
***************
*** 194,207 ****
    (string-match "\\`\\(?:\\([^']+\\)?'\\([^']+\\)?'\\)?\\(.+\\)" string)
    (let ((coding-system (mm-charset-to-coding-system (match-string 1 string)))
  	;;(language (match-string 2 string))
! 	(value (match-string 3 string)))
      (mm-with-multibyte-buffer
        (insert value)
        (goto-char (point-min))
        (while (search-forward "%" nil t)
  	(insert
  	 (prog1
! 	     (string-to-number (buffer-substring (point) (+ (point) 2)) 16)
  	   (delete-region (1- (point)) (+ (point) 2)))))
        ;; Decode using the charset, if any.
        (unless (memq coding-system '(nil ascii))
--- 194,213 ----
    (string-match "\\`\\(?:\\([^']+\\)?'\\([^']+\\)?'\\)?\\(.+\\)" string)
    (let ((coding-system (mm-charset-to-coding-system (match-string 1 string)))
  	;;(language (match-string 2 string))
! 	(value (match-string 3 string))
! 	number)
      (mm-with-multibyte-buffer
        (insert value)
        (goto-char (point-min))
        (while (search-forward "%" nil t)
+ 	(setq number (string-to-number (buffer-substring (point) (+ (point) 2)) 16))
  	(insert
  	 (prog1
! 	   (if (>= emacs-major-version 23)
! 	       (if (> number #x7F)
! 		   (+ number #x3fff00) 
! 		 number) ;; why (unibyte-char-to-multibyte ch) does not work?
! 	     number)
  	   (delete-region (1- (point)) (+ (point) 2)))))
        ;; Decode using the charset, if any.
        (unless (memq coding-system '(nil ascii))
***************
*** 219,226 ****
  	(num -1)
  	;; Don't make lines exceeding 76 column.
  	(limit (- 74 (length param)))
! 	spacep encodep charsetp charset broken)
!     (with-temp-buffer
        (insert value)
        (goto-char (point-min))
        (while (not (eobp))
--- 225,232 ----
  	(num -1)
  	;; Don't make lines exceeding 76 column.
  	(limit (- 74 (length param)))
! 	spacep encodep charsetp charset broken following-char)
!     (mm-with-multibyte-buffer
        (insert value)
        (goto-char (point-min))
        (while (not (eobp))
***************
*** 245,260 ****
  	(goto-char (point-min))
  	(insert (symbol-name (or charset 'us-ascii)) "''")
  	(while (not (eobp))
! 	  (if (or (not (memq (following-char) ascii))
! 		  (memq (following-char) control)
! 		  (memq (following-char) tspecial)
! 		  (memq (following-char) special)
! 		  (eq (following-char) ? ))
  	      (progn
  		(when (>= (current-column) (1- limit))
  		  (insert ";\n")
  		  (setq broken t))
! 		(insert "%" (format "%02x" (following-char)))
  		(delete-char 1))
  	    (when (> (current-column) limit)
  	      (insert ";\n")
--- 251,270 ----
  	(goto-char (point-min))
  	(insert (symbol-name (or charset 'us-ascii)) "''")
  	(while (not (eobp))
! 	  (if (>= emacs-major-version 23)
! 	      (setq following-char
! 		    (multibyte-char-to-unibyte (following-char)))
! 	    (setq following-char (following-char)))
! 	  (if (or (not (memq following-char ascii))
! 		  (memq following-char control)
! 		  (memq following-char tspecial)
! 		  (memq following-char special)
! 		  (eq following-char ? ))
  	      (progn
  		(when (>= (current-column) (1- limit))
  		  (insert ";\n")
  		  (setq broken t))
! 		(insert "%" (format "%02x" following-char))
  		(delete-char 1))
  	    (when (> (current-column) limit)
  	      (insert ";\n")

-- 
Zhang Wei or Brep
<brep@smth.org>

Re: gnus doesn't encode non-ascii attachment filenames

[Kenichi Handa]

In article <87irriras8.fsf@emacsfans.org>, Zhang Wei <id.brep@gmail.com> writes:

[...]
> Sorry, false alarm. This bug is only with the emacs-unicode-2
> branch. No problem with emacs 22.

> The difference of internal character representation between emacs23
> and emacs22, makes gnus can't handle non-ascii attachment filename
> correctly.

> To generate a rfc2231 encoded filename such as "%d6%d0%ce%c4",
> function (following-char) returns "d6" for the first char of the
> filename "d6d0cec4" in emacs22, while in emacs23 this function returns
> "3fffd6".

Ah, I see.

> This is an ugly hack to solve this problem:

For Emacs 23, we can have a better/simpler fix.  But, could
someone tell me the maintenance policy of Gnus codes?
Should we modify codes in cvs HEAD so that it works also
with emacs-unicode-2 (perhaps with version check here and
there)?  Or, should we directly modify codes in cvs
emacs-unicode-2?

---
Kenichi Handa
handa@m17n.org

Re: gnus doesn't encode non-ascii attachment filenames

[Miles Bader]

Kenichi Handa <handa@m17n.org> writes:
> For Emacs 23, we can have a better/simpler fix.  But, could someone
> tell me the maintenance policy of Gnus codes?  Should we modify codes
> in cvs HEAD so that it works also with emacs-unicode-2 (perhaps with
> version check here and there)?  Or, should we directly modify codes in
> cvs emacs-unicode-2?

It seems to me better keep the code in both branches the same, if
possible.

Changes to Emacs 22 will be propagated to Gnus 5.10 and the Gnus
development branch (No Gnus); as people may wish to use the Gnus
development branch with Emacs 23, it would be a good thing to avoid
changes which are only in the Emacs 23 sources.

-Miles
-- 
Next to fried food, the South has suffered most from oratory.
  			-- Walter Hines Page

Gnus compatibility with Emacs 23 (unicode branch) (was: gnus doesn't encode non-ascii attachment filenames)

[Reiner Steib]

On Tue, Feb 14 2006, Miles Bader wrote:

> Kenichi Handa <handa@m17n.org> writes:
>> For Emacs 23, we can have a better/simpler fix.  But, could someone
>> tell me the maintenance policy of Gnus codes?  Should we modify codes
>> in cvs HEAD so that it works also with emacs-unicode-2 (perhaps with
>> version check here and there)?  

(Or feature check, if possible.)  But please don't break compatibility
with Emacs 21 and Emacs 20.7 [1].

>> Or, should we directly modify codes in cvs emacs-unicode-2?
>
> It seems to me better keep the code in both branches the same, if
> possible.
>
> Changes to Emacs 22 will be propagated to Gnus 5.10 and the Gnus
> development branch (No Gnus); as people may wish to use the Gnus
> development branch with Emacs 23, it would be a good thing to avoid
> changes which are only in the Emacs 23 sources.

I agree.  AFAICS, Dave Love already installed some changes to make
Gnus work with emacs-unicode-2 (often referred as "Emacs 22" [2] in
code comments).  But there are probably also some open issues:
E.g. "Wrong for Emacs 23" in `message-fix-before-sending'.

Bye, Reiner.

[1] We are currently discussing if we should drop Emacs 20.7
    compatibility also for the stable branch (Gnus 5.10, v5-10) as we
    have already done for the development version of Gnus (No Gnus).

[2] We should probably check out the before-renaming-to-emacs-22
    version, look for strings like "Emacs 22" and replace the comments
    with "Emacs 23 (unicode)" 
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus compatibility with Emacs 23 (unicode branch)

[Reiner Steib]

On Tue, Feb 14 2006, Reiner Steib wrote:

> AFAICS, Dave Love already installed some changes to make Gnus work
> with emacs-unicode-2 (often referred as "Emacs 22" [2] in code
> comments).  But there are probably also some open issues:
> E.g. "Wrong for Emacs 23" in `message-fix-before-sending'.
[...]
> [2] We should probably check out the before-renaming-to-emacs-22
>     version, look for strings like "Emacs 22" and replace the comments
>     with "Emacs 23 (unicode)" 

Done in Gnus' CVS.  (I looked at the Gnus 5.10.6 release files.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: gnus doesn't encode non-ascii attachment filenames

[Kenichi Handa]

I'm sorry for the late response on this matter.

In article <buo64ni392v.fsf@dhapc248.dev.necel.com>, Miles Bader <miles.bader@necel.com> writes:

> Kenichi Handa <handa@m17n.org> writes:
>> For Emacs 23, we can have a better/simpler fix.  But, could someone
>> tell me the maintenance policy of Gnus codes?  Should we modify codes
>> in cvs HEAD so that it works also with emacs-unicode-2 (perhaps with
>> version check here and there)?  Or, should we directly modify codes in
>> cvs emacs-unicode-2?

> It seems to me better keep the code in both branches the same, if
> possible.

> Changes to Emacs 22 will be propagated to Gnus 5.10 and the Gnus
> development branch (No Gnus); as people may wish to use the Gnus
> development branch with Emacs 23, it would be a good thing to avoid
> changes which are only in the Emacs 23 sources.

I've just installed the attached change in HEAD (it doesn't
check versions nor existence of a specific function).  But,
as I'm not using Gnus, could Gnus users please check if it
doesn't break anything?

To Emacs 23 users; the change will be propagated to Emacs 23
in a week or so.  If you can't wait for that, please apply
the patch by yourself and check if it fixes the current
problem.

---
Kenichi Handa
handa@m17n.org

2006-03-23  Kenichi Handa  <handa@m17n.org>

	* rfc2231.el (rfc2231-decode-encoded-string): Work on unibyte
	buffer and then decode the buffer text if necessary.
	(rfc2231-encode-string): Be sure to work on multibyte buffer at
	first, and after mm-encode-body, change the buffer to unibyte.
	
*** rfc2231.el	20 Feb 2006 09:51:27 +0900	1.4.14.11
--- rfc2231.el	22 Mar 2006 22:07:21 +0900	
***************
*** 227,233 ****
    (let ((coding-system (mm-charset-to-coding-system (match-string 2 string)))
  	;;(language (match-string 3 string))
  	(value (match-string 4 string)))
!     (mm-with-multibyte-buffer
        (insert value)
        (goto-char (point-min))
        (while (search-forward "%" nil t)
--- 227,233 ----
    (let ((coding-system (mm-charset-to-coding-system (match-string 2 string)))
  	;;(language (match-string 3 string))
  	(value (match-string 4 string)))
!     (mm-with-unibyte-buffer
        (insert value)
        (goto-char (point-min))
        (while (search-forward "%" nil t)
***************
*** 236,244 ****
  	     (string-to-number (buffer-substring (point) (+ (point) 2)) 16)
  	   (delete-region (1- (point)) (+ (point) 2)))))
        ;; Decode using the charset, if any.
!       (unless (memq coding-system '(nil ascii))
! 	(mm-decode-coding-region (point-min) (point-max) coding-system))
!       (buffer-string))))
  
  (defun rfc2231-encode-string (param value)
    "Return and PARAM=VALUE string encoded according to RFC2231.
--- 236,244 ----
  	     (string-to-number (buffer-substring (point) (+ (point) 2)) 16)
  	   (delete-region (1- (point)) (+ (point) 2)))))
        ;; Decode using the charset, if any.
!       (if (memq coding-system '(nil ascii))
! 	  (buffer-string)
! 	(mm-decode-coding-string (buffer-string) coding-system)))))
  
  (defun rfc2231-encode-string (param value)
    "Return and PARAM=VALUE string encoded according to RFC2231.
***************
*** 252,258 ****
  	;; Don't make lines exceeding 76 column.
  	(limit (- 74 (length param)))
  	spacep encodep charsetp charset broken)
!     (with-temp-buffer
        (insert value)
        (goto-char (point-min))
        (while (not (eobp))
--- 252,258 ----
  	;; Don't make lines exceeding 76 column.
  	(limit (- 74 (length param)))
  	spacep encodep charsetp charset broken)
!     (mm-with-multibyte-buffer
        (insert value)
        (goto-char (point-min))
        (while (not (eobp))
***************
*** 268,273 ****
--- 268,274 ----
  	(forward-char 1))
        (when charsetp
  	(setq charset (mm-encode-body)))
+       (set-buffer-multibyte nil)
        (cond
         ((or encodep charsetp
  	    (progn