bug#21471: 25.0.50; REGRESSION: bug report with text from Info has spurious escape chars

bug#21471: 25.0.50; REGRESSION: bug report with text from Info has spurious escape chars

[Drew Adams]

Dunno whether this regression is perhaps limited to MS Windows or
mail client Outlook.  I am using both.

emacs -Q
C-h i ; Choose Elisp
g font and TAB RET

Select the last paragraph about parameter `alpha' and copy it
using `M-w'.

M-x report-emacs-bug

Yank the copied text into the bug-report buffer.

Hit `C-c C-c'.  Choose to use an external mail client to
send the report.

When the mail client message window opens, select the
boiler-plate text:

*** E-Mail body has been placed on clipboard, please paste it here! ***

Paste the copied report-message text, replacing the selected
boiler-plate text.

This is what you get in the email message, to send as the
bug report:

     The =A1=AEalpha=A1=AF frame parameter can also be a cons cell =A1=AE(=
=A1=AEactive=A1=AF .
     =A1=AEinactive=A1=AF)=A1=AF, where =A1=AEactive=A1=AF is the opacity o=
f the frame when it is
     selected, and =A1=AEinactive=A1=AF is the opacity when it is not selec=
ted.

Curly quotes in the Info buffer have been replaced with
escape sequences such as =A1=AE.  This is in spite of the
fact that the mail-client window into which the text is
pasted is perfectly capable of handling Unicode chars such
as curly quotes.

(Just one more undesirable and unforeseen consequence of the
curly-quote-mania virus, I guess.  Time for yet another
hack-job workaround?)


In GNU Emacs 25.0.50.1 (i686-pc-mingw32)
 of 2015-09-05
Repository revision: 2330ca33a97867f2ea1123bcf7bfe5cfcc030b36
Windowing system distributor 'Microsoft Corp.', version 6.1.7601
Configured using:
 'configure --host=3Di686-pc-mingw32 --enable-checking=3Dyes,glyphs'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Info

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Composing main Info directory...done
Mark set [2 times]
Type C-x 1 to delete the help window, C-M-v to scroll help.
Char: =A1=AE (8216, #o20030, #x2018, file ...) point=3D1897434 of 3604579 (=
53%) <1894203-1899121> column=3D9

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
help-fns mail-prsvr mail-utils pp wid-edit descr-text help-mode
cl-loaddefs pcase cl-lib info easymenu dired time-date mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 8 201120 12359)
 (symbols 32 29890 0)
 (miscs 32 66 267)
 (strings 16 33256 8135)
 (string-bytes 1 757007)
 (vectors 8 14837)
 (vector-slots 4 840990 7268)
 (floats 8 133 188)
 (intervals 28 34710 2066)
 (buffers 516 15))

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Paul Eggert]

This bug is not a regression, as I get similar behavior with Emacs 24.5 on 
Ubuntu 15.04 (email client Thunderbird 38.2.0) as follows:

emacs -Q
C-h i m elisp RET
C-x h M-w
M-x report-emacs-bug RET
test subject RET
C-y
C-c C-c yes RET
mail client RET

The mail that's sent includes strings like this:

Copyright =C2=A9 1990=E2=80=931996, 1998=E2=80=932014 Free Software Foun=
dation, Inc.

even though it's marked "Content-type: text/plain; charset=utf-8; 
format=flowed".  This problem occurs with all non-ASCII characters, not merely 
with curved quotes.

bug#21471: 25.0.50; REGRESSION: bug report with text from Info has spurious escape chars

[Eli Zaretskii]

> Date: Sun, 13 Sep 2015 08:07:42 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> Dunno whether this regression is perhaps limited to MS Windows or
> mail client Outlook.

It isn't.  You can paste into Notepad, or even another Emacs session.

> This is what you get in the email message, to send as the
> bug report:
> 
>      The =A1=AEalpha=A1=AF frame parameter can also be a cons cell =A1=AE(=
> =A1=AEactive=A1=AF .
>      =A1=AEinactive=A1=AF)=A1=AF, where =A1=AEactive=A1=AF is the opacity o=
> f the frame when it is
>      selected, and =A1=AEinactive=A1=AF is the opacity when it is not selec=
> ted.
> 
> Curly quotes in the Info buffer have been replaced with
> escape sequences such as =A1=AE.  This is in spite of the
> fact that the mail-client window into which the text is
> pasted is perfectly capable of handling Unicode chars such
> as curly quotes.

This is called quoted-printable representation of non-ASCII
characters, and is a feature.  I guess whoever wrote that didn't want
to trust mailers on user systems to be configured for non-ASCII.

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Drew Adams]

> This bug is not a regression, as I get similar behavior with Emacs 24.5 on
> Ubuntu 15.04 (email client Thunderbird 38.2.0) as follows:
...
> The mail that's sent includes strings like this:
> 
> Copyright =C2=A9 1990=E2=80=931996, 1998=E2=80=932014 Free Software Foun=
> dation, Inc.
> 
> even though it's marked "Content-type: text/plain; charset=utf-8;
> format=flowed".  This problem occurs with all non-ASCII characters, not
> merely with curved quotes.

OK, so it's not a regression, in that Unicode chars have
apparently long been copied and pasted incorrectly in this
context.

But Unicode chars were not used all over the place previously,
which makes it a regression of sorts, in observed behavior.
This will bite lots more users a lot more, even if the problem
was potentially present previously as well.

It's a bad bug, interfering considerably with usability,
regardless of whether we want to call it a regression.

bug#21471: 25.0.50; REGRESSION: bug report with text from Info has spurious escape chars

[Drew Adams]

> This is called quoted-printable representation of non-ASCII
> characters, and is a feature.  I guess whoever wrote that didn't want
> to trust mailers on user systems to be configured for non-ASCII.

At the very least, if Emacs wants to consider this a feature,
it should be a user option.  Users should be include Unicode
chars normally in bug reports.  Especially now that Emacs
supports Unicode so well and Unicode is used throughout the
manuals.

I'd argue that we should have a user option, and that the
default behavior should be to copy+paste normally.  Anyone
who really needs this "feature" can customize the option
accordingly.

Is there some reason not to proceed that way?  Am I missing
something?

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Eli Zaretskii]

> Date: Sun, 13 Sep 2015 13:55:42 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 21471@debbugs.gnu.org
> 
> > This bug is not a regression, as I get similar behavior with Emacs 24.5 on
> > Ubuntu 15.04 (email client Thunderbird 38.2.0) as follows:
> ...
> > The mail that's sent includes strings like this:
> > 
> > Copyright =C2=A9 1990=E2=80=931996, 1998=E2=80=932014 Free Software Foun=
> > dation, Inc.
> > 
> > even though it's marked "Content-type: text/plain; charset=utf-8;
> > format=flowed".  This problem occurs with all non-ASCII characters, not
> > merely with curved quotes.
> 
> OK, so it's not a regression, in that Unicode chars have
> apparently long been copied and pasted incorrectly in this
> context.

Not "Unicode", any non-ASCII characters.

> But Unicode chars were not used all over the place previously,
> which makes it a regression of sorts, in observed behavior.
> This will bite lots more users a lot more, even if the problem
> was potentially present previously as well.
> 
> It's a bad bug, interfering considerably with usability,
> regardless of whether we want to call it a regression.

You can always copy/paste manually, replacing the text that Emacs put
in the clipboard for you, if you care.

Or you can make the text an attachment.

And please recall that the instructions for writing the bug report
explicitly asked for avoiding non-ASCII characters.

bug#21471: 25.0.50; REGRESSION: bug report with text from Info has spurious escape chars

[Eli Zaretskii]

> Date: Sun, 13 Sep 2015 13:58:57 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 21471@debbugs.gnu.org
> 
> Is there some reason not to proceed that way?  Am I missing
> something?

Reliability.  The command to report a bug must be 100% reliable, and
it must work out of the box in "emacs -Q".

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Drew Adams]

> > But Unicode chars were not used all over the place previously,
> > which makes it a regression of sorts, in observed behavior.
> > This will bite lots more users a lot more, even if the problem
> > was potentially present previously as well.
> >
> > It's a bad bug, interfering considerably with usability,
> > regardless of whether we want to call it a regression.
> 
> You can always copy/paste manually, replacing the text that Emacs put
> in the clipboard for you, if you care.
> 
> Or you can make the text an attachment.
> 
> And please recall that the instructions for writing the bug report
> explicitly asked for avoiding non-ASCII characters.

Excuse me?  That sounds like a complete cop-out, to me.

But if you're fine with pasted text from manuals not being
readable, who am I to say that this is a problem?

Telling users to avoid non-ASCII characters flies in the
face of spreading non-ASCII characters all over the manuals.

Do you honestly expect users to strip out all of the curly
quotes or replace them all with ASCII quotes, just to be
able to cite text in the manuals?

bug#21471: 25.0.50; REGRESSION: bug report with text from Info has spurious escape chars

[Drew Adams]

> > Is there some reason not to proceed that way?  Am I missing
> > something?
> 
> Reliability.  The command to report a bug must be 100% reliable, and
> it must work out of the box in "emacs -Q".

We already let users configure how bug reporting interacts
with their mail environment.  Why not provide a user option
that handles this correctly?

Even in `emacs -Q', I should be able to copy+paste non-ASCII
text, assuming my mail client can handle it.

I'm sorry to say it, but your response about this really
sounds like a cop-out.  This might not be the most urgent
bug, but I cannot see that this is something that should
be foisted on the user as being her problem (just don't
use `C-c C-c' and instead create a mail-client message
by hand and paste copied non-ASCII text into it manually).

That is not a reasonable answer, IMHO.  But whatever.

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

It is a longstanding double-encoding bug in mailclient-encode-string-as-url, and 
I installed the attached attempt to fix it.  Please give this a try.  I am 
boldly marking the bug as fixed.

I expect our more-extensive use of curved quotes to shake out other bugs like 
this in Emacs, and it is a good thing to fix them.

[-- Attachment #2: 0001-Don-t-double-encode-non-ASCII-for-mail-client.patch --]
[-- Type: text/plain, Size: 4369 bytes --]

From d76d8aa460927924ab35b119c49be09f8aa84dd3 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 14 Sep 2015 09:31:23 -0700
Subject: [PATCH] =?UTF-8?q?Don=E2=80=99t=20double-encode=20non-ASCII=20for?=
 =?UTF-8?q?=20mail=20client?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/mail/mailclient.el (mailclient-encode-string-as-url):
Use RFC 6068’s list of unreserved characters.
(mailclient-send-it): When encoding the body as a URL,
first decode it as per Content-Type: and Content-Transfer-Encoding:,
as URLs must use percent-encoded UTF-8 (Bug#21471).

* doc/misc/url.texi (mailto): Update RFC number.
---
 doc/misc/url.texi       |  2 +-
 lisp/mail/mailclient.el | 42 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 5 deletions(-)

diff --git a/doc/misc/url.texi b/doc/misc/url.texi
index d94b19f..95fe5ea 100644
--- a/doc/misc/url.texi
+++ b/doc/misc/url.texi
@@ -593,7 +593,7 @@ sending a message to @samp{foo@@bar.com}.  The ``retrieval method''
 for such URLs is to open a mail composition buffer in which the
 appropriate content (e.g., the recipient address) has been filled in.
 
-  As defined in RFC 2368, a @code{mailto} URL has the form
+  As defined in RFC 6068, a @code{mailto} URL can have the form
 
 @example
 @samp{mailto:@var{mailbox}[?@var{header}=@var{contents}[&@var{header}=@var{contents}]]}
diff --git a/lisp/mail/mailclient.el b/lisp/mail/mailclient.el
index 31da6ee..bef2038 100644
--- a/lisp/mail/mailclient.el
+++ b/lisp/mail/mailclient.el
@@ -62,10 +62,9 @@ supported.  Defaults to non-nil on Windows, nil otherwise."
 	 (mapcar
 	  (lambda (char)
 	    (cond
-	     ((eq char ?\x20) "%20")   ;; space
 	     ((eq char ?\n) "%0D%0A")  ;; newline
-	     ((string-match "[-a-zA-Z0-9_:/.@]" (char-to-string char))
-	      (char-to-string char))   ;; printable
+	     ((string-match "[-a-zA-Z0-9._~]" (char-to-string char))
+	      (char-to-string char))   ;; unreserved as per RFC 6068
 	     (t                        ;; everything else
 	      (format "%%%02x" char))))	;; escape
 	  ;; Convert string to list of chars
@@ -125,6 +124,13 @@ The mail client is taken to be the handler of mailto URLs."
 		      (< (point) delimline))
 	    (replace-match "\n"))
 	  (let ((case-fold-search t)
+		(mime-charset-pattern
+		 (concat
+		  "^content-type:[ \t]*text/plain;"
+		  "\\(?:[ \t\n]*\\(?:format\\|delsp\\)=\"?[-a-z0-9]+\"?;\\)*"
+		  "[ \t\n]*charset=\"?\\([^ \t\n\";]+\\)\"?"))
+		coding-system
+		character-coding
 		;; Use the external browser function to send the
 		;; message.
 		(browse-url-mailto-function nil))
@@ -135,6 +141,15 @@ The mail client is taken to be the handler of mailto URLs."
 	     (concat
 	      (save-excursion
 		(narrow-to-region (point-min) delimline)
+		(goto-char (point-min))
+		(setq coding-system
+		      (if (re-search-forward mime-charset-pattern nil t)
+			  (coding-system-from-name (match-string 1))
+			'undecided))
+		(setq character-coding
+		      (mail-fetch-field "content-transfer-encoding"))
+		(when character-coding
+		  (setq character-coding (downcase character-coding)))
 		(concat
 		 "mailto:"
 		 ;; some of the headers according to RFC822
@@ -171,7 +186,26 @@ The mail client is taken to be the handler of mailto URLs."
 		       "*** E-Mail body has been placed on clipboard, "
 		       "please paste it here! ***"))
 		  ;; else
-		  (buffer-substring (+ 1 delimline) (point-max))))))))))))
+		  (let ((body (buffer-substring (+ 1 delimline) (point-max))))
+		    (if (null character-coding)
+			body
+		      ;; mailto: requires UTF-8 and cannot deal with
+		      ;; Content-Transfer-Encoding or Content-Type.
+		      ;; FIXME: There is a lot of code duplication here
+		      ;; with rmail.el.
+		      (erase-buffer)
+		      (set-buffer-multibyte nil)
+		      (insert body)
+		      (cond
+		       ((string= character-coding "quoted-printable")
+			(mail-unquote-printable-region (point-min) (point-max)
+						       nil nil 'unibyte))
+		       ((string= character-coding "base64")
+			(base64-decode-region (point-min) (point-max)))
+		       (t (error "unsupported Content-Transfer-Encoding: %s"
+				 character-coding)))
+		      (decode-coding-region (point-min) (point-max)
+					    coding-system t)))))))))))))
 
 (provide 'mailclient)
 
-- 
2.1.4

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Eli Zaretskii]

> Cc: 21471-done@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 14 Sep 2015 09:38:29 -0700
> 
> It is a longstanding double-encoding bug in mailclient-encode-string-as-url, and 
> I installed the attached attempt to fix it.  Please give this a try.  I am 
> boldly marking the bug as fixed.

What was this fix supposed to change?  I still see quoted-printable
encoding of non-ASCII characters in the text pasted from the clipboard
into a mail client.

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

Eli Zaretskii wrote:
> What was this fix supposed to change?

It was supposed to fix the two scenarios mentioned (one mine, the other Drew's). 
  I didn't know the clipboard didn't work.  Fixed, I hope, with the attached patch.

[-- Attachment #2: 0001-Don-t-double-encode-non-ASCII-mail-clipboard.patch --]
[-- Type: text/plain, Size: 3242 bytes --]

From 59f0b53c1092b1528d34d9c4edebcf034be628a4 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 14 Sep 2015 14:09:24 -0700
Subject: [PATCH] =?UTF-8?q?Don=E2=80=99t=20double-encode=20non-ASCII=20mai?=
 =?UTF-8?q?l=20clipboard?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/mail/mailclient.el (mailclient-send-it):
Also fix the case when mailclient-place-body-on-clipboard-flag
is non-nil.  Problem reported by Eli Zaretskii (Bug#21471#37).
---
 lisp/mail/mailclient.el | 56 ++++++++++++++++++++++---------------------------
 1 file changed, 25 insertions(+), 31 deletions(-)

diff --git a/lisp/mail/mailclient.el b/lisp/mail/mailclient.el
index bef2038..2fb944b 100644
--- a/lisp/mail/mailclient.el
+++ b/lisp/mail/mailclient.el
@@ -175,37 +175,31 @@ The mail client is taken to be the handler of mailto URLs."
 			       (mailclient-encode-string-as-url subj))
 		     ""))))
 	      ;; body
-	      (concat
-	       (mailclient-url-delim) "body="
-	       (mailclient-encode-string-as-url
-		(if mailclient-place-body-on-clipboard-flag
-		    (progn
-		      (clipboard-kill-ring-save
-		       (+ 1 delimline) (point-max))
-		      (concat
-		       "*** E-Mail body has been placed on clipboard, "
-		       "please paste it here! ***"))
-		  ;; else
-		  (let ((body (buffer-substring (+ 1 delimline) (point-max))))
-		    (if (null character-coding)
-			body
-		      ;; mailto: requires UTF-8 and cannot deal with
-		      ;; Content-Transfer-Encoding or Content-Type.
-		      ;; FIXME: There is a lot of code duplication here
-		      ;; with rmail.el.
-		      (erase-buffer)
-		      (set-buffer-multibyte nil)
-		      (insert body)
-		      (cond
-		       ((string= character-coding "quoted-printable")
-			(mail-unquote-printable-region (point-min) (point-max)
-						       nil nil 'unibyte))
-		       ((string= character-coding "base64")
-			(base64-decode-region (point-min) (point-max)))
-		       (t (error "unsupported Content-Transfer-Encoding: %s"
-				 character-coding)))
-		      (decode-coding-region (point-min) (point-max)
-					    coding-system t)))))))))))))
+	      (mailclient-url-delim) "body="
+	      (progn
+		(delete-region (point-min) delimline)
+		(unless (null character-coding)
+		  ;; mailto: and clipboard need UTF-8 and cannot deal with
+		  ;; Content-Transfer-Encoding or Content-Type.
+		  ;; FIXME: There is code duplication here with rmail.el.
+		  (set-buffer-multibyte nil)
+		  (cond
+		   ((string= character-coding "base64")
+		    (base64-decode-region (point-min) (point-max)))
+		   ((string= character-coding "quoted-printable")
+		    (mail-unquote-printable-region (point-min) (point-max)
+						   nil nil t))
+		   (t (error "unsupported Content-Transfer-Encoding: %s"
+			     character-coding)))
+		  (decode-coding-region (point-min) (point-max) coding-system))
+		(mailclient-encode-string-as-url
+		 (if mailclient-place-body-on-clipboard-flag
+		     (progn
+		       (clipboard-kill-ring-save (point-min) (point-max))
+		       (concat
+			"*** E-Mail body has been placed on clipboard, "
+			"please paste it here! ***"))
+		   (buffer-string)))))))))))
 
 (provide 'mailclient)
 
-- 
2.1.4

bug#21471: 25.0.50; bug report with text from Info has spurious escape chars

[Eli Zaretskii]

> Cc: drew.adams@oracle.com, 21471@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 14 Sep 2015 14:10:51 -0700
> 
>   I didn't know the clipboard didn't work.  Fixed, I hope, with the attached patch.

Works here, thanks.