[PATCH v1 0/2] Fixes for some 0.22 problems reported on the mailing list.

[PATCH v1 0/2] Fixes for some 0.22 problems reported on the mailing list.

[David Edmondson]

Fixes for some 0.22 problems reported on the mailing list.


David Edmondson (2):
  emacs: Observe the charset of MIME parts when reading them.
  emacs: Tell `message-mode' that outgoing messages are email.

 emacs/notmuch-lib.el | 16 +++++++++++++++-
 emacs/notmuch-mua.el |  5 ++++-
 2 files changed, 19 insertions(+), 2 deletions(-)

-- 
2.7.1

[PATCH v1 1/2] emacs: Observe the charset of MIME parts when reading them.

[David Edmondson]

`notmuch--get-bodypart-raw' previously assumed that all non-binary MIME
parts could be successfully read by assuming that they were UTF-8
encoded. This was demonstrated to be wrong, specifically when a part was
marked as ISO8859-1 and included accented characters (which were
incorrectly rendered as a result).

Rather than assuming UTF-8, attempt to use the part's declared charset
when reading it, falling back to US-ASCII if the declared charset is
unknown, unsupported or invalid.
---
 emacs/notmuch-lib.el | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 78978ee..f05ded6 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -23,6 +23,7 @@
 
 ;;; Code:
 
+(require 'mm-util)
 (require 'mm-view)
 (require 'mm-decode)
 (require 'cl)
@@ -572,7 +573,20 @@ the given type."
 				   ,@(when process-crypto '("--decrypt"))
 				   ,(notmuch-id-to-query (plist-get msg :id))))
 			   (coding-system-for-read
-			    (if binaryp 'no-conversion 'utf-8)))
+			    (if binaryp 'no-conversion
+			      (let ((coding-system (mm-charset-to-coding-system
+						    (plist-get part :content-charset))))
+				;; Sadly,
+				;; `mm-charset-to-coding-system' seems
+				;; to return things that are not
+				;; considered acceptable values for
+				;; `coding-system-for-read'.
+				(if (coding-system-p coding-system)
+				    coding-system
+				  ;; RFC 2047 says that the default
+				  ;; charset is US-ASCII. RFC6657
+				  ;; complicates this somewhat.
+				  'us-ascii)))))
 		       (apply #'call-process notmuch-command nil '(t nil) nil args)
 		       (buffer-string))))))
     (when (and cache data)
-- 
2.7.1

[PATCH v1 2/2] emacs: Tell `message-mode' that outgoing messages are email.

[David Edmondson]

When composing messages (including replies, etc.), indicate to
`message-mode' definitively that the message is email (as opposed to
Usenet news) rather than having it attempt to determine this for itself.

This causes `message-mode' to observe such variables as
`message-default-mail-headers', which previously happened haphazardly.
---
 emacs/notmuch-mua.el | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 0445975..399e138 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -338,7 +338,10 @@ modified. This function is notmuch addaptation of
 	  ;; We need to convert any string input, eg from rmail-start-mail.
 	  (dolist (h other-headers other-headers)
 	    (if (stringp (car h)) (setcar h (intern (capitalize (car h))))))))
-	(args (list yank-action send-actions)))
+	(args (list yank-action send-actions))
+	;; Cause `message-setup-1' to do things relevant for mail,
+	;; such as observe `message-default-mail-headers'.
+	(message-this-is-mail t))
     ;; message-setup-1 in Emacs 23 does not accept return-action
     ;; argument. Pass it only if it is supplied by the caller. This
     ;; will never be the case when we're called by `compose-mail' in
-- 
2.7.1

Re: [PATCH v1 0/2] Fixes for some 0.22 problems reported on the mailing list.

[Tomi Ollila]

On Sat, Apr 30 2016, David Edmondson <dme@dme.org> wrote:

> Fixes for some 0.22 problems reported on the mailing list.

LGTM. applies, does not seem to break things and renders
right at least one 
  Content-Type: text/html; charset=iso-8859-1
  Content-Transfer-Encoding: quoted-printable
message

>
>
> David Edmondson (2):
>   emacs: Observe the charset of MIME parts when reading them.
>   emacs: Tell `message-mode' that outgoing messages are email.
>
>  emacs/notmuch-lib.el | 16 +++++++++++++++-
>  emacs/notmuch-mua.el |  5 ++++-
>  2 files changed, 19 insertions(+), 2 deletions(-)
>
> -- 
> 2.7.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v1 0/2] Fixes for some 0.22 problems reported on the mailing list.

[David Bremner]

David Edmondson <dme@dme.org> writes:

> Fixes for some 0.22 problems reported on the mailing list.
>

I've merged these to release and master. I guess we can think about
0.22.1 with these, and maybe some similar small fixes.

d

Re: [PATCH v1 1/2] emacs: Observe the charset of MIME parts when reading them.

[Mark Walters]

On Sat, 30 Apr 2016, David Edmondson <dme@dme.org> wrote:
> `notmuch--get-bodypart-raw' previously assumed that all non-binary MIME
> parts could be successfully read by assuming that they were UTF-8
> encoded. This was demonstrated to be wrong, specifically when a part was
> marked as ISO8859-1 and included accented characters (which were
> incorrectly rendered as a result).
>
> Rather than assuming UTF-8, attempt to use the part's declared charset
> when reading it, falling back to US-ASCII if the declared charset is
> unknown, unsupported or invalid.

As this seemed hard to test (if I understand the bug correctly it didn't
show up on my test of the entire of the entire performance corpus -- of
course my testing could have been wrong) would it be possible to add a test
for it?

Best wishes

Mark


> ---
>  emacs/notmuch-lib.el | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> index 78978ee..f05ded6 100644
> --- a/emacs/notmuch-lib.el
> +++ b/emacs/notmuch-lib.el
> @@ -23,6 +23,7 @@
>  
>  ;;; Code:
>  
> +(require 'mm-util)
>  (require 'mm-view)
>  (require 'mm-decode)
>  (require 'cl)
> @@ -572,7 +573,20 @@ the given type."
>  				   ,@(when process-crypto '("--decrypt"))
>  				   ,(notmuch-id-to-query (plist-get msg :id))))
>  			   (coding-system-for-read
> -			    (if binaryp 'no-conversion 'utf-8)))
> +			    (if binaryp 'no-conversion
> +			      (let ((coding-system (mm-charset-to-coding-system
> +						    (plist-get part :content-charset))))
> +				;; Sadly,
> +				;; `mm-charset-to-coding-system' seems
> +				;; to return things that are not
> +				;; considered acceptable values for
> +				;; `coding-system-for-read'.
> +				(if (coding-system-p coding-system)
> +				    coding-system
> +				  ;; RFC 2047 says that the default
> +				  ;; charset is US-ASCII. RFC6657
> +				  ;; complicates this somewhat.
> +				  'us-ascii)))))
>  		       (apply #'call-process notmuch-command nil '(t nil) nil args)
>  		       (buffer-string))))))
>      (when (and cache data)
> -- 
> 2.7.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v1 1/2] emacs: Observe the charset of MIME parts when reading them.

[David Edmondson]

On Mon, May 02 2016, Mark Walters wrote:
> On Sat, 30 Apr 2016, David Edmondson <dme@dme.org> wrote:
>> `notmuch--get-bodypart-raw' previously assumed that all non-binary MIME
>> parts could be successfully read by assuming that they were UTF-8
>> encoded. This was demonstrated to be wrong, specifically when a part was
>> marked as ISO8859-1 and included accented characters (which were
>> incorrectly rendered as a result).
>>
>> Rather than assuming UTF-8, attempt to use the part's declared charset
>> when reading it, falling back to US-ASCII if the declared charset is
>> unknown, unsupported or invalid.
>
> As this seemed hard to test (if I understand the bug correctly it didn't
> show up on my test of the entire of the entire performance corpus -- of
> course my testing could have been wrong) would it be possible to add a test
> for it?

I agree that we should have a test, and I will make one.

(/me lives in fear of the day when bremner introduces a "no fixes
without tests" policy...)