* [PATCH] Output unmodified Content-Type header value for JSON format. @ 2011-11-18 23:45 Dmitry Kurochkin 2011-11-19 1:58 ` Jameson Graef Rollins 2011-11-19 4:18 ` [PATCH v2] " Dmitry Kurochkin 0 siblings, 2 replies; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-18 23:45 UTC (permalink / raw) To: notmuch Before the change, notmuch used g_mime_content_type_to_string(3) function to output Content-Type header value. Turns out it outputs only "type/subtype" part and ignores all parameters. Also, if there is no Content-Type header, default "text/plain" value is used. JSON is supposed to be a "low-level" structured format and should not add missing values or throw away information. The patch changes notmuch show to use unmodified Content-Type value for JSON format. Also, no default value is added if the header is missing. Corresponding changes to Emacs UI are made to handle full Content-Type header values. The header is parsed using MIME `mail-header-parse-content-type' function. All message part rendering functions have access to full Content-Type value. In particular, this is important for `notmuch-show-mm-display-part-inline' which uses `mm-display-part' to display parts that notmuch-show does not handle. Expected results for the tests are updated accordingly. --- emacs/notmuch-show.el | 28 ++++++++++++++++++---------- notmuch-show.c | 14 ++++++++++++-- test/crypto | 23 ++++++++--------------- test/json | 6 +++--- test/maildir-sync | 1 - test/multipart | 36 ++++++++++++++++++------------------ 6 files changed, 59 insertions(+), 49 deletions(-) diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el index d5c95d8..d2c2fa3 100644 --- a/emacs/notmuch-show.el +++ b/emacs/notmuch-show.el @@ -261,120 +261,120 @@ message at DEPTH in the current thread." (if (and header-value (not (string-equal "" header-value))) (notmuch-show-insert-header header header-value)))) notmuch-message-headers) (save-excursion (save-restriction (narrow-to-region start (point-max)) (run-hooks 'notmuch-show-markup-headers-hook))))) (define-button-type 'notmuch-show-part-button-type 'action 'notmuch-show-part-button-action 'follow-link t 'face 'message-mml) (defun notmuch-show-insert-part-header (nth content-type declared-type &optional name comment) (let ((button)) (setq button (insert-button (concat "[ " (if name (concat name ": ") "") - declared-type - (if (not (string-equal declared-type content-type)) - (concat " (as " content-type ")") + (car declared-type) + (if (not (string-equal (car declared-type) (car content-type))) + (concat " (as " (car content-type) ")") "") (or comment "") " ]") :type 'notmuch-show-part-button-type :notmuch-part nth :notmuch-filename name)) (insert "\n") ;; return button button)) ;; Functions handling particular MIME parts. (defun notmuch-show-save-part (message-id nth &optional filename) (let ((process-crypto notmuch-show-process-crypto)) (with-temp-buffer (setq notmuch-show-process-crypto process-crypto) ;; Always acquires the part via `notmuch part', even if it is ;; available in the JSON output. (insert (notmuch-show-get-bodypart-internal message-id nth)) (let ((file (read-file-name "Filename to save as: " (or mailcap-download-directory "~/") nil nil filename))) ;; Don't re-compress .gz & al. Arguably we should make ;; `file-name-handler-alist' nil, but that would chop ;; ange-ftp, which is reasonable to use here. (mm-write-region (point-min) (point-max) file nil nil nil 'no-conversion t))))) (defun notmuch-show-mm-display-part-inline (msg part content-type content) "Use the mm-decode/mm-view functions to display a part in the current buffer, if possible." (let ((display-buffer (current-buffer))) (with-temp-buffer (insert content) - (let ((handle (mm-make-handle (current-buffer) (list content-type)))) + (let ((handle (mm-make-handle (current-buffer) content-type))) (set-buffer display-buffer) (if (and (mm-inlinable-p handle) (mm-inlined-p handle)) (progn (mm-display-part handle) t) nil))))) (defvar notmuch-show-multipart/alternative-discouraged '( ;; Avoid HTML parts. "text/html" ;; multipart/related usually contain a text/html part and some associated graphics. "multipart/related" )) (defun notmuch-show-multipart/*-to-list (part) - (mapcar '(lambda (inner-part) (plist-get inner-part :content-type)) + (mapcar '(lambda (inner-part) (car (notmuch-show-get-content-type inner-part))) (plist-get part :content))) (defun notmuch-show-multipart/alternative-choose (types) ;; Based on `mm-preferred-alternative-precedence'. (let ((seq types)) (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged)) (dolist (elem (copy-sequence seq)) (when (string-match pref elem) (setq seq (nconc (delete elem seq) (list elem)))))) seq)) (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type) (notmuch-show-insert-part-header nth declared-type content-type nil) (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part)))) (inner-parts (plist-get part :content)) (start (point))) ;; This inserts all parts of the chosen type rather than just one, ;; but it's not clear that this is the wrong thing to do - which ;; should be chosen if there are more than one that match? (mapc (lambda (inner-part) - (let ((inner-type (plist-get inner-part :content-type))) + (let ((inner-type (notmuch-show-get-content-type inner-part))) (if (or notmuch-show-all-multipart/alternative-parts - (string= chosen-type inner-type)) + (string= chosen-type (car inner-type))) (notmuch-show-insert-bodypart msg inner-part depth) (notmuch-show-insert-part-header (plist-get inner-part :id) inner-type inner-type nil " (not shown)")))) inner-parts) (when notmuch-show-indent-multipart (indent-rigidly start (point) 1))) t) (defun notmuch-show-setup-w3m () "Instruct w3m how to retrieve content from a \"related\" part of a message." (interactive) (if (boundp 'w3m-cid-retrieve-function-alist) (unless (assq 'notmuch-show-mode w3m-cid-retrieve-function-alist) (push (cons 'notmuch-show-mode 'notmuch-show-w3m-cid-retrieve) w3m-cid-retrieve-function-alist))) (setq mm-inline-text-html-with-images t)) (defvar w3m-current-buffer) ;; From `w3m.el'. (defvar notmuch-show-w3m-cid-store nil) (make-variable-buffer-local 'notmuch-show-w3m-cid-store) @@ -557,41 +557,42 @@ current buffer, if possible." (set-buffer (get-file-buffer file)) (setq result (buffer-substring (point-min) (point-max))) (set-buffer-modified-p nil) (kill-buffer (current-buffer)) (delete-file file) result))) t) (defun notmuch-show-insert-part-application/octet-stream (msg part content-type nth depth declared-type) ;; If we can deduce a MIME type from the filename of the attachment, ;; do so and pass it on to the handler for that type. (if (plist-get part :filename) (let ((extension (file-name-extension (plist-get part :filename))) mime-type) (if extension (progn (mailcap-parse-mimetypes) (setq mime-type (mailcap-extension-to-mime extension)) (if (and mime-type (not (string-equal mime-type "application/octet-stream"))) - (notmuch-show-insert-bodypart-internal msg part mime-type nth depth content-type) + (notmuch-show-insert-bodypart-internal msg part (list mime-type) + nth depth content-type) nil)) nil)))) (defun notmuch-show-insert-part-application/* (msg part content-type nth depth declared-type ) ;; do not render random "application" parts (notmuch-show-insert-part-header nth content-type declared-type (plist-get part :filename))) (defun notmuch-show-insert-part-*/* (msg part content-type nth depth declared-type) ;; This handler _must_ succeed - it is the handler of last resort. (notmuch-show-insert-part-header nth content-type declared-type (plist-get part :filename)) (let ((content (notmuch-show-get-bodypart-content msg part nth))) (if content (notmuch-show-mm-display-part-inline msg part content-type content))) t) ;; Functions for determining how to handle MIME parts. (defun notmuch-show-split-content-type (content-type) (split-string content-type "/")) @@ -618,51 +619,51 @@ current buffer, if possible." (defun notmuch-show-get-bodypart-internal (message-id part-number) (let ((args '("show" "--format=raw")) (part-arg (format "--part=%s" part-number))) (setq args (append args (list part-arg))) (if notmuch-show-process-crypto (setq args (append args '("--decrypt")))) (setq args (append args (list message-id))) (with-temp-buffer (let ((coding-system-for-read 'no-conversion)) (progn (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args)) (buffer-string)))))) (defun notmuch-show-get-bodypart-content (msg part nth) (or (plist-get part :content) (notmuch-show-get-bodypart-internal (concat "id:" (plist-get msg :id)) nth))) ;; \f (defun notmuch-show-insert-bodypart-internal (msg part content-type nth depth declared-type) - (let ((handlers (notmuch-show-handlers-for content-type))) + (let ((handlers (notmuch-show-handlers-for (car content-type)))) ;; Run the content handlers until one of them returns a non-nil ;; value. (while (and handlers (not (funcall (car handlers) msg part content-type nth depth declared-type))) (setq handlers (cdr handlers)))) t) (defun notmuch-show-insert-bodypart (msg part depth) "Insert the body part PART at depth DEPTH in the current thread." - (let ((content-type (downcase (plist-get part :content-type))) + (let ((content-type (notmuch-show-get-content-type part)) (nth (plist-get part :id))) (notmuch-show-insert-bodypart-internal msg part content-type nth depth content-type)) ;; Some of the body part handlers leave point somewhere up in the ;; part, so we make sure that we're down at the end. (goto-char (point-max)) ;; Ensure that the part ends with a carriage return. (if (not (bolp)) (insert "\n"))) (defun notmuch-show-insert-body (msg body depth) "Insert the body BODY at depth DEPTH in the current thread." (mapc '(lambda (part) (notmuch-show-insert-bodypart msg part depth)) body)) (defun notmuch-show-make-symbol (type) (make-symbol (concat "notmuch-show-" type))) (defun notmuch-show-strip-re (string) (replace-regexp-in-string "\\([Rr]e: *\\)+" "" string)) (defvar notmuch-show-previous-subject "") @@ -1054,40 +1055,47 @@ All currently available key bindings: (save-excursion (notmuch-show-move-to-message-top) (get-text-property (point) :notmuch-message-properties))) (defun notmuch-show-set-prop (prop val &optional props) (let ((inhibit-read-only t) (props (or props (notmuch-show-get-message-properties)))) (plist-put props prop val) (notmuch-show-set-message-properties props))) (defun notmuch-show-get-prop (prop &optional props) (let ((props (or props (notmuch-show-get-message-properties)))) (plist-get props prop))) (defun notmuch-show-get-message-id () "Return the message id of the current message." (concat "id:\"" (notmuch-show-get-prop :id) "\"")) +(defun notmuch-show-get-content-type (&optional props) + "Return parsed Content-Type of the given message, or part, or +current message if no `props` is given. If there is no +Content-Type header, it defaults to \"text/plain\"." + (mail-header-parse-content-type (or (notmuch-show-get-prop :content-type props) + "text/plain"))) + ;; dme: Would it make sense to use a macro for many of these? (defun notmuch-show-get-filename () "Return the filename of the current message." (notmuch-show-get-prop :filename)) (defun notmuch-show-get-header (header) "Return the named header of the current message, if any." (plist-get (notmuch-show-get-prop :headers) header)) (defun notmuch-show-get-cc () (notmuch-show-get-header :Cc)) (defun notmuch-show-get-date () (notmuch-show-get-header :Date)) (defun notmuch-show-get-from () (notmuch-show-get-header :From)) (defun notmuch-show-get-subject () diff --git a/notmuch-show.c b/notmuch-show.c index 603992a..da3e87f 100644 --- a/notmuch-show.c +++ b/notmuch-show.c @@ -3,40 +3,42 @@ * Copyright © 2009 Carl Worth * * This program is free software: you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation, either version 3 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see http://www.gnu.org/licenses/ . * * Author: Carl Worth <cworth@cworth.org> */ #include "notmuch-client.h" +static const char HEADER_CONTENT_TYPE[] = "Content-Type"; + static void format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent); static void format_headers_text (const void *ctx, notmuch_message_t *message); static void format_headers_message_part_text (GMimeMessage *message); static void format_part_start_text (GMimeObject *part, int *part_count); static void format_part_content_text (GMimeObject *part); static void format_part_end_text (GMimeObject *part); @@ -640,42 +642,50 @@ format_part_sigstatus_json (const GMimeSignatureValidity* validity) } printf ("}"); signer = signer->next; } printf ("]"); talloc_free (ctx_quote); } static void format_part_content_json (GMimeObject *part) { GMimeContentType *content_type = g_mime_object_get_content_type (GMIME_OBJECT (part)); GMimeStream *stream_memory = g_mime_stream_mem_new (); const char *cid = g_mime_object_get_content_id (part); void *ctx = talloc_new (NULL); GByteArray *part_content; - printf (", \"content-type\": %s", - json_quote_str (ctx, g_mime_content_type_to_string (content_type))); + { + /* Output full Content-Type header value, + * g_mime_content_type_to_string(3) does not include + * parameters. Content-Type header may be missing, + * g_mime_object_get_content_type(3) defaults to "text/plain" + * in this case. */ + const char *const h = g_mime_object_get_header (part, HEADER_CONTENT_TYPE); + if (h) + printf (", \"content-type\": %s", json_quote_str (ctx, h)); + } if (cid != NULL) printf(", \"content-id\": %s", json_quote_str (ctx, cid)); if (GMIME_IS_PART (part)) { const char *filename = g_mime_part_get_filename (GMIME_PART (part)); if (filename) printf (", \"filename\": %s", json_quote_str (ctx, filename)); } if (g_mime_content_type_is_type (content_type, "text", "*") && !g_mime_content_type_is_type (content_type, "text", "html")) { show_text_part_content (part, stream_memory); part_content = g_mime_stream_mem_get_byte_array (GMIME_STREAM_MEM (stream_memory)); printf (", \"content\": %s", json_quote_chararray (ctx, (char *) part_content->data, part_content->len)); } else if (g_mime_content_type_is_type (content_type, "multipart", "*")) diff --git a/test/crypto b/test/crypto index 0af4aa8..b923d22 100755 --- a/test/crypto +++ b/test/crypto @@ -40,111 +40,108 @@ test_expect_success 'emacs delivery of signed message' \ test_begin_subtest "signature verification" output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "good", "fingerprint": "'$FINGERPRINT'", "created": 946728000}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "signature verification with full owner trust" # give the key full owner trust echo "${FINGERPRINT}:6:" | gpg --no-tty --import-ownertrust >>"$GNUPGHOME"/trust.log 2>&1 gpg --no-tty --check-trustdb >>"$GNUPGHOME"/trust.log 2>&1 output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "good", "fingerprint": "'$FINGERPRINT'", "created": 946728000, "userid": " Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)"}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "signature verification with signer key unavailable" # move the gnupghome temporarily out of the way mv "${GNUPGHOME}"{,.bak} output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "error", "keyid": "'$(echo $FINGERPRINT | cut -c 25-)'", "errors": 2}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" mv "${GNUPGHOME}"{.bak,} # create a test encrypted message with attachment cat <<EOF >TESTATTACHMENT This is a test file. EOF test_expect_success 'emacs delivery of encrypted message with attachment' \ 'emacs_deliver_message \ "test encrypted message 001" \ "This is a test encrypted message.\n" \ "(mml-attach-file \"TESTATTACHMENT\") (mml-secure-message-encrypt)"' test_begin_subtest "decryption, --format=text" @@ -181,138 +178,135 @@ test_expect_equal \ test_begin_subtest "decryption, --format=json" output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["encrypted","inbox"], "headers": {"Subject": "test encrypted message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "encstatus": [{"status": "good"}], "sigstatus": [], - "content-type": "multipart/encrypted", + "content-type": "multipart/encrypted; boundary=\"==-=-=\";\tprotocol=\"application/pgp-encrypted\"", "content": [{"id": 2, "content-type": "application/pgp-encrypted"}, {"id": 3, - "content-type": "multipart/mixed", + "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [{"id": 4, - "content-type": "text/plain", "content": "This is a test encrypted message.\n"}, {"id": 5, "content-type": "application/octet-stream", "filename": "TESTATTACHMENT"}]}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "decryption, --format=json, --part=4" output=$(notmuch show --format=json --part=4 --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='{"id": 4, - "content-type": "text/plain", "content": "This is a test encrypted message.\n"}' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "decrypt attachment (--part=5 --format=raw)" notmuch show \ --format=raw \ --part=5 \ --decrypt \ subject:"test encrypted message 001" >OUTPUT test_expect_equal_file OUTPUT TESTATTACHMENT test_begin_subtest "decryption failure with missing key" mv "${GNUPGHOME}"{,.bak} output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["encrypted","inbox"], "headers": {"Subject": "test encrypted message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "encstatus": [{"status": "bad"}], - "content-type": "multipart/encrypted", + "content-type": "multipart/encrypted; boundary=\"==-=-=\";\tprotocol=\"application/pgp-encrypted\"", "content": [{"id": 2, "content-type": "application/pgp-encrypted"}, {"id": 3, "content-type": "application/octet-stream"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" mv "${GNUPGHOME}"{.bak,} test_expect_success 'emacs delivery of encrypted + signed message' \ 'emacs_deliver_message \ "test encrypted message 002" \ "This is another test encrypted message.\n" \ "(mml-secure-message-sign-encrypt)"' test_begin_subtest "decryption + signature verification" output=$(notmuch show --format=json --decrypt subject:"test encrypted message 002" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["encrypted","inbox"], "headers": {"Subject": "test encrypted message 002", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "encstatus": [{"status": "good"}], "sigstatus": [{"status": "good", "fingerprint": "'$FINGERPRINT'", "created": 946728000, "userid": " Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)"}], - "content-type": "multipart/encrypted", + "content-type": "multipart/encrypted; boundary=\"=-=-=\";\tprotocol=\"application/pgp-encrypted\"", "content": [{"id": 2, "content-type": "application/pgp-encrypted"}, {"id": 3, - "content-type": "text/plain", "content": "This is another test encrypted message.\n"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "reply to encrypted message" output=$(notmuch reply --decrypt subject:"test encrypted message 002" \ | grep -v -e '^In-Reply-To:' -e '^References:') expected='From: Notmuch Test Suite <test_suite@notmuchmail.org> Subject: Re: test encrypted message 002 On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote: > This is another test encrypted message.' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "signature verification with revoked key" # generate revocation certificate and load it to revoke key @@ -327,32 +321,31 @@ y | gpg --no-tty --quiet --import output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "error", "keyid": "6D92612D94E46381", "errors": 8}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_done diff --git a/test/json b/test/json index 592b068..64f35cf 100755 --- a/test/json +++ b/test/json @@ -1,50 +1,50 @@ #!/usr/bin/env bash test_description="--format=json output" . ./test-lib.sh test_begin_subtest "Show message: json" add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-show-message\"" output=$(notmuch show --format=json "json-show-message") -test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]" +test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content\": \"json-show-message\n\"}]}, []]]]" test_begin_subtest "Search message: json" add_message "[subject]=\"json-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-search-message\"" output=$(notmuch search --format=json "json-search-message" | notmuch_search_sanitize) test_expect_equal "$output" "[{\"thread\": \"XXX\", \"timestamp\": 946728000, \"matched\": 1, \"total\": 1, \"authors\": \"Notmuch Test Suite\", \"subject\": \"json-search-subject\", \"tags\": [\"inbox\", \"unread\"]}]" test_begin_subtest "Show message: json, utf-8" add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\"" output=$(notmuch show --format=json "jsön-show-méssage") -test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]" +test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content\": \"jsön-show-méssage\n\"}]}, []]]]" test_begin_subtest "Show message: json, inline attachment filename" subject='json-show-inline-attachment-filename' id="json-show-inline-attachment-filename@notmuchmail.org" emacs_deliver_message \ "$subject" \ 'This is a test message with inline attachment with a filename' \ "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\") (message-goto-eoh) (insert \"Message-ID: <$id>\n\")" output=$(notmuch show --format=json "id:$id") filename=$(notmuch search --output=files "id:$id") -test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]" +test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed; boundary=\\\"=-=-=\\\"\", \"content\": [{\"id\": 2, \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]" test_begin_subtest "Search message: json, utf-8" add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\"" output=$(notmuch search --format=json "jsön-search-méssage" | notmuch_search_sanitize) test_expect_equal "$output" "[{\"thread\": \"XXX\", \"timestamp\": 946728000, \"matched\": 1, \"total\": 1, \"authors\": \"Notmuch Test Suite\", \"subject\": \"json-search-utf8-body-sübjéct\", \"tags\": [\"inbox\", \"unread\"]}]" test_done diff --git a/test/maildir-sync b/test/maildir-sync index a60854f..c7ca22f 100755 --- a/test/maildir-sync +++ b/test/maildir-sync @@ -41,41 +41,40 @@ add_message [subject]='"Adding replied tag"' [filename]='adding-replied-tag:2,S' notmuch tag +replied subject:"Adding replied tag" output=$(cd ${MAIL_DIR}/cur; ls -1 adding-replied*) test_expect_equal "$output" "adding-replied-tag:2,RS" test_begin_subtest "notmuch show works with renamed file (without notmuch new)" output=$(notmuch show --format=json id:${gen_msg_id} | filter_show_json) test_expect_equal "$output" '[[[{"id": "adding-replied-tag@notmuch-test-suite", "match": true, "filename": "MAIL_DIR/cur/adding-replied-tag:2,RS", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox","replied"], "headers": {"Subject": "Adding replied tag", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "Notmuch Test Suite <test_suite@notmuchmail.org>", "Cc": "", "Bcc": "", "Date": "Tue, 05 Jan 2001 15:43:57 -0000"}, "body": [{"id": 1, -"content-type": "text/plain", "content": "This is just a test message (#3)\n"}]}, []]]]' test_expect_success 'notmuch reply works with renamed file (without notmuch new)' 'notmuch reply id:${gen_msg_id}' test_begin_subtest "notmuch new detects no file rename after tag->flag synchronization" output=$(NOTMUCH_NEW) test_expect_equal "$output" "No new mail." test_begin_subtest "When read, message moved from new to cur" add_message [subject]='"Message to move to cur"' [date]='"Sat, 01 Jan 2000 12:00:00 -0000"' [filename]='message-to-move-to-cur' [dir]=new notmuch tag -unread subject:"Message to move to cur" output=$(cd "$MAIL_DIR/cur"; ls message-to-move*) test_expect_equal "$output" "message-to-move-to-cur:2,S" test_begin_subtest "No rename should be detected by notmuch new" output=$(NOTMUCH_NEW) test_expect_equal "$output" "No new mail." # (*) If notmuch new was not run we've got "Processed 1 file in almost # no time" here. The reason is that removing unread tag in a previous diff --git a/test/multipart b/test/multipart index f83526b..ca4db71 100755 --- a/test/multipart +++ b/test/multipart @@ -306,140 +306,140 @@ test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=text --part=9, pgp signature (unverified)" notmuch show --format=text --part=9 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT cat <<EOF >EXPECTED \fpart{ ID: 9, Content-type: application/pgp-signature Non-text part: application/pgp-signature \fpart} EOF test_expect_equal_file OUTPUT EXPECTED test_expect_success \ "--format=text --part=8, no part, expect error" \ "notmuch show --format=text --part=8 'id:87liy5ap00.fsf@yoom.home.cworth.org'" test_begin_subtest "--format=json --part=0, full message" notmuch show --format=json --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Cc": "", "Bcc": "", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [ -{"id": 1, "content-type": "multipart/signed", "content": [ -{"id": 2, "content-type": "multipart/mixed", "content": [ +{"id": 1, "content-type": "multipart/signed; boundary=\"==-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [ +{"id": 2, "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [ {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"}, +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"}]}, {"id": 9, "content-type": "application/pgp-signature"}]}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=1, message body" notmuch show --format=json --part=1 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 1, "content-type": "multipart/signed", "content": [ -{"id": 2, "content-type": "multipart/mixed", "content": [ +{"id": 1, "content-type": "multipart/signed; boundary=\"==-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [ +{"id": 2, "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [ {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"}, +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"}]}, {"id": 9, "content-type": "application/pgp-signature"}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=2, multipart/mixed" notmuch show --format=json --part=2 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 2, "content-type": "multipart/mixed", "content": [ +{"id": 2, "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [ {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]} +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"}, +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=3, rfc822 part" notmuch show --format=json --part=3 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=4, rfc822's multipart/alternative" notmuch show --format=json --part=4 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=5, rfc822's html part" notmuch show --format=json --part=5 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 5, "content-type": "text/html"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=6, rfc822's text part" notmuch show --format=json --part=6 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=7, inline attachment" notmuch show --format=json --part=7 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"} +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=8, plain text part" notmuch show --format=json --part=8 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"} +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=9, pgp signature (unverified)" notmuch show --format=json --part=9 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 9, "content-type": "application/pgp-signature"} EOF test_expect_equal_file OUTPUT EXPECTED test_expect_success \ "--format=json --part=10, no part, expect error" \ "notmuch show --format=json --part=10 'id:87liy5ap00.fsf@yoom.home.cworth.org'" test_begin_subtest "--format=raw" notmuch show --format=raw 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT test_expect_equal_file OUTPUT "${MAIL_DIR}"/multipart -- 1.7.7.3 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-18 23:45 [PATCH] Output unmodified Content-Type header value for JSON format Dmitry Kurochkin @ 2011-11-19 1:58 ` Jameson Graef Rollins 2011-11-19 2:42 ` Dmitry Kurochkin 2011-11-19 4:18 ` [PATCH v2] " Dmitry Kurochkin 1 sibling, 1 reply; 24+ messages in thread From: Jameson Graef Rollins @ 2011-11-19 1:58 UTC (permalink / raw) To: Dmitry Kurochkin, notmuch [-- Attachment #1: Type: text/plain, Size: 1286 bytes --] On Sat, 19 Nov 2011 03:45:05 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > Before the change, notmuch used g_mime_content_type_to_string(3) > function to output Content-Type header value. Turns out it outputs > only "type/subtype" part and ignores all parameters. Also, if there > is no Content-Type header, default "text/plain" value is used. Hi, Dmitry. Can you explain under what circumstances you would need the extra content-type parameters? It just seems like a lot of extra noise in the output to me, but that's partially because I can't think of any reason why something that is receiving pre-parsed mime content would need it. Maybe there's a better way to handle what you're trying to get to. I think it would help a lot if you could submit some sort of test modification that demonstrates the issue. This is one of the reasons we keep emphasizing that it's good to first have tests in hand that demonstrate issues before patches that address them. > "content": [{"id": 2, > - "content-type": "text/plain", > "content": "This is a test signed message.\n"}, Without figuring out what's going on, I notice that some of the tests have been modified to remove the content-type fields on a bunch of parts. I think that is probably not right. jamie. [-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 1:58 ` Jameson Graef Rollins @ 2011-11-19 2:42 ` Dmitry Kurochkin 2011-11-19 4:59 ` Austin Clements 2011-11-19 10:49 ` Jameson Graef Rollins 0 siblings, 2 replies; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-19 2:42 UTC (permalink / raw) To: Jameson Graef Rollins, notmuch Hi Jamie. On Fri, 18 Nov 2011 17:58:52 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote: > On Sat, 19 Nov 2011 03:45:05 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > Before the change, notmuch used g_mime_content_type_to_string(3) > > function to output Content-Type header value. Turns out it outputs > > only "type/subtype" part and ignores all parameters. Also, if there > > is no Content-Type header, default "text/plain" value is used. > > Hi, Dmitry. Can you explain under what circumstances you would need the > extra content-type parameters? Charset is an example of a parameter which you need to render a part correctly. > It just seems like a lot of extra noise > in the output to me, but that's partially because I can't think of any > reason why something that is receiving pre-parsed mime content would > need it. Maybe there's a better way to handle what you're trying to get > to. > Why extra output in JSON is an issue? The parameters are there for a reason. They are part of the content-type and are needed to handle the body properly. If you say that the parameters are not needed by notmuch users, that implies that they are handled by notmuch. Which is just not possible. > I think it would help a lot if you could submit some sort of test > modification that demonstrates the issue. This is one of the reasons we > keep emphasizing that it's good to first have tests in hand that > demonstrate issues before patches that address them. > The fact that this change happens to fix an issue with HTML charsets for me is just a side effect. The real issue is that JSON format throws away information which is required to properly render a part. I do not think we need to add a dedicated test to check that JSON outputs charsets with parameters, considering that it is already present in many other tests. I do not think it was intended that notmuch outputs stripped Content-Type values. It was just a side effect of using g_mime_content_type_to_string(3) which gone unnoticed. > > "content": [{"id": 2, > > - "content-type": "text/plain", > > "content": "This is a test signed message.\n"}, > > Without figuring out what's going on, I notice that some of the tests > have been modified to remove the content-type fields on a bunch of > parts. I think that is probably not right. > I tried to explain this in the preable. These parts do not have Content-Type in the original message. So I think it is wrong for notmuch JSON format to add it. Regards, Dmitry > jamie. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 2:42 ` Dmitry Kurochkin @ 2011-11-19 4:59 ` Austin Clements 2011-11-19 5:26 ` Dmitry Kurochkin 2011-11-19 10:49 ` Jameson Graef Rollins 1 sibling, 1 reply; 24+ messages in thread From: Austin Clements @ 2011-11-19 4:59 UTC (permalink / raw) To: Dmitry Kurochkin; +Cc: notmuch Quoth Dmitry Kurochkin on Nov 19 at 6:42 am: > Hi Jamie. > > On Fri, 18 Nov 2011 17:58:52 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote: > > On Sat, 19 Nov 2011 03:45:05 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > > Before the change, notmuch used g_mime_content_type_to_string(3) > > > function to output Content-Type header value. Turns out it outputs > > > only "type/subtype" part and ignores all parameters. Also, if there > > > is no Content-Type header, default "text/plain" value is used. > > > > Hi, Dmitry. Can you explain under what circumstances you would need the > > extra content-type parameters? > > Charset is an example of a parameter which you need to render a part > correctly. Can notmuch convert to a common charset, given that, otherwise, every client is going to have to implement this conversion anyway? (And are there other examples of useful things in the content type?) ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 4:59 ` Austin Clements @ 2011-11-19 5:26 ` Dmitry Kurochkin 2011-11-19 18:58 ` Austin Clements 0 siblings, 1 reply; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-19 5:26 UTC (permalink / raw) To: Austin Clements; +Cc: notmuch On Fri, 18 Nov 2011 23:59:57 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > Quoth Dmitry Kurochkin on Nov 19 at 6:42 am: > > Hi Jamie. > > > > On Fri, 18 Nov 2011 17:58:52 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote: > > > On Sat, 19 Nov 2011 03:45:05 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > > > Before the change, notmuch used g_mime_content_type_to_string(3) > > > > function to output Content-Type header value. Turns out it outputs > > > > only "type/subtype" part and ignores all parameters. Also, if there > > > > is no Content-Type header, default "text/plain" value is used. > > > > > > Hi, Dmitry. Can you explain under what circumstances you would need the > > > extra content-type parameters? > > > > Charset is an example of a parameter which you need to render a part > > correctly. > > Can notmuch convert to a common charset, given that, otherwise, every > client is going to have to implement this conversion anyway? > Notmuch can handle charset (and any other) parameters but only for known media types (i.e. text/*). I think it would be useful (especially for human-readable output formats). But it is a separate issue. Notmuch can not convert other types it does not know how to handle. E.g. HTML charset conversion is not as simple as for plain text. AFAIK standard defines charset parameter just for few types. So in general, charset parameter can have any meaning for some custom media type. > (And are there other examples of useful things in the content type?) What is meant by useful? All parameters do have some use. The fact that notmuch does not handle them does not mean they are useless. And notmuch can not handle all parameters just because the list of parameters is not defined. So there is no choice but to let notmuch users see and use these parameters. Regards, Dmitry ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 5:26 ` Dmitry Kurochkin @ 2011-11-19 18:58 ` Austin Clements 2011-11-20 18:52 ` Dmitry Kurochkin 0 siblings, 1 reply; 24+ messages in thread From: Austin Clements @ 2011-11-19 18:58 UTC (permalink / raw) To: Dmitry Kurochkin; +Cc: notmuch Quoth Dmitry Kurochkin on Nov 19 at 9:26 am: > On Fri, 18 Nov 2011 23:59:57 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > > Quoth Dmitry Kurochkin on Nov 19 at 6:42 am: > > > Hi Jamie. > > > > > > On Fri, 18 Nov 2011 17:58:52 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote: > > > > On Sat, 19 Nov 2011 03:45:05 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > > > > Before the change, notmuch used g_mime_content_type_to_string(3) > > > > > function to output Content-Type header value. Turns out it outputs > > > > > only "type/subtype" part and ignores all parameters. Also, if there > > > > > is no Content-Type header, default "text/plain" value is used. > > > > > > > > Hi, Dmitry. Can you explain under what circumstances you would need the > > > > extra content-type parameters? > > > > > > Charset is an example of a parameter which you need to render a part > > > correctly. > > > > Can notmuch convert to a common charset, given that, otherwise, every > > client is going to have to implement this conversion anyway? > > > > Notmuch can handle charset (and any other) parameters but only for known > media types (i.e. text/*). I think it would be useful (especially for > human-readable output formats). But it is a separate issue. > > Notmuch can not convert other types it does not know how to handle. > E.g. HTML charset conversion is not as simple as for plain text. > > AFAIK standard defines charset parameter just for few types. So in > general, charset parameter can have any meaning for some custom media > type. Interesting. I hadn't realized the content-type specification was so open-ended. However, there are many things that *could* be included in the JSON format but aren't; what's included is primarily driven by what consumers actually need and it seems like the actual need here is charset handling. Maybe the JSON format *shouldn't* evolve this way, but I think it should either be driven by its needs like it is now, or we should be taking bigger steps like providing *all* of the headers (essentially, a JSON-ification of the MIME structure), which would subsume more specific generalizations like exposing just the full content-type header. Regarding charset, specifically, though, the JSON format only includes part bodies for text/* types and, according to RFC 2045, For example, the "charset" parameter is applicable to any subtype of "text", ... Section 4.1.2 (Charset Parameter) of RFC 2046 beats around the bush, but I think it's saying essentially the same thing in a lot more detail. Given that, I think it does make sense for notmuch to handle the charset parameter and re-coding. > > (And are there other examples of useful things in the content type?) > > What is meant by useful? All parameters do have some use. The fact > that notmuch does not handle them does not mean they are useless. And > notmuch can not handle all parameters just because the list of > parameters is not defined. So there is no choice but to let notmuch > users see and use these parameters. Yes, I now agree with this, modulo my statements about generality above. > Regards, > Dmitry > -- Austin Clements MIT/'06/PhD/CSAIL amdragon@mit.edu http://web.mit.edu/amdragon Somewhere in the dream we call reality you will find me, searching for the reality we call dreams. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 18:58 ` Austin Clements @ 2011-11-20 18:52 ` Dmitry Kurochkin 0 siblings, 0 replies; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-20 18:52 UTC (permalink / raw) To: Austin Clements; +Cc: notmuch On Sat, 19 Nov 2011 13:58:18 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > Quoth Dmitry Kurochkin on Nov 19 at 9:26 am: > > On Fri, 18 Nov 2011 23:59:57 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > > > Quoth Dmitry Kurochkin on Nov 19 at 6:42 am: > > > > Hi Jamie. > > > > > > > > On Fri, 18 Nov 2011 17:58:52 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote: > > > > > On Sat, 19 Nov 2011 03:45:05 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > > > > > Before the change, notmuch used g_mime_content_type_to_string(3) > > > > > > function to output Content-Type header value. Turns out it outputs > > > > > > only "type/subtype" part and ignores all parameters. Also, if there > > > > > > is no Content-Type header, default "text/plain" value is used. > > > > > > > > > > Hi, Dmitry. Can you explain under what circumstances you would need the > > > > > extra content-type parameters? > > > > > > > > Charset is an example of a parameter which you need to render a part > > > > correctly. > > > > > > Can notmuch convert to a common charset, given that, otherwise, every > > > client is going to have to implement this conversion anyway? > > > > > > > Notmuch can handle charset (and any other) parameters but only for known > > media types (i.e. text/*). I think it would be useful (especially for > > human-readable output formats). But it is a separate issue. > > > > Notmuch can not convert other types it does not know how to handle. > > E.g. HTML charset conversion is not as simple as for plain text. > > > > AFAIK standard defines charset parameter just for few types. So in > > general, charset parameter can have any meaning for some custom media > > type. > > Interesting. I hadn't realized the content-type specification was so > open-ended. However, there are many things that *could* be included > in the JSON format but aren't; what's included is primarily driven by > what consumers actually need and it seems like the actual need here is > charset handling. Maybe the JSON format *shouldn't* evolve this way, > but I think it should either be driven by its needs like it is now, or > we should be taking bigger steps like providing *all* of the headers > (essentially, a JSON-ification of the MIME structure), which would > subsume more specific generalizations like exposing just the full > content-type header. > I think it is a good idea to provide all headers in JSON output. Still I believe this patch is still valid. It is a simple change, which makes the JSON format simpler and we have consumers that need it. Providing all headers would be a bigger change (and I expect it to be much more difficult to get accepted). What I definately do not like, is adding an exception for charset parameter and inventing complex rules for JSON format instead of keeping it simple. > Regarding charset, specifically, though, the JSON format only includes > part bodies for text/* types and, according to RFC 2045, > > For example, the "charset" parameter is applicable to any subtype of > "text", ... > > Section 4.1.2 (Charset Parameter) of RFC 2046 beats around the bush, > but I think it's saying essentially the same thing in a lot more > detail. Given that, I think it does make sense for notmuch to handle > the charset parameter and re-coding. > I think it may be a good idea but it is not trivial to do right. We should not just convert all text parts unconditionally to locale or UTF-8. > > > (And are there other examples of useful things in the content type?) > > > > What is meant by useful? All parameters do have some use. The fact > > that notmuch does not handle them does not mean they are useless. And > > notmuch can not handle all parameters just because the list of > > parameters is not defined. So there is no choice but to let notmuch > > users see and use these parameters. > > Yes, I now agree with this, modulo my statements about generality above. > Thanks. Regards, Dmitry > > Regards, > > Dmitry > > > > -- > Austin Clements MIT/'06/PhD/CSAIL > amdragon@mit.edu http://web.mit.edu/amdragon > Somewhere in the dream we call reality you will find me, > searching for the reality we call dreams. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 2:42 ` Dmitry Kurochkin 2011-11-19 4:59 ` Austin Clements @ 2011-11-19 10:49 ` Jameson Graef Rollins 2011-11-20 18:32 ` Dmitry Kurochkin 1 sibling, 1 reply; 24+ messages in thread From: Jameson Graef Rollins @ 2011-11-19 10:49 UTC (permalink / raw) To: Dmitry Kurochkin, notmuch [-- Attachment #1: Type: text/plain, Size: 2734 bytes --] On Sat, 19 Nov 2011 06:42:00 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > The parameters are there for a reason. They are part of the > content-type and are needed to handle the body properly. If you say > that the parameters are not needed by notmuch users, that implies that > they are handled by notmuch. Which is just not possible. Hey, Dimitry. At least some of the parameters in the content-type are actually related to the mime structure itself. A good example is: boundary=\"=-=-=\" This parameter is there to tell the MIME parser how to parse the content that follows. It is meant for the first level parser and no more. Once the MIME has been separated into it's constituent parts, there's no need for any further clients to know anything about boundary string. I would argue that notmuch is acting as the first level parser. As far as I can tell, most of the rest of the parameters I've seen should only be useful to the those first-level parsers. As Austin mentioned, is it not possible for notmuch itself to act on the parameter to give a properly formatted output to its clients? > The fact that this change happens to fix an issue with HTML charsets for > me is just a side effect. But isn't that actually a large part of the issue? If this patch fixes something that you think notmuch is doing improperly, could there not be a test for it? However, based on your patches and as far as I can tell, this change adds more than a boundary parameter to only crypto parts (application/signed and application/encrypted). However, I don't think any of the crypto functionality needs any of the extra information provided in the extended output. If there was a test for the functionality you think is missing, it would help bolster the case for the additional output. > > > "content": [{"id": 2, > > > - "content-type": "text/plain", > > > "content": "This is a test signed message.\n"}, > > > > Without figuring out what's going on, I notice that some of the tests > > have been modified to remove the content-type fields on a bunch of > > parts. I think that is probably not right. > > > > I tried to explain this in the preable. These parts do not have > Content-Type in the original message. So I think it is wrong for > notmuch JSON format to add it. Ah, ok, I think I understand this point. I think this is actually a separate issue than the one the rest of the patch set is for, though. One part of the patch is that content-type parameters are also about, and another part is that parts without content-type shouldn't be assigned one automatically. I personally think those should be separate patches. jamie. [-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-19 10:49 ` Jameson Graef Rollins @ 2011-11-20 18:32 ` Dmitry Kurochkin 2011-11-20 20:10 ` Jameson Graef Rollins 0 siblings, 1 reply; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-20 18:32 UTC (permalink / raw) To: Jameson Graef Rollins, notmuch Hi Jameson. On Sat, 19 Nov 2011 02:49:43 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote: > On Sat, 19 Nov 2011 06:42:00 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > The parameters are there for a reason. They are part of the > > content-type and are needed to handle the body properly. If you say > > that the parameters are not needed by notmuch users, that implies that > > they are handled by notmuch. Which is just not possible. > > Hey, Dimitry. At least some of the parameters in the content-type are > actually related to the mime structure itself. A good example is: > > boundary=\"=-=-=\" > > This parameter is there to tell the MIME parser how to parse the content > that follows. It is meant for the first level parser and no more. Once > the MIME has been separated into it's constituent parts, there's no need > for any further clients to know anything about boundary string. > Yes, at least in most cases. On the other hand, if you can make notmuch show raw multipart part (you can, right?), then it seems natural that notmuch provides enough information to parse it. > I would argue that notmuch is acting as the first level parser. As far > as I can tell, most of the rest of the parameters I've seen should only > be useful to the those first-level parsers. > I do not think first-level parser is a standard term. As I understand, you mean that notmuch parses MIME recursively until the leaf non-MIME parts. Correct? I do not know what parameters you have seen. The most common example of a useful parameter for second-level parsers is a charset. I do not understand why do we try to come up with excuses for not providing useful information to users. Current assumption that all parameters that notmuch does not handle are useless is plain invalid. > As Austin mentioned, is it not possible for notmuch itself to act on the > parameter to give a properly formatted output to its clients? > Please see my answer to Austin. I explained why this is not an option in general case. As for parameters that notmuch already handles, like "boundary", I just do not understand why we should invent some artificial exceptions and decide for our users what parameters are useful or useless for them instead of implementing a simple and kind of expected approach: content-type in JSON is original Content-Type header value. It makes both the code and the format simpler. > > The fact that this change happens to fix an issue with HTML charsets for > > me is just a side effect. > > But isn't that actually a large part of the issue? If this patch fixes > something that you think notmuch is doing improperly, could there not be > a test for it? > No. It just happens to be how I found the problem. The issue is: notmuch JSON format mangles Content-Type header value by throwing away useful information in some cases and adding info that was not there in others. Note that I do not mention any single parameter name here. It is a general issue, not a "charset" or "boundary" parameter issue. > However, based on your patches and as far as I can tell, this change > adds more than a boundary parameter to only crypto parts > (application/signed and application/encrypted). However, I don't think > any of the crypto functionality needs any of the extra information > provided in the extended output. If there was a test for the > functionality you think is missing, it would help bolster the case for > the additional output. > Again, the patch is not about "add boundary and other useless crypto parameters". The patch is about stop throwing away useless information. > > > > "content": [{"id": 2, > > > > - "content-type": "text/plain", > > > > "content": "This is a test signed message.\n"}, > > > > > > Without figuring out what's going on, I notice that some of the tests > > > have been modified to remove the content-type fields on a bunch of > > > parts. I think that is probably not right. > > > > > > > I tried to explain this in the preable. These parts do not have > > Content-Type in the original message. So I think it is wrong for > > notmuch JSON format to add it. > > Ah, ok, I think I understand this point. I think this is actually a > separate issue than the one the rest of the patch set is for, though. > One part of the patch is that content-type parameters are also about, > and another part is that parts without content-type shouldn't be > assigned one automatically. I personally think those should be separate > patches. > The implementation makes it not practical to separate these changes. They come as a result of the same code change. Regards, Dmitry > jamie. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-20 18:32 ` Dmitry Kurochkin @ 2011-11-20 20:10 ` Jameson Graef Rollins 2011-11-23 3:40 ` Austin Clements 0 siblings, 1 reply; 24+ messages in thread From: Jameson Graef Rollins @ 2011-11-20 20:10 UTC (permalink / raw) To: Dmitry Kurochkin, notmuch [-- Attachment #1: Type: text/plain, Size: 2811 bytes --] On Sun, 20 Nov 2011 22:32:53 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > Yes, at least in most cases. On the other hand, if you can make notmuch > show raw multipart part (you can, right?), then it seems natural that > notmuch provides enough information to parse it. This is kind of an unresolved issue, actually. Currently headers are only included in the "raw" format output of an entire message. Otherwise, when raw output is requested of an individual part only the body is output. For multipart parts, the raw format output includes all body parts concatenated together, still without any headers. This "raw" multipart output clearly doesn't really make much sense and we need to figure that out. dkg wrote a good breakdown of the issue here: id:"4E09072A.7040906@fifthhorseman.net" However, this only for "raw" output. It's definitely not the same as the json output. For json the parts are all parsed by notmuch and placed into separate json elements. The receiver is not going to do any further parsing since all the mime structure parsing has been done. We need to keep clear the distinction between "parsing" the mime structure, and "encoding" the content of the part. Confusion seems to be coming from the fact that the Content-Type header includes information needed for both mime parsing and content encoding. However, I don't think that means that we need to just include everything in the output. Parameters that have to do with mime parsing should be dropped, since that information has already been used in the mime parsing and can't is no longer useful to the consumer. It's just noise, and I don't think notmuch should be outputting useless noise. The open question seems to be how we handle the content encoding parameters. My argument is that those should either be used by notmuch to properly encode the content for the consumer. If that's not possible, then just those parameters needed by the consumer to decode the content should be output. > > But isn't that actually a large part of the issue? If this patch fixes > > something that you think notmuch is doing improperly, could there not be > > a test for it? > > No. It just happens to be how I found the problem. The issue is: > notmuch JSON format mangles Content-Type header value by throwing away > useful information in some cases and adding info that was not there in > others. Note that I do not mention any single parameter name here. It > is a general issue, not a "charset" or "boundary" parameter issue. I'm sorry, but I still don't believe it's not possible to test for this issue. If there's a problem that you're seeing, then you must of identified it somehow, and therefore there must be a way to test for it. jamie. [-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-20 20:10 ` Jameson Graef Rollins @ 2011-11-23 3:40 ` Austin Clements 2012-01-12 17:07 ` Pieter Praet 0 siblings, 1 reply; 24+ messages in thread From: Austin Clements @ 2011-11-23 3:40 UTC (permalink / raw) To: Jameson Graef Rollins; +Cc: notmuch Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm: > The open question seems to be how we handle the content encoding > parameters. My argument is that those should either be used by notmuch > to properly encode the content for the consumer. If that's not > possible, then just those parameters needed by the consumer to decode > the content should be output. If notmuch is going to include part content in the JSON output (which perhaps it shouldn't, as per recent IRC discussions), then it must handle content encodings because JSON must be Unicode and therefore the content strings in the JSON must be Unicode. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2011-11-23 3:40 ` Austin Clements @ 2012-01-12 17:07 ` Pieter Praet 2012-01-12 17:28 ` Austin Clements 0 siblings, 1 reply; 24+ messages in thread From: Pieter Praet @ 2012-01-12 17:07 UTC (permalink / raw) To: Austin Clements, Jameson Graef Rollins; +Cc: notmuch On Tue, 22 Nov 2011 22:40:21 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm: > > The open question seems to be how we handle the content encoding > > parameters. My argument is that those should either be used by notmuch > > to properly encode the content for the consumer. If that's not > > possible, then just those parameters needed by the consumer to decode > > the content should be output. > > If notmuch is going to include part content in the JSON output (which > perhaps it shouldn't, as per recent IRC discussions), then it must > handle content encodings because JSON must be Unicode and therefore > the content strings in the JSON must be Unicode. Having missed the IRC discussions: what is the rationale for not including (specific types of?) part content in the JSON output ? Eg. how about inline attached text/x-patch ? > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch Peace -- Pieter ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2012-01-12 17:07 ` Pieter Praet @ 2012-01-12 17:28 ` Austin Clements 2012-01-14 9:19 ` Pieter Praet 0 siblings, 1 reply; 24+ messages in thread From: Austin Clements @ 2012-01-12 17:28 UTC (permalink / raw) To: Pieter Praet; +Cc: notmuch Quoth Pieter Praet on Jan 12 at 6:07 pm: > On Tue, 22 Nov 2011 22:40:21 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > > Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm: > > > The open question seems to be how we handle the content encoding > > > parameters. My argument is that those should either be used by notmuch > > > to properly encode the content for the consumer. If that's not > > > possible, then just those parameters needed by the consumer to decode > > > the content should be output. > > > > If notmuch is going to include part content in the JSON output (which > > perhaps it shouldn't, as per recent IRC discussions), then it must > > handle content encodings because JSON must be Unicode and therefore > > the content strings in the JSON must be Unicode. > > Having missed the IRC discussions: what is the rationale for not > including (specific types of?) part content in the JSON output ? > Eg. how about inline attached text/x-patch ? Technically the IRC discussion was about not including *any* part content in the JSON output, and always using show --format=raw or similar to retrieve desired parts. Currently, notmuch includes part content in the JSON only for text/*, *except* when it's text/html. I assume non-text parts are omitted because binary data is hard to represent in JSON and text/html is omitted because some people don't need it. However, this leads to some peculiar asymmetry in the Emacs code where sometimes it pulls part content out of the JSON and sometimes it retrieves it using show --format=raw. This in turn leads to asymmetry in content encoding handling, since notmuch handles content encoding for parts included in the JSON (and there's no good way around that since JSON is Unicode), but not for parts retrieved as raw. The idea discussed on IRC was to remove all part content from the JSON output and to always use show to retrieve it, possibly beefing up show's support for content decoding (and possibly introducing a way to retrieve multiple raw parts at once to avoid re-parsing). This would get the JSON format out of the business of guessing what consumers need, simplify the Emacs code, and normalize content encoding handling. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2012-01-12 17:28 ` Austin Clements @ 2012-01-14 9:19 ` Pieter Praet 2012-01-15 11:52 ` David Edmondson 0 siblings, 1 reply; 24+ messages in thread From: Pieter Praet @ 2012-01-14 9:19 UTC (permalink / raw) To: Austin Clements; +Cc: notmuch On Thu, 12 Jan 2012 12:28:40 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > Quoth Pieter Praet on Jan 12 at 6:07 pm: > > On Tue, 22 Nov 2011 22:40:21 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > > > Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm: > > > > The open question seems to be how we handle the content encoding > > > > parameters. My argument is that those should either be used by notmuch > > > > to properly encode the content for the consumer. If that's not > > > > possible, then just those parameters needed by the consumer to decode > > > > the content should be output. > > > > > > If notmuch is going to include part content in the JSON output (which > > > perhaps it shouldn't, as per recent IRC discussions), then it must > > > handle content encodings because JSON must be Unicode and therefore > > > the content strings in the JSON must be Unicode. > > > > Having missed the IRC discussions: what is the rationale for not > > including (specific types of?) part content in the JSON output ? > > Eg. how about inline attached text/x-patch ? > > Technically the IRC discussion was about not including *any* part > content in the JSON output, and always using show --format=raw or > similar to retrieve desired parts. Currently, notmuch includes part > content in the JSON only for text/*, *except* when it's text/html. I > assume non-text parts are omitted because binary data is hard to > represent in JSON and text/html is omitted because some people don't > need it. However, this leads to some peculiar asymmetry in the Emacs > code where sometimes it pulls part content out of the JSON and > sometimes it retrieves it using show --format=raw. This in turn leads > to asymmetry in content encoding handling, since notmuch handles > content encoding for parts included in the JSON (and there's no good > way around that since JSON is Unicode), but not for parts retrieved as > raw. > > The idea discussed on IRC was to remove all part content from the JSON > output and to always use show to retrieve it, possibly beefing up > show's support for content decoding (and possibly introducing a way to > retrieve multiple raw parts at once to avoid re-parsing). This would > get the JSON format out of the business of guessing what consumers > need, simplify the Emacs code, and normalize content encoding > handling. Full ACK. One concern though (IIUC): Due to the prevalence of retarded MUA's, not outputting 'text/plain' and/or 'text/html' parts is unfortunately all too often equivalent to not outputting anything at all, so wouldn't we, in essence, be reducing `show --format=json' to an ever-so-slightly augmented `search --format=json' ? Peace -- Pieter ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2012-01-14 9:19 ` Pieter Praet @ 2012-01-15 11:52 ` David Edmondson 2012-01-15 17:58 ` Austin Clements 0 siblings, 1 reply; 24+ messages in thread From: David Edmondson @ 2012-01-15 11:52 UTC (permalink / raw) To: Pieter Praet, Austin Clements; +Cc: notmuch [-- Attachment #1: Type: text/plain, Size: 1680 bytes --] > Technically the IRC discussion was about not including *any* part > content in the JSON output, and always using show --format=raw or > similar to retrieve desired parts. Currently, notmuch includes part > content in the JSON only for text/*, *except* when it's text/html. I > assume non-text parts are omitted because binary data is hard to > represent in JSON and text/html is omitted because some people don't > need it. However, this leads to some peculiar asymmetry in the Emacs > code where sometimes it pulls part content out of the JSON and > sometimes it retrieves it using show --format=raw. This in turn leads > to asymmetry in content encoding handling, since notmuch handles > content encoding for parts included in the JSON (and there's no good > way around that since JSON is Unicode), but not for parts retrieved as > raw. Including the text output in the JSON results in significantly fewer calls to 'notmuch' during the building of a typical `notmuch-show-mode' buffer. Someone with one of those older, crankier computers could easily test how much effect this has by changing `notmuch-show-get-bodypart-content' slightly. > The idea discussed on IRC was to remove all part content from the JSON > output and to always use show to retrieve it, possibly beefing up > show's support for content decoding (and possibly introducing a way to > retrieve multiple raw parts at once to avoid re-parsing). This would > get the JSON format out of the business of guessing what consumers > need, simplify the Emacs code, and normalize content encoding > handling. Is there a real problem being solved here? Having a clean structure is nice, except when it's not. [-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2012-01-15 11:52 ` David Edmondson @ 2012-01-15 17:58 ` Austin Clements 2012-01-16 8:49 ` David Edmondson 0 siblings, 1 reply; 24+ messages in thread From: Austin Clements @ 2012-01-15 17:58 UTC (permalink / raw) To: David Edmondson, Pieter Praet; +Cc: notmuch On Sun, 15 Jan 2012 11:52:40 +0000, David Edmondson <dme@dme.org> wrote: > > Technically the IRC discussion was about not including *any* part > > content in the JSON output, and always using show --format=raw or > > similar to retrieve desired parts. Currently, notmuch includes part > > content in the JSON only for text/*, *except* when it's text/html. I > > assume non-text parts are omitted because binary data is hard to > > represent in JSON and text/html is omitted because some people don't > > need it. However, this leads to some peculiar asymmetry in the Emacs > > code where sometimes it pulls part content out of the JSON and > > sometimes it retrieves it using show --format=raw. This in turn leads > > to asymmetry in content encoding handling, since notmuch handles > > content encoding for parts included in the JSON (and there's no good > > way around that since JSON is Unicode), but not for parts retrieved as > > raw. > > Including the text output in the JSON results in significantly fewer > calls to 'notmuch' during the building of a typical `notmuch-show-mode' > buffer. Someone with one of those older, crankier computers could easily > test how much effect this has by changing > `notmuch-show-get-bodypart-content' slightly. Yes. I was mostly reiterating the IRC discussion for Pieter. Since this discussion, I've stabilized on the pre-fetching notion I described in id:"20120115003617.GH1801@mit.edu", though I do think we should make this clear in the code: that the rule for whether the JSON includes a "content" key for a leaf part is internal to the CLI and that consumers should be prepared to use it if it's there and to retrieve the content separately if it's not. This is exactly how the Emacs code happens to work, it just hasn't been codified anywhere. Looking at it this way gives us more flexibility than the current code takes advantage of; for example we could omit content from the JSON if it's over some size threshold since the cost of sending that to a client that doesn't need it is high while the cost of having the client retrieve it for itself is relatively low. > > The idea discussed on IRC was to remove all part content from the JSON > > output and to always use show to retrieve it, possibly beefing up > > show's support for content decoding (and possibly introducing a way to > > retrieve multiple raw parts at once to avoid re-parsing). This would > > get the JSON format out of the business of guessing what consumers > > need, simplify the Emacs code, and normalize content encoding > > handling. > > Is there a real problem being solved here? Having a clean structure is > nice, except when it's not. The "real" problem is the asymmetry in encoding handling that started this discussion. Content included in the JSON is re-encoded by the CLI, while content retrieved via raw needs to be re-encoded by the client. OTOH, I don't understand the encoding story for HTML, since the encoding can come from either a header or from the body of the HTML. Does this make it strictly necessary for the client to handle the encoding? ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2012-01-15 17:58 ` Austin Clements @ 2012-01-16 8:49 ` David Edmondson 2012-01-16 13:18 ` Tomi Ollila 0 siblings, 1 reply; 24+ messages in thread From: David Edmondson @ 2012-01-16 8:49 UTC (permalink / raw) To: Austin Clements, Pieter Praet; +Cc: notmuch [-- Attachment #1: Type: text/plain, Size: 1912 bytes --] On Sun, 15 Jan 2012 12:58:40 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > Yes. I was mostly reiterating the IRC discussion for Pieter. Since > this discussion, I've stabilized on the pre-fetching notion I described > in id:"20120115003617.GH1801@mit.edu", Will read when I get there. > though I do think we should make this clear in the code: that the rule > for whether the JSON includes a "content" key for a leaf part is > internal to the CLI and that consumers should be prepared to use it if > it's there and to retrieve the content separately if it's not. This > is exactly how the Emacs code happens to work, it just hasn't been > codified anywhere. It's a bit more than 'happens to work' :-) > Looking at it this way gives us more flexibility than the current code > takes advantage of; for example we could omit content from the JSON if > it's over some size threshold since the cost of sending that to a > client that doesn't need it is high while the cost of having the > client retrieve it for itself is relatively low. This, I suspect, brings us back to what may have been Dmitry's original concern. If we codify the current behaviour then we're actually _forcing_ clients to use inline content if it's present, because otherwise they have no way to discover the charset/encoding used for the raw part. That seems as though it could be a problem for some clients. > OTOH, I don't understand the encoding story for HTML, since the encoding > can come from either a header or from the body of the HTML. Does this > make it strictly necessary for the client to handle the encoding? If it can be specified by the content of a part rather than the part headers, then I think that the client will have to be prepared to handle it. Even if not, it might still be more effective to choose that approach - it would remove the need to add arbitrary encoding support to the CLI application. [-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] Output unmodified Content-Type header value for JSON format. 2012-01-16 8:49 ` David Edmondson @ 2012-01-16 13:18 ` Tomi Ollila 0 siblings, 0 replies; 24+ messages in thread From: Tomi Ollila @ 2012-01-16 13:18 UTC (permalink / raw) To: David Edmondson, Austin Clements, Pieter Praet; +Cc: notmuch On Mon, 16 Jan 2012 08:49:03 +0000, David Edmondson <dme@dme.org> wrote: > On Sun, 15 Jan 2012 12:58:40 -0500, Austin Clements <amdragon@MIT.EDU> wrote: > > This, I suspect, brings us back to what may have been Dmitry's original > concern. If we codify the current behaviour then we're actually > _forcing_ clients to use inline content if it's present, because > otherwise they have no way to discover the charset/encoding used for the > raw part. > > That seems as though it could be a problem for some clients. > > > OTOH, I don't understand the encoding story for HTML, since the encoding > > can come from either a header or from the body of the HTML. Does this > > make it strictly necessary for the client to handle the encoding? Either from a header or from the body of the *HTTP*. Html meta tag is part of the header of the HTML document, body of the HTTP message. > If it can be specified by the content of a part rather than the part > headers, then I think that the client will have to be prepared to handle > it. > > Even if not, it might still be more effective to choose that approach - > it would remove the need to add arbitrary encoding support to the CLI > application. In case of w3m interface if charset is not told it defaults to iso-8859-1 as both input and output encoding. That is no problem in w3m -- it just doesn't do any conversion. But emacs thinks that it gets input in iso-8859-1 format and will attempt to convert that to whatever charset the window user has is using. If input was utf8 but emacs thinks input was latin1 then we get this infamous 'double-utf8'ied output (a subset of wtf-8 charset ;) (in case window charset is utf8) In case of w3m the content feed to w3m could be pre-encoded to utf-8 and w3m interface is told that charset is utf-8 -- w3m will now called using utf-8 as input and output encoding -- and at the end emacs does conversion from utf-8 to the window encoding (if needed). As mentioned in IRC: 2012-01-10 11:46 (UTC) xxxXX indeed, the headers should take precedence to meta tag, as defined in HTML4 So, complying clients takes the precedence from command line options (which, in case of command line clients makes perfect sense). Tomi ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2] Output unmodified Content-Type header value for JSON format. 2011-11-18 23:45 [PATCH] Output unmodified Content-Type header value for JSON format Dmitry Kurochkin 2011-11-19 1:58 ` Jameson Graef Rollins @ 2011-11-19 4:18 ` Dmitry Kurochkin 2011-11-19 12:59 ` David Bremner 1 sibling, 1 reply; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-19 4:18 UTC (permalink / raw) To: notmuch Before the change, notmuch used g_mime_content_type_to_string(3) function to output Content-Type header value. Turns out it outputs only "type/subtype" part and ignores all parameters. Also, if there is no Content-Type header, default "text/plain" value is used. JSON is supposed to be a "low-level" structured format and should not add missing values or throw away information. The patch changes notmuch show to use unmodified Content-Type value for JSON format. Also, no default value is added if the header is missing. Corresponding changes to Emacs UI are made to handle full Content-Type header values. The header is parsed using MIME `mail-header-parse-content-type' function. All message part rendering functions have access to full Content-Type value. In particular, this is important for `notmuch-show-mm-display-part-inline' which uses `mm-display-part' to display parts that notmuch-show does not handle. Expected results for the tests are updated accordingly. --- Changes in v2 since v1: * Use "text/plain; charset=us-ascii" for default Content-Type value in `notmuch-show-get-content-type' function as defined in RFC 2045. emacs/notmuch-show.el | 29 +++++++++++++++++++---------- notmuch-show.c | 14 ++++++++++++-- test/crypto | 23 ++++++++--------------- test/json | 6 +++--- test/maildir-sync | 1 - test/multipart | 36 ++++++++++++++++++------------------ 6 files changed, 60 insertions(+), 49 deletions(-) diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el index d5c95d8..cb801ae 100644 --- a/emacs/notmuch-show.el +++ b/emacs/notmuch-show.el @@ -261,120 +261,120 @@ message at DEPTH in the current thread." (if (and header-value (not (string-equal "" header-value))) (notmuch-show-insert-header header header-value)))) notmuch-message-headers) (save-excursion (save-restriction (narrow-to-region start (point-max)) (run-hooks 'notmuch-show-markup-headers-hook))))) (define-button-type 'notmuch-show-part-button-type 'action 'notmuch-show-part-button-action 'follow-link t 'face 'message-mml) (defun notmuch-show-insert-part-header (nth content-type declared-type &optional name comment) (let ((button)) (setq button (insert-button (concat "[ " (if name (concat name ": ") "") - declared-type - (if (not (string-equal declared-type content-type)) - (concat " (as " content-type ")") + (car declared-type) + (if (not (string-equal (car declared-type) (car content-type))) + (concat " (as " (car content-type) ")") "") (or comment "") " ]") :type 'notmuch-show-part-button-type :notmuch-part nth :notmuch-filename name)) (insert "\n") ;; return button button)) ;; Functions handling particular MIME parts. (defun notmuch-show-save-part (message-id nth &optional filename) (let ((process-crypto notmuch-show-process-crypto)) (with-temp-buffer (setq notmuch-show-process-crypto process-crypto) ;; Always acquires the part via `notmuch part', even if it is ;; available in the JSON output. (insert (notmuch-show-get-bodypart-internal message-id nth)) (let ((file (read-file-name "Filename to save as: " (or mailcap-download-directory "~/") nil nil filename))) ;; Don't re-compress .gz & al. Arguably we should make ;; `file-name-handler-alist' nil, but that would chop ;; ange-ftp, which is reasonable to use here. (mm-write-region (point-min) (point-max) file nil nil nil 'no-conversion t))))) (defun notmuch-show-mm-display-part-inline (msg part content-type content) "Use the mm-decode/mm-view functions to display a part in the current buffer, if possible." (let ((display-buffer (current-buffer))) (with-temp-buffer (insert content) - (let ((handle (mm-make-handle (current-buffer) (list content-type)))) + (let ((handle (mm-make-handle (current-buffer) content-type))) (set-buffer display-buffer) (if (and (mm-inlinable-p handle) (mm-inlined-p handle)) (progn (mm-display-part handle) t) nil))))) (defvar notmuch-show-multipart/alternative-discouraged '( ;; Avoid HTML parts. "text/html" ;; multipart/related usually contain a text/html part and some associated graphics. "multipart/related" )) (defun notmuch-show-multipart/*-to-list (part) - (mapcar '(lambda (inner-part) (plist-get inner-part :content-type)) + (mapcar '(lambda (inner-part) (car (notmuch-show-get-content-type inner-part))) (plist-get part :content))) (defun notmuch-show-multipart/alternative-choose (types) ;; Based on `mm-preferred-alternative-precedence'. (let ((seq types)) (dolist (pref (reverse notmuch-show-multipart/alternative-discouraged)) (dolist (elem (copy-sequence seq)) (when (string-match pref elem) (setq seq (nconc (delete elem seq) (list elem)))))) seq)) (defun notmuch-show-insert-part-multipart/alternative (msg part content-type nth depth declared-type) (notmuch-show-insert-part-header nth declared-type content-type nil) (let ((chosen-type (car (notmuch-show-multipart/alternative-choose (notmuch-show-multipart/*-to-list part)))) (inner-parts (plist-get part :content)) (start (point))) ;; This inserts all parts of the chosen type rather than just one, ;; but it's not clear that this is the wrong thing to do - which ;; should be chosen if there are more than one that match? (mapc (lambda (inner-part) - (let ((inner-type (plist-get inner-part :content-type))) + (let ((inner-type (notmuch-show-get-content-type inner-part))) (if (or notmuch-show-all-multipart/alternative-parts - (string= chosen-type inner-type)) + (string= chosen-type (car inner-type))) (notmuch-show-insert-bodypart msg inner-part depth) (notmuch-show-insert-part-header (plist-get inner-part :id) inner-type inner-type nil " (not shown)")))) inner-parts) (when notmuch-show-indent-multipart (indent-rigidly start (point) 1))) t) (defun notmuch-show-setup-w3m () "Instruct w3m how to retrieve content from a \"related\" part of a message." (interactive) (if (boundp 'w3m-cid-retrieve-function-alist) (unless (assq 'notmuch-show-mode w3m-cid-retrieve-function-alist) (push (cons 'notmuch-show-mode 'notmuch-show-w3m-cid-retrieve) w3m-cid-retrieve-function-alist))) (setq mm-inline-text-html-with-images t)) (defvar w3m-current-buffer) ;; From `w3m.el'. (defvar notmuch-show-w3m-cid-store nil) (make-variable-buffer-local 'notmuch-show-w3m-cid-store) @@ -557,41 +557,42 @@ current buffer, if possible." (set-buffer (get-file-buffer file)) (setq result (buffer-substring (point-min) (point-max))) (set-buffer-modified-p nil) (kill-buffer (current-buffer)) (delete-file file) result))) t) (defun notmuch-show-insert-part-application/octet-stream (msg part content-type nth depth declared-type) ;; If we can deduce a MIME type from the filename of the attachment, ;; do so and pass it on to the handler for that type. (if (plist-get part :filename) (let ((extension (file-name-extension (plist-get part :filename))) mime-type) (if extension (progn (mailcap-parse-mimetypes) (setq mime-type (mailcap-extension-to-mime extension)) (if (and mime-type (not (string-equal mime-type "application/octet-stream"))) - (notmuch-show-insert-bodypart-internal msg part mime-type nth depth content-type) + (notmuch-show-insert-bodypart-internal msg part (list mime-type) + nth depth content-type) nil)) nil)))) (defun notmuch-show-insert-part-application/* (msg part content-type nth depth declared-type ) ;; do not render random "application" parts (notmuch-show-insert-part-header nth content-type declared-type (plist-get part :filename))) (defun notmuch-show-insert-part-*/* (msg part content-type nth depth declared-type) ;; This handler _must_ succeed - it is the handler of last resort. (notmuch-show-insert-part-header nth content-type declared-type (plist-get part :filename)) (let ((content (notmuch-show-get-bodypart-content msg part nth))) (if content (notmuch-show-mm-display-part-inline msg part content-type content))) t) ;; Functions for determining how to handle MIME parts. (defun notmuch-show-split-content-type (content-type) (split-string content-type "/")) @@ -618,51 +619,51 @@ current buffer, if possible." (defun notmuch-show-get-bodypart-internal (message-id part-number) (let ((args '("show" "--format=raw")) (part-arg (format "--part=%s" part-number))) (setq args (append args (list part-arg))) (if notmuch-show-process-crypto (setq args (append args '("--decrypt")))) (setq args (append args (list message-id))) (with-temp-buffer (let ((coding-system-for-read 'no-conversion)) (progn (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args)) (buffer-string)))))) (defun notmuch-show-get-bodypart-content (msg part nth) (or (plist-get part :content) (notmuch-show-get-bodypart-internal (concat "id:" (plist-get msg :id)) nth))) ;; \f (defun notmuch-show-insert-bodypart-internal (msg part content-type nth depth declared-type) - (let ((handlers (notmuch-show-handlers-for content-type))) + (let ((handlers (notmuch-show-handlers-for (car content-type)))) ;; Run the content handlers until one of them returns a non-nil ;; value. (while (and handlers (not (funcall (car handlers) msg part content-type nth depth declared-type))) (setq handlers (cdr handlers)))) t) (defun notmuch-show-insert-bodypart (msg part depth) "Insert the body part PART at depth DEPTH in the current thread." - (let ((content-type (downcase (plist-get part :content-type))) + (let ((content-type (notmuch-show-get-content-type part)) (nth (plist-get part :id))) (notmuch-show-insert-bodypart-internal msg part content-type nth depth content-type)) ;; Some of the body part handlers leave point somewhere up in the ;; part, so we make sure that we're down at the end. (goto-char (point-max)) ;; Ensure that the part ends with a carriage return. (if (not (bolp)) (insert "\n"))) (defun notmuch-show-insert-body (msg body depth) "Insert the body BODY at depth DEPTH in the current thread." (mapc '(lambda (part) (notmuch-show-insert-bodypart msg part depth)) body)) (defun notmuch-show-make-symbol (type) (make-symbol (concat "notmuch-show-" type))) (defun notmuch-show-strip-re (string) (replace-regexp-in-string "\\([Rr]e: *\\)+" "" string)) (defvar notmuch-show-previous-subject "") @@ -1054,40 +1055,48 @@ All currently available key bindings: (save-excursion (notmuch-show-move-to-message-top) (get-text-property (point) :notmuch-message-properties))) (defun notmuch-show-set-prop (prop val &optional props) (let ((inhibit-read-only t) (props (or props (notmuch-show-get-message-properties)))) (plist-put props prop val) (notmuch-show-set-message-properties props))) (defun notmuch-show-get-prop (prop &optional props) (let ((props (or props (notmuch-show-get-message-properties)))) (plist-get props prop))) (defun notmuch-show-get-message-id () "Return the message id of the current message." (concat "id:\"" (notmuch-show-get-prop :id) "\"")) +(defun notmuch-show-get-content-type (&optional props) + "Return parsed Content-Type of the given message, or part, or +current message if no `props` is given. If there is no +Content-Type header, it defaults to +\"text/plain; charset=us-ascii\" as defined in RFC 2045." + (mail-header-parse-content-type (or (notmuch-show-get-prop :content-type props) + "text/plain; charset=us-ascii"))) + ;; dme: Would it make sense to use a macro for many of these? (defun notmuch-show-get-filename () "Return the filename of the current message." (notmuch-show-get-prop :filename)) (defun notmuch-show-get-header (header) "Return the named header of the current message, if any." (plist-get (notmuch-show-get-prop :headers) header)) (defun notmuch-show-get-cc () (notmuch-show-get-header :Cc)) (defun notmuch-show-get-date () (notmuch-show-get-header :Date)) (defun notmuch-show-get-from () (notmuch-show-get-header :From)) (defun notmuch-show-get-subject () diff --git a/notmuch-show.c b/notmuch-show.c index 603992a..da3e87f 100644 --- a/notmuch-show.c +++ b/notmuch-show.c @@ -3,40 +3,42 @@ * Copyright © 2009 Carl Worth * * This program is free software: you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation, either version 3 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see http://www.gnu.org/licenses/ . * * Author: Carl Worth <cworth@cworth.org> */ #include "notmuch-client.h" +static const char HEADER_CONTENT_TYPE[] = "Content-Type"; + static void format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent); static void format_headers_text (const void *ctx, notmuch_message_t *message); static void format_headers_message_part_text (GMimeMessage *message); static void format_part_start_text (GMimeObject *part, int *part_count); static void format_part_content_text (GMimeObject *part); static void format_part_end_text (GMimeObject *part); @@ -640,42 +642,50 @@ format_part_sigstatus_json (const GMimeSignatureValidity* validity) } printf ("}"); signer = signer->next; } printf ("]"); talloc_free (ctx_quote); } static void format_part_content_json (GMimeObject *part) { GMimeContentType *content_type = g_mime_object_get_content_type (GMIME_OBJECT (part)); GMimeStream *stream_memory = g_mime_stream_mem_new (); const char *cid = g_mime_object_get_content_id (part); void *ctx = talloc_new (NULL); GByteArray *part_content; - printf (", \"content-type\": %s", - json_quote_str (ctx, g_mime_content_type_to_string (content_type))); + { + /* Output full Content-Type header value, + * g_mime_content_type_to_string(3) does not include + * parameters. Content-Type header may be missing, + * g_mime_object_get_content_type(3) defaults to "text/plain" + * in this case. */ + const char *const h = g_mime_object_get_header (part, HEADER_CONTENT_TYPE); + if (h) + printf (", \"content-type\": %s", json_quote_str (ctx, h)); + } if (cid != NULL) printf(", \"content-id\": %s", json_quote_str (ctx, cid)); if (GMIME_IS_PART (part)) { const char *filename = g_mime_part_get_filename (GMIME_PART (part)); if (filename) printf (", \"filename\": %s", json_quote_str (ctx, filename)); } if (g_mime_content_type_is_type (content_type, "text", "*") && !g_mime_content_type_is_type (content_type, "text", "html")) { show_text_part_content (part, stream_memory); part_content = g_mime_stream_mem_get_byte_array (GMIME_STREAM_MEM (stream_memory)); printf (", \"content\": %s", json_quote_chararray (ctx, (char *) part_content->data, part_content->len)); } else if (g_mime_content_type_is_type (content_type, "multipart", "*")) diff --git a/test/crypto b/test/crypto index 0af4aa8..b923d22 100755 --- a/test/crypto +++ b/test/crypto @@ -40,111 +40,108 @@ test_expect_success 'emacs delivery of signed message' \ test_begin_subtest "signature verification" output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "good", "fingerprint": "'$FINGERPRINT'", "created": 946728000}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "signature verification with full owner trust" # give the key full owner trust echo "${FINGERPRINT}:6:" | gpg --no-tty --import-ownertrust >>"$GNUPGHOME"/trust.log 2>&1 gpg --no-tty --check-trustdb >>"$GNUPGHOME"/trust.log 2>&1 output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "good", "fingerprint": "'$FINGERPRINT'", "created": 946728000, "userid": " Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)"}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "signature verification with signer key unavailable" # move the gnupghome temporarily out of the way mv "${GNUPGHOME}"{,.bak} output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "error", "keyid": "'$(echo $FINGERPRINT | cut -c 25-)'", "errors": 2}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" mv "${GNUPGHOME}"{.bak,} # create a test encrypted message with attachment cat <<EOF >TESTATTACHMENT This is a test file. EOF test_expect_success 'emacs delivery of encrypted message with attachment' \ 'emacs_deliver_message \ "test encrypted message 001" \ "This is a test encrypted message.\n" \ "(mml-attach-file \"TESTATTACHMENT\") (mml-secure-message-encrypt)"' test_begin_subtest "decryption, --format=text" @@ -181,138 +178,135 @@ test_expect_equal \ test_begin_subtest "decryption, --format=json" output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["encrypted","inbox"], "headers": {"Subject": "test encrypted message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "encstatus": [{"status": "good"}], "sigstatus": [], - "content-type": "multipart/encrypted", + "content-type": "multipart/encrypted; boundary=\"==-=-=\";\tprotocol=\"application/pgp-encrypted\"", "content": [{"id": 2, "content-type": "application/pgp-encrypted"}, {"id": 3, - "content-type": "multipart/mixed", + "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [{"id": 4, - "content-type": "text/plain", "content": "This is a test encrypted message.\n"}, {"id": 5, "content-type": "application/octet-stream", "filename": "TESTATTACHMENT"}]}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "decryption, --format=json, --part=4" output=$(notmuch show --format=json --part=4 --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='{"id": 4, - "content-type": "text/plain", "content": "This is a test encrypted message.\n"}' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "decrypt attachment (--part=5 --format=raw)" notmuch show \ --format=raw \ --part=5 \ --decrypt \ subject:"test encrypted message 001" >OUTPUT test_expect_equal_file OUTPUT TESTATTACHMENT test_begin_subtest "decryption failure with missing key" mv "${GNUPGHOME}"{,.bak} output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["encrypted","inbox"], "headers": {"Subject": "test encrypted message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "encstatus": [{"status": "bad"}], - "content-type": "multipart/encrypted", + "content-type": "multipart/encrypted; boundary=\"==-=-=\";\tprotocol=\"application/pgp-encrypted\"", "content": [{"id": 2, "content-type": "application/pgp-encrypted"}, {"id": 3, "content-type": "application/octet-stream"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" mv "${GNUPGHOME}"{.bak,} test_expect_success 'emacs delivery of encrypted + signed message' \ 'emacs_deliver_message \ "test encrypted message 002" \ "This is another test encrypted message.\n" \ "(mml-secure-message-sign-encrypt)"' test_begin_subtest "decryption + signature verification" output=$(notmuch show --format=json --decrypt subject:"test encrypted message 002" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["encrypted","inbox"], "headers": {"Subject": "test encrypted message 002", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "encstatus": [{"status": "good"}], "sigstatus": [{"status": "good", "fingerprint": "'$FINGERPRINT'", "created": 946728000, "userid": " Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)"}], - "content-type": "multipart/encrypted", + "content-type": "multipart/encrypted; boundary=\"=-=-=\";\tprotocol=\"application/pgp-encrypted\"", "content": [{"id": 2, "content-type": "application/pgp-encrypted"}, {"id": 3, - "content-type": "text/plain", "content": "This is another test encrypted message.\n"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "reply to encrypted message" output=$(notmuch reply --decrypt subject:"test encrypted message 002" \ | grep -v -e '^In-Reply-To:' -e '^References:') expected='From: Notmuch Test Suite <test_suite@notmuchmail.org> Subject: Re: test encrypted message 002 On 01 Jan 2000 12:00:00 -0000, Notmuch Test Suite <test_suite@notmuchmail.org> wrote: > This is another test encrypted message.' test_expect_equal \ "$output" \ "$expected" test_begin_subtest "signature verification with revoked key" # generate revocation certificate and load it to revoke key @@ -327,32 +321,31 @@ y | gpg --no-tty --quiet --import output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ | sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "XXXXX", "match": true, "filename": "YYYYY", "timestamp": 946728000, "date_relative": "2000-01-01", "tags": ["inbox","signed"], "headers": {"Subject": "test signed message 001", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "test_suite@notmuchmail.org", "Cc": "", "Bcc": "", "Date": "01 Jan 2000 12:00:00 -0000"}, "body": [{"id": 1, "sigstatus": [{"status": "error", "keyid": "6D92612D94E46381", "errors": 8}], - "content-type": "multipart/signed", + "content-type": "multipart/signed; boundary=\"=-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [{"id": 2, - "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature"}]}]}, []]]]' test_expect_equal \ "$output" \ "$expected" test_done diff --git a/test/json b/test/json index 592b068..64f35cf 100755 --- a/test/json +++ b/test/json @@ -1,50 +1,50 @@ #!/usr/bin/env bash test_description="--format=json output" . ./test-lib.sh test_begin_subtest "Show message: json" add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-show-message\"" output=$(notmuch show --format=json "json-show-message") -test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]" +test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content\": \"json-show-message\n\"}]}, []]]]" test_begin_subtest "Search message: json" add_message "[subject]=\"json-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-search-message\"" output=$(notmuch search --format=json "json-search-message" | notmuch_search_sanitize) test_expect_equal "$output" "[{\"thread\": \"XXX\", \"timestamp\": 946728000, \"matched\": 1, \"total\": 1, \"authors\": \"Notmuch Test Suite\", \"subject\": \"json-search-subject\", \"tags\": [\"inbox\", \"unread\"]}]" test_begin_subtest "Show message: json, utf-8" add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\"" output=$(notmuch show --format=json "jsön-show-méssage") -test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]" +test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content\": \"jsön-show-méssage\n\"}]}, []]]]" test_begin_subtest "Show message: json, inline attachment filename" subject='json-show-inline-attachment-filename' id="json-show-inline-attachment-filename@notmuchmail.org" emacs_deliver_message \ "$subject" \ 'This is a test message with inline attachment with a filename' \ "(mml-attach-file \"$TEST_DIRECTORY/README\" nil nil \"inline\") (message-goto-eoh) (insert \"Message-ID: <$id>\n\")" output=$(notmuch show --format=json "id:$id") filename=$(notmuch search --output=files "id:$id") -test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]" +test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed; boundary=\\\"=-=-=\\\"\", \"content\": [{\"id\": 2, \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]" test_begin_subtest "Search message: json, utf-8" add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\"" output=$(notmuch search --format=json "jsön-search-méssage" | notmuch_search_sanitize) test_expect_equal "$output" "[{\"thread\": \"XXX\", \"timestamp\": 946728000, \"matched\": 1, \"total\": 1, \"authors\": \"Notmuch Test Suite\", \"subject\": \"json-search-utf8-body-sübjéct\", \"tags\": [\"inbox\", \"unread\"]}]" test_done diff --git a/test/maildir-sync b/test/maildir-sync index a60854f..c7ca22f 100755 --- a/test/maildir-sync +++ b/test/maildir-sync @@ -41,41 +41,40 @@ add_message [subject]='"Adding replied tag"' [filename]='adding-replied-tag:2,S' notmuch tag +replied subject:"Adding replied tag" output=$(cd ${MAIL_DIR}/cur; ls -1 adding-replied*) test_expect_equal "$output" "adding-replied-tag:2,RS" test_begin_subtest "notmuch show works with renamed file (without notmuch new)" output=$(notmuch show --format=json id:${gen_msg_id} | filter_show_json) test_expect_equal "$output" '[[[{"id": "adding-replied-tag@notmuch-test-suite", "match": true, "filename": "MAIL_DIR/cur/adding-replied-tag:2,RS", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["inbox","replied"], "headers": {"Subject": "Adding replied tag", "From": "Notmuch Test Suite <test_suite@notmuchmail.org>", "To": "Notmuch Test Suite <test_suite@notmuchmail.org>", "Cc": "", "Bcc": "", "Date": "Tue, 05 Jan 2001 15:43:57 -0000"}, "body": [{"id": 1, -"content-type": "text/plain", "content": "This is just a test message (#3)\n"}]}, []]]]' test_expect_success 'notmuch reply works with renamed file (without notmuch new)' 'notmuch reply id:${gen_msg_id}' test_begin_subtest "notmuch new detects no file rename after tag->flag synchronization" output=$(NOTMUCH_NEW) test_expect_equal "$output" "No new mail." test_begin_subtest "When read, message moved from new to cur" add_message [subject]='"Message to move to cur"' [date]='"Sat, 01 Jan 2000 12:00:00 -0000"' [filename]='message-to-move-to-cur' [dir]=new notmuch tag -unread subject:"Message to move to cur" output=$(cd "$MAIL_DIR/cur"; ls message-to-move*) test_expect_equal "$output" "message-to-move-to-cur:2,S" test_begin_subtest "No rename should be detected by notmuch new" output=$(NOTMUCH_NEW) test_expect_equal "$output" "No new mail." # (*) If notmuch new was not run we've got "Processed 1 file in almost # no time" here. The reason is that removing unread tag in a previous diff --git a/test/multipart b/test/multipart index f83526b..ca4db71 100755 --- a/test/multipart +++ b/test/multipart @@ -306,140 +306,140 @@ test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=text --part=9, pgp signature (unverified)" notmuch show --format=text --part=9 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT cat <<EOF >EXPECTED \fpart{ ID: 9, Content-type: application/pgp-signature Non-text part: application/pgp-signature \fpart} EOF test_expect_equal_file OUTPUT EXPECTED test_expect_success \ "--format=text --part=8, no part, expect error" \ "notmuch show --format=text --part=8 'id:87liy5ap00.fsf@yoom.home.cworth.org'" test_begin_subtest "--format=json --part=0, full message" notmuch show --format=json --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Cc": "", "Bcc": "", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [ -{"id": 1, "content-type": "multipart/signed", "content": [ -{"id": 2, "content-type": "multipart/mixed", "content": [ +{"id": 1, "content-type": "multipart/signed; boundary=\"==-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [ +{"id": 2, "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [ {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"}, +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"}]}, {"id": 9, "content-type": "application/pgp-signature"}]}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=1, message body" notmuch show --format=json --part=1 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 1, "content-type": "multipart/signed", "content": [ -{"id": 2, "content-type": "multipart/mixed", "content": [ +{"id": 1, "content-type": "multipart/signed; boundary=\"==-=-=\";\tmicalg=pgp-sha1; protocol=\"application/pgp-signature\"", "content": [ +{"id": 2, "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [ {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"}, +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"}]}, {"id": 9, "content-type": "application/pgp-signature"}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=2, multipart/mixed" notmuch show --format=json --part=2 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 2, "content-type": "multipart/mixed", "content": [ +{"id": 2, "content-type": "multipart/mixed; boundary=\"=-=-=\"", "content": [ {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]} +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"}, +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=3, rfc822 part" notmuch show --format=json --part=3 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [ -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=4, rfc822's multipart/alternative" notmuch show --format=json --part=4 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 4, "content-type": "multipart/alternative", "content": [ +{"id": 4, "content-type": "multipart/alternative; boundary=\"==-=-==\"", "content": [ {"id": 5, "content-type": "text/html"}, {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=5, rfc822's html part" notmuch show --format=json --part=5 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 5, "content-type": "text/html"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=6, rfc822's text part" notmuch show --format=json --part=6 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=7, inline attachment" notmuch show --format=json --part=7 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"} +{"id": 7, "filename": "attachment", "content": "This is a text attachment.\n"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=8, plain text part" notmuch show --format=json --part=8 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED -{"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"} +{"id": 8, "content": "And this message is signed.\n\n-Carl\n"} EOF test_expect_equal_file OUTPUT EXPECTED test_begin_subtest "--format=json --part=9, pgp signature (unverified)" notmuch show --format=json --part=9 'id:87liy5ap00.fsf@yoom.home.cworth.org' | sed 's|{"id":|\n{"id":|g' >OUTPUT echo >>OUTPUT # expect *no* newline at end of output cat <<EOF >EXPECTED {"id": 9, "content-type": "application/pgp-signature"} EOF test_expect_equal_file OUTPUT EXPECTED test_expect_success \ "--format=json --part=10, no part, expect error" \ "notmuch show --format=json --part=10 'id:87liy5ap00.fsf@yoom.home.cworth.org'" test_begin_subtest "--format=raw" notmuch show --format=raw 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT test_expect_equal_file OUTPUT "${MAIL_DIR}"/multipart -- 1.7.7.3 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2] Output unmodified Content-Type header value for JSON format. 2011-11-19 4:18 ` [PATCH v2] " Dmitry Kurochkin @ 2011-11-19 12:59 ` David Bremner 2011-11-20 18:35 ` Dmitry Kurochkin 0 siblings, 1 reply; 24+ messages in thread From: David Bremner @ 2011-11-19 12:59 UTC (permalink / raw) To: Dmitry Kurochkin, notmuch On Sat, 19 Nov 2011 08:18:41 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > Before the change, notmuch used g_mime_content_type_to_string(3) > function to output Content-Type header value. Turns out it outputs > only "type/subtype" part and ignores all parameters. Also, if there > is no Content-Type header, default "text/plain" value is used. Hi Dmitry; I haven't analyzed the substance of your patch yet, but I did have a couple thoughts while reading your mail. - It seems that every time we change the json format, we have a round of suffering because people are unable to detect a mismatch between their emacs code and the cli. Not that this is your problem necessarily, but it would be nice if someone (TM), would come up with some version info for the json output, and a patch to check it on the emacs side. - The previous point is a bit of a counterargument to this, but in general, I think I prefer patches that modify the core seperate from those that do emacs (or python, or ...) stuff. - I understand you want to make your patches reviewable without applying by including lots of context, but at a certain point it has actually the opposite effect for me. I just don't read 900+ line emails ;). Of course, I can still apply the patch and look at it, so it's really up to you. d ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v2] Output unmodified Content-Type header value for JSON format. 2011-11-19 12:59 ` David Bremner @ 2011-11-20 18:35 ` Dmitry Kurochkin 2012-01-12 17:08 ` Pieter Praet 0 siblings, 1 reply; 24+ messages in thread From: Dmitry Kurochkin @ 2011-11-20 18:35 UTC (permalink / raw) To: David Bremner, notmuch On Sat, 19 Nov 2011 08:59:29 -0400, David Bremner <david@tethera.net> wrote: > On Sat, 19 Nov 2011 08:18:41 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > Before the change, notmuch used g_mime_content_type_to_string(3) > > function to output Content-Type header value. Turns out it outputs > > only "type/subtype" part and ignores all parameters. Also, if there > > is no Content-Type header, default "text/plain" value is used. > > Hi Dmitry; > > I haven't analyzed the substance of your patch yet, but I did have a > couple thoughts while reading your mail. > > - It seems that every time we change the json format, we have a round of > suffering because people are unable to detect a mismatch between their > emacs code and the cli. Not that this is your problem necessarily, but > it would be nice if someone (TM), would come up with some version info > for the json output, and a patch to check it on the emacs side. > IMO this is a good idea. > - The previous point is a bit of a counterargument to this, but in > general, I think I prefer patches that modify the core seperate from > those that do emacs (or python, or ...) stuff. > I couls separate it. I made is a single patch to avoid having a revision with broken emacs UI (and tests). Regards, Dmitry > - I understand you want to make your patches reviewable without applying > by including lots of context, but at a certain point it has actually > the opposite effect for me. I just don't read 900+ line emails ;). Of > course, I can still apply the patch and look at it, so it's really up > to you. > > d ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v2] Output unmodified Content-Type header value for JSON format. 2011-11-20 18:35 ` Dmitry Kurochkin @ 2012-01-12 17:08 ` Pieter Praet 2012-01-13 3:16 ` David Bremner 0 siblings, 1 reply; 24+ messages in thread From: Pieter Praet @ 2012-01-12 17:08 UTC (permalink / raw) To: Dmitry Kurochkin, David Bremner, notmuch On Sun, 20 Nov 2011 22:35:42 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > On Sat, 19 Nov 2011 08:59:29 -0400, David Bremner <david@tethera.net> wrote: > > On Sat, 19 Nov 2011 08:18:41 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote: > > > Before the change, notmuch used g_mime_content_type_to_string(3) > > > function to output Content-Type header value. Turns out it outputs > > > only "type/subtype" part and ignores all parameters. Also, if there > > > is no Content-Type header, default "text/plain" value is used. > > > > Hi Dmitry; > > > > I haven't analyzed the substance of your patch yet, but I did have a > > couple thoughts while reading your mail. > > > > - It seems that every time we change the json format, we have a round of > > suffering because people are unable to detect a mismatch between their > > emacs code and the cli. Not that this is your problem necessarily, but > > it would be nice if someone (TM), would come up with some version info > > for the json output, and a patch to check it on the emacs side. > > > > IMO this is a good idea. > > > - The previous point is a bit of a counterargument to this, but in > > general, I think I prefer patches that modify the core seperate from > > those that do emacs (or python, or ...) stuff. > > > > I couls separate it. I made is a single patch to avoid having a > revision with broken emacs UI (and tests). > I'd like to propose to always apply patch series on a *topic* branch which would then be merged back into 'master', thus avoiding this issue altogether whilst making it more obvious which patches belong together (eg. for easier cross-referencing with the ML). > Regards, > Dmitry > > > - I understand you want to make your patches reviewable without applying > > by including lots of context, but at a certain point it has actually > > the opposite effect for me. I just don't read 900+ line emails ;). Of > > course, I can still apply the patch and look at it, so it's really up > > to you. > > > > d > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch Peace -- Pieter ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v2] Output unmodified Content-Type header value for JSON format. 2012-01-12 17:08 ` Pieter Praet @ 2012-01-13 3:16 ` David Bremner 2012-01-14 8:59 ` Pieter Praet 0 siblings, 1 reply; 24+ messages in thread From: David Bremner @ 2012-01-13 3:16 UTC (permalink / raw) To: Pieter Praet, notmuch On Thu, 12 Jan 2012 18:08:08 +0100, Pieter Praet <pieter@praet.org> wrote: > > > - The previous point is a bit of a counterargument to this, but in > > I couls separate it. I made is a single patch to avoid having a > > revision with broken emacs UI (and tests). > > > I'd like to propose to always apply patch series on a *topic* branch > which would then be merged back into 'master', thus avoiding this issue > altogether whilst making it more obvious which patches belong together > (eg. for easier cross-referencing with the ML). Hi Pieter; Sorry, I must have lost the thread somewhere. What application are you thinking about, and what issue do you think that topic branches avoid? d ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v2] Output unmodified Content-Type header value for JSON format. 2012-01-13 3:16 ` David Bremner @ 2012-01-14 8:59 ` Pieter Praet 0 siblings, 0 replies; 24+ messages in thread From: Pieter Praet @ 2012-01-14 8:59 UTC (permalink / raw) To: David Bremner, notmuch On Thu, 12 Jan 2012 23:16:29 -0400, David Bremner <david@tethera.net> wrote: > On Thu, 12 Jan 2012 18:08:08 +0100, Pieter Praet <pieter@praet.org> wrote: > > > > - The previous point is a bit of a counterargument to this, but in > > > > I couls separate it. I made is a single patch to avoid having a > > > revision with broken emacs UI (and tests). > > > > > > I'd like to propose to always apply patch series on a *topic* branch > > which would then be merged back into 'master', thus avoiding this issue > > altogether whilst making it more obvious which patches belong together > > (eg. for easier cross-referencing with the ML). > > Hi Pieter; > > Sorry, I must have lost the thread somewhere. What application are you > thinking about, and what issue do you think that topic branches avoid? > No specific application; In the case of patch series, it may occur that an individual commit leaves the code in a broken state. One might argue that devs need to ensure that the project builds cleanly at every commit, but this can be prohibitively time-consuming or even simply impossible due to the need to keep changes atomic and area-specific (eg. changes to the binary and the Emacs UI should be separate commits irrespective of their relation). So it would make sense to consistently apply (certain, or perhaps all?) patch series on an ad-hoc branch which would then be merged back into 'master', instead of being directly applied on 'master'. Aside from preventing broken builds (as was Dmitry's reason to consolidate a series touching on multiple distinct "spheres" of the code into a single patch), series of related commits would also be clearly delineated/clustered, facilitating source archaeology. (and all without making the commit log exude the odor of entropy incarnate) > d > Peace -- Pieter ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2012-01-16 13:18 UTC | newest] Thread overview: 24+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-11-18 23:45 [PATCH] Output unmodified Content-Type header value for JSON format Dmitry Kurochkin 2011-11-19 1:58 ` Jameson Graef Rollins 2011-11-19 2:42 ` Dmitry Kurochkin 2011-11-19 4:59 ` Austin Clements 2011-11-19 5:26 ` Dmitry Kurochkin 2011-11-19 18:58 ` Austin Clements 2011-11-20 18:52 ` Dmitry Kurochkin 2011-11-19 10:49 ` Jameson Graef Rollins 2011-11-20 18:32 ` Dmitry Kurochkin 2011-11-20 20:10 ` Jameson Graef Rollins 2011-11-23 3:40 ` Austin Clements 2012-01-12 17:07 ` Pieter Praet 2012-01-12 17:28 ` Austin Clements 2012-01-14 9:19 ` Pieter Praet 2012-01-15 11:52 ` David Edmondson 2012-01-15 17:58 ` Austin Clements 2012-01-16 8:49 ` David Edmondson 2012-01-16 13:18 ` Tomi Ollila 2011-11-19 4:18 ` [PATCH v2] " Dmitry Kurochkin 2011-11-19 12:59 ` David Bremner 2011-11-20 18:35 ` Dmitry Kurochkin 2012-01-12 17:08 ` Pieter Praet 2012-01-13 3:16 ` David Bremner 2012-01-14 8:59 ` Pieter Praet
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).