* [PATCH v4 0/3] indicate length of omitted body content
@ 2012-12-09 12:56 Mark Walters
2012-12-09 12:56 ` [PATCH v4 1/3] test: normalize only message filenames in show json Mark Walters
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Mark Walters @ 2012-12-09 12:56 UTC (permalink / raw)
To: notmuch
This is a (trivially) rebased version of
id:1344428872-12374-1-git-send-email-novalazy@gmail.com (There was one
bit of context clash, and a small change to one of the sexp tests to
mirror the same change for the json test)
The patch is Peter's not mine so as a reviewer +1 from me.
Best wishes
Mark
Peter Wang (3):
test: normalize only message filenames in show json
show: indicate length, encoding of omitted body content
test: conform to content length, encoding fields
devel/schemata | 9 ++++++++-
notmuch-show.c | 14 ++++++++++++++
test/crypto | 30 +++++++++++++++++++++---------
test/json | 4 +++-
test/multipart | 11 ++++++-----
test/sexp | 2 +-
test/test-lib.sh | 2 +-
7 files changed, 54 insertions(+), 18 deletions(-)
--
1.7.9.1
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v4 1/3] test: normalize only message filenames in show json 2012-12-09 12:56 [PATCH v4 0/3] indicate length of omitted body content Mark Walters @ 2012-12-09 12:56 ` Mark Walters 2012-12-09 12:56 ` [PATCH v4 2/3] show: indicate length, encoding of omitted body content Mark Walters ` (2 subsequent siblings) 3 siblings, 0 replies; 8+ messages in thread From: Mark Walters @ 2012-12-09 12:56 UTC (permalink / raw) To: notmuch From: Peter Wang <novalazy@gmail.com> notmuch_json_show_sanitize replaced "filename" field values even in part structures, where the value is predictable. Make it only normalize the filename value if it is an absolute path (begins with slash), which is true of the Maildir filenames that were intended to be normalized away. --- test/multipart | 2 +- test/test-lib.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/test/multipart b/test/multipart index 0527f84..344ed81 100755 --- a/test/multipart +++ b/test/multipart @@ -630,7 +630,7 @@ cat <<EOF >EXPECTED "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, {"id": 7, "content-type": "text/plain", - "filename": "YYYYY", + "filename": "attachment", "content": "This is a text attachment.\n"}, {"id": 8, "content-type": "text/plain", diff --git a/test/test-lib.sh b/test/test-lib.sh index fd64736..6ce3b31 100644 --- a/test/test-lib.sh +++ b/test/test-lib.sh @@ -600,7 +600,7 @@ notmuch_json_show_sanitize () { sed \ -e 's|"id": "[^"]*",|"id": "XXXXX",|g' \ - -e 's|"filename": "[^"]*",|"filename": "YYYYY",|g' + -e 's|"filename": "/[^"]*",|"filename": "YYYYY",|g' } # End of notmuch helper functions -- 1.7.9.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v4 2/3] show: indicate length, encoding of omitted body content 2012-12-09 12:56 [PATCH v4 0/3] indicate length of omitted body content Mark Walters 2012-12-09 12:56 ` [PATCH v4 1/3] test: normalize only message filenames in show json Mark Walters @ 2012-12-09 12:56 ` Mark Walters 2012-12-09 12:56 ` [PATCH v4 3/3] test: conform to content length, encoding fields Mark Walters 2012-12-09 19:21 ` [PATCH v4 0/3] indicate length of omitted body content Mark Walters 3 siblings, 0 replies; 8+ messages in thread From: Mark Walters @ 2012-12-09 12:56 UTC (permalink / raw) To: notmuch From: Peter Wang <novalazy@gmail.com> If a leaf part's body content is omitted, return the encoded length and transfer encoding in --format=json output. This information may be used by the consumer, e.g. to decide whether to download a large attachment over a slow link. Returning the _encoded_ content length is more efficient than returning the _decoded_ content length. Returning the transfer encoding allows the consumer to estimate the decoded content length. --- devel/schemata | 9 ++++++++- notmuch-show.c | 14 ++++++++++++++ 2 files changed, 22 insertions(+), 1 deletions(-) diff --git a/devel/schemata b/devel/schemata index d1ab983..7d167be 100644 --- a/devel/schemata +++ b/devel/schemata @@ -75,7 +75,14 @@ part = { # A leaf part's body content is optional, but may be included if # it can be correctly encoded as a string. Consumers should use # this in preference to fetching the part content separately. - content?: string + content?: string, + # If a leaf part's body content is not included, the length of + # the encoded content (in bytes) may be given instead. + content-length?: int, + # If a leaf part's body content is not included, its transfer encoding + # may be given. Using this and the encoded content length, it is + # possible for the consumer to estimate the decoded content length. + content-transfer-encoding?: string } # The headers of a message or part (format_headers_sprinter with reply = FALSE) diff --git a/notmuch-show.c b/notmuch-show.c index a83fef9..4aeb949 100644 --- a/notmuch-show.c +++ b/notmuch-show.c @@ -692,6 +692,20 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node, sp->map_key (sp, "content"); sp->string_len (sp, (char *) part_content->data, part_content->len); g_object_unref (stream_memory); + } else { + GMimeDataWrapper *wrapper = g_mime_part_get_content_object (GMIME_PART (node->part)); + GMimeStream *stream = g_mime_data_wrapper_get_stream (wrapper); + ssize_t length = g_mime_stream_length (stream); + const char *cte = g_mime_object_get_header (meta, "content-transfer-encoding"); + + if (length >= 0) { + sp->map_key (sp, "content-length"); + sp->integer (sp, length); + } + if (cte) { + sp->map_key (sp, "content-transfer-encoding"); + sp->string (sp, cte); + } } } else if (GMIME_IS_MULTIPART (node->part)) { sp->map_key (sp, "content"); -- 1.7.9.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v4 3/3] test: conform to content length, encoding fields 2012-12-09 12:56 [PATCH v4 0/3] indicate length of omitted body content Mark Walters 2012-12-09 12:56 ` [PATCH v4 1/3] test: normalize only message filenames in show json Mark Walters 2012-12-09 12:56 ` [PATCH v4 2/3] show: indicate length, encoding of omitted body content Mark Walters @ 2012-12-09 12:56 ` Mark Walters 2012-12-09 19:21 ` [PATCH v4 0/3] indicate length of omitted body content Mark Walters 3 siblings, 0 replies; 8+ messages in thread From: Mark Walters @ 2012-12-09 12:56 UTC (permalink / raw) To: notmuch From: Peter Wang <novalazy@gmail.com> Update tests to expect content-length and content-transfer-encoding fields in show --format=json output, for leaf parts with omitted body content. --- test/crypto | 30 +++++++++++++++++++++--------- test/json | 4 +++- test/multipart | 9 +++++---- test/sexp | 2 +- 4 files changed, 30 insertions(+), 15 deletions(-) diff --git a/test/crypto b/test/crypto index 5dd14c4..aa96ec2 100755 --- a/test/crypto +++ b/test/crypto @@ -61,7 +61,8 @@ expected='[[[{"id": "XXXXX", "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, - "content-type": "application/pgp-signature"}]}]}, + "content-type": "application/pgp-signature", + "content-length": 315}]}]}, []]]]' test_expect_equal_json \ "$output" \ @@ -95,7 +96,8 @@ expected='[[[{"id": "XXXXX", "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, - "content-type": "application/pgp-signature"}]}]}, + "content-type": "application/pgp-signature", + "content-length": 315}]}]}, []]]]' test_expect_equal_json \ "$output" \ @@ -127,7 +129,8 @@ expected='[[[{"id": "XXXXX", "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, - "content-type": "application/pgp-signature"}]}]}, + "content-type": "application/pgp-signature", + "content-length": 315}]}]}, []]]]' test_expect_equal_json \ "$output" \ @@ -196,7 +199,8 @@ expected='[[[{"id": "XXXXX", "sigstatus": [], "content-type": "multipart/encrypted", "content": [{"id": 2, - "content-type": "application/pgp-encrypted"}, + "content-type": "application/pgp-encrypted", + "content-length": 11}, {"id": 3, "content-type": "multipart/mixed", "content": [{"id": 4, @@ -204,6 +208,8 @@ expected='[[[{"id": "XXXXX", "content": "This is a test encrypted message.\n"}, {"id": 5, "content-type": "application/octet-stream", + "content-length": 28, + "content-transfer-encoding": "base64", "filename": "TESTATTACHMENT"}]}]}]}, []]]]' test_expect_equal_json \ @@ -231,9 +237,11 @@ test_expect_equal_file OUTPUT TESTATTACHMENT test_begin_subtest "decryption failure with missing key" mv "${GNUPGHOME}"{,.bak} +# The length of the encrypted attachment varies so must be normalized. output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ - | sed -e 's|"created": [1234567890]*|"created": 946728000|') + | sed -e 's|"created": [1234567890]*|"created": 946728000|' \ + | sed -e 's|"content-length": 6[1234567890]*|"content-length": 652|') expected='[[[{"id": "XXXXX", "match": true, "excluded": false, @@ -249,9 +257,11 @@ expected='[[[{"id": "XXXXX", "encstatus": [{"status": "bad"}], "content-type": "multipart/encrypted", "content": [{"id": 2, - "content-type": "application/pgp-encrypted"}, + "content-type": "application/pgp-encrypted", + "content-length": 11}, {"id": 3, - "content-type": "application/octet-stream"}]}]}, + "content-type": "application/octet-stream", + "content-length": 652}]}]}, []]]]' test_expect_equal_json \ "$output" \ @@ -287,7 +297,8 @@ expected='[[[{"id": "XXXXX", "userid": " Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)"}], "content-type": "multipart/encrypted", "content": [{"id": 2, - "content-type": "application/pgp-encrypted"}, + "content-type": "application/pgp-encrypted", + "content-length": 11}, {"id": 3, "content-type": "text/plain", "content": "This is another test encrypted message.\n"}]}]}, @@ -342,7 +353,8 @@ expected='[[[{"id": "XXXXX", "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, - "content-type": "application/pgp-signature"}]}]}, + "content-type": "application/pgp-signature", + "content-length": 315}]}]}, []]]]' test_expect_equal_json \ "$output" \ diff --git a/test/json b/test/json index bfafd55..ff7fbe1 100755 --- a/test/json +++ b/test/json @@ -45,7 +45,9 @@ emacs_deliver_message \ (insert \"Message-ID: <$id>\n\")" output=$(notmuch show --format=json "id:$id") filename=$(notmuch search --output=files "id:$id") -test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]" +# Get length of README after base64-encoding, minus additional newline. +attachment_length=$(( $(base64 $TEST_DIRECTORY/README | wc -c) - 1 )) +test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"content-length\": $attachment_length, \"content-transfer-encoding\": \"base64\", \"filename\": \"README\"}]}]}, []]]]" test_begin_subtest "Search message: json, utf-8" add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\"" diff --git a/test/multipart b/test/multipart index 344ed81..497f3ec 100755 --- a/test/multipart +++ b/test/multipart @@ -330,7 +330,7 @@ cat <<EOF >EXPECTED {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, {"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, {"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, -{"id": 9, "content-type": "application/pgp-signature"}]}]} +{"id": 9, "content-type": "application/pgp-signature", "content-length": 197}]}]} EOF test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)" @@ -345,7 +345,7 @@ cat <<EOF >EXPECTED {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, {"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, {"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, -{"id": 9, "content-type": "application/pgp-signature"}]} +{"id": 9, "content-type": "application/pgp-signature", "content-length": 197}]} EOF test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)" @@ -412,7 +412,7 @@ test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)" test_begin_subtest "--format=json --part=9, pgp signature (unverified)" notmuch show --format=json --part=9 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT cat <<EOF >EXPECTED -{"id": 9, "content-type": "application/pgp-signature"} +{"id": 9, "content-type": "application/pgp-signature", "content-length": 197} EOF test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)" @@ -636,7 +636,8 @@ cat <<EOF >EXPECTED "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, {"id": 9, - "content-type": "application/pgp-signature"}]}]}} + "content-type": "application/pgp-signature", + "content-length": 197}]}]}} EOF test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)" diff --git a/test/sexp b/test/sexp index b804942..dc52e79 100755 --- a/test/sexp +++ b/test/sexp @@ -37,7 +37,7 @@ emacs_deliver_message \ (insert \"Message-ID: <$id>\n\")" output=$(notmuch show --format=sexp "id:$id") filename=$(notmuch search --output=files "id:$id") -test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename \"$filename\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\") :headers (:Subject \"sexp-show-inline-attachment-filename\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"test_suite@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"multipart/mixed\" :content ((:id 2 :content-type \"text/plain\" :content \"This is a test message with inline attachment with a filename\") (:id 3 :content-type \"application/octet-stream\" :filename \"README\"))))) ())))" +test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename \"$filename\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\") :headers (:Subject \"sexp-show-inline-attachment-filename\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"test_suite@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"multipart/mixed\" :content ((:id 2 :content-type \"text/plain\" :content \"This is a test message with inline attachment with a filename\") (:id 3 :content-type \"application/octet-stream\" :filename \"README\" :content-length 12555 :content-transfer-encoding \"base64\"))))) ())))" test_begin_subtest "Search message: sexp, utf-8" add_message "[subject]=\"sexp-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\"" -- 1.7.9.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v4 0/3] indicate length of omitted body content 2012-12-09 12:56 [PATCH v4 0/3] indicate length of omitted body content Mark Walters ` (2 preceding siblings ...) 2012-12-09 12:56 ` [PATCH v4 3/3] test: conform to content length, encoding fields Mark Walters @ 2012-12-09 19:21 ` Mark Walters 2012-12-13 12:23 ` Peter Wang 3 siblings, 1 reply; 8+ messages in thread From: Mark Walters @ 2012-12-09 19:21 UTC (permalink / raw) To: notmuch, Peter Wang On Sun, 09 Dec 2012, Mark Walters <markwalters1009@gmail.com> wrote: > This is a (trivially) rebased version of > id:1344428872-12374-1-git-send-email-novalazy@gmail.com (There was one > bit of context clash, and a small change to one of the sexp tests to > mirror the same change for the json test) > > > The patch is Peter's not mine so as a reviewer +1 from me. > Hi (sorry Peter I had assumed git send-email would have cc'd you on my rebase earlier today) There was some discussion about this patch series on irc and the general view was favourable but that, for consistency, all omitted parts should received a content-length (including text/html). This is a little messy to code into the if else stuff: Jani suggested putting content-length/content-encoding into a small function. Anyway I had hope it was just a trivial rebase and could go in, but the above will take a bit longer to do neatly so I will leave it for now. I think it could go in relatively easily if resubmitted with this change but I will mark this version `moreinfo'. Best wishes Mark > Best wishes > > Mark > > > Peter Wang (3): > test: normalize only message filenames in show json > show: indicate length, encoding of omitted body content > test: conform to content length, encoding fields > > devel/schemata | 9 ++++++++- > notmuch-show.c | 14 ++++++++++++++ > test/crypto | 30 +++++++++++++++++++++--------- > test/json | 4 +++- > test/multipart | 11 ++++++----- > test/sexp | 2 +- > test/test-lib.sh | 2 +- > 7 files changed, 54 insertions(+), 18 deletions(-) > > -- > 1.7.9.1 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v4 0/3] indicate length of omitted body content 2012-12-09 19:21 ` [PATCH v4 0/3] indicate length of omitted body content Mark Walters @ 2012-12-13 12:23 ` Peter Wang 2012-12-13 12:52 ` Mark Walters 0 siblings, 1 reply; 8+ messages in thread From: Peter Wang @ 2012-12-13 12:23 UTC (permalink / raw) To: Mark Walters; +Cc: notmuch On Sun, 09 Dec 2012 19:21:18 +0000, Mark Walters <markwalters1009@gmail.com> wrote: > > Hi (sorry Peter I had assumed git send-email would have cc'd you on my > rebase earlier today) > > There was some discussion about this patch series on irc and the general > view was favourable but that, for consistency, all omitted parts should > received a content-length (including text/html). This is a little messy > to code into the if else stuff: Jani suggested putting > content-length/content-encoding into a small function. > > Anyway I had hope it was just a trivial rebase and could go in, but the > above will take a bit longer to do neatly so I will leave it for now. Ok. To save any potential duplicated work: I've updated the patch here, save for test/sexp. Waiting for that to be pushed before updating it. Peter ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v4 0/3] indicate length of omitted body content 2012-12-13 12:23 ` Peter Wang @ 2012-12-13 12:52 ` Mark Walters 2012-12-13 13:31 ` Peter Wang 0 siblings, 1 reply; 8+ messages in thread From: Mark Walters @ 2012-12-13 12:52 UTC (permalink / raw) To: Peter Wang; +Cc: notmuch On Thu, 13 Dec 2012, Peter Wang <novalazy@gmail.com> wrote: > On Sun, 09 Dec 2012 19:21:18 +0000, Mark Walters <markwalters1009@gmail.com> wrote: >> >> Hi (sorry Peter I had assumed git send-email would have cc'd you on my >> rebase earlier today) >> >> There was some discussion about this patch series on irc and the general >> view was favourable but that, for consistency, all omitted parts should >> received a content-length (including text/html). This is a little messy >> to code into the if else stuff: Jani suggested putting >> content-length/content-encoding into a small function. >> >> Anyway I had hope it was just a trivial rebase and could go in, but the >> above will take a bit longer to do neatly so I will leave it for now. > > Ok. To save any potential duplicated work: I've updated the patch here, > save for test/sexp. Waiting for that to be pushed before updating it. Sorry I am confused: isn't test/sexp already in master? Best wishes Mark ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v4 0/3] indicate length of omitted body content 2012-12-13 12:52 ` Mark Walters @ 2012-12-13 13:31 ` Peter Wang 0 siblings, 0 replies; 8+ messages in thread From: Peter Wang @ 2012-12-13 13:31 UTC (permalink / raw) To: Mark Walters; +Cc: notmuch On Thu, 13 Dec 2012 12:52:50 +0000, Mark Walters <markwalters1009@gmail.com> wrote: > > Sorry I am confused: isn't test/sexp already in master? You're right, somehow I missed it. Peter ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-12-13 13:31 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-12-09 12:56 [PATCH v4 0/3] indicate length of omitted body content Mark Walters 2012-12-09 12:56 ` [PATCH v4 1/3] test: normalize only message filenames in show json Mark Walters 2012-12-09 12:56 ` [PATCH v4 2/3] show: indicate length, encoding of omitted body content Mark Walters 2012-12-09 12:56 ` [PATCH v4 3/3] test: conform to content length, encoding fields Mark Walters 2012-12-09 19:21 ` [PATCH v4 0/3] indicate length of omitted body content Mark Walters 2012-12-13 12:23 ` Peter Wang 2012-12-13 12:52 ` Mark Walters 2012-12-13 13:31 ` Peter Wang
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).