unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* [PATCH 1/1] Fixing failing JSON tests.
@ 2012-12-04 13:46 Peter Feigl
  2012-12-04 14:35 ` Austin Clements
  0 siblings, 1 reply; 7+ messages in thread
From: Peter Feigl @ 2012-12-04 13:46 UTC (permalink / raw)
  To: notmuch

Two --format=json tests were failing, both due changes in the output
format introduced by the structured formatting code. This patch updates
the expected output.
---
 test/json | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/test/json b/test/json
index bfafd55..75b732f 100755
--- a/test/json
+++ b/test/json
@@ -32,7 +32,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
 test_begin_subtest "Show message: json, utf-8"
 add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
 output=$(notmuch show --format=json "jsön-show-méssage")
-test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
+test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\", \"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
 
 test_begin_subtest "Show message: json, inline attachment filename"
 subject='json-show-inline-attachment-filename'
@@ -50,14 +50,6 @@ test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"exclud
 test_begin_subtest "Search message: json, utf-8"
 add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
 output=$(notmuch search --format=json "jsön-search-méssage" | notmuch_search_sanitize)
-test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
- \"timestamp\": 946728000,
- \"date_relative\": \"2000-01-01\",
- \"matched\": 1,
- \"total\": 1,
- \"authors\": \"Notmuch Test Suite\",
- \"subject\": \"json-search-utf8-body-sübjéct\",
- \"tags\": [\"inbox\",
- \"unread\"]}]"
+test_expect_equal_json "$output" "[{\"thread\": \"XXX\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"matched\": 1, \"total\": 1, \"authors\": \"Notmuch Test Suite\", \"subject\": \"json-search-utf8-body-sübjéct\", \"tags\": [\"inbox\", \"unread\"]}]"
 
 test_done
-- 
1.8.0

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH 1/1] Fixing failing JSON tests.
  2012-12-04 13:46 [PATCH 1/1] Fixing failing JSON tests Peter Feigl
@ 2012-12-04 14:35 ` Austin Clements
  2012-12-04 14:43   ` Peter Feigl
  0 siblings, 1 reply; 7+ messages in thread
From: Austin Clements @ 2012-12-04 14:35 UTC (permalink / raw)
  To: Peter Feigl; +Cc: notmuch

I must be missing something...  These both look like whitespace-only
changes, which test_expect_equal_json normalizes.  Did you change
something else, too?

Quoth Peter Feigl on Dec 04 at  2:46 pm:
> Two --format=json tests were failing, both due changes in the output
> format introduced by the structured formatting code. This patch updates
> the expected output.
> ---
>  test/json | 12 ++----------
>  1 file changed, 2 insertions(+), 10 deletions(-)
> 
> diff --git a/test/json b/test/json
> index bfafd55..75b732f 100755
> --- a/test/json
> +++ b/test/json
> @@ -32,7 +32,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
>  test_begin_subtest "Show message: json, utf-8"
>  add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
>  output=$(notmuch show --format=json "jsön-show-méssage")
> -test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
> +test_expect_equal_json "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\", \"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
>  
>  test_begin_subtest "Show message: json, inline attachment filename"
>  subject='json-show-inline-attachment-filename'
> @@ -50,14 +50,6 @@ test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"exclud
>  test_begin_subtest "Search message: json, utf-8"
>  add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
>  output=$(notmuch search --format=json "jsön-search-méssage" | notmuch_search_sanitize)
> -test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> - \"timestamp\": 946728000,
> - \"date_relative\": \"2000-01-01\",
> - \"matched\": 1,
> - \"total\": 1,
> - \"authors\": \"Notmuch Test Suite\",
> - \"subject\": \"json-search-utf8-body-sübjéct\",
> - \"tags\": [\"inbox\",
> - \"unread\"]}]"
> +test_expect_equal_json "$output" "[{\"thread\": \"XXX\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"matched\": 1, \"total\": 1, \"authors\": \"Notmuch Test Suite\", \"subject\": \"json-search-utf8-body-sübjéct\", \"tags\": [\"inbox\", \"unread\"]}]"
>  
>  test_done

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 1/1] Fixing failing JSON tests.
  2012-12-04 14:35 ` Austin Clements
@ 2012-12-04 14:43   ` Peter Feigl
  2012-12-04 15:24     ` [PATCH] test: Fix UTF-8 JSON tests in Python 3 Austin Clements
  0 siblings, 1 reply; 7+ messages in thread
From: Peter Feigl @ 2012-12-04 14:43 UTC (permalink / raw)
  To: Austin Clements; +Cc: notmuch

On Tue, 4 Dec 2012 09:35:41 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> I must be missing something...  These both look like whitespace-only
> changes, which test_expect_equal_json normalizes.  Did you change
> something else, too?

They are, bremner on IRC was so kind to point out that my using Python
3.3 may be connected to the problem. I'll investigate further on why
exactly I get this behaviour.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH] test: Fix UTF-8 JSON tests in Python 3
  2012-12-04 14:43   ` Peter Feigl
@ 2012-12-04 15:24     ` Austin Clements
  2012-12-06 21:15       ` Tomi Ollila
                         ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Austin Clements @ 2012-12-04 15:24 UTC (permalink / raw)
  To: notmuch

test_expect_equal_json uses json.tool from the system Python.  While
Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
stdin strictly according to the environment.  Since we set LC_ALL=C
for the tests, Python 3's json.tool was assuming stdin would be in
ASCII and aborting when it couldn't decode the UTF-8 characters from
some of the JSON tests.  This patch sets the PYTHONIOENCODING
environment variable to utf-8 when invoking json.tool to override
Python's default encoding choice.
---
 test/test-lib.sh |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index f169785..9487526 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -534,8 +534,13 @@ test_expect_equal_file ()
 # canonicalized before diff'ing.  If an argument cannot be parsed, it
 # is used unchanged so that there's something to diff against.
 test_expect_equal_json () {
-    output=$(echo "$1" | python -mjson.tool || echo "$1")
-    expected=$(echo "$2" | python -mjson.tool || echo "$2")
+    # The test suite forces LC_ALL=C, but this causes Python 3 to
+    # decode stdin as ASCII.  We need to read JSON in UTF-8, so
+    # override Python's stdio encoding defaults.
+    output=$(echo "$1" | PYTHONIOENCODING=utf-8 python -mjson.tool \
+        || echo "$1")
+    expected=$(echo "$2" | PYTHONIOENCODING=utf-8 python -mjson.tool \
+        || echo "$2")
     shift 2
     test_expect_equal "$output" "$expected" "$@"
 }
-- 
1.7.10.4

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH] test: Fix UTF-8 JSON tests in Python 3
  2012-12-04 15:24     ` [PATCH] test: Fix UTF-8 JSON tests in Python 3 Austin Clements
@ 2012-12-06 21:15       ` Tomi Ollila
  2012-12-07 13:14       ` David Bremner
  2012-12-08 13:37       ` David Bremner
  2 siblings, 0 replies; 7+ messages in thread
From: Tomi Ollila @ 2012-12-06 21:15 UTC (permalink / raw)
  To: Austin Clements, notmuch

On Tue, Dec 04 2012, Austin Clements <amdragon@MIT.EDU> wrote:

> test_expect_equal_json uses json.tool from the system Python.  While
> Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
> stdin strictly according to the environment.  Since we set LC_ALL=C
> for the tests, Python 3's json.tool was assuming stdin would be in
> ASCII and aborting when it couldn't decode the UTF-8 characters from
> some of the JSON tests.  This patch sets the PYTHONIOENCODING
> environment variable to utf-8 when invoking json.tool to override
> Python's default encoding choice.
> ---

LGTM.

Tomi

>  test/test-lib.sh |    9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index f169785..9487526 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -534,8 +534,13 @@ test_expect_equal_file ()
>  # canonicalized before diff'ing.  If an argument cannot be parsed, it
>  # is used unchanged so that there's something to diff against.
>  test_expect_equal_json () {
> -    output=$(echo "$1" | python -mjson.tool || echo "$1")
> -    expected=$(echo "$2" | python -mjson.tool || echo "$2")
> +    # The test suite forces LC_ALL=C, but this causes Python 3 to
> +    # decode stdin as ASCII.  We need to read JSON in UTF-8, so
> +    # override Python's stdio encoding defaults.
> +    output=$(echo "$1" | PYTHONIOENCODING=utf-8 python -mjson.tool \
> +        || echo "$1")
> +    expected=$(echo "$2" | PYTHONIOENCODING=utf-8 python -mjson.tool \
> +        || echo "$2")
>      shift 2
>      test_expect_equal "$output" "$expected" "$@"
>  }
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] test: Fix UTF-8 JSON tests in Python 3
  2012-12-04 15:24     ` [PATCH] test: Fix UTF-8 JSON tests in Python 3 Austin Clements
  2012-12-06 21:15       ` Tomi Ollila
@ 2012-12-07 13:14       ` David Bremner
  2012-12-08 13:37       ` David Bremner
  2 siblings, 0 replies; 7+ messages in thread
From: David Bremner @ 2012-12-07 13:14 UTC (permalink / raw)
  To: Austin Clements, notmuch

Austin Clements <amdragon@MIT.EDU> writes:

> test_expect_equal_json uses json.tool from the system Python.  While
> Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
> stdin strictly according to the environment.  Since we set LC_ALL=C
> for the tests, Python 3's json.tool was assuming stdin would be in
> ASCII and aborting when it couldn't decode the UTF-8 characters from
> some of the JSON tests.  This patch sets the PYTHONIOENCODING
> environment variable to utf-8 when invoking json.tool to override
> Python's default encoding choice.

This looks sensible; marking ready

d

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] test: Fix UTF-8 JSON tests in Python 3
  2012-12-04 15:24     ` [PATCH] test: Fix UTF-8 JSON tests in Python 3 Austin Clements
  2012-12-06 21:15       ` Tomi Ollila
  2012-12-07 13:14       ` David Bremner
@ 2012-12-08 13:37       ` David Bremner
  2 siblings, 0 replies; 7+ messages in thread
From: David Bremner @ 2012-12-08 13:37 UTC (permalink / raw)
  To: Austin Clements, notmuch

Austin Clements <amdragon@MIT.EDU> writes:

> This patch sets the PYTHONIOENCODING
> environment variable to utf-8 when invoking json.tool to override
> Python's default encoding choice.

Pushed. One day I'd like to experience, "yes, python really got that
aspect of encoding right" ;).

d

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2012-12-08 13:37 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-04 13:46 [PATCH 1/1] Fixing failing JSON tests Peter Feigl
2012-12-04 14:35 ` Austin Clements
2012-12-04 14:43   ` Peter Feigl
2012-12-04 15:24     ` [PATCH] test: Fix UTF-8 JSON tests in Python 3 Austin Clements
2012-12-06 21:15       ` Tomi Ollila
2012-12-07 13:14       ` David Bremner
2012-12-08 13:37       ` David Bremner

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).