BUG: Backward compatibility breakage by --sort argument

BUG: Backward compatibility breakage by --sort argument

[YUE Daian]

Hi Notmuch mail list,

I noticed that the Emacs notmuch tree stopped working after a recent (?)
upgrade.

I checked the code and found that it adds "--sort=newest-first" as
sort-arg to "notmuch show" command.

Machine environment:
Emacs: 27.2
Emasc Notmuch version: notmuch-20210805.1158 installed via package.el
Notmuch binrary version: notmuch 0.32.2

I could remove the sort-arg as a workaround, but a fix would be highly
appreciated! :-)


Regards
Daian

Re: BUG: Backward compatibility breakage by --sort argument

[David Bremner]

YUE Daian <sheepduke@gmail.com> writes:

> Hi Notmuch mail list,
>
> I noticed that the Emacs notmuch tree stopped working after a recent (?)
> upgrade.
>

I think the most we could do is have notmuch-emacs print a more
informative message telling the user to upgrade notmuch (patches for
that would be welcome; I haven't decided if it makes sense to bump the
notmuch schema version).

Quoting https://melpa.org/#/notmuch

    Note for MELPA users (and others tracking the development version
    of notmuch-emacs):

    This emacs package needs a fairly closely matched version of the
    notmuch program. If you use the MELPA version of notmuch.el (as
    opposed to MELPA stable), you should be prepared to track the
    master development branch (i.e. build from git) for the notmuch
    program as well. Upgrading notmuch-emacs too far beyond the notmuch
    program can CAUSE YOUR EMAIL TO STOP WORKING.

    TL;DR: notmuch-emacs from MELPA and notmuch from distro packages is
    NOT SUPPORTED.

Re: BUG: Backward compatibility breakage by --sort argument

[YUE Daian]

On 2021-08-14 12:19, David Bremner <david@tethera.net> wrote:
> YUE Daian <sheepduke@gmail.com> writes:
>
>> Hi Notmuch mail list,
>>
>> I noticed that the Emacs notmuch tree stopped working after a recent (?)
>> upgrade.
>>
>
> I think the most we could do is have notmuch-emacs print a more
> informative message telling the user to upgrade notmuch (patches for
> that would be welcome; I haven't decided if it makes sense to bump the
> notmuch schema version).
>
> Quoting https://melpa.org/#/notmuch
>
>     Note for MELPA users (and others tracking the development version
>     of notmuch-emacs):
>
>     This emacs package needs a fairly closely matched version of the
>     notmuch program. If you use the MELPA version of notmuch.el (as
>     opposed to MELPA stable), you should be prepared to track the
>     master development branch (i.e. build from git) for the notmuch
>     program as well. Upgrading notmuch-emacs too far beyond the notmuch
>     program can CAUSE YOUR EMAIL TO STOP WORKING.
>
>     TL;DR: notmuch-emacs from MELPA and notmuch from distro packages is
>     NOT SUPPORTED.
>
Maybe check version compatibility when notmuch is invoked from Emacs?

So the user (like me) immediately know that it is time to upgrade my
system...

Re: BUG: Backward compatibility breakage by --sort argument

[David Bremner]

YUE Daian <sheepduke@gmail.com> writes:

>>
> Maybe check version compatibility when notmuch is invoked from Emacs?
>
> So the user (like me) immediately know that it is time to upgrade my
> system...

Yes, that would be fine if someone wants to implement it.

d

Re: BUG: Backward compatibility breakage by --sort argument

[Tomi Ollila]

On Sun, Aug 15 2021, David Bremner wrote:

> YUE Daian <sheepduke@gmail.com> writes:
>
>>>
>> Maybe check version compatibility when notmuch is invoked from Emacs?
>>
>> So the user (like me) immediately know that it is time to upgrade my
>> system...
>
> Yes, that would be fine if someone wants to implement it.

Who volunteer to maintain it ? ;/

>
> d

Tomi

Re: BUG: Backward compatibility breakage by --sort argument

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Sun, Aug 15 2021, David Bremner wrote:
>
>> YUE Daian <sheepduke@gmail.com> writes:
>>
>>>>
>>> Maybe check version compatibility when notmuch is invoked from Emacs?
>>>
>>> So the user (like me) immediately know that it is time to upgrade my
>>> system...
>>
>> Yes, that would be fine if someone wants to implement it.
>
> Who volunteer to maintain it ? ;/
>

It's a fair point, but if we just bump the schema version, there is not
too much maintenance (just a bit of search and replace in the elisp to
bump the version). Of course it's a different from the other schema
version bumps (where we added new keys), but maybe that doesn't matter.

d

[PATCH] CLI: define and use format version 5

[David Bremner]

This is a bit of a cheat, since the format does not actually
change. On the other hand it is fairly common to do something like
this to shared libary SONAMEs when the ABI changes in some subtle way.
It does rely on the format-version argument being early enough on the
command line to generate a sensible error message.
---
 devel/schemata           | 6 +++++-
 emacs/notmuch-address.el | 2 +-
 emacs/notmuch-mua.el     | 2 +-
 emacs/notmuch-query.el   | 2 +-
 emacs/notmuch-tree.el    | 2 +-
 emacs/notmuch.el         | 2 +-
 notmuch-client.h         | 2 +-
 test/T310-emacs.sh       | 2 +-
 test/T450-emacs-show.sh  | 2 +-
 9 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index ae84a528..01e3a3df 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -14,7 +14,7 @@ are interleaved. Keys are printed as keywords (symbols preceded by a
 colon), e.g. (:id "123" :time 54321 :from "foobar"). Null is printed as
 nil, true as t and false as nil.
 
-This is version 4 of the structured output format.
+This is version 5 of the structured output format.
 
 Version history
 ---------------
@@ -36,6 +36,10 @@ v4
 - (notmuch 0.29) added message.crypto to identify overall message
   cryptographic state
 
+v5
+- sorting support for notmuch show (no change to actual schema,
+  just new command line argument)
+
 Common non-terminals
 --------------------
 
diff --git a/emacs/notmuch-address.el b/emacs/notmuch-address.el
index f0af6667..9fc13bc5 100644
--- a/emacs/notmuch-address.el
+++ b/emacs/notmuch-address.el
@@ -308,7 +308,7 @@ execution, CALLBACK is called when harvesting finishes."
 			    (and config-query
 				 (format " and (%s)" config-query)))
 		  from-or-to-me-query))
-	 (args `("address" "--format=sexp" "--format-version=4"
+	 (args `("address" "--format=sexp" "--format-version=5"
 		 ,(if sent "--output=recipients" "--output=sender")
 		 "--deduplicate=address"
 		 ,query)))
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 17954fb3..f510c043 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -235,7 +235,7 @@ Typically this is added to `notmuch-mua-send-hook'."
 ;;; Mua reply
 
 (defun notmuch-mua-reply (query-string &optional sender reply-all)
-  (let ((args '("reply" "--format=sexp" "--format-version=4"))
+  (let ((args '("reply" "--format=sexp" "--format-version=5"))
 	(process-crypto notmuch-show-process-crypto)
 	reply
 	original)
diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
index d7349b77..5c7f4f8d 100644
--- a/emacs/notmuch-query.el
+++ b/emacs/notmuch-query.el
@@ -31,7 +31,7 @@
 A thread is a forest or list of trees. A tree is a two element
 list where the first element is a message, and the second element
 is a possibly empty forest of replies."
-  (let ((args '("show" "--format=sexp" "--format-version=4")))
+  (let ((args '("show" "--format=sexp" "--format-version=5")))
     (when notmuch-show-process-crypto
       (setq args (append args '("--decrypt=true"))))
     (setq args (append args search-terms))
diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el
index ef1ca4c5..ec803294 100644
--- a/emacs/notmuch-tree.el
+++ b/emacs/notmuch-tree.el
@@ -1101,7 +1101,7 @@ the same as for the function notmuch-tree."
     (notmuch-tag-clear-cache)
     (let ((proc (notmuch-start-notmuch
 		 "notmuch-tree" (current-buffer) #'notmuch-tree-process-sentinel
-		 "show" "--body=false" "--format=sexp" "--format-version=4"
+		 "show" "--body=false" "--format=sexp" "--format-version=5"
 		 sort-arg message-arg search-args))
 	  ;; Use a scratch buffer to accumulate partial output.
 	  ;; This buffer will be killed by the sentinel, which
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 351334aa..739cb93b 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -1020,7 +1020,7 @@ the configured default sort order."
       (save-excursion
 	(let ((proc (notmuch-start-notmuch
 		     "notmuch-search" buffer #'notmuch-search-process-sentinel
-		     "search" "--format=sexp" "--format-version=4"
+		     "search" "--format=sexp" "--format-version=5"
 		     (if oldest-first
 			 "--sort=oldest-first"
 		       "--sort=newest-first")
diff --git a/notmuch-client.h b/notmuch-client.h
index 8227fea4..8643a63f 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -137,7 +137,7 @@ chomp_newline (char *str)
  * this.  New (required) map fields can be added without increasing
  * this.
  */
-#define NOTMUCH_FORMAT_CUR 4
+#define NOTMUCH_FORMAT_CUR 5
 /* The minimum supported structured output format version.  Requests
  * for format versions below this will return an error. */
 #define NOTMUCH_FORMAT_MIN 1
diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
index 7b70175b..d69d94a3 100755
--- a/test/T310-emacs.sh
+++ b/test/T310-emacs.sh
@@ -1060,7 +1060,7 @@ End of search results.
 YYY/notmuch_fail exited with status 1 (see *Notmuch errors* for more details)
 === ERROR ===
 YYY/notmuch_fail exited with status 1
-command: YYY/notmuch_fail search --format\=sexp --format-version\=4 --sort\=newest-first tag\:inbox
+command: YYY/notmuch_fail search --format\=sexp --format-version\=5 --sort\=newest-first tag\:inbox
 exit status: 1"
 
 test_begin_subtest "Search handles subprocess warnings"
diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
index 9d08d2cc..a750cc4d 100755
--- a/test/T450-emacs-show.sh
+++ b/test/T450-emacs-show.sh
@@ -192,7 +192,7 @@ test_expect_equal "$(notmuch_emacs_error_sanitize notmuch_fail OUTPUT MESSAGES E
 This is an error (see *Notmuch errors* for more details)
 === ERROR ===
 This is an error
-command: YYY/notmuch_fail show --format\\=sexp --format-version\\=4 --decrypt\\=true --exclude\\=false \\' \\* \\'
+command: YYY/notmuch_fail show --format\\=sexp --format-version\\=5 --decrypt\\=true --exclude\\=false \\' \\* \\'
 exit status: 1
 stderr:
 This is an error
-- 
2.32.0

Re: [PATCH] CLI: define and use format version 5

[Tomi Ollila]

On Sat, Aug 21 2021, David Bremner wrote:

> This is a bit of a cheat, since the format does not actually
> change. On the other hand it is fairly common to do something like
> this to shared libary SONAMEs when the ABI changes in some subtle way.
> It does rely on the format-version argument being early enough on the
> command line to generate a sensible error message.

looks 0.33 to me.

Tomi


> ---
>  devel/schemata           | 6 +++++-
>  emacs/notmuch-address.el | 2 +-
>  emacs/notmuch-mua.el     | 2 +-
>  emacs/notmuch-query.el   | 2 +-
>  emacs/notmuch-tree.el    | 2 +-
>  emacs/notmuch.el         | 2 +-
>  notmuch-client.h         | 2 +-
>  test/T310-emacs.sh       | 2 +-
>  test/T450-emacs-show.sh  | 2 +-
>  9 files changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/devel/schemata b/devel/schemata
> index ae84a528..01e3a3df 100644
> --- a/devel/schemata
> +++ b/devel/schemata
> @@ -14,7 +14,7 @@ are interleaved. Keys are printed as keywords (symbols preceded by a
>  colon), e.g. (:id "123" :time 54321 :from "foobar"). Null is printed as
>  nil, true as t and false as nil.
>  
> -This is version 4 of the structured output format.
> +This is version 5 of the structured output format.
>  
>  Version history
>  ---------------
> @@ -36,6 +36,10 @@ v4
>  - (notmuch 0.29) added message.crypto to identify overall message
>    cryptographic state
>  
> +v5
> +- sorting support for notmuch show (no change to actual schema,
> +  just new command line argument)
> +
>  Common non-terminals
>  --------------------
>  
> diff --git a/emacs/notmuch-address.el b/emacs/notmuch-address.el
> index f0af6667..9fc13bc5 100644
> --- a/emacs/notmuch-address.el
> +++ b/emacs/notmuch-address.el
> @@ -308,7 +308,7 @@ execution, CALLBACK is called when harvesting finishes."
>  			    (and config-query
>  				 (format " and (%s)" config-query)))
>  		  from-or-to-me-query))
> -	 (args `("address" "--format=sexp" "--format-version=4"
> +	 (args `("address" "--format=sexp" "--format-version=5"
>  		 ,(if sent "--output=recipients" "--output=sender")
>  		 "--deduplicate=address"
>  		 ,query)))
> diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
> index 17954fb3..f510c043 100644
> --- a/emacs/notmuch-mua.el
> +++ b/emacs/notmuch-mua.el
> @@ -235,7 +235,7 @@ Typically this is added to `notmuch-mua-send-hook'."
>  ;;; Mua reply
>  
>  (defun notmuch-mua-reply (query-string &optional sender reply-all)
> -  (let ((args '("reply" "--format=sexp" "--format-version=4"))
> +  (let ((args '("reply" "--format=sexp" "--format-version=5"))
>  	(process-crypto notmuch-show-process-crypto)
>  	reply
>  	original)
> diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
> index d7349b77..5c7f4f8d 100644
> --- a/emacs/notmuch-query.el
> +++ b/emacs/notmuch-query.el
> @@ -31,7 +31,7 @@
>  A thread is a forest or list of trees. A tree is a two element
>  list where the first element is a message, and the second element
>  is a possibly empty forest of replies."
> -  (let ((args '("show" "--format=sexp" "--format-version=4")))
> +  (let ((args '("show" "--format=sexp" "--format-version=5")))
>      (when notmuch-show-process-crypto
>        (setq args (append args '("--decrypt=true"))))
>      (setq args (append args search-terms))
> diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el
> index ef1ca4c5..ec803294 100644
> --- a/emacs/notmuch-tree.el
> +++ b/emacs/notmuch-tree.el
> @@ -1101,7 +1101,7 @@ the same as for the function notmuch-tree."
>      (notmuch-tag-clear-cache)
>      (let ((proc (notmuch-start-notmuch
>  		 "notmuch-tree" (current-buffer) #'notmuch-tree-process-sentinel
> -		 "show" "--body=false" "--format=sexp" "--format-version=4"
> +		 "show" "--body=false" "--format=sexp" "--format-version=5"
>  		 sort-arg message-arg search-args))
>  	  ;; Use a scratch buffer to accumulate partial output.
>  	  ;; This buffer will be killed by the sentinel, which
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index 351334aa..739cb93b 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -1020,7 +1020,7 @@ the configured default sort order."
>        (save-excursion
>  	(let ((proc (notmuch-start-notmuch
>  		     "notmuch-search" buffer #'notmuch-search-process-sentinel
> -		     "search" "--format=sexp" "--format-version=4"
> +		     "search" "--format=sexp" "--format-version=5"
>  		     (if oldest-first
>  			 "--sort=oldest-first"
>  		       "--sort=newest-first")
> diff --git a/notmuch-client.h b/notmuch-client.h
> index 8227fea4..8643a63f 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -137,7 +137,7 @@ chomp_newline (char *str)
>   * this.  New (required) map fields can be added without increasing
>   * this.
>   */
> -#define NOTMUCH_FORMAT_CUR 4
> +#define NOTMUCH_FORMAT_CUR 5
>  /* The minimum supported structured output format version.  Requests
>   * for format versions below this will return an error. */
>  #define NOTMUCH_FORMAT_MIN 1
> diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh
> index 7b70175b..d69d94a3 100755
> --- a/test/T310-emacs.sh
> +++ b/test/T310-emacs.sh
> @@ -1060,7 +1060,7 @@ End of search results.
>  YYY/notmuch_fail exited with status 1 (see *Notmuch errors* for more details)
>  === ERROR ===
>  YYY/notmuch_fail exited with status 1
> -command: YYY/notmuch_fail search --format\=sexp --format-version\=4 --sort\=newest-first tag\:inbox
> +command: YYY/notmuch_fail search --format\=sexp --format-version\=5 --sort\=newest-first tag\:inbox
>  exit status: 1"
>  
>  test_begin_subtest "Search handles subprocess warnings"
> diff --git a/test/T450-emacs-show.sh b/test/T450-emacs-show.sh
> index 9d08d2cc..a750cc4d 100755
> --- a/test/T450-emacs-show.sh
> +++ b/test/T450-emacs-show.sh
> @@ -192,7 +192,7 @@ test_expect_equal "$(notmuch_emacs_error_sanitize notmuch_fail OUTPUT MESSAGES E
>  This is an error (see *Notmuch errors* for more details)
>  === ERROR ===
>  This is an error
> -command: YYY/notmuch_fail show --format\\=sexp --format-version\\=4 --decrypt\\=true --exclude\\=false \\' \\* \\'
> +command: YYY/notmuch_fail show --format\\=sexp --format-version\\=5 --decrypt\\=true --exclude\\=false \\' \\* \\'
>  exit status: 1
>  stderr:
>  This is an error
> -- 
> 2.32.0
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: [PATCH] CLI: define and use format version 5

[David Bremner]

David Bremner <david@tethera.net> writes:

> This is a bit of a cheat, since the format does not actually
> change. On the other hand it is fairly common to do something like
> this to shared libary SONAMEs when the ABI changes in some subtle way.
> It does rely on the format-version argument being early enough on the
> command line to generate a sensible error message.

applied to master