[PATCH v3] emacs: postpone/resume support

[PATCH v3] emacs: postpone/resume support

[Mark Walters]

This provides preliminary support for postponing and resuming in the
emacs frontend. On postponing it uses notmuch insert to put the
message in the notmuch database; resume gets the raw file from notmuch
and using the emacs function mime-to-mml reconstructs the message
(including attachments).

Current bindings are C-x C-s to save a draft, C-c C-p to postpone a
draft (save and exit compose buffer), and e to resume a draft from
show or tree mode.

Previous drafts get tagged deleted on subsequent saves, or on the
message being sent.

Each draft gets its own message-id, and we use the namespace
draft-.... for draft message ids (so, at least for most people, drafts
are easily distinguisable).
---

Sorry to be rather spamming the list. This is another version of the
postpone/resume series. This replaces the third patch in the series at
id:1464976195-23134-1-git-send-email-markwalters1009@gmail.com (so
should be applied on top of the first two).

There are three main changes --

1) It seems that editing an already sent message does work -- as it is
not heavily tested we warn before doing it. But now when you send the
new version it does not tag the old version as deleted (we only tag
drafts deleted).

2) We quote secure mml tags before saving. This avoids problems with
signing the wrong message, stale signatures, and using the wrong keys
for encryption. Note the draft message will be stored in the mail
store unencrypted.

3) You can choose to quote more mml tags than just secure; there is a
custom variable notmuch-message-quoted-tags under notmuch-send which
should be a list of tags to quote. If you set it to '("secure" "part")
then attachments won't be saved with the draft. This may be desired in
some cases (but may break things like postponing rfc822 forwarded
messages).  Anyway the option is there for anyone who wants to test!

Best wishes

Mark




emacs/notmuch-message.el | 190 +++++++++++++++++++++++++++++++++++++++++++++++
 emacs/notmuch-mua.el     |   4 +
 emacs/notmuch-show.el    |   9 +++
 emacs/notmuch-tree.el    |   1 +
 4 files changed, 204 insertions(+)

diff --git a/emacs/notmuch-message.el b/emacs/notmuch-message.el
index d437b85..6a137b5 100644
--- a/emacs/notmuch-message.el
+++ b/emacs/notmuch-message.el
@@ -25,6 +25,8 @@
 (require 'notmuch-tag)
 (require 'notmuch-mua)
 
+(declare-function notmuch-show-get-message-id "notmuch-show" (&optional bare))
+
 (defcustom notmuch-message-replied-tags '("+replied")
   "List of tag changes to apply to a message when it has been replied to.
 
@@ -38,6 +40,49 @@ the \"inbox\" and \"todo\" tags, you would set:
   :type '(repeat string)
   :group 'notmuch-send)
 
+(defcustom notmuch-message-draft-tags '("+draft")
+  "List of tags changes to apply to a draft message when it is saved in the database.
+
+Tags starting with \"+\" (or not starting with either \"+\" or
+\"-\") in the list will be added, and tags starting with \"-\"
+will be removed from the message being stored.
+
+For example, if you wanted to give the message a \"draft\" tag
+but not the (normally added by default) \"inbox\" tag, you would
+set:
+    (\"+draft\" \"-inbox\")"
+  :type '(repeat string)
+  :group 'notmuch-send)
+
+(defcustom notmuch-message-draft-folder "drafts"
+  "Folder to save draft messages in.
+
+This should be specified relative to the root of the notmuch
+database. It will be created if necessary."
+  :type 'string
+  :group 'notmuch-send)
+
+(defcustom notmuch-message-quoted-tags '("secure")
+  "Mml tags to quote.
+
+This should be a list of mml tags to quote before saving. It is
+recommended that the list includes \"secure\".
+
+If you include \"part\" then attachments will not be saved with
+the draft -- if not then they will be saved with the draft. The
+former means the attachments may not still exist when you resume
+the message, the latter means that the attachments as they were
+when you postponed will be sent with the resumed message.
+
+Note you may get strange results if you change this between
+postponing and resuming a message."
+  :type '(repeat string)
+  :group 'notmuch-send)
+
+(defvar notmuch-message-draft-id nil
+  "Message-id of the most recent saved draft of this message")
+(make-variable-buffer-local 'notmuch-message-draft-id)
+
 (defun notmuch-message-mark-replied ()
   ;; get the in-reply-to header and parse it for the message id.
   (let ((rep (mail-header-parse-addresses (message-field-value "In-Reply-To"))))
@@ -45,7 +90,152 @@ the \"inbox\" and \"todo\" tags, you would set:
       (notmuch-tag (notmuch-id-to-query (car (car rep)))
 	       (notmuch-tag-change-list notmuch-message-replied-tags)))))
 
+(defun notmuch-message-mark-draft-deleted ()
+  "Tag the last saved draft deleted.
+
+Used when a new version is saved, or the message is sent."
+  (when notmuch-message-draft-id
+    (notmuch-tag notmuch-message-draft-id '("+deleted"))))
+
+(defun notmuch-message-quote-some-mml ()
+  "Quote the mml tags in `notmuch-message-quoted-tags`."
+  ;; This is copied from mml-quote-region but only quotes the
+  ;; specified tags.
+  (when notmuch-message-quoted-tags
+    (save-excursion
+      (let ((re (concat "<#!*/?\\("
+			(mapconcat 'identity notmuch-message-quoted-tags "\\|")
+			"\\)")))
+	(message-goto-body)
+	(while (re-search-forward re nil t)
+	  ;; Insert ! after the #.
+	  (goto-char (+ (match-beginning 0) 2))
+	  (insert "!"))))))
+
+(defun notmuch-message-unquote-some-mml ()
+  "Unquote the mml tags in `notmuch-message-quoted-tags`."
+  (when notmuch-message-quoted-tags
+    (save-excursion
+      (let ((re (concat "<#!+/?\\("
+			(mapconcat 'identity notmuch-message-quoted-tags "\\|")
+			"\\)")))
+	(message-goto-body)
+	(while (re-search-forward re nil t)
+	  ;; Remove one ! from after the #.
+	  (goto-char (+ (match-beginning 0) 2))
+	  (delete-char 1))))))
+
+(defun notmuch-message-save-draft ()
+  "Save the current draft message in the notmuch database.
+
+This saves the current message in the database with tags
+`notmuch-message-draft-tags` (in addition to any default tags
+applied to newly inserted messages)."
+  (interactive)
+
+  ;; This is based on message-do-fcc but modified for our needs.
+  (let ((case-fold-search t)
+	(buf (current-buffer))
+	(mml-externalize-attachments nil)
+	;; We generate a message id now as we will need it later. Note
+	;; message-make-message-id gives the id inside a "<" ">" pair,
+	;; but notmuch doesn't want that form, so remove them.
+	(id (concat "draft-" (substring (message-make-message-id) 1 -1))))
+    (with-current-buffer (get-buffer-create " *message temp*")
+      (erase-buffer)
+      (insert-buffer-substring buf)
+      ;; We insert a Date header and a Message-ID header, the former
+      ;; so that it is easier to search for the message, and the
+      ;; latter so we have a way of accessing the saved message (for
+      ;; example to delete it at a later time). We check that the
+      ;; user has these in `message-deletable-headers` (the default)
+      ;; as otherwise they are doing something strange and we
+      ;; shouldn't interfere. Note, since we are doing this in a new
+      ;; buffer we don't change the version in the compose buffer.
+      (if (member 'Message-ID message-deletable-headers)
+	  (progn
+	    (message-remove-header "Message-ID")
+	    (message-add-header (concat "Message-ID: <" id ">")))
+	(message "You have customized emacs so Message-ID is not a deletable header, so not changing it")
+	(setq id nil))
+      (if (member 'Date message-deletable-headers)
+	  (progn
+	    (message-remove-header "Date")
+	    (message-add-header (concat "Date: " (message-make-date))))
+	(message "You have customized emacs so Date is not a deletable header, so not changing it"))
+      (notmuch-message-quote-some-mml)
+      ;; Back to following message-do-fcc
+      (message-encode-message-body)
+      (save-restriction
+	(message-narrow-to-headers)
+	(let ((mail-parse-charset message-default-charset)
+	      (rfc2047-header-encoding-alist
+	       (cons '("Newsgroups" . default)
+		     rfc2047-header-encoding-alist)))
+	  (mail-encode-encoded-word-buffer)))
+      (goto-char (point-min))
+      (when (re-search-forward
+	     (concat "^" (regexp-quote mail-header-separator) "$")
+	     nil t)
+	(replace-match "" t t ))
+
+      (apply 'notmuch-call-notmuch-process :stdin-string (buffer-string)
+	     "insert" "--create-folder"
+	     (concat "--folder=" notmuch-message-draft-folder)
+	     notmuch-message-draft-tags))
+    ;; We are now back in the original compose buffer. Note the
+    ;; function notmuch-call-notmuch-process signals an error on
+    ;; failure, so to get to this point it must have succeeded. Note
+    ;; notmuch-message-draft-id is still the id of the previous draft,
+    ;; so it is safe to mark it deleted.
+    (notmuch-message-mark-draft-deleted)
+    (setq notmuch-message-draft-id (concat "id:" id))
+    (set-buffer-modified-p nil)))
+
+(defun notmuch-message-postpone ()
+  "Save the draft message in the notmuch database and exit buffer."
+  (interactive)
+  (notmuch-message-save-draft)
+  (kill-buffer))
+
+(defun notmuch-message-resume (id)
+  "Resume editing of message with id ID."
+  (let* ((tags (process-lines notmuch-command "search" "--output=tags"
+			      "--exclude=false" id))
+	 (draft (equal tags (notmuch-update-tags tags notmuch-message-draft-tags))))
+    (when (or draft
+	      (yes-or-no-p "Message does not appear to be a draft: really resume? "))
+      (switch-to-buffer (get-buffer-create (concat "*notmuch-draft-" id "*")))
+      (setq buffer-read-only nil)
+      (erase-buffer)
+      (let ((coding-system-for-read 'no-conversion))
+	(call-process notmuch-command nil t nil "show" "--format=raw" id))
+      (mime-to-mml)
+      (goto-char (point-min))
+      (when (re-search-forward "^$" nil t)
+	(replace-match mail-header-separator t t))
+      ;; Remove our added Date and Message-ID headers (unless the user has
+      ;; explicitly customized emacs to tell us not to).
+      (save-restriction
+	(message-narrow-to-headers)
+	(when (member 'Message-ID message-deletable-headers)
+	  (message-remove-header "Message-ID"))
+	(when (member 'Date message-deletable-headers)
+	  (message-remove-header "Date")))
+      ;; If the message does not appear to be a draft, the postpone
+      ;; code probably didn't write it, so it should not be unquoted.
+      (when draft
+	(notmuch-message-unquote-some-mml))
+      (notmuch-message-mode)
+      (set-buffer-modified-p nil)
+      ;; If the resumed message was a draft then set the draft
+      ;; message-id so that we can delete the current saved draft if the
+      ;; message is resaved or sent.
+      (setq notmuch-message-draft-id (when draft id)))))
+
+
 (add-hook 'message-send-hook 'notmuch-message-mark-replied)
+(add-hook 'message-send-hook 'notmuch-message-mark-draft-deleted)
 
 (provide 'notmuch-message)
 
diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el
index 399e138..3118e5d 100644
--- a/emacs/notmuch-mua.el
+++ b/emacs/notmuch-mua.el
@@ -33,6 +33,8 @@
 (declare-function notmuch-show-insert-body "notmuch-show" (msg body depth))
 (declare-function notmuch-fcc-header-setup "notmuch-maildir-fcc" ())
 (declare-function notmuch-fcc-handler "notmuch-maildir-fcc" (destdir))
+(declare-function notmuch-message-postpone "notmuch-message" ())
+(declare-function notmuch-message-save-draft "notmuch-message" ())
 
 ;;
 
@@ -283,6 +285,8 @@ mutiple parts get a header."
 
 (define-key notmuch-message-mode-map (kbd "C-c C-c") #'notmuch-mua-send-and-exit)
 (define-key notmuch-message-mode-map (kbd "C-c C-s") #'notmuch-mua-send)
+(define-key notmuch-message-mode-map (kbd "C-c C-p") #'notmuch-message-postpone)
+(define-key notmuch-message-mode-map (kbd "C-x C-s") #'notmuch-message-save-draft)
 
 (defun notmuch-mua-pop-to-buffer (name switch-function)
   "Pop to buffer NAME, and warn if it already exists and is
diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index f33096c..12b21c9 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -38,6 +38,7 @@
 (require 'notmuch-mua)
 (require 'notmuch-crypto)
 (require 'notmuch-print)
+(require 'notmuch-message)
 
 (declare-function notmuch-call-notmuch-process "notmuch" (&rest args))
 (declare-function notmuch-search-next-thread "notmuch" nil)
@@ -1425,6 +1426,7 @@ reset based on the original query."
     (define-key map "|" 'notmuch-show-pipe-message)
     (define-key map "w" 'notmuch-show-save-attachments)
     (define-key map "V" 'notmuch-show-view-raw-message)
+    (define-key map "e" 'notmuch-show-resume-message)
     (define-key map "c" 'notmuch-show-stash-map)
     (define-key map "h" 'notmuch-show-toggle-visibility-headers)
     (define-key map "*" 'notmuch-show-tag-all)
@@ -1955,6 +1957,13 @@ to show, nil otherwise."
     (setq buffer-read-only t)
     (view-buffer buf 'kill-buffer-if-not-modified)))
 
+(defun notmuch-show-resume-message ()
+  "Resume EDITING the current draft message."
+  (interactive)
+  (let ((id (notmuch-show-get-message-id)))
+    (when id
+      (notmuch-message-resume id))))
+
 (put 'notmuch-show-pipe-message 'notmuch-doc
      "Pipe the contents of the current message to a command.")
 (put 'notmuch-show-pipe-message 'notmuch-prefix-doc
diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el
index 6c35543..c759290 100644
--- a/emacs/notmuch-tree.el
+++ b/emacs/notmuch-tree.el
@@ -261,6 +261,7 @@ FUNC."
     (define-key map "r" (notmuch-tree-close-message-pane-and #'notmuch-show-reply-sender))
     (define-key map "R" (notmuch-tree-close-message-pane-and #'notmuch-show-reply))
     (define-key map "V" (notmuch-tree-close-message-pane-and #'notmuch-show-view-raw-message))
+    (define-key map "e" (notmuch-tree-close-message-pane-and #'notmuch-show-resume-message))
 
     ;; The main tree view bindings
     (define-key map (kbd "RET") 'notmuch-tree-show-message)
-- 
2.1.4

Re: [PATCH v3] emacs: postpone/resume support

[Tomi Ollila]

On Sat, Jun 04 2016, Mark Walters <markwalters1009@gmail.com> wrote:

> This provides preliminary support for postponing and resuming in the
> emacs frontend. On postponing it uses notmuch insert to put the
> message in the notmuch database; resume gets the raw file from notmuch
> and using the emacs function mime-to-mml reconstructs the message
> (including attachments).
>
> Current bindings are C-x C-s to save a draft, C-c C-p to postpone a

According to the discussion on IRC this feature seems like a thing
many people are enthusiastically eager to have...

... but I am not -- I've been happy with the current draft handling
(yes, I had to set message-save-directory)...

... but I think I am not biased when I think that if ctrl-x ctrl-s
does something more magical than simple (save-buffer) in buffers that
looks like text editing buffer, that is going to be unexpected disturbing
functionality to someone (else like me)...

... I can easily add (and will do if this functionaly lands... :):

(put 'notmuch-message-save-draft 'disabled t)
(define-key notmuch-message-mode-map (kbd "C-x C-s") #'save-buffer)

to my $HOME/.emacs.d/notmuch-config.el

(or actually my init-notmuch.el which is in my git-clonable dotfiles so it
goes automatically everywhere; notmuch-config.el is for system-local configs)

but how are others protected?

It might be that no-one else notices nor cares (or they can handle the
situation themselves). anyway, now if this lands and we get tons of
angry responses then I can say 'I said so' >;/

Tomi

PS: the brief look I had to the code I can say Mark has done pretty good
work with it.


> draft (save and exit compose buffer), and e to resume a draft from
> show or tree mode.
>
> Previous drafts get tagged deleted on subsequent saves, or on the
> message being sent.
>
> Each draft gets its own message-id, and we use the namespace
> draft-.... for draft message ids (so, at least for most people, drafts
> are easily distinguisable).
> ---
>
> Sorry to be rather spamming the list. This is another version of the
> postpone/resume series. This replaces the third patch in the series at
> id:1464976195-23134-1-git-send-email-markwalters1009@gmail.com (so
> should be applied on top of the first two).
>
> There are three main changes --
>
> 1) It seems that editing an already sent message does work -- as it is
> not heavily tested we warn before doing it. But now when you send the
> new version it does not tag the old version as deleted (we only tag
> drafts deleted).
>
> 2) We quote secure mml tags before saving. This avoids problems with
> signing the wrong message, stale signatures, and using the wrong keys
> for encryption. Note the draft message will be stored in the mail
> store unencrypted.
>
> 3) You can choose to quote more mml tags than just secure; there is a
> custom variable notmuch-message-quoted-tags under notmuch-send which
> should be a list of tags to quote. If you set it to '("secure" "part")
> then attachments won't be saved with the draft. This may be desired in
> some cases (but may break things like postponing rfc822 forwarded
> messages).  Anyway the option is there for anyone who wants to test!
>
> Best wishes
>
> Mark
>
>

Re: [PATCH v3] emacs: postpone/resume support

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> ... but I think I am not biased when I think that if ctrl-x ctrl-s
> does something more magical than simple (save-buffer) in buffers that
> looks like text editing buffer, that is going to be unexpected disturbing
> functionality to someone (else like me)...

This is a valid point. On the other hand the binding is copied from
Gnus, which also includes at least _some_ magic

      http://www.gnu.org/software/emacs/manual/html_node/gnus/Drafts.html

Of course we don't need to slavishly follow Gnus (that way lies
madness), but there is also a group of users familiar with its draft
handling.

Re: [PATCH v3] emacs: postpone/resume support

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

On Sun 2016-06-05 07:18:25 -0400, David Bremner wrote:
> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>> ... but I think I am not biased when I think that if ctrl-x ctrl-s
>> does something more magical than simple (save-buffer) in buffers that
>> looks like text editing buffer, that is going to be unexpected disturbing
>> functionality to someone (else like me)...
>
> This is a valid point. On the other hand the binding is copied from
> Gnus, which also includes at least _some_ magic
>
>       http://www.gnu.org/software/emacs/manual/html_node/gnus/Drafts.html

gnus isn't the only thing that does something more "magical" than saving
a raw buffer to a file.

try the following:

   echo test | gpg --output test.gpg --armor --default-recipient-self  --encrypt
   emacs test.gpg

emacs will prompt you to decrypt the file, then let you edit the
cleartext buffer.

a subsequent "C-x C-s" will re-save it encrypted.  (at least it does for
me!)

[ i note several weirdnesses here around the round-trip cleanliness and
  safety of this operation which aren't relevant to notmuch, but that's
  a different story, and i've reported them to emacs upstream ]

Of course, this functionality appears to also happen within
"save-buffer" itself -- so arguably it's not more "magical" than
save-buffer, since it *is* save-buffer.


Tomi, would you object if Mark's patches were somehow integrated into
save-buffer itself, instead of just re-mapping C-x C-s ?  (i don't know
elisp well enough to know how feasible that would be)

My bottom line:

Mark's series here works well for me, and represents the best draft
handling i've seen yet for notmuch-emacs.  It would be really good to
incorporate something approximating this functionality in the mainline.

        --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

Re: [PATCH v3] emacs: postpone/resume support

[Tomi Ollila]

On Fri, Jul 08 2016, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:

> [ Unknown signature status ]
> On Sun 2016-06-05 07:18:25 -0400, David Bremner wrote:
>> Tomi Ollila <tomi.ollila@iki.fi> writes:
>>
>>> ... but I think I am not biased when I think that if ctrl-x ctrl-s
>>> does something more magical than simple (save-buffer) in buffers that
>>> looks like text editing buffer, that is going to be unexpected disturbing
>>> functionality to someone (else like me)...
>>
>> This is a valid point. On the other hand the binding is copied from
>> Gnus, which also includes at least _some_ magic
>>
>>       http://www.gnu.org/software/emacs/manual/html_node/gnus/Drafts.html
>
> gnus isn't the only thing that does something more "magical" than saving
> a raw buffer to a file.
>
> try the following:
>
>    echo test | gpg --output test.gpg --armor --default-recipient-self  --encrypt
>    emacs test.gpg
>
> emacs will prompt you to decrypt the file, then let you edit the
> cleartext buffer.
>
> a subsequent "C-x C-s" will re-save it encrypted.  (at least it does for
> me!)
>
> [ i note several weirdnesses here around the round-trip cleanliness and
>   safety of this operation which aren't relevant to notmuch, but that's
>   a different story, and i've reported them to emacs upstream ]
>
> Of course, this functionality appears to also happen within
> "save-buffer" itself -- so arguably it's not more "magical" than
> save-buffer, since it *is* save-buffer.
>
>
> Tomi, would you object if Mark's patches were somehow integrated into
> save-buffer itself, instead of just re-mapping C-x C-s ?  (i don't know
> elisp well enough to know how feasible that would be)

customizable hook might be tolerable ;)

(quick reply writing coat on)

>
> My bottom line:
>
> Mark's series here works well for me, and represents the best draft
> handling i've seen yet for notmuch-emacs.  It would be really good to
> incorporate something approximating this functionality in the mainline.
>
>         --dkg

Re: Emacs: postponing messages

[Brian Sniffen]

I just found this thread from June while looking to resume a draft saved
from another program (dovecot) and which was given the 'draft' tag by
synchronization from Maildir flags.  Mark, thanks!

I *think* all I have to do to deal with other clients accessing the
Maildir through IMAP is to remove 'Message-ID from message-deletable
headers, because other clients may be watching for that---though this
only matters if the draft is resumed and postponed again, rather than
deleted.

Does that sound right?

      (when draft
       (notmuch-message-unquote-some-mml))

One comment here: anyone using maildir flag synchronization will have
items with tag:draft but no quoted MML.  To help them reasonably
*discuss* quoted MML, even when working from a mix of notmuch and other
MUAs, can this use a distinct tag?  For example, tag:quoted-mml?

Thanks,
Brian

Re: Emacs: postponing messages

[David Bremner]

Brian Sniffen <bts@evenmere.org> writes:

> I just found this thread from June while looking to resume a draft saved
> from another program (dovecot) and which was given the 'draft' tag by
> synchronization from Maildir flags.  Mark, thanks!
>
> I *think* all I have to do to deal with other clients accessing the
> Maildir through IMAP is to remove 'Message-ID from message-deletable
> headers, because other clients may be watching for that---though this
> only matters if the draft is resumed and postponed again, rather than
> deleted.
>
> Does that sound right?
>
>       (when draft
>        (notmuch-message-unquote-some-mml))
>
> One comment here: anyone using maildir flag synchronization will have
> items with tag:draft but no quoted MML.  To help them reasonably
> *discuss* quoted MML, even when working from a mix of notmuch and other
> MUAs, can this use a distinct tag?  For example, tag:quoted-mml?

I'm not sure if Mark saw this, as it wasn't connected to the original
thread.

d