Re: [BUG] notmuch-emacs: spoils sendmail -f with emacs 27 commit 3a59cc84069376802ba8fd731b524d78db58262c and later

[Gregor Zattler <telegraph@gmx.net>]

Hi David,
* David Edmondson <dme@dme.org> [2019-12-03; 18:43]:
> On Sunday, 2019-12-01 at 18:10:59 +01, Gregor Zattler wrote:
>> I use notmuch-emacs configured with
>> (setq mail-specify-envelope-from t)
>> (setq mail-envelope-from 'header)
>> (setq send-mail-function 'sendmail-send-it)
>>
>> this should/used to invoke sendmail (in my case exim4) with the email
>> address given in the From: header of the message buffer as
>> argument to sendmails -f option.
>
> Wow, this was, err, “interesting”.
>
> Comments below.
>
>> Since emacs 27 commit 3a59cc84069376802ba8fd731b524d78db58262c
>>
>>      3a59cc84069376802ba8fd731b524d78db58262c
>>      Author:     Stefan Monnier <monnier@iro.umontreal.ca>
>>      AuthorDate: Tue Jul 30 16:37:01 2019 -0400
>>      Commit:     Stefan Monnier <monnier@iro.umontreal.ca>
>>      CommitDate: Tue Jul 30 16:37:01 2019 -0400
>>
>>      Parent:     add146f09f * lisp/bindings.el (mode-line-defining-kbd-macro): New defvar.
>>      Contained:  master
>>      Follows:    emacs-26.1 (6691)
>>
>>      * lisp/gnus/message.el: Reduce redundancy with send-mail-function
>>
>>      (message-send-mail-function) <function>: Remove `local-library` tests
>>      for libs distributed with Emacs.
>>      (message-use-send-mail-function): New function.
>>      (message-default-send-mail-function): Default to it, and remove cases
>>      already handled by it.
>>      (message--default-send-mail-function): New function.
>>      (message-send-mail-function) <variable>: Use it as new default.
>>      (message-sendmail-f-is-evil): Obey mail-specify-envelope-from if available.
>>      (message-check, message-with-reply-buffer): Use `declare`.
>>      (message-smtpmail-send-it): smtpmail accepts mail-header-separator,
>>      so simplify and declare obsolete.
>>      (message-send-mail-with-mailclient): Declare obsolete.
>>      (message-check-news-body-syntax): Don't presume that the checksum is
>>      a fixnum.
>>
>>
>> this actually invokes sendmail -f with an address derived from the
>> EMAIL environment variable.
>>
>>
>> It took me some time to understand, that there is no problem with
>> compose-mail/message-send-and-exit which are bound to ^X m and ^C
>> ^C respectively within a barely configured emacs which I used for
>> testing.
>> If instead notmuch is loaded compose-mail/message-send-and-exit
>> and notmuch-mua-new-mail/notmuch-mua-send behave different.
>>
>> With this command (with test.el as attached):
>>
>> ~/src/emacs/src/emacs -Q -nw -l /tmp/test.el
>>
>> compose-mail/message-send-and-exit rightly show
>> -f example@example.org in the file /tmp/result (and then there is
>> an error because the notmuch commands are not known).
>>
>> While this command
>>
>> ~/src/emacs/src/emacs -Q -nw -L ~/src/notmuch/emacs/ --eval "(require 'notmuch)" -f notmuch -l /tmp/test.el
>>
>> wrongly show some other value for -f, depending on your EMAIL
>> environment variable, for both compose-mail/message-send-and-exit
>> and notmuch-mua-new-mail/notmuch-mua-send.

>> (find-file "/tmp/sent")
>> (insert "empty")
>> (kill-region (point-min) (point-max))
>> (write-file "/tmp/sent")
>> (kill-buffer)
>>
>> (find-file "/tmp/result")
>> (kill-region (point-min) (point-max))
>> (write-file "/tmp/result")
>> (kill-buffer)
>>
>> (find-file "/tmp/show-sendmail-args.sh")
>> (kill-region (point-min) (point-max))
>> (insert "#!/bin/sh
>> echo $* >> /tmp/result")
>> (write-file "/tmp/show-sendmail-args.sh")
>> (chmod "/tmp/show-sendmail-args.sh" 448)
>>
>> (setq sendmail-program "/tmp/show-sendmail-args.sh")
>> (setq mail-specify-envelope-from t)
>> (setq mail-envelope-from 'header)
>> (setq send-mail-function 'sendmail-send-it)
>>
>> (compose-mail "Echo Mail Server <echo@univie.ac.at>" "test"
>>                '((From: . "example@example.org")))
>
> This example, and the one below, are broken - you can't use “From:”
> here, it has to be “From”. Changing that doesn't fix it, though.

OK, thanks.

>> (message-send-and-exit)
>>
>> (notmuch-mua-mail "Echo Mail Server <echo@univie.ac.at>" "test"
>>                '((From: . "example@example.org")))
>> (notmuch-mua-send-and-exit)
>>
>> (save-buffers-kill-terminal)
>
> As best I can determine, this relates to the order in which things are
> loaded.
>
> If you load message.el before setting `mail-specify-envelope-from',
> things are broken (sorry for the long lines):
>
> disaster-area ~/s/emacs % ./src/emacs -Q -nw -batch --eval "(progn (require 'message) (setq mail-specify-envelope-from t mail-envelope-from 'header) (message \"%s\" message-sendmail-envelope-from))"
> nil
> disaster-area ~/s/emacs % 
>
> ...but if you load it after, things work fine:
>
> disaster-area ~/s/emacs % ./src/emacs -Q -nw -batch --eval "(progn (setq mail-specify-envelope-from t mail-envelope-from 'header) (require 'message) (message \"%s\" message-sendmail-envelope-from))"
> header
> disaster-area ~/s/emacs %
>
> This doesn't appear to be new behaviour - 26.1 does the same.

this is astonishing, since my test script behaves different with
emacs build with different last commits!?

> This is related to the way that `message-sendmail-envelope-from' is
> initialised from `mail-envelope-from', and it's
> `message-sendmail-envelope-from' that matters, because you end up in
> `message-send-mail-with-sendmail'.

At the moment I cannot follow...

> message.el is being loaded by “(require 'notmuch)” in your example,
> which is happening before `mail-specify-envelope-from' is set, so you
> see the failure mode described.

... but in my example the setq forms precede every call to
message/notmuch (as in your example)!? [And in my emacs
configuration the customizations made with emacs' customization
interface (among others the three variables which are setq in my
test script) are loaded before almost anything else, especially
message/notmuch, see below]

> The example you provided is obviously contrived to show the problem -
> are you hitting it in normal use?

Absolutely.  I struggle with this at least since 2nd of August and have
reproduced this behaviour dozens of times.  It took my several
attempts to isolate the problem.

I use https://github.com/xundeenergie/exim4-multiaccount version
1.57 as of 2018-06-05 in order to send emails through different
smarthosts.

I do this every day and with emacs from sources newer than commit
01739625704aaaea6831cef459a4a53171689513 I'm e.g. not able to send
emails with my work email address ot to this mailing list.  This
is especially annoying since sending fails silently.  So
relatively important emails as e.g. announcements of downtimes did
not reach their intended audience.


The whole thing is above my elisp knowledge.  Is it possible you
describe something which is not the problem I described?

Ciao; Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-