bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[rameiko87]

Disclaimer: I use Rmail on Emacs, and Message-Mode.

Bug Part 1:

The From: header is ignored (from the message as saved in 'queued email 
folder') when sending with smtpmail-send-queued-mail, and takes the 
value of user-mail-address when this wasn't previously customized by the 
user. This is really worrisome! What else is ignored and rewritten 
without warning when invoking smtpmail-send-queued-mail?

Followup to Bug Part 1:

It would be interesting to figure out whether this happens also when 
user-mail-address had been previously customized by the user.

Bug Part 2:

This discrepancy between the behaviour of queued-sending and immediate 
sending (I use the default functions for the most current version of 
Emacs as of today) I consider to be a bug on itself. Because not doing 
things consistently confuses the user (even not doing the *wrong* thing 
consistently confuses the user. In this case the wrong thing is 
rewriting the From: header of every email).

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

rameiko87@posteo.net writes:

> Disclaimer: I use Rmail on Emacs, and Message-Mode.
>
> Bug Part 1:
>
> The From: header is ignored (from the message as saved in 'queued
> email folder') when sending with smtpmail-send-queued-mail, and takes
> the value of user-mail-address when this wasn't previously customized
> by the user. This is really worrisome! What else is ignored and
> rewritten without warning when invoking smtpmail-send-queued-mail?

This is not what I observe.  Like you, I have a one word (without dot)
hostname: "computer".  Then, I do:

   - emacs -Q
   - M-: (setq send-mail-function 'smtpmail-send-it)
   - M-: (setq smtpmail-queue-mail t)
   - M-x message-mail
  [ here the "From:" is set to
    manuel@computer.mail-host-address-is-not-set]
   - I fill "To:" and 'C-c C-c'
  [ now another mail… ]
   - M-x message-mail
  [ I fill "To:" and change the "From:" to manuel@elsewhere ]
   - 'C-c C-c'
  [ do send now… ]
   - M-x smtpmail-send-queued-mail

Both mails are sent and when I retrieve them both have the "From:"
exactly how it was: respectively
manuel@computer.mail-host-address-is-not-set and manuel@elsewhere

> Followup to Bug Part 1:
>
> It would be interesting to figure out whether this happens also when
> user-mail-address had been previously customized by the user.
>
> Bug Part 2:
>
> This discrepancy between the behaviour of queued-sending and immediate
> sending (I use the default functions for the most current version of
> Emacs as of today) I consider to be a bug on itself. Because not doing
> things consistently confuses the user (even not doing the *wrong*
> thing consistently confuses the user. In this case the wrong thing is
> rewriting the From: header of every email).

I have tested the same recipe as above (without setting
smtpmail-queue-mail to t) and it works correctly for me and the "From:"
of the two retrieved mails are also the same.  Could you test this
recipe?
-- 
Manuel Giraud

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[rameiko87]

Try the following to reproduce the core of the bug which I'm talking 
about on GNU Emacs 29.1:

Open emacs as follows:
emacs -Q

For each of the following lines, do M-: and then paste the line, press 
enter.

(setq message-mail-user-agent t)  BUT I THINK EVEN IF YOU SKIP THIS, THE 
SAME RESULT OCCURS

(setq send-mail-function 'smtpmail-send-it)

(setq smtpmail-queue-mail t)

(setq smtpmail-smtp-server "INSERT YOUR SMTP SERVER HERE")

(setq smtpmail-smtp-service 465)

(setq smtpmail-stream-type 'ssl)

Now generate a new message:
C-x m

Now fill in the headers To: to one of your addresses, From: to another 
of your addresses (mine have different domain, and the To: is the same 
domain as the smtp server), Subject: test, Body: test

C-c C-s

M-x smtpmail-send-queued-mail

Press enter, and I get the following message:

smtpmail-send-queued-mail: Sending failed: 504 5.5.2 <user@fedora>: 
Sender address rejected: need fully-qualified address


>> Bug Part 1:
>> 
>> The From: header is ignored (from the message as saved in 'queued
>> email folder') when sending with smtpmail-send-queued-mail, and takes
>> the value of user-mail-address when this wasn't previously customized
>> by the user. This is really worrisome! What else is ignored and
>> rewritten without warning when invoking smtpmail-send-queued-mail?
>> 
>> Followup to Bug Part 1:
>> 
>> It would be interesting to figure out whether this happens also when
>> user-mail-address had been previously customized by the user.
>> 
>> Bug Part 2:
>> 
>> This discrepancy between the behaviour of queued-sending and immediate
>> sending (I use the default functions for the most current version of
>> Emacs as of today) I consider to be a bug on itself. Because not doing
>> things consistently confuses the user (even not doing the *wrong*
>> thing consistently confuses the user. In this case the wrong thing is
>> rewriting the From: header of every email).

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

rameiko87@posteo.net writes:

[...]

> smtpmail-send-queued-mail: Sending failed: 504 5.5.2 <user@fedora>:
> Sender address rejected: need fully-qualified address

This is strange because I have followed your recipe (manually modifying
To: and From: to manuel@ledu-giraud.fr) and it works as expected: the
message is delivered and there is no trace of the local user/hostname in
it.
-- 
Manuel Giraud

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[rameiko87]

We need to get to the bottom of this! If the steps are the same then you 
must be using a different version... Are you using Emacs 29.1?

On 25.01.2024 09:31, Manuel Giraud wrote:
> 
> [...]
> 
>> smtpmail-send-queued-mail: Sending failed: 504 5.5.2 <user@fedora>:
>> Sender address rejected: need fully-qualified address
> 
> This is strange because I have followed your recipe (manually modifying
> To: and From: to manuel@ledu-giraud.fr) and it works as expected: the
> message is delivered and there is no trace of the local user/hostname 
> in
> it.

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

rameiko87@posteo.net writes:

> We need to get to the bottom of this! If the steps are the same then
> you must be using a different version... Are you using Emacs 29.1?

Yes I'm on recent master.  I'll test on 29.1.
-- 
Manuel Giraud

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[rameiko87]

Please do, because 29.2 is not packaged for guix, so I don't know how to 
install it. Otherwise I would test myself with 29.2...

On 25.01.2024 11:48, Manuel Giraud wrote:
> 
>> We need to get to the bottom of this! If the steps are the same then
>> you must be using a different version... Are you using Emacs 29.1?
> 
> Yes I'm on recent master.  I'll test on 29.1.

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

rameiko87@posteo.net writes:

> Please do, because 29.2 is not packaged for guix, so I don't know how
> to install it. Otherwise I would test myself with 29.2...

Hi,

So I did the following test on 29.1 (built from source from the
emacs-29.1 branch):

      - emacs -Q
      - M-: (setq send-mail-function 'smtpmail-send-it
                  smtpmail-queue-mail t
                  smtpmail-smtp-server "ledu-giraud.fr")
      - M-: (compose-mail)
      [ manually change "To:" and "From:" to manuel@ledu-giraud.fr ]
      - C-c C-c
      - M-: (smtpmail-send-queued-mail)

And this mail is correctly delivered.  The only trace left of the local
hostname is in "Message-ID:" which was set to
<87dvxoapt1.fsf@computer.mail-host-address-is-not-set>

So, I cannot see evidence of a bug in Emacs here.
-- 
Manuel Giraud

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[rameiko87]

On 26.01.2024 16:37, Manuel Giraud wrote:
> 
>> Please do, because 29.2 is not packaged for guix, so I don't know how
>> to install it. Otherwise I would test myself with 29.2...
> 
> Hi,
> 
> So I did the following test on 29.1 (built from source from the
> emacs-29.1 branch):
> 
>       - emacs -Q
>       - M-: (setq send-mail-function 'smtpmail-send-it
>                   smtpmail-queue-mail t
>                   smtpmail-smtp-server "ledu-giraud.fr")
>       - M-: (compose-mail)
>       [ manually change "To:" and "From:" to manuel@ledu-giraud.fr ]
>       - C-c C-c
>       - M-: (smtpmail-send-queued-mail)
> 
> And this mail is correctly delivered.  The only trace left of the local
> hostname is in "Message-ID:" which was set to
> <87dvxoapt1.fsf@computer.mail-host-address-is-not-set>
> 
> So, I cannot see evidence of a bug in Emacs here.

Ok I think I solved the conundrum! It has to do with the "envelope 
 From", and I think the reason why you can't reproduce the error is 
because your smtp server allows envelope-from which are different from 
the From: header (and my smtp server doesn't). So you don't _see_ the 
bug. Let's keep start emacs -Q and let's set the parameters we 
discussed.

For some reason, the envelope-from when sending via 
smtpmail-send-queued-mail is assigned to the value of user-mail-address 
(which for me was user@fedora) independently of the From: header. But 
with "immediate sending" this doesn't happen (I wonder if it's assigned 
to the same value as the header, or is not assigned at all and the smtp 
server assigns it? I don't know much about how these protocols work). In 
fact, if I just set a valid email for user-mail-address, then I get a 
different error, that is "envelope-from mismatches the From: header". 
Now, I could solve it by fiddling with

'(mail-envelope-from 'header)
'(mail-specify-envelope-from t)

and at this point I can send queued email, but this is really a fishy 
business and this fact that the value assigned to envelope-from changes 
between using "immediate sending" and queued-sending I think is the real 
core of the bug. Because I don't want the envelopes which I prescribed 
above to insert a header X-Authentication-Warning in my message, then I 
set '(message-sendmail-f-is-evil t), and looking at the description of 
this variable I discover it mentions bug#36937 which is very similar to 
the strange behaviour I described.

I will try to explore, with more calm and over the next few weeks, what 
is going on. But this is really a fishy business!

bug#68686: [Bug] smtpmail-send-queued-mail ignores some headers of the message it's sending!

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

rameiko87@posteo.net writes:

> On 26.01.2024 16:37, Manuel Giraud wrote:
>> 
>>> Please do, because 29.2 is not packaged for guix, so I don't know how
>>> to install it. Otherwise I would test myself with 29.2...
>> Hi,
>> So I did the following test on 29.1 (built from source from the
>> emacs-29.1 branch):
>>       - emacs -Q
>>       - M-: (setq send-mail-function 'smtpmail-send-it
>>                   smtpmail-queue-mail t
>>                   smtpmail-smtp-server "ledu-giraud.fr")
>>       - M-: (compose-mail)
>>       [ manually change "To:" and "From:" to manuel@ledu-giraud.fr ]
>>       - C-c C-c
>>       - M-: (smtpmail-send-queued-mail)
>> And this mail is correctly delivered.  The only trace left of the
>> local
>> hostname is in "Message-ID:" which was set to
>> <87dvxoapt1.fsf@computer.mail-host-address-is-not-set>
>> So, I cannot see evidence of a bug in Emacs here.
>
> Ok I think I solved the conundrum! It has to do with the "envelope
> From", and I think the reason why you can't reproduce the error is
> because your smtp server allows envelope-from which are different from
> the From: header (and my smtp server doesn't). So you don't _see_ the
> bug. Let's keep start emacs -Q and let's set the parameters we
> discussed.
>
> For some reason, the envelope-from when sending via
> smtpmail-send-queued-mail is assigned to the value of
> user-mail-address (which for me was user@fedora) independently of the
> From: header. But with "immediate sending" this doesn't happen (I
> wonder if it's assigned to the same value as the header, or is not
> assigned at all and the smtp server assigns it? I don't know much
> about how these protocols work). In fact, if I just set a valid email
> for user-mail-address, then I get a different error, that is
> "envelope-from mismatches the From: header". Now, I could solve it by
> fiddling with
>
> '(mail-envelope-from 'header)
> '(mail-specify-envelope-from t)
>
> and at this point I can send queued email, but this is really a fishy
> business and this fact that the value assigned to envelope-from
> changes between using "immediate sending" and queued-sending I think
> is the real core of the bug.

Good.  It seems that you have tracked this issue down.  You're right
that the behaviour between "queued sending" and "immediate sending"
should not diverge here.  But on this matter it seems that there is a
lot of possible customization: for example, in message.el, there is also
'message-sendmail-envelope-from'.
-- 
Manuel Giraud