Default for the envelope From in smtpmail.el

Default for the envelope From in smtpmail.el

[Lars Magne Ingebrigtsen]

Currently, the default envelope from address (aka. SMTP "MAIL FROM") in
smtpmail is `user-mail-address'.  The user can change this by changing
some rather confusing variables called `mail-specify-envelope-from' and
`mail-envelope-from'.

I wonder whether a more sensible default here would be to use the From:
address in the buffer being sent.  Some MTAs filter on the MAIL FROM,
and only allow access based on a filter, and people adjust their From:
headers to reflect that.

For instance, when they are at work with their laptops, they use "From:
foo@workplace.com", and at home "From: foo@gmail.com".

I have no idea whether changing this default will break more stuff than
it would help, though.  Thoughts?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Default for the envelope From in smtpmail.el

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 05 Jul 2011 17:16:22 +0200
> 
> Currently, the default envelope from address (aka. SMTP "MAIL FROM") in
> smtpmail is `user-mail-address'.  The user can change this by changing
> some rather confusing variables called `mail-specify-envelope-from' and
> `mail-envelope-from'.
> 
> I wonder whether a more sensible default here would be to use the From:
> address in the buffer being sent.  Some MTAs filter on the MAIL FROM,
> and only allow access based on a filter, and people adjust their From:
> headers to reflect that.
> 
> For instance, when they are at work with their laptops, they use "From:
> foo@workplace.com", and at home "From: foo@gmail.com".
> 
> I have no idea whether changing this default will break more stuff than
> it would help, though.  Thoughts?

I would suggest not to make any new-featurish changes at this point
that are not absolutely necessary.  We are just a couple of weeks from
the pretest -- do we really want to make Emacs as unstable as humanly
possible till then?

Re: Default for the envelope From in smtpmail.el

[Juanma Barranquero]

On Tue, Jul 5, 2011 at 19:16, Eli Zaretskii <eliz@gnu.org> wrote:

> We are just a couple of weeks from
> the pretest -- do we really want to make Emacs as unstable as humanly
> possible till then?

Yes, of course. As a challenge. Then we can try to beat the 333 days
between 21.0.90 and 21.1.

    Juanma

Re: Default for the envelope From in smtpmail.el

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Tue, 5 Jul 2011 21:40:06 +0200
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org
> 
> On Tue, Jul 5, 2011 at 19:16, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > We are just a couple of weeks from
> > the pretest -- do we really want to make Emacs as unstable as humanly
> > possible till then?
> 
> Yes, of course. As a challenge. Then we can try to beat the 333 days
> between 21.0.90 and 21.1.

We will break it anyway.  Wait until I dump all the new bidi stuff on
everyone and turn that on by default...

Re: Default for the envelope From in smtpmail.el

[Tim Cross]

On Wed, Jul 6, 2011 at 1:16 AM, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> Currently, the default envelope from address (aka. SMTP "MAIL FROM") in
> smtpmail is `user-mail-address'.  The user can change this by changing
> some rather confusing variables called `mail-specify-envelope-from' and
> `mail-envelope-from'.
>
> I wonder whether a more sensible default here would be to use the From:
> address in the buffer being sent.  Some MTAs filter on the MAIL FROM,
> and only allow access based on a filter, and people adjust their From:
> headers to reflect that.
>
> For instance, when they are at work with their laptops, they use "From:
> foo@workplace.com", and at home "From: foo@gmail.com".
>
> I have no idea whether changing this default will break more stuff than
> it would help, though.  Thoughts?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>  bloggy blog http://lars.ingebrigtsen.no/
>
>
>

Given the short time to pretest, I would wait.

I can't remember the full details, but I do distinctly remember
running into all sorts of issues with respect to envelope from and
mail servers configured to do masquerading and I vaguely remember some
issues relating to fetchmail and multi-drop mailboxes etc. I suspect
changes in this area may have some unexpected side effects that may
take some time to be realised/identified, so you would probably want
as long a test period as possible.

Tim

Re: Default for the envelope From in smtpmail.el

[Glenn Morris]

Lars Magne Ingebrigtsen wrote:

> Currently, the default envelope from address (aka. SMTP "MAIL FROM") in
> smtpmail is `user-mail-address'.  The user can change this by changing
> some rather confusing variables called `mail-specify-envelope-from' and
> `mail-envelope-from'.
>
> I wonder whether a more sensible default here would be to use the From:
> address in the buffer being sent.  Some MTAs filter on the MAIL FROM,
> and only allow access based on a filter, and people adjust their From:
> headers to reflect that.

This was an eminently sensible suggestion from July last year.
Please could it be now be implemented.

The current default is dumb and can lead to confusing failure modes
("553 From: address not verified").

Re: Default for the envelope From in smtpmail.el

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Fri, 10 Aug 2012 21:15:10 -0400
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>
> 
> Lars Magne Ingebrigtsen wrote:
> 
> > Currently, the default envelope from address (aka. SMTP "MAIL FROM") in
> > smtpmail is `user-mail-address'.  The user can change this by changing
> > some rather confusing variables called `mail-specify-envelope-from' and
> > `mail-envelope-from'.
> >
> > I wonder whether a more sensible default here would be to use the From:
> > address in the buffer being sent.  Some MTAs filter on the MAIL FROM,
> > and only allow access based on a filter, and people adjust their From:
> > headers to reflect that.
> 
> This was an eminently sensible suggestion from July last year.
> Please could it be now be implemented.
> 
> The current default is dumb and can lead to confusing failure modes
> ("553 From: address not verified").

Sorry, I cannot figure this out, neither the nature of the problem nor
the details of the proposed solution.  Can you spell that out?

In particular, is the proposal to stop using user-mail-address, or
stop using it in smtpmail.el only?  If the former, I certainly object,
because at least in my setup it is the source of the 'From:' header in
the first place.

If the proposal is for not using it only in smtpmail.el, then we need
to verify that there are no valid use cases without a 'From:' header
in the current buffer.  Or maybe use 'From:' if it exists and fall
back on the existing machinery if not.

Re: Default for the envelope From in smtpmail.el

[Glenn Morris]

Eli Zaretskii wrote:

> Sorry, I cannot figure this out, neither the nature of the problem nor
> the details of the proposed solution.  Can you spell that out?

You are writing an email. The From: header has whatever value it has.
Either it came from user-mail-address, or you edited it in the mail
composition buffer to say what you wanted it to say.

Now you send it. The envelope from address (which is used eg for the
MAIL FROM command sent to the stmp server) should default to whatever
was in the From: header. At the moment, it effectively defaults to
user-mail-address. If this is not the same as From:, you can get
confusing failures to send the mail.

If From: is the same as user-mail-address,, then the change has no
impact on you.

> In particular, is the proposal to stop using user-mail-address, or
> stop using it in smtpmail.el only?

No.

> If the proposal is for not using it only in smtpmail.el, then we need
> to verify that there are no valid use cases without a 'From:' header
> in the current buffer.  Or maybe use 'From:' if it exists and fall
> back on the existing machinery if not.

The proposal is effectively:

(setq mail-specify-envelope-from t
      mail-envelope-from 'header)

If you look at the start of smtpmail-send-it, this would do what you
suggest.

Re: Default for the envelope From in smtpmail.el

[Glenn Morris]

To put it another way: if an email has a From: header, default to using
that for the Envelope-From. I think the case of wanting to use something
different is much less common, and should be the one that requires
Emacs configuration.

Re: Default for the envelope From in smtpmail.el

[Stephen J. Turnbull]

Glenn Morris writes:

 > Now you send it. The envelope from address (which is used eg for the
 > MAIL FROM command sent to the stmp server) should default to whatever
 > was in the From: header.

This is a bad idea.  Emacs should set the envelope address only on
specific request from the user.  Provide an easily customized option,
yes.  Default, no.  The default is to let the MTA figure it out.

Users who change the From address had better know what they are doing.

 > At the moment, it effectively defaults to user-mail-address.

Don't you have this backward?  The envelope address is set by the MTA,
which defaults to using the mailbox of the logged-in user.  This
corresponds to user-mail-address because user-mail-address defaults to
the same value.  (I don't know or care what smtpmail.el currently
does, but here it is functioning as an MTA.)

Note that on some systems unprivileged users using the system MTA may
not be able to set the envelope address at all.

 > > If the proposal is for not using it only in smtpmail.el, then we need
 > > to verify that there are no valid use cases without a 'From:' header
 > > in the current buffer.

A mail message must have a From field to conform to RFC 5322 (see
table on p. 20 of that RFC).

Steve

Re: Default for the envelope From in smtpmail.el

[Rasmus]

Glenn Morris <rgm@gnu.org> writes:

> To put it another way: if an email has a From: header, default to using
> that for the Envelope-From. I think the case of wanting to use something
> different is much less common, and should be the one that requires
> Emacs configuration.

It is perhaps also worth noting that quite a few universities host their
mail on outlook.com or on a local outlook server (which I'd guess run an
Exchange server underneath).  These will /only/ allow you to send mails
using the above 'fix'.

–Rasmus

-- 
May the Force be with you

Re: Default for the envelope From in smtpmail.el

[chad]

On 11 Aug 2012, at 17:19, Stephen J. Turnbull <stephen@xemacs.org> wrote:

> Glenn Morris writes:
> 
>> Now you send it. The envelope from address (which is used eg for the
>> MAIL FROM command sent to the stmp server) should default to whatever
>> was in the From: header.
> 
> This is a bad idea.  Emacs should set the envelope address only on
> specific request from the user.  Provide an easily customized option,
> yes.  Default, no.  The default is to let the MTA figure it out.

I suspect that Glenn is talking about the case where Emacs is
pretending to be an MTA. Then again, I might be wrong. The large
number of possible mail configurations for emacs makes these
conversations tricky.

*Chad

Re: Default for the envelope From in smtpmail.el

[Glenn Morris]

chad wrote:

> I suspect that Glenn is talking about the case where Emacs is
> pretending to be an MTA.

Not as far as I know. I'm talking about plain old talking to an SMTP
server. smtpmail-via-smtp (abbreviated) does this:

(smtpmail-send-command "MAIL FROM:" envelope-from ...)

Currently envelope-from defaults to user-mail-address, which may bear no
relation to the From: header, which I argue is what the majority will
want to use. This is particularly likely to cause problems if you have
multiple email accounts, since there is only one value of
user-mail-address, which cannot be right for all of them. But I find it
hard to think of any case where you want to sent to MAIL FROM != From:.
They might exist, but should not be the default.

Re: Default for the envelope From in smtpmail.el

[Glenn Morris]

Glenn Morris wrote:

> I'm talking about plain old talking to an SMTP server.

Maybe I'm only talking about smtpmail.el, I haven't really looked at
sendmail.el. The _only_ thing smtpmail.el uses "envelope-from" for is to
send the MAIL FROM command to the server. It seems totally obvious to me
that the only sane default for this is whatever is in the From: header.
(Of course the option to set it to whatever you like should remain.)

Re: Default for the envelope From in smtpmail.el

[chad]

On 11 Aug 2012, at 19:08, Glenn Morris <rgm@gnu.org> wrote:

>> I suspect that Glenn is talking about the case where Emacs is
>> pretending to be an MTA.
> 
> Not as far as I know. I'm talking about plain old talking to an SMTP
> server.

Probably poor terminology on my part; I learned most of this when MSA and MTA meant basically the same thing (if they were even coined). This is the case that I meant: emacs speaking a network protocol, as opposed to formatting a file and handing it off to a locale program for delivery.

In the case Stephen was talking about, local users needed privileges to request an envelope-from (via comand-line switch or From: header) that's not the username. Those systems are still out there, but many people just use network services instead, entirely ignoring the local mail systems.  This is basically what emacs does when it talks directly to an SMTP server.

I'd be surprised that the current default ever worked, but with all the changes to the SMTP infrastructure in the past several months, it seems likely that everyone who cares fixes it for themselves (I used to do so myself, with some simple wrapper code around the mail sending code).

I was hoping to clarify for Stephen that you guys aren't talking about the same cases (at least, it seemed to me that you weren't).  I do agree with you about the better default choice, although I also imagine that it's still manually fixed by almost everyone who cares.

I hope that helps.  Sorry for adding to the noise.
*Chad

Re: Default for the envelope From in smtpmail.el

[Stephen J. Turnbull]

chad writes:
 > I was hoping to clarify for Stephen that you guys aren't talking
 > about the same cases (at least, it seemed to me that you weren't).

That's true.  AFAICT I was talking about a superset of the cases that
Glenn was, and that's important.

My basic point is that there are many different cases, and that Emacs
(when acting as an MTA, which is more or less equivalent to speaking
SMTP as it does via smtpmail.el) should try to make them all look the
same to users.  That means that smtpmail.el should *default* to using
the logged-in user's mailbox for the envelope sender.

It is true that there are fascist MTAs out there that will insist that
Sender and envelope sender be consistent (I doubt that From matters if
Sender is specified, but not very strongly ... Microsoft's sins in the
area of polluting the mail system are varied and often vicious).  As
long as they accept Sender == envelope sender, that's what I would
recommend.

Many, perhaps most, users don't know anything about Sender, and maybe
Exchange is so broken that the rule is From == envelope sender.  So
what?  As I wrote earlier, if the user is stuck with such a system and
chooses to change the From address, he'd better know what he's doing.
Among other things, he may be violating the rules of his network by
spoofing the envelope sender (eg, last time I checked this was true
for mine), or even by spoofing the From address.

Re: Default for the envelope From in smtpmail.el

[chad]

On 12 Aug 2012, at 08:02, Stephen J. Turnbull <stephen@xemacs.org> wrote:

> chad writes:
>> I was hoping to clarify for Stephen that you guys aren't talking
>> about the same cases (at least, it seemed to me that you weren't).
> 
> That's true.  AFAICT I was talking about a superset of the cases that
> Glenn was, and that's important.

It's not a superset in the meaningful case; it's a subset - of the users.

> My basic point is that there are many different cases, and that Emacs
> (when acting as an MTA, which is more or less equivalent to speaking
> SMTP as it does via smtpmail.el) should try to make them all look the
> same to users.  That means that smtpmail.el should *default* to using
> the logged-in user's mailbox for the envelope sender.

This is a default that will be wrong the vast majority of the time. It
seems silly to choose to default to an assumption that stopped being
valid ~15 years ago instead of the value that the user already picked.

The cases where a user's login name on an emacs-capable machine is the
same as a workable email address are a small minority, even of emacs
users, and a fair chunk of those cases *could* use login-name, but
don't want to. Work addresses use a company server. School addresses
use a university server or service. ISPs, webmail, etc - none of them
use the user name.

Conversely, user-mail-address has been used for decades, and it
already documented to be exactly what we want:

	Specify your own email address, if Emacs can't figure it out
	correctly.
          (setq user-mail-address "cheney@torture.gov")

	Various Emacs packages, such as Message mode, consult
	user-mail-address when they need to know your email address. 
	See	Mail Headers.

If the user hasn't told emacs about an email address, then using the
login name is a reasonable choice. If the user *has* told emacs their
email address, emacs should use it.

Re: Default for the envelope From in smtpmail.el

[Stephen J. Turnbull]

chad writes:

 > It's not a superset in the meaningful case; it's a subset - of the users.

Yes, it is, and no, it's not.  My mistake was elsewhere.

 > > That means that smtpmail.el should *default* to using the
 > > logged-in user's mailbox for the envelope sender.
 > 
 > This is a default that will be wrong the vast majority of the time.

Not at all.  `user-mail-address' is an inherited non-default setting
in my opinion.  My mistake was not explicitly acknowledging that your
opinion is different and at least as valid as mine.

 > The cases where a user's login name on an emacs-capable machine is the
 > same as a workable email address are a small minority,

Probably true.  Although I've never bothered on a personal GNU or BSD
system, since I've always been behind the relevant firewall where
relaying was open (not to mention that, until recently, so were
outgoing calls to port 25 :-( ).  I suppose a "sufficiently smart"
network security committee would prohibit both of those, though.  (Do
any of those exist?  But I digress....)

 > Conversely, user-mail-address has been used for decades,

And is the setting that is currently used by smtpmail.el according to
Glenn.  But he wants something else!  Glenn writes:

 >> To put it another way: if an email has a From: header, default to
 >> using that for the Envelope-From. I think the case of wanting to
 >> use something different is much less common, and should be the one
 >> that requires Emacs configuration.

That's what I was arguing against.  So in the end you and I are on the
same team, although I was way too fuzzy in my own head about where the
default mailbox needs to come from and how Emacs might find out what
it is.

 > If the user hasn't told emacs about an email address, then using the
 > login name is a reasonable choice. If the user *has* told emacs their
 > email address, emacs should use it.

And if the user decides to mess with From, he's on his own, at least
as far as Emacs proper is concerned.  No?

I suppose it would be possible to enhance user-mail-address to
something like user-mail-configurations, which would be a list of
lists like (ADDRESS FULLNAME LOGIN AUTHENTICATION EXTRA-HEADERS).  Of
course Gnus already supports this kind of thing (I forget the Gnus
name for the feature, as does VM ("personality crisis").

IIRC, Glenn was specifically referring to Gnus.  So if he rephrases
slightly to specify that MUA "personality" customizations should offer
this option and documentation explaining when it makes sense, I think
everybody would be happy.

In the long run, it might be nice if Emacs would offer this as part of
the compose-mail facility, and MUA packages would call back to
themselves (by default) through this facility.  Probably not going to
happen, though.

Re: Default for the envelope From in smtpmail.el

[Nix]

On 13 Aug 2012, Stephen J. Turnbull stated:
> I suppose it would be possible to enhance user-mail-address to
> something like user-mail-configurations, which would be a list of
> lists like (ADDRESS FULLNAME LOGIN AUTHENTICATION EXTRA-HEADERS).  Of
> course Gnus already supports this kind of thing (I forget the Gnus
> name for the feature, as does VM ("personality crisis").

By coincidence I just had to do exactly that:

,----
| (defvar nix-overriding-smtp-servers '(("nick.alcock@oracle.com" "[work.mailserver.host.censored]" 465 ssl))
|   "List of SMTP servers to override the default with.
| 
| An alist with each element of the form (EMAIL-ADDRESS SMTP-SERVER SMTP-PORT STREAM-TYPE).")
| 
| (defun nix-get-overriding-smtp-servers ()
|   (save-excursion
|     (loop with from = (save-restriction
|                         (message-narrow-to-headers)
|                         (message-fetch-field "from"))
|           for (addr server port stream-type) in nix-overriding-smtp-servers
|           when (string-match addr from) return (list server port stream-type)
|           finally return (list smtpmail-smtp-server smtpmail-smtp-service smtpmail-stream-type))))
| 
| (defadvice smtpmail-via-smtp
|   (around change-smtp-by-message-from-field (recipient buffer &optional ask) activate)
|   (with-current-buffer buffer
|     (progv '(smtpmail-smtp-server smtpmail-smtp-service smtpmail-stream-type)
|         (nix-get-overriding-smtp-servers)
|       ad-do-it)))
`----

Not killingly difficult, but something it would be nice if we didn't
have to do every time.

> IIRC, Glenn was specifically referring to Gnus.  So if he rephrases
> slightly to specify that MUA "personality" customizations should offer
> this option and documentation explaining when it makes sense, I think
> everybody would be happy.

Yes. As seen above, you can do it yourself, but you shouldn't need to,
not really: like gnus-posting-styles, this should be something Gnus (or
message mode) does for you.

-- 
NULL && (void)

Re: Default for the envelope From in smtpmail.el

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> The proposal is effectively:
>
> (setq mail-specify-envelope-from t
>       mail-envelope-from 'header)
>
> If you look at the start of smtpmail-send-it, this would do what you
> suggest.

Doesn't it make more sense to just make smtpmail default to using the
From header, and then people who don't want that, for some reason or
other, can just set `mail-envelope-from' explicitly to whatever they
want?

Looking at smtpmail.el, I think two `M-t's are all that are needed for
that to happen.  And a notice in NEWS.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Emacs

Re: Default for the envelope From in smtpmail.el

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Doesn't it make more sense to just make smtpmail default to using the
> From header, and then people who don't want that, for some reason or
> other, can just set `mail-envelope-from' explicitly to whatever they
> want?

I've now committed this.  If there's unintended consequences here, it
should be trivial enough to revert...

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Emacs

Re: Default for the envelope From in smtpmail.el

[Stephen J. Turnbull]

Nix writes:

 > Yes. As seen above, you can do it yourself, but you shouldn't need to,
 > not really: like gnus-posting-styles, this should be something Gnus (or
 > message mode) does for you.

That would suck. :-)  Seriously, I don't use Gnus, and I don't always
use message mode.  It's not like this kind of facility is Gnus-specific.

Emacs should do it.  This is something Python (and I suppose Perl,
Ruby, et al) does pretty well.  It's one thing that XEmacs was aiming
for (and missed badly, in my opinion) in splitting out the packages.

I understand why in practice every app has to reinvent these
facilities for itself (this kind of refactoring is not amazingly fun
work and all the benefit goes to third parties), but it's a shame.

Steve

Re: Default for the envelope From in smtpmail.el

[Nix]

On 5 Sep 2012, Stephen J. Turnbull uttered the following:

> Nix writes:
>
>  > Yes. As seen above, you can do it yourself, but you shouldn't need to,
>  > not really: like gnus-posting-styles, this should be something Gnus (or
>  > message mode) does for you.
>
> That would suck. :-)  Seriously, I don't use Gnus, and I don't always
> use message mode.  It's not like this kind of facility is Gnus-specific.

Well, at least Gnus needs to be able to talk to other modes, and if
message mode doesn't do it, who will? Substitute 'Gnus, or message mode,
or whatever weird modes exist for people who want to send messages some
other way', if you like, but I'm not sure this *can* be hardwired into
just smtpmail.el, and if it requires cooperation from other widely-used
modes, so be it.

> I understand why in practice every app has to reinvent these
> facilities for itself (this kind of refactoring is not amazingly fun
> work and all the benefit goes to third parties), but it's a shame.

What seems to happen is that significant packages (such as Gnus) sprout
modes for their own internal use that can then be reused by other
people, and are eventually split out of the significant packages into
their own thing. This requires a modicum of decent internal interface
design, but larsi has always shown himself equal to the task before now.

-- 
NULL && (void)