gmail+SMTP(only) (oauth2)

gmail+SMTP(only) (oauth2)

[Uwe Brauer]

Hi

Today I got the unofficial approval to forward my email to an account of
my choice, if I can't access my mail with TLS (SSL or the gmail app
password anymore)

However I have been warned that the message should have a from field
with a domain of my university. (@mat.ucm.es for example).

Now I could either use

    1. The smtpmail (or sendmail) program of my linux machine (not sure
       about MacOS

    2. Or use a SMTP service that allows me to use a different form
       field, once the address has been verified. Gmail did this in the
       past (and maybe still does it).

In any case I have been warned that my mails could be blacklisted.

Anybody has experience with such a service/setting?

Regards

Uwe Brauer 

-- 
I strongly condemn Putin's war of aggression against the Ukraine.
I support to deliver weapons to Ukraine's military. 
I support the ban of Russia from SWIFT.
I support the EU membership of the Ukraine. 

gmail+SMTP(only) (oauth2)

[Bob Rogers]

   From: Uwe Brauer <oub@mat.ucm.es>
   Date: Tue, 17 May 2022 18:34:39 +0200

   Hi

   Today I got the unofficial approval to forward my email to an account
   of my choice, if I can't access my mail with TLS (SSL or the gmail
   app password anymore)

   However I have been warned that the message should have a from field
   with a domain of my university. (@mat.ucm.es for example).

   Now I could either use

       1. The smtpmail (or sendmail) program of my linux machine (not
	  sure about MacOS

I have implemented this for my own domains (principally rgrjr.com) for
over a year now with acceptable results.  The only catch is that emails
forwarded from Google have to be relayed through a VPS in order to avoid
my ISP's embargo on port 25, since Google does not offer forwarding to
an arbitrary port.  But this setup should work just as well for a domain
for which you are not the admin, and will give you complete control over
how to process messages on your home GNU/Linux system.  And the setup
for outgoing email is no different than before.

   . . .

   In any case I have been warned that my mails could be blacklisted.

   Anybody has experience with such a service/setting?

   Regards

   Uwe Brauer 

My emails do have some tendency to end up in peoples' spam folders.  But
whether that's more or less than for other senders (including some GMail
clients who have tried to send to me) is harder to say.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: gmail+SMTP(only) (oauth2)

[Tim Cross]

Uwe Brauer <oub@mat.ucm.es> writes:

> Hi
>
> Today I got the unofficial approval to forward my email to an account of
> my choice, if I can't access my mail with TLS (SSL or the gmail app
> password anymore)

So are you saying your institution won't allow you to use app passwords?
If they do, I would highly recommend going that route as the most
reliable and easily maintained approach. Note that even with the app
password solution, you still are using TLS both with imap and smtp. 

>
> However I have been warned that the message should have a from field
> with a domain of my university. (@mat.ucm.es for example).
>
> Now I could either use
>
>     1. The smtpmail (or sendmail) program of my linux machine (not sure
>        about MacOS

No, at least not on its own. You may well use smtpmail to communicate
with an SMTP server, but you will need to identify a new smtp server you
can use which will allow you to use a from header which is not the
'standard' for that server. You cannot just use an SMTP server running
on your local Linux desktop - at least not if you want to ensure
reliable mail service and avoid having your messages dropped into
blackholes and anti-spam quarantine systems. Some services might allow
you to configure your local desktop sendmail (or postfix or whatever) as
a satelite/smarhost which relays messages to the main SMTP service, but
your really just adding complexity with little benefit and will likely
run into all sorts of ISP issues (especially if your system is a laptop
which connects via different networks).  

>
>     2. Or use a SMTP service that allows me to use a different form
>        field, once the address has been verified. Gmail did this in the
>        past (and maybe still does it).
>

Yes, you will need this. You will likely communicate with this server
using smtpmail. However, in addition to allowing you to set the from
field, you probably also need a service which will handle the SPF and
various other anti-spam headers correctly. This can be very complex and
few services seem to do this correctly. 

Creating such setups when you actually own the email domain is somewhat
easier. Doing it when you are just a client of that domain (as in your
case) is much much more difficult. 

> In any case I have been warned that my mails could be blacklisted.
>

Yes, almost certainly will. The real challenge will be in getting the
various spam prevention schemes to work correctly. This is very
difficult when your using an SMTP server which is not part of the domain
your email appears to come from. When you own the domain, there are
various things you can do with DNS records to allow messages sent via a
different domain, but you don't own the domain and so cannot manage such
records. This would have to be done by the University and they almost
certainly won't be willing to do that. 

> Anybody has experience with such a service/setting?

I have run into this issue for various clients in the past. It is a pain
to get sorted out and you need a really accommodating SMTP service
provider (very difficult to find unless you are willing to fork out real
cash). 

I would check with your institution to see if they would be agreeable to
having a reply-to: header which points to your institutional address
rahter than putting it in the from: header. This would cause less issues
and would still ensure the replies to your messages go through the
University mail system (which is what I'm assuming is the reason for
their request to have their domain as the from line). The downside wiht
reply-to: is that it depends on the message recipient's mail client
honouring that header. While most clients will, some do not. 

I would strongly encourage you to try and use the app password solution.
The alternative you are proposing is possible, but is not for the faint
hearted and requires significant deep understanding of both SMTP and the
various evolving spam prevention frameworks as well as an SMTP service
partner who actually has the in-house expertise to run a correctly
configured and maintained SMTP server.   

Bottom line: the app passwords solution 'just works'. Just generate the
app passwords and use them instead of your normal password for imap and
smtp. The altgernative you are proposing will be unlikely to work 100%,
will require on-going maintenance and will likely be fragile and result
in a higher percentage of your messages being blocked or quarantined.

Re: gmail+SMTP(only) (oauth2)

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 2768 bytes --]

>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:

> Uwe Brauer <oub@mat.ucm.es> writes:

>> Hi
>> 
>> Today I got the unofficial approval to forward my email to an account of
>> my choice, if I can't access my mail with TLS (SSL or the gmail app
>> password anymore)

> So are you saying your institution won't allow you to use app passwords?
> If they do, I would highly recommend going that route as the most
> reliable and easily maintained approach. Note that even with the app
> password solution, you still are using TLS both with imap and smtp. 


No, I did not say that, but I have seen rumors that google will drop also
the app password approach, in which case this route is not any longer
possible.

However, I don't remember where I have seen it.

Do you have information regarding this subject?

>> 
>> However I have been warned that the message should have a from field
>> with a domain of my university. (@mat.ucm.es for example).
>> 
>> Now I could either use
>> 
>> 1. The smtpmail (or sendmail) program of my linux machine (not sure
>> about MacOS

> No, at least not on its own. You may well use smtpmail to communicate
> with an SMTP server, but you will need to identify a new smtp server you
> can use which will allow you to use a from header which is not the
> 'standard' for that server. You cannot just use an SMTP server running
> on your local Linux desktop - at least not if you want to ensure
> reliable mail service and avoid having your messages dropped into
> blackholes and anti-spam quarantine systems. Some services might allow
> you to configure your local desktop sendmail (or postfix or whatever) as
> a satelite/smarhost which relays messages to the main SMTP service, but
> your really just adding complexity with little benefit and will likely
> run into all sorts of ISP issues (especially if your system is a laptop
> which connects via different networks).  

I see, I might be able that my machine will be registered in our DNS
databank as 

machine-name.mat.ucm.es

I am not sure that it will help if I use my Linux SMTP server, [1].

Especially if I am using my laptop on any other institutions or at home,
since I then will have an IP address that is not within the range of the
IP addresses my university uses. I could use VPN but that slows down
things and, it also does not allow me to use newservers since that is
blocked when using non static university's IP, sigh.

If this approach (registering my machine) does not help, I have to pray
that google will stick to app passwords for the foreseeable future.

Regards

Uwe 

Footnotes:
[1]  (I need the from to be oub@mat.ucm.es, because otherwise my SMIME
     certificate is not working)


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: gmail+SMTP(only) (oauth2)

[Tim Cross]

Uwe Brauer <oub@mat.ucm.es> writes:

> [[S/MIME Signed Part:Undecided]]
>>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:
>
>> Uwe Brauer <oub@mat.ucm.es> writes:
>
>>> Hi
>>> 
>>> Today I got the unofficial approval to forward my email to an account of
>>> my choice, if I can't access my mail with TLS (SSL or the gmail app
>>> password anymore)
>
>> So are you saying your institution won't allow you to use app passwords?
>> If they do, I would highly recommend going that route as the most
>> reliable and easily maintained approach. Note that even with the app
>> password solution, you still are using TLS both with imap and smtp. 
>
>
> No, I did not say that, but I have seen rumors that google will drop also
> the app password approach, in which case this route is not any longer
> possible.
>
> However, I don't remember where I have seen it.
>
> Do you have information regarding this subject?
>

My recollection is that when Google originally announced the intended
changes (back in 2017 or 2018), they indicated they would be removing
app passwords. However, when I go through their current announcements
and documentation on their support site, I cannot find any mention of
removing them. Furthermore, much of their documentation appears to have
been updated to state that app passwords are the preferred solution for
clients who cannot do oauth2/2FA. My guess is that there was sufficient
push-back after the original announcement that they reviewed their
proposal and are going to leave app passwords in place for the time
being. Even if they do decide to remove them, I would expect they will
give a fairly lengthy notice period (It has been 4+ years since they
flagged the removal of being able to use your normal google password). 

I would just use app passwords and wait to see what happens. A lot can
change in a couple of years. 

>>> 
>>> However I have been warned that the message should have a from field
>>> with a domain of my university. (@mat.ucm.es for example).
>>> 
>>> Now I could either use
>>> 
>>> 1. The smtpmail (or sendmail) program of my linux machine (not sure
>>> about MacOS
>
>> No, at least not on its own. You may well use smtpmail to communicate
>> with an SMTP server, but you will need to identify a new smtp server you
>> can use which will allow you to use a from header which is not the
>> 'standard' for that server. You cannot just use an SMTP server running
>> on your local Linux desktop - at least not if you want to ensure
>> reliable mail service and avoid having your messages dropped into
>> blackholes and anti-spam quarantine systems. Some services might allow
>> you to configure your local desktop sendmail (or postfix or whatever) as
>> a satelite/smarhost which relays messages to the main SMTP service, but
>> your really just adding complexity with little benefit and will likely
>> run into all sorts of ISP issues (especially if your system is a laptop
>> which connects via different networks).  
>
> I see, I might be able that my machine will be registered in our DNS
> databank as 
>
> machine-name.mat.ucm.es

That might help if the initial sending SMTP server is on your machine
and it is registered in the global DNS (not just an internal only DNS).
It means your machine has a real IP address (not a non-routed address
like 192.168.*.* or 10.*.*.* etc). This would have a higher likelihood
of working if your University still runs its own SMTP server and you are
able to relay through it (some institutions do this to support internal
applications which need to send mail). However, it is very likely the
University will not allow connections out to external SMTP servers
(often, spam is generated by a compromised machine connecting to an SMTP
server and relaying spam etc, so standard practice is to block those
ports). However, it probably won't work if your just connecting to an
external SMTP server (using smtpmail or some other user mail agent).

>
> I am not sure that it will help if I use my Linux SMTP server, [1].
>
> Especially if I am using my laptop on any other institutions or at home,
> since I then will have an IP address that is not within the range of the
> IP addresses my university uses. I could use VPN but that slows down
> things and, it also does not allow me to use newservers since that is
> blocked when using non static university's IP, sigh.

Yes, that is one reason fewer people run their own SMTP server these
days. In addition to all the extra hassles associated with spam
prevention there is the complexity added by being mobile and connecting
via different networks. 

>
> If this approach (registering my machine) does not help, I have to pray
> that google will stick to app passwords for the foreseeable future.
>

One of the reasons this is so hard to work out is because there are just
so many different options and moving parts and the constant need to
evolve and adapt anti-spam measures as the spammers find new loopholes
and ways to circumvent spam prevention. I suspect it is one reason many
organisations have decided to adopt an external service provider and
drop in-house SMTP services. 

Re: gmail+SMTP(only) (oauth2)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It looks like we are making progress on the case of using gmail only
to receive incoming mail (by forwarding it).  Assuming you can get an email
account in a place with better policies than Google's.

Do schools and employers ever require that you _send_ them mail
from the Google account you're ordered to use?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: gmail+SMTP(only) (oauth2)

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

>>> "RS" == Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> It looks like we are making progress on the case of using gmail only
> to receive incoming mail (by forwarding it).  Assuming you can get an email
> account in a place with better policies than Google's.

To be completely clear here. 

    1. That information was a bit unofficial.

    2. Most likely the legal department would warn the user, that in the
       case of missing/disappearing messages it is entirely the user's
       fault, since he/she forwarded the messages. That enough will
       scare most of the users.

> Do schools and employers ever require that you _send_ them mail
> from the Google account you're ordered to use?

I can only speak for my university and the answer is «Yes» it does do,
and that is the core of the problem.

I thought I could modify my «From field» and use either some external SMTP
servers or a SMTP server ones installed on my own machine), however
messages by users more knowledgeable than me convinced me that such
messages will be either blocked or marked as SPAM.

So for the moment I only see two alternatives.

    1. Use google app-password (for this approach one can still use gnus
       (and I presume rmail, vm, etc) vanilla).

    2. Switch to oauth2 (with all the hassle involved).


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: gmail+SMTP(only) (oauth2)

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 2189 bytes --]


> Uwe Brauer <oub@mat.ucm.es> writes:


> My recollection is that when Google originally announced the intended
> changes (back in 2017 or 2018), they indicated they would be removing
> app passwords. However, when I go through their current announcements
> and documentation on their support site, I cannot find any mention of
> removing them. Furthermore, much of their documentation appears to have
> been updated to state that app passwords are the preferred solution for
> clients who cannot do oauth2/2FA. My guess is that there was sufficient
> push-back after the original announcement that they reviewed their
> proposal and are going to leave app passwords in place for the time
> being. Even if they do decide to remove them, I would expect they will
> give a fairly lengthy notice period (It has been 4+ years since they
> flagged the removal of being able to use your normal google password). 

Aha, thanks, htat is helpful, and good to know.

> I would just use app passwords and wait to see what happens. A lot can
> change in a couple of years. 

Right.

> That might help if the initial sending SMTP server is on your machine
> and it is registered in the global DNS (not just an internal only DNS).
> It means your machine has a real IP address (not a non-routed address
> like 192.168.*.* or 10.*.*.* etc). 
[Snip]...


> One of the reasons this is so hard to work out is because there are just
> so many different options and moving parts and the constant need to
> evolve and adapt anti-spam measures as the spammers find new loopholes
> and ways to circumvent spam prevention. I suspect it is one reason many
> organisations have decided to adopt an external service provider and
> drop in-house SMTP services. 

Right, that makes sense. I am now convinced that this idea of using my
own SMTP server will fail and I will therefore give the app-password a
try.

Thanks for you detailed explanation.

Uwe 


-- 
I strongly condemn Putin's war of aggression against the Ukraine.
I support to deliver weapons to Ukraine's military. 
I support the ban of Russia from SWIFT.
I support the EU membership of the Ukraine. 

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: gmail+SMTP(only) (oauth2)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > However I have been warned that the message should have a from field
  > with a domain of my university. (@mat.ucm.es for example).

  > Now I could either use

  >     1. The smtpmail (or sendmail) program of my linux machine (not sure
  >        about MacOS

  >     2. Or use a SMTP service that allows me to use a different form
  >        field, once the address has been verified. Gmail did this in the
  >        past (and maybe still does it).

  > In any case I have been warned that my mails could be blacklisted.

That is rather unclear.  Do you mean that
(1) if you fail to have the right From field, your emails could
be blocked, or
(2) regardless of the From field, your emails could be blocked?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: gmail+SMTP(only) (oauth2)

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > However I have been warned that the message should have a from field
>   > with a domain of my university. (@mat.ucm.es for example).
>
>   > Now I could either use
>
>   >     1. The smtpmail (or sendmail) program of my linux machine (not sure
>   >        about MacOS
>
>   >     2. Or use a SMTP service that allows me to use a different form
>   >        field, once the address has been verified. Gmail did this in the
>   >        past (and maybe still does it).
>
>   > In any case I have been warned that my mails could be blacklisted.
>
> That is rather unclear.  Do you mean that
> (1) if you fail to have the right From field, your emails could
> be blocked, or
> (2) regardless of the From field, your emails could be blocked?

This is all related to various spam prevention schemes used by SMTP
servers these days. In a nutshell, there are various widely used schemes
which verify the IP address and/or domain of the originating SMTP server
is part of or associated with the domain in the from header address.
This is often achieved based on DNS records, such as an MX record. 

The result is, if there isn't some verifiable relationship between the
SMTP server and the domain in the from header, it is likely the message
will be blocked by many SMTP servers. The matter is made worse in that
many of the SMTP servers which will allow you to set an arbitrary domain
in your from header are already considered suspect as these are also
often servers legitimately associated with sending spam. Many SMTP
servers won't allow you to set an arbitrary from header or will only
allow you to set the header to a specific domain based on your IP
address. 
 
Just to try and be clear for the very last time here. There is NO 100%
free/libre solution to using gmail for email. There are some solutions
which will minimise the amount/frequency of need to use non-free/libre
software, but none which eliminate it. However, adopting Google as the
email provider for an organisation does NOT mean the organisation uses
Google's 'standard' gmail authentication/authorisation infrastructure. Often,
especially for larger organisations, the organisations own identity
management infrastructure will be used. Unfortunately, few identity
management solutions are free/libre and only a handful exist which are
classified as open source. This means there is no solution which will
work across the board. Each organisation will need to be assessed
individually. 

Using Google to host your email services is NOT the same as using
Google's gmail service. When an organisation announces they will be
moving their email service to google, you cannot assume this will mean
it is going to be equivalent to Google's gmail service. (same holds for
organisations which adopt Office365 as their email service provider).
Therre will likely be some commonality, but there could also be
significant differences.  

Technically, email can be forwarded to a different service. This is an
optional feature which may or may not be allowed by the owning
organisation. Most organisations will not permit this and it is an
option which is generally discouraged by most security professionals.
Even when allowed, it tends not to work well due to the way modern spam
prevention techniques work. 

If an organisation adopts Google as their email service provider, it is
not equivalent to using gmail as your email service provider. While the
data may use/share some of the same physical infrastructure, the
policies and procedures, as well as the authentication and authorisation
processes, can be significantly different.  

If you are using 'real' gmail, you can minimise your use of
non-free/libre software by setting up application passwords. This only
needs to be setup once and after that, you can use whatever SMTP and
IMAP client you want. It is likely Google will remove application
password support at some point in the future, but at this time, they
have not flagged any definite plans to do this. It would be expected
they will provide a significant transition period if and when such plans
are announced. 

Application passwords may or may not be available with organisations
which use Google to host their email services. This will depend on the
policies of the organisation, the level of integration and the
authentication/authorisation approach adopted by the organisation. 

Re: gmail+SMTP(only) (oauth2)

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:

> Richard Stallman <rms@gnu.org> writes:

>> 
>> That is rather unclear.  Do you mean that
>> (1) if you fail to have the right From field, your emails could
>> be blocked, or
>> (2) regardless of the From field, your emails could be blocked?

 
> Just to try and be clear for the very last time here. There is NO 100%



Thanks for this very detailed and clear explanation. Very helpful.


> If an organisation adopts Google as their email service provider, it is
> not equivalent to using gmail as your email service provider. While the
> data may use/share some of the same physical infrastructure, the
> policies and procedures, as well as the authentication and authorisation
> processes, can be significantly different.  


> Application passwords may or may not be available with organisations
> which use Google to host their email services. This will depend on the
> policies of the organisation, the level of integration and the
> authentication/authorisation approach adopted by the organisation. 

Just one last remark: according to my university, it is google itself,
who is setting up the its password authentication policy, there is no
room for negotiation. If they decide to not allow app password, there is
nothing the university can do about it. I find this strange, but that is
what I have been told.



-- 
I strongly condemn Putin's war of aggression against the Ukraine.
I support to deliver weapons to Ukraine's military. 
I support the ban of Russia from SWIFT.
I support the EU membership of the Ukraine. 

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: gmail+SMTP(only) (oauth2)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Do schools and employers ever require that you _send_ them mail
  > > from the Google account you're ordered to use?

  > I can only speak for my university and the answer is «Yes» it does do,
  > and that is the core of the problem.

Have you tried sending these "official" messages to them on paper by
snail mail?  It will be difficult for the university to claim that a
paper letter was not received.  If Correos has a service where the
recipient has to sign for the letter, even better.

Staff might vent anger at you, but it may be hard for them to do
anything to you.  This is clearly sufficient for the university
to prove you sent it.

Once they can't actually punish you, you can offer negotiate some
other digital method to use, one that is ethical for you and that they
can handle easily enough in practice.

You could send such messages in both ways in parallel.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)