Can't send mail with smtpmail "any more". (Gmail and other providers)

Can't send mail with smtpmail "any more". (Gmail and other providers)

[Vladimir Nikishkin]

`Hello, everyone.

I had a setup for sending mail with Emacs' smtpmail:

```
(setq send-mail-function 'smtpmail-send-it)
(setq smtpmail-smtp-server "smtp.gmail.com")
(setq smtpmail-default-smtp-server "smtp.gmail.com")
(setq smtpmail-smtp-service 587)
(setq smtpmail-stream-type 'starttls)
(setq smtpmail-starttls-credentials '("smtp.gmail.com" 587))
(setq smtpmail-smtp-user "lockywolf")
(setq smtpmail-auth-credentials '("smtp.gmail.com" 587))
(require 'starttls)
(setq starttls-use-gnutls t)
(setq smtpmail-debug-info t)
(setq smtpmail-debug-verb t)
(require 'smtpmail)
```

This used to work until today.

Today, this setup stopped working for some reason. Initially I blamed
it on GMail, but it seems that other providers don't work tool.

The symptom is the following:

I open message mode by M-x message-mail RET , type in the from: and
to: as the same, my own, address, and send it with C-c C-c.

The system displays the message:
```
Sending via mail...
```
and freezes. This freeze is "quitable" with C-g.
If I set "toggle-debug-on-quit", the stack trace is the following:

```
Debugger entered--Lisp error: (quit)
  make-network-process(:name "smtpmail" :buffer #<buffer *trace of
SMTP session to smtp.gmail.com*> :host "smtp.gmail.com" :service 587)
  network-stream-open-starttls("smtpmail" #<buffer *trace of SMTP
session to smtp.gmail.com*> "smtp.gmail.com" 587 (:type starttls
:return-list t :warn-unless-encrypted nil :capability-command "EHLO
delllaptop.lockywolf.net\15\n" :end-of-command "^[0-9]+ .*\15\n"
:success "^2.*\n" :always-query-capabilities t :starttls-function
#f(compiled-function (capabilities) #<bytecode 0x6fb3e5>)
:client-certificate t :use-starttls-if-possible t))
  open-network-stream("smtpmail" #<buffer *trace of SMTP session to
smtp.gmail.com*> "smtp.gmail.com" 587 :type starttls :return-list t
:warn-unless-encrypted nil :capability-command "EHLO
delllaptop.lockywolf.net\15\n" :end-of-command "^[0-9]+ .*\15\n"
:success "^2.*\n" :always-query-capabilities t :starttls-function
#f(compiled-function (capabilities) #<bytecode 0x6fb3e5>)
:client-certificate t :use-starttls-if-possible t)
  smtpmail-via-smtp(("<redacted>") #<buffer  smtpmail temp>)
  smtpmail-send-it()
  message-use-send-mail-function()
  message--default-send-mail-function()
  message-multi-smtp-send-mail()
  message--send-mail-maybe-partially()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
  funcall-interactively(message-send-and-exit nil)
  call-interactively(message-send-and-exit nil nil)
  command-execute(message-send-and-exit)
```
I have redacted out the email address.

Connecting to the process with gdb and asking for a stack trace produces:

```
lockywolf@delllaptop:~$ gdb -p $(pgrep emacs)
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-slackware-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 13559
[New LWP 13560]
[New LWP 13561]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--c
0x00007f268a51207e in pselect () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 3 (Thread 0x7f268380f700 (LWP 13561)):
#0  0x00007f268a50fe9f in poll () at /lib64/libc.so.6
#1  0x00007f268c4f4bce in  () at /usr/lib64/libglib-2.0.so.0
#2  0x00007f268c4f4f33 in g_main_loop_run () at /usr/lib64/libglib-2.0.so.0
#3  0x00007f268c73b936 in  () at /usr/lib64/libgio-2.0.so.0
#4  0x00007f268c51c61d in  () at /usr/lib64/libglib-2.0.so.0
#5  0x00007f268a6e406b in start_thread () at /lib64/libpthread.so.0
#6  0x00007f268a51b72f in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7f2688975700 (LWP 13560)):
#0  0x00007f268a50fe9f in poll () at /lib64/libc.so.6
#1  0x00007f268c4f4bce in  () at /usr/lib64/libglib-2.0.so.0
#2  0x00007f268c4f4cef in g_main_context_iteration () at
/usr/lib64/libglib-2.0.so.0
#3  0x00007f268c4f4d41 in  () at /usr/lib64/libglib-2.0.so.0
#4  0x00007f268c51c61d in  () at /usr/lib64/libglib-2.0.so.0
#5  0x00007f268a6e406b in start_thread () at /lib64/libpthread.so.0
#6  0x00007f268a51b72f in clone () at /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c

Thread 1 (Thread 0x7f2689455640 (LWP 13559)):
#0  0x00007f268a51207e in pselect () at /lib64/libc.so.6
#1  0x00000000005aef85 in  ()
#2  0x00000000005af9d2 in  ()
#3  0x0000000000572883 in  ()
#4  0x00000000005a6bb8 in  ()
#5  0x00000000005727d7 in  ()
#6  0x00000000005a6bb8 in  ()
#7  0x00000000005727d7 in  ()
#8  0x00000000005a6bb8 in  ()
#9  0x00000000005727d7 in  ()
#10 0x00000000005a6bb8 in  ()
#11 0x00000000005727d7 in  ()
#12 0x00000000005a6bb8 in  ()
#13 0x00000000005727d7 in  ()
#14 0x00000000005a6bb8 in  ()
#15 0x00000000005727d7 in  ()
#16 0x00000000005a6bb8 in  ()
#17 0x00000000005727d7 in  ()
#18 0x00000000005a6bb8 in  ()
#19 0x00000000005727d7 in  ()
#20 0x00000000005a6bb8 in  ()
#21 0x00000000005727d7 in  ()
#22 0x00000000005a6bb8 in  ()
#23 0x00000000005727d7 in  ()
#24 0x00000000005a6bb8 in  ()
#25 0x00000000005727d7 in  ()
#26 0x00000000005a6bb8 in  ()
#27 0x00000000005727d7 in  ()
#28 0x000000000056f2e4 in  ()
#29 0x0000000000572883 in  ()
#30 0x0000000000570590 in  ()
#31 0x0000000000572883 in  ()
#32 0x00000000005a6bb8 in  ()
#33 0x00000000005727d7 in  ()
#34 0x000000000057292a in  ()
#35 0x000000000050c45d in  ()
#36 0x0000000000571bef in  ()
#37 0x00000000004fd648 in  ()
#38 0x0000000000571b53 in  ()
#39 0x00000000004fd5d6 in  ()
#40 0x0000000000500aef in  ()
#41 0x0000000000500e0d in  ()
#42 0x0000000000426642 in  ()
#43 0x00007f268a428e5b in __libc_start_main () at /lib64/libc.so.6
#44 0x0000000000426dda in  ()
(gdb)
```

Sometimes (unpredictably), the connection times out and produces the
following message:

```
network-stream-open-starttls: make client process failed: Network is
unreachable, :name, smtpmail, :buffer, *trace of SMTP session to
smtp.gmail.com*, :host, smtp.gmail.com, :service, 587
```

Trying to make an MWE,
 1. I launched emacs -Q
 2. M-x message-mail RET
 3. Typed in :to and :from
 4. M-x toggle-debug-on-quit RET
 5. C-c C-c
 6. smtp RET
 7. smtp.gmail.com RET
And received an identical freeze.
After:
 8. C-g (once) -> nothing happens (no stack trace)
 9. C-g (second time) -> a message appears:
`Port number to use when contacting smtp.gmail.com?`
 10. 587 -> freeze
 11. C-g -> `smtpmail-query-smtp-server: Couldn’t contact an SMTP server`

If I connect to emacs with gdb after step 7:

```
(gdb) thread apply all bt

Thread 3 (Thread 0x7fa7558b2700 (LWP 9546)):
#0  0x00007fa75858be9f in poll () at /lib64/libc.so.6
#1  0x00007fa75a570bce in  () at /usr/lib64/libglib-2.0.so.0
#2  0x00007fa75a570f33 in g_main_loop_run () at /usr/lib64/libglib-2.0.so.0
#3  0x00007fa75a7b7936 in  () at /usr/lib64/libgio-2.0.so.0
#4  0x00007fa75a59861d in  () at /usr/lib64/libglib-2.0.so.0
#5  0x00007fa75876006b in start_thread () at /lib64/libpthread.so.0
#6  0x00007fa75859772f in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7fa7569f1700 (LWP 9545)):
#0  0x00007fa75858be9f in poll () at /lib64/libc.so.6
#1  0x00007fa75a570bce in  () at /usr/lib64/libglib-2.0.so.0
#2  0x00007fa75a570cef in g_main_context_iteration () at
/usr/lib64/libglib-2.0.so.0
#3  0x00007fa75a570d41 in  () at /usr/lib64/libglib-2.0.so.0
#4  0x00007fa75a59861d in  () at /usr/lib64/libglib-2.0.so.0
#5  0x00007fa75876006b in start_thread () at /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--c
#6  0x00007fa75859772f in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7fa7574d1640 (LWP 9544)):
#0  0x00007fa75858e07e in pselect () at /lib64/libc.so.6
#1  0x00000000005aef85 in  ()
#2  0x00000000005af9d2 in  ()
#3  0x0000000000572883 in  ()
#4  0x00000000005a6bb8 in  ()
#5  0x00000000005727d7 in  ()
#6  0x00000000005a6bb8 in  ()
#7  0x00000000005727d7 in  ()
#8  0x00000000005a6bb8 in  ()
#9  0x00000000005727d7 in  ()
#10 0x00000000005a6bb8 in  ()
#11 0x00000000005727d7 in  ()
#12 0x00000000005a6bb8 in  ()
#13 0x00000000005727d7 in  ()
#14 0x00000000005a6bb8 in  ()
#15 0x00000000005727d7 in  ()
#16 0x00000000005a6bb8 in  ()
#17 0x00000000005727d7 in  ()
#18 0x00000000005a6bb8 in  ()
#19 0x00000000005727d7 in  ()
#20 0x00000000005a6bb8 in  ()
#21 0x00000000005727d7 in  ()
#22 0x00000000005a6bb8 in  ()
#23 0x00000000005727d7 in  ()
#24 0x00000000005a6bb8 in  ()
#25 0x00000000005727d7 in  ()
#26 0x00000000005a6bb8 in  ()
#27 0x00000000005727d7 in  ()
#28 0x000000000056f2e4 in  ()
#29 0x0000000000572883 in  ()
#30 0x0000000000570590 in  ()
#31 0x0000000000572883 in  ()
#32 0x00000000005a6bb8 in  ()
#33 0x00000000005727d7 in  ()
#34 0x000000000057292a in  ()
#35 0x000000000050c45d in  ()
#36 0x0000000000571bef in  ()
#37 0x00000000004fd648 in  ()
#38 0x0000000000571b53 in  ()
#39 0x00000000004fd5d6 in  ()
#40 0x0000000000500aef in  ()
#41 0x0000000000500e0d in  ()
#42 0x0000000000426642 in  ()
#43 0x00007fa7584a4e5b in __libc_start_main () at /lib64/libc.so.6
#44 0x0000000000426dda in  ()
(gdb)
```
(require 'starttls) doesn't seem to change anything, except give a
warning about starttls being obsolete.


System:
Emacs 27.1
Kernel: 5.4.72 #1 SMP Sat Oct 17 14:26:06 CDT 2020 x86_64 Intel(R)
Core(TM) i5-2520M CPU @ 2.50GHz GenuineIntel GNU/Linux
glib2-2.66.2
glibc-2.30
gtk+3-3.24.23

What can I do else to make smtp work as yesterday, or at least debuggable?

--
Yours sincerely, Vladimir Nikishkin
(Sent from GMail web interface.)

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Eli Zaretskii]

> From: Vladimir Nikishkin <lockywolf@gmail.com>
> Date: Tue, 27 Oct 2020 12:49:24 +0800
> 
> Sometimes (unpredictably), the connection times out and produces the
> following message:
> 
> ```
> network-stream-open-starttls: make client process failed: Network is
> unreachable, :name, smtpmail, :buffer, *trace of SMTP session to
> smtp.gmail.com*, :host, smtp.gmail.com, :service, 587
> ```

This seems to be self-explanatory: Emacs is unable to connect to that
address and that port.  Are you sure there isn't some connectivity
problem on your side?  Like some firewall settings, or maybe actual
connectivity issue?

smtpmail.el has a debug facility, whereby you can set a variable and
get a buffer with the details of the connection session.  If the above
doesn't give a clue, try looking at those details.

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Vladimir Nikishkin]

I worked around this issue by switching to port 465+ssl/tls

It almost seems like port 587 got blackholed by something.

Debugging facilities did not illuminate the situation, the deadlock happens
before anything is written there.

--
Yours sincerely, Vladimir Nikishkin
(Sent with Google mail mobile.)

Vladimir Nikishkin <lockywolf@gmail.com> 于 2020年10月27日周二 23:43写道：

> I worked around this issue by switching to port 465+ssl/tls
>
> It almost seems like port 587 got blackholed by something.
>
> Debugging facilities did not illuminate the situation, the deadlock
> happens before anything is written there.
>
> --
> Yours sincerely, Vladimir Nikishkin
> (Sent with Google mail mobile.)
>
> Eli Zaretskii <eliz@gnu.org> 于 2020年10月27日周二 23:28写道：
>
>> > From: Vladimir Nikishkin <lockywolf@gmail.com>
>> > Date: Tue, 27 Oct 2020 12:49:24 +0800
>> >
>> > Sometimes (unpredictably), the connection times out and produces the
>> > following message:
>> >
>> > ```
>> > network-stream-open-starttls: make client process failed: Network is
>> > unreachable, :name, smtpmail, :buffer, *trace of SMTP session to
>> > smtp.gmail.com*, :host, smtp.gmail.com, :service, 587
>> > ```
>>
>> This seems to be self-explanatory: Emacs is unable to connect to that
>> address and that port.  Are you sure there isn't some connectivity
>> problem on your side?  Like some firewall settings, or maybe actual
>> connectivity issue?
>>
>> smtpmail.el has a debug facility, whereby you can set a variable and
>> get a buffer with the details of the connection session.  If the above
>> doesn't give a clue, try looking at those details.
>>
>>

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Robert Pluim]

Vladimir Nikishkin <lockywolf@gmail.com> writes:

> I worked around this issue by switching to port 465+ssl/tls
>

That in itself is a good thing :-)

> It almost seems like port 587 got blackholed by something.

Itʼs not uncommon for ISPs to block outgoing port 25, I guess somebody
decided to do the same for 587

> Debugging facilities did not illuminate the situation, the deadlock happens
> before anything is written there.

Itʼs not a deadlock, itʼs just that whoever is doing the blocking just
drops the packets rather than sending a reset, so Emacs is waiting for
a response that will never come (and eventually TCP will time out).

Robert

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Saša Janiška]

On Tue, 27 Oct 2020 17:15:56 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> smtpmail.el has a debug facility, whereby you can set a variable and
> get a buffer with the details of the connection session.

What about adding some regular logging facility to smtpmail.el? It is
the main reason why I'm using msmtp instead?

(This email is still sent by Claws until I decide which client to use: Gnus or mu4e.)


Sincerely,
Gour

-- 
Not by merely abstaining from work can one achieve freedom
from reaction, nor by renunciation alone can one attain perfection.

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Jean Louis]

* Saša Janiška <gour@atmarama.com> [2020-11-01 15:35]:
> On Tue, 27 Oct 2020 17:15:56 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > smtpmail.el has a debug facility, whereby you can set a variable and
> > get a buffer with the details of the connection session.
> 
> What about adding some regular logging facility to smtpmail.el? It is
> the main reason why I'm using msmtp instead?
> 
> (This email is still sent by Claws until I decide which client to
> use: Gnus or mu4e.)

M-x rmail is quite good client for email handling in Emacs.

Then there is:

M-x mh-rmail which may have many good power users' options

For me, due to nature of my email handling and having over 50000
Maildir directories inside of ~/Maildir and that none of Emacs clients
is handling Maildirs, I am using mutt email client. That is one of
most efficient for me. And I am launching it inside of Emacs and
editing files with Emacs by using emacsclient

(defun mutt-emacs ()
  "Opens mutt in ansi-term buffer within GNU Emacs"
  (interactive)
  (let* ((default-directory (getenv "HOME")))
    (ansi-term "mutt")))

But my M-x becomes not usable when I am in ansi-term, I would like
that it is usable. 

I made this function to open up mutt in vterm and I still use it. But
that was workaround from time when ansi-term was not handling best
some terminal activities. Bugs have been polished and now ansi-term
works just fine.

(defun mutt-vterm ()
  "Opens mutt in vterm buffer within GNU Emacs"
  (interactive)
  (let* ((default-directory (getenv "HOME"))
	 (buffer-exist (get-buffer "mutt"))
	 (vterm-kill-buffer-on-exit t)
	 (vterm-shell "mutt")
	 (last-key (last-key)))
    (if buffer-exist
	(if (and (not (symbolp last-key))
		 (not (= last-key 8388717)))
	    (vterm "mutt")
	  (if (and (not (string-match (buffer-name (current-buffer)) "mutt"))
		   (y-or-n-p "Buffer `mutt' exists, do you wish to switch?"))
	      (switch-to-buffer "mutt")))
      (vterm "mutt"))))

(global-set-key (kbd "s-m") 'mutt-you-name-it)

For fetching emails I recommend using GNU mailtools and setting up
crontab to regularly fetch emails to local computer.

For sending, I recommend using the MTA on the system if such exists on
your system, for example exim, courier MTA, OpenSMTPD, Postfix and
similar. Those are handling the queues well if you have good
connection to Internet.

msmtp is excellent tool and handles queues of email even if connection
to Internet is sporadic. MTAs need to be configured to handled mail in
queues for longer time and for that reason some emails could be lost
or returned back.

I am using those scripts for msmtp, like list queue and similar. So
the script below is at ~/bin/sendmail and my PATH is ~/bin as well so
Emacs will recognize it as sendmail and dispatch email without any
further settings to msmtp. That is very handy when using it with emacs
-Q when one needs to M-x report-emacs-bug

~/bin/sendmail follows below:

#!/usr/bin/env bash

QUEUEDIR=$HOME/.msmtpqueue

# Set secure permissions on created directories and files
umask 077

# Change to queue directory (create it if necessary)
if [ ! -d "$QUEUEDIR" ]; then
	mkdir -p "$QUEUEDIR" || exit 1
fi
cd "$QUEUEDIR" || exit 1

# Create new unique filenames of the form
# MAILFILE:  ccyy-mm-dd-hh.mm.ss[-x].mail
# MSMTPFILE: ccyy-mm-dd-hh.mm.ss[-x].msmtp
# where x is a consecutive number only appended if you send more than one 
# mail per second.
BASE="`/bin/date +%Y-%m-%d-%H.%M.%S`"
if [ -f "$BASE.mail" -o -f "$BASE.msmtp" ]; then
	TMP="$BASE"
	i=1
	while [ -f "$TMP-$i.mail" -o -f "$TMP-$i.msmtp" ]; do
		i=`expr $i + 1`
	done
	BASE="$BASE-$i"
fi
MAILFILE="$BASE.mail"
MSMTPFILE="$BASE.msmtp"

# Write command line to $MSMTPFILE
echo "$@" > "$MSMTPFILE" || exit 1

# Write the mail to $MAILFILE
cat > "$MAILFILE" || exit 1

# If we are online, run the queue immediately.
# Replace the test with something suitable for your site.
ping -c 1 -w 2 stw1.rcdrun.com > /dev/null 
if [ $? -eq 0 ]; then
	msmtp-runqueue.sh > /dev/null &
fi

exit 0

Re: Can't send mail with smtpmail "any more" (Gmail and other providers)

[Saša Janiška]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

On Sun, 1 Nov 2020 16:05:01 +0300
Jean Louis <bugs@gnu.support> wrote:

> M-x rmail is quite good client for email handling in Emacs.
> 
> Then there is:
> 
> M-x mh-rmail which may have many good power users' options

Yeah, I heard goof things about those, but they seem a bit isolated
when/if one wants to switch to something else?

> For me, due to nature of my email handling and having over 50000
> Maildir directories inside of ~/Maildir and that none of Emacs clients
> is handling Maildirs, I am using mutt email client. 

What about notmuch or mu4e which can use Maildirs (with the help of
e.g. mbsync)?

> That is one of most efficient for me. And I am launching it inside of
> Emacs and editing files with Emacs by using emacsclient

I'd be happy with (neo)mutt, but would miss (better) integration with
org-mode?

> msmtp is excellent tool and handles queues of email even if connection
> to Internet is sporadic. MTAs need to be configured to handled mail in
> queues for longer time and for that reason some emails could be lost
> or returned back.

I believe that for my volume of mail, msmtp is good-enough. :-)


Sincerely,
Gour

-- 
The spirit soul bewildered by the influence of false ego thinks
himself the doer of activities that are in actuality carried out
by the three modes of material nature.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Can't send mail with smtpmail "any more" (Gmail and other providers)

[Jean Louis]

* Saša Janiška <gour@atmarama.com> [2020-11-01 16:31]:
> On Sun, 1 Nov 2020 16:05:01 +0300
> Jean Louis <bugs@gnu.support> wrote:
> 
> > M-x rmail is quite good client for email handling in Emacs.
> > 
> > Then there is:
> > 
> > M-x mh-rmail which may have many good power users' options
> 
> Yeah, I heard goof things about those, but they seem a bit isolated
> when/if one wants to switch to something else?

When you wish to change to anything you can do that. Every email
format can be changed to other format.

By using mutt I can change from mh to maildir or mbox format, whatever
exists. I can open one file and save it into other file blazing fast
by T . RET and then ; s to save it, those are keybindings in mutt.

By using GNU mailtools I can quickly move from one email file to other
as there is command sieve:

For example:

function save-maildir-by-to () {
    sieve -f /home/admin/Maildir/Sent.mbox.2020 "/home/data1/protected/Programming/System Administration/sieve/save-by-to.sieve" && echo Sieve DONE.
}

/home/data1/protected/Programming/System Administration/sieve/save-by-to.sieve:

require [ "fileinto", "variables" ];
if address :matches [ "to" ] "*" {
  set "recipient" :lower "${1}";
  fileinto "~/Maildir/${recipient}";
            }

I am often sending emails from Emacs, they are stored in
~/Maildir/Sent.mbox.2020 and from time to time I sieve those emails
into specific directories per user, for example
~/Maildir/user-one@example.com and ~/Maildir/user-two@example.com

Because each email address has its directory I can quickly see all
previous conversation related to that email address without even using
any search tools.

> > For me, due to nature of my email handling and having over 50000
> > Maildir directories inside of ~/Maildir and that none of Emacs clients
> > is handling Maildirs, I am using mutt email client. 
> 
> What about notmuch or mu4e which can use Maildirs (with the help of
> e.g. mbsync)?

notmuch never worked for me, it was terribly slow, but it may work for
you. There was one bug related to bits, whatever 32 or 64 I do not
know. Maybe it would work today but I got familiar with mu.

mu4e is great for Emacs, I would say the best if you do not have too
many emails like my side. mu4e is using background tool mu

Because I have too many emails mu4e is not much useful for me. It is
very slow. If you have not get hundreds of thousands of emails, it is
good choice.

What Emacs needs is mu4e that can directly access Maildirs and not
just database. With mutt I am directly accessing Maildirs so my email
system is blazing fast.

Let us say I am looking into email X and there is something I would
like to look into previous conversation. Because each email is sorted
in separate Maildir directoris ~/Maildir/somebody@example.com this
line in mutt configuration is opening new mutt instance over the other
one and showing me past emails related to same person.

macro index,pager \ev ":set wait_key=no<enter><pipe-entry>ef > ~/tmp/ef-email<enter>!mutt-emailsof.lisp<enter>:set wait_key=yes<enter>" "Quickly view maildir for contact ID"

I could give you scripts. ef is C script made by author of GNU
Mailtools and it just extracts the email address from email so that
same address can be used to open up
~/Maildir/thatmailaddress@example.com

Then I click q and I am back in previous listing of emails. That is
one example.

I wish Emacs would have Maildir handlings. There is one package for
maildir and I am sometimes using it but it is slow and not polished.

If I wish to search for emails in mutt I am also using mu command,
which is behind mu4e tools. So I click F3 and enter search terms and I
get new list with search results:

macro index <F3> "<enter-command>unset wait_key<enter><shell-escape>mu-find.lisp<enter><change-folder-readonly>~/Maildir/search" "mu find"

That small macro makes mutt search all emails.

Maildir is most safe format for emails. You could even use just
ssh/rsync to move emails from remote Maildir folder to local one.

Those tools like mbsync are alright. I do not use them as I find
enough versatility in GNU Mailutils, so I am using movemail
utility. If Emacs finds movemail on system it is not going to use the
built-in function when compiling. So internally Emacs can use movemail
as well to fetch emails.

From manual:

  Apart from that use, 'movemail' proved to be a useful tool for
incorporating mail from remote mailboxes into the local one.  See
Fetching Mail with Movemail
http://mailutils.org/wiki/Fetching_Mail_with_Movemail, for a detailed
discussion with usage recipes.

> > That is one of most efficient for me. And I am launching it inside of
> > Emacs and editing files with Emacs by using emacsclient
> 
> I'd be happy with (neo)mutt, but would miss (better) integration with
> org-mode?

That neomutt I do not know. I follow main stream mutt. Neomutt is full
of problems and lack of care in programming.

You let me know what you wish and I can maybe help you as I have
made some functions for mutt and Emacs. For example I can send Org
subtreet to one of staff members straight.

I like the concept of Android innovation that everything could be or
should be shared to other communication channels. This concept is so
much absent on desktop.

For example it is most logical that when using dired that I wish to
send some files to somebody, but such function is not yet there.

I just press Super-m and marked files are attached to mutt and sent to
desired person. I am choosing person from a database. i would not like
spending time attaching one by one file to email, or how I would be
even choosing them? By mouse? I would die just by selecting
files. Sometimes I send 24 or 40 images to somebody.

In Org I am preparing a TASK but task has to be assigned to somebody,
so I have to be able to send exact that task to somebody.

Which Org integration you need?

> > msmtp is excellent tool and handles queues of email even if connection
> > to Internet is sporadic. MTAs need to be configured to handled mail in
> > queues for longer time and for that reason some emails could be lost
> > or returned back.
> 
> I believe that for my volume of mail, msmtp is good-enough. :-)

It is good for large volume, it does not matter. It will be stored in
the queue.


-- 
There are 59 messages yet in my incoming mailbox.

Re: Can't send mail with smtpmail "any more" (Gmail and other providers)

[Saša Janiška]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On Sun, 1 Nov 2020 18:30:16 +0300
Jean Louis <bugs@gnu.support> wrote:
> * Saša Janiška <gour@atmarama.com> [2020-11-01 16:31]:

> When you wish to change to anything you can do that. Every email
> format can be changed to other format.

Even from rmail?

> By using mutt I can change from mh to maildir or mbox format, whatever
> exists. I can open one file and save it into other file blazing fast
> by T . RET and then ; s to save it, those are keybindings in mutt.

For Mutt, I now since I was using it for mbox, MH, Maildir...

> mu4e is great for Emacs, I would say the best if you do not have too
> many emails like my side. mu4e is using background tool mu

OK. Thanks.

> Those tools like mbsync are alright. I do not use them as I find
> enough versatility in GNU Mailutils, so I am using movemail
> utility. If Emacs finds movemail on system it is not going to use the
> built-in function when compiling. So internally Emacs can use movemail
> as well to fetch emails.

I see...

> You let me know what you wish and I can maybe help you as I have
> made some functions for mutt and Emacs. 

OK. Let me do my own homework first. ;)

> Which Org integration you need?

Just ability to create the task from within mutt and after completing it in org 
to jump back to mutt...


Sincerely,
Gour

-- 
What is night for all beings is the time of awakening
for the self-controlled; and the time of awakening for
all beings is night for the introspective sage.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Can't send mail with smtpmail "any more" (Gmail and other providers)

[Jean Louis]

* Saša Janiška <gour@atmarama.com> [2020-11-01 22:03]:
> On Sun, 1 Nov 2020 18:30:16 +0300
> Jean Louis <bugs@gnu.support> wrote:
> > * Saša Janiška <gour@atmarama.com> [2020-11-01 16:31]:
> 
> > When you wish to change to anything you can do that. Every email
> > format can be changed to other format.
> 
> Even from rmail?

My sieve recipe shown is doing it from rmail saved mail box.

> > Which Org integration you need?
> 
> Just ability to create the task from within mutt and after completing it in org 
> to jump back to mutt...

My key binding {C-c c} is bound to `org-capture' that opens various
capture templates that use in turn various Org files, I can quickly
add a task by using template and quickly come back to whatever buffer
I was. Would that be something for you?

On the other hand sometimes I simply want to record TODO in the
database straight from mutt, by simply pressing F12:

macro index,pager <F12> "<pipe-entry>/home/data1/protected/bin/rcd/email-pipe-simpletodo.lisp<enter>" "Quickly add TODO to RCD Business"

So that records for me the contact and subject of email, so instead of
keeping email in inbox to represent unhandled tasks, I transfer the
TODO into the database.

That principle could be used to transform email into Org task without
any further interaction or with interaction if necessary.

Simple example without interaction (without asking user for anything):

macro index,pager <F12> "<pipe-entry>cat | grep \"^Subject: \\|^From: \\|Date: \" | sed -E '1,1s/^(.*)/** TODO \\1/' >> my-file.org<enter>" "Quickly add TODO to my org file"

-- 
There are 57 messages yet in my incoming mailbox.

Re: Can't send mail with smtpmail "any more" (Gmail and other providers)

[Saša Janiška]

[-- Attachment #1: Type: text/plain, Size: 1311 bytes --]

On Mon, 2 Nov 2020 00:19:38 +0300
Jean Louis <bugs@gnu.support> wrote:

> My key binding {C-c c} is bound to `org-capture' that opens various
> capture templates that use in turn various Org files, I can quickly
> add a task by using template and quickly come back to whatever buffer
> I was. Would that be something for you?

Yes, that's cool.

> 
> On the other hand sometimes I simply want to record TODO in the
> database straight from mutt, by simply pressing F12:
> 
> macro index,pager <F12>
> "<pipe-entry>/home/data1/protected/bin/rcd/email-pipe-simpletodo.lisp<enter>"
> "Quickly add TODO to RCD Business"

Nice.

> That principle could be used to transform email into Org task without
> any further interaction or with interaction if necessary.

Very nice!

> 
> Simple example without interaction (without asking user for anything):
> 
> macro index,pager <F12> "<pipe-entry>cat | grep \"^Subject: \\|^From:
> \\|Date: \" | sed -E '1,1s/^(.*)/** TODO \\1/' >> my-file.org<enter>"
> "Quickly add TODO to my org file"


Thank you very much!


Sincerely,
Gour

-- 
A self-realized man has no purpose to fulfill in the discharge
of his prescribed duties, nor has he any reason not to perform
such work. Nor has he any need to depend on any other living being.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Vladimir Nikishkin]

mu4e can handle maildirs. Notmuch can also handle maildirs.

Moreover, you can use both on the same maildir that mutt already has, at
the same time, and use any of those as a search engine for mutt.

--
Yours sincerely, Vladimir Nikishkin
(Sent with Google mail mobile.)

Jean Louis <bugs@gnu.support> 于 2020年11月1日周日 21:06写道：

> * Saša Janiška <gour@atmarama.com> [2020-11-01 15:35]:
> > On Tue, 27 Oct 2020 17:15:56 +0200
> > Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > smtpmail.el has a debug facility, whereby you can set a variable and
> > > get a buffer with the details of the connection session.
> >
> > What about adding some regular logging facility to smtpmail.el? It is
> > the main reason why I'm using msmtp instead?
> >
> > (This email is still sent by Claws until I decide which client to
> > use: Gnus or mu4e.)
>
> M-x rmail is quite good client for email handling in Emacs.
>
> Then there is:
>
> M-x mh-rmail which may have many good power users' options
>
> For me, due to nature of my email handling and having over 50000
> Maildir directories inside of ~/Maildir and that none of Emacs clients
> is handling Maildirs, I am using mutt email client. That is one of
> most efficient for me. And I am launching it inside of Emacs and
> editing files with Emacs by using emacsclient
>
> (defun mutt-emacs ()
>   "Opens mutt in ansi-term buffer within GNU Emacs"
>   (interactive)
>   (let* ((default-directory (getenv "HOME")))
>     (ansi-term "mutt")))
>
> But my M-x becomes not usable when I am in ansi-term, I would like
> that it is usable.
>
> I made this function to open up mutt in vterm and I still use it. But
> that was workaround from time when ansi-term was not handling best
> some terminal activities. Bugs have been polished and now ansi-term
> works just fine.
>
> (defun mutt-vterm ()
>   "Opens mutt in vterm buffer within GNU Emacs"
>   (interactive)
>   (let* ((default-directory (getenv "HOME"))
>          (buffer-exist (get-buffer "mutt"))
>          (vterm-kill-buffer-on-exit t)
>          (vterm-shell "mutt")
>          (last-key (last-key)))
>     (if buffer-exist
>         (if (and (not (symbolp last-key))
>                  (not (= last-key 8388717)))
>             (vterm "mutt")
>           (if (and (not (string-match (buffer-name (current-buffer))
> "mutt"))
>                    (y-or-n-p "Buffer `mutt' exists, do you wish to
> switch?"))
>               (switch-to-buffer "mutt")))
>       (vterm "mutt"))))
>
> (global-set-key (kbd "s-m") 'mutt-you-name-it)
>
> For fetching emails I recommend using GNU mailtools and setting up
> crontab to regularly fetch emails to local computer.
>
> For sending, I recommend using the MTA on the system if such exists on
> your system, for example exim, courier MTA, OpenSMTPD, Postfix and
> similar. Those are handling the queues well if you have good
> connection to Internet.
>
> msmtp is excellent tool and handles queues of email even if connection
> to Internet is sporadic. MTAs need to be configured to handled mail in
> queues for longer time and for that reason some emails could be lost
> or returned back.
>
> I am using those scripts for msmtp, like list queue and similar. So
> the script below is at ~/bin/sendmail and my PATH is ~/bin as well so
> Emacs will recognize it as sendmail and dispatch email without any
> further settings to msmtp. That is very handy when using it with emacs
> -Q when one needs to M-x report-emacs-bug
>
> ~/bin/sendmail follows below:
>
> #!/usr/bin/env bash
>
> QUEUEDIR=$HOME/.msmtpqueue
>
> # Set secure permissions on created directories and files
> umask 077
>
> # Change to queue directory (create it if necessary)
> if [ ! -d "$QUEUEDIR" ]; then
>         mkdir -p "$QUEUEDIR" || exit 1
> fi
> cd "$QUEUEDIR" || exit 1
>
> # Create new unique filenames of the form
> # MAILFILE:  ccyy-mm-dd-hh.mm.ss[-x].mail
> # MSMTPFILE: ccyy-mm-dd-hh.mm.ss[-x].msmtp
> # where x is a consecutive number only appended if you send more than one
> # mail per second.
> BASE="`/bin/date +%Y-%m-%d-%H.%M.%S`"
> if [ -f "$BASE.mail" -o -f "$BASE.msmtp" ]; then
>         TMP="$BASE"
>         i=1
>         while [ -f "$TMP-$i.mail" -o -f "$TMP-$i.msmtp" ]; do
>                 i=`expr $i + 1`
>         done
>         BASE="$BASE-$i"
> fi
> MAILFILE="$BASE.mail"
> MSMTPFILE="$BASE.msmtp"
>
> # Write command line to $MSMTPFILE
> echo "$@" > "$MSMTPFILE" || exit 1
>
> # Write the mail to $MAILFILE
> cat > "$MAILFILE" || exit 1
>
> # If we are online, run the queue immediately.
> # Replace the test with something suitable for your site.
> ping -c 1 -w 2 stw1.rcdrun.com > /dev/null
> if [ $? -eq 0 ]; then
>         msmtp-runqueue.sh > /dev/null &
> fi
>
> exit 0
>
>

notmuch or mu for Emacs - Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Jean Louis]

* Vladimir Nikishkin <lockywolf@gmail.com> [2020-11-01 18:21]:
> mu4e can handle maildirs. Notmuch can also handle maildirs.
> 
> Moreover, you can use both on the same maildir that mutt already has, at
> the same time, and use any of those as a search engine for mutt.

I have spent hours and hours yesterday trying notmuch to make it, I
will try again. If maildir is deleted during indexing it breaks.

-- 
There are 57 messages yet in my incoming mailbox.

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Eli Zaretskii]

> From: Saša Janiška <gour@atmarama.com>
> Date: Sun, 1 Nov 2020 13:34:43 +0100
> 
> On Tue, 27 Oct 2020 17:15:56 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > smtpmail.el has a debug facility, whereby you can set a variable and
> > get a buffer with the details of the connection session.
> 
> What about adding some regular logging facility to smtpmail.el?

How would that be different from what smtpmail already has?

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Saša Janiška]

On Sun, 01 Nov 2020 17:24:41 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> How would that be different from what smtpmail already has?

When I was playing with smtpmail I wanted to check whether the email was 
delivered, but have a feeling that there is only verbose debug output which is 
more than I usually need just to check if email went through.


Sincerely,
Gour

-- 
Bewildered by the modes of material nature, the ignorant fully
engage themselves in material activities and become attached. But
the wise should not unsettle them, although these duties are inferior
due to the performers' lack of knowledge.

Re: Can't send mail with smtpmail "any more". (Gmail and other providers)

[Eli Zaretskii]

> From: Saša Janiška <gour@atmarama.com>
> Date: Sun, 1 Nov 2020 20:05:46 +0100
> 
> On Sun, 01 Nov 2020 17:24:41 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > How would that be different from what smtpmail already has?
> 
> When I was playing with smtpmail I wanted to check whether the email was 
> delivered

This is announced in the echo-area by default.  No need for any logs.