How I am handling msmtp queues

How I am handling msmtp queues

[Jean Louis]

I wish to describe how I use Emacs Lisp to send emails.

msmtp is about sending emails from computer:

Description     : A mini smtp client
URL             : https://marlam.de/msmtp/

For longer time I have used msmtp by cron job. However, that somehow
blocked itself, some colliding msmtp processes were in memory and I
had to kill them all before running msmtp-runqueue.sh script

Often I have thousands of emails in the queue, and sending them
reliably is important. Cron job for some unknown reasons did not do
well. 

Then I have made this Emacs Lisp function.

;;;; MSMTP

(defun msmtp-count-remaining ()
  "Count and send remaining MSMTP messages in the queue"
  (interactive)
  (let ((count (length (directory-files "~/.msmtpqueue" nil "\\.mail"))))
    (if (> count 0)
	(progn
	  (async-shell-command "msmtp-runqueue.sh" "*msmtp*")
	  (message "There is %s e-mails in queue." count))
      (message "No emails."))))

And I run the function with timer so that it is repeated every
minute. If there are some emails in the queue, the function will
dispatch them.

;; (run-with-timer 0 60 'msmtp-count-remaining)

Additionally I have adjusted the variable `display-buffer-alist' so
that buffers matching "msmtp" have function `display-buffer-no-window'
so that the buffer does not appear on screen when running the queue.

I can run the above Lisp with M-x msmtp-count-remaining or the timer
will run the script and send any messages in the queue.

And original msmtp script is here:

msmtp-runqueue.sh
=================

#!/usr/bin/env bash

QUEUEDIR="$HOME/.msmtpqueue"
LOCKFILE="$QUEUEDIR/.lock"
MAXWAIT=120

OPTIONS=$@

# wait for a lock that another instance has set
WAIT=0
while [ -e "$LOCKFILE" -a "$WAIT" -lt "$MAXWAIT" ]; do
	sleep 1
	WAIT="`expr "$WAIT" + 1`"
done
if [ -e "$LOCKFILE" ]; then
	echo "Cannot use $QUEUEDIR: waited $MAXWAIT seconds for"
	echo "lockfile $LOCKFILE to vanish, giving up."
	echo "If you are sure that no other instance of this script is"
	echo "running, then delete the lock file."
	exit 1
fi

# change into $QUEUEDIR 
cd "$QUEUEDIR" || exit 1

# check for empty queuedir
if [ "`echo *.mail`" = '*.mail' ]; then
	echo "No mails in $QUEUEDIR"
	exit 0
fi

# lock the $QUEUEDIR
touch "$LOCKFILE" || exit 1

# process all mails
for MAILFILE in *.mail; do
	MSMTPFILE="`echo $MAILFILE | sed -e 's/mail/msmtp/'`"
	echo "*** Sending $MAILFILE to `sed -e 's/^.*-- \(.*$\)/\1/' $MSMTPFILE` ..."
	if [ ! -f "$MSMTPFILE" ]; then
		echo "No corresponding file $MSMTPFILE found"
		echo "FAILURE"
		continue
	fi
	msmtp $OPTIONS `cat "$MSMTPFILE"` < "$MAILFILE"
	if [ $? -eq 0 ]; then
		rm "$MAILFILE" "$MSMTPFILE"
		echo "$MAILFILE sent successfully"
	else
		echo "FAILURE"
	fi
done

# remove the lock
rm -f "$LOCKFILE"

exit 0


Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: How I am handling msmtp queues

[Felix Dietrich]

Jean Louis <bugs@gnu.support> writes:

> I wish to describe how I use Emacs Lisp to send emails.
>
> msmtp is about sending emails from computer:
>
> Description     : A mini smtp client
> URL             : https://marlam.de/msmtp/
>
> Often I have thousands of emails in the queue, and sending them
> reliably is important. 

At these numbers, shouldnʼt you consider a somewhat less “mini” MTA like
Exim, postfix, or opensmtpd?  The latter, especially, seems to have a
nice and streamlined configuration syntax.

-- 
Felix Dietrich

Re: How I am handling msmtp queues

[Jean Louis]

* Felix Dietrich <felix.dietrich@sperrhaken.name> [2022-03-10 19:33]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > I wish to describe how I use Emacs Lisp to send emails.
> >
> > msmtp is about sending emails from computer:
> >
> > Description     : A mini smtp client
> > URL             : https://marlam.de/msmtp/
> >
> > Often I have thousands of emails in the queue, and sending them
> > reliably is important. 
> 
> At these numbers, shouldnʼt you consider a somewhat less “mini” MTA like
> Exim, postfix, or opensmtpd?  The latter, especially, seems to have a
> nice and streamlined configuration syntax.

Of course, I use them but on remote servers. 

Long time I have used Courier MTA.

The Courier Mail Server
http://www.courier-mta.org/

And I still use it online. It is way easier to configure than Exim,
Postfix. Though I find the simplest MSMTP good, there are no many
dependencies for local computer. And I just configure it in private
directory, no need for system wide configuration.

OpenSMTPD is good of course. Though it does not handle queue as well
as Courier MTA and has some problems, it is still in development. I
have many emails, many people, so it gives me experience with it.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: How I am handling msmtp queues

[Felix Dietrich]

Jean Louis <bugs@gnu.support> writes:

> * Felix Dietrich <felix.dietrich@sperrhaken.name> [2022-03-10 19:33]:
>> Jean Louis <bugs@gnu.support> writes:

>>> msmtp-runqueue.sh
>>> =================
>>>
>>> #!/usr/bin/env bash
>>>
>>> QUEUEDIR="$HOME/.msmtpqueue"
>>> LOCKFILE="$QUEUEDIR/.lock"
>>> MAXWAIT=120
>>>
>>> OPTIONS=$@
>>>
>>> # wait for a lock that another instance has set
>>> WAIT=0
>>> while [ -e "$LOCKFILE" -a "$WAIT" -lt "$MAXWAIT" ]; do
>>>
>>> # […]
>>>
>>> # lock the $QUEUEDIR
>>> touch "$LOCKFILE" || exit 1
>>>
>>> # […]

I think your lock file handling is not race free.  There are lock file
helpers that you may find useful.  On Debian, for example:

  $ apt-file find bin/lockfile
  lockfile-progs: /usr/bin/lockfile-check   
  lockfile-progs: /usr/bin/lockfile-create
  lockfile-progs: /usr/bin/lockfile-remove
  lockfile-progs: /usr/bin/lockfile-touch
  procmail:       /usr/bin/lockfile

>>> while [ -e "$LOCKFILE" -a "$WAIT" -lt "$MAXWAIT" ]; do

The general advice for the test commandʼs operators "-a" and "-o" is to
not use them (they have been marked obsolescent) but instead the shellʼs
operators "&&" and "||", respectively [1]:

  while [ -e "$LOCKFILE" ] && [ "$WAIT" -lt "$MAXWAIT" ]; do

> OpenSMTPD is good of course. Though it does not handle queue as well
> as Courier MTA and has some problems, it is still in development. I
> have many emails, many people, so it gives me experience with it.

What issues did you encounter with OpenSMTPD?


Footnotes:
[1]  See the note at the description for "-a" and "-o" as well as the
     section “APPLICATION USAGE” in the following document:

     https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html

-- 
Felix Dietrich

Re: How I am handling msmtp queues

[Jean Louis]

* Felix Dietrich <felix.dietrich@sperrhaken.name> [2022-03-11 21:06]:
> I think your lock file handling is not race free.  There are lock file
> helpers that you may find useful.  On Debian, for example:

Yes, I have to work on it. The script for msmtp was adopted without
thinking, I guess from Internet.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: How I am handling msmtp queues

[Tomas Hlavaty]

On Fri 11 Mar 2022 at 15:55, Felix Dietrich <felix.dietrich@sperrhaken.name> wrote:
> Jean Louis <bugs@gnu.support> writes:
>> * Felix Dietrich <felix.dietrich@sperrhaken.name> [2022-03-10 19:33]:
>>> Jean Louis <bugs@gnu.support> writes:
>>>> # wait for a lock that another instance has set
>>>> WAIT=0
>>>> while [ -e "$LOCKFILE" -a "$WAIT" -lt "$MAXWAIT" ]; do
>>>>
>>>> # […]
>>>>
>>>> # lock the $QUEUEDIR
>>>> touch "$LOCKFILE" || exit 1
> I think your lock file handling is not race free.  There are lock file
> helpers that you may find useful.  On Debian, for example:
>
>   $ apt-file find bin/lockfile
>   lockfile-progs: /usr/bin/lockfile-check   
>   lockfile-progs: /usr/bin/lockfile-create
>   lockfile-progs: /usr/bin/lockfile-remove
>   lockfile-progs: /usr/bin/lockfile-touch
>   procmail:       /usr/bin/lockfile

you probably want

   flock

or

   flock -n

see (man "flock")