bug#25436: 25.1; sendmail does not add a Date: header in FCC

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Francesco Potortì]

When mail-user-agent is set to 'sendmail-user-agent, email saved to FCC
files does not include a Date: header.  Don't know if this happens for
other user agents as well.

I work around this bug by doing:

;; Work around 25.1 bug, who does not add a Date: header
(add-hook 'mail-send-hook
	  (lambda ()
	    (save-excursion
	      (goto-char (point-min))
	      (when (not (re-search-forward
			  "^Date: " (search-forward mail-header-separator) t))
		(insert "Date: " (mail-rfc822-date) "\n"))))
	  'append)



In GNU Emacs 25.1.1 (x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2016-10-24, modified by Debian built on trouble
Windowing system distributor 'The X.Org Foundation', version 11.0.11900000
System Description:	Debian GNU/Linux testing (stretch)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --with-x=yes --with-x-toolkit=lucid
 --with-toolkit-scroll-bars --without-gconf --without-gsettings
 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs25-25.1+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS NOTIFY ACL
LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
  value of $LC_COLLATE: it_IT.UTF-8
  value of $LC_CTYPE: it_IT.UTF-8
  value of $LC_NUMERIC: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  global-visual-fill-column-mode: t
  diff-auto-refine-mode: t
  TeX-PDF-mode: t
  desktop-save-mode: t
  epa-global-mail-mode: t
  shell-dirtrack-mode: t
  openwith-mode: t
  xterm-mouse-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: do-auto-fill

Recent messages:
(ask-if-fcc ask-for-from-address mime-send-hook-function (lambda nil (save-excursion (save-restriction ... ...))) (lambda nil (save-excursion (goto-char ...) (insert "Date: " ... "
") (save-restriction ... ...))))
Mark set
Make carbon copy before sending? (y or n) y
Unfrobbing selective-display...
Sending...
Added to /tmp/a
Added to /home/pot/Mail/backup/SMAIL
Sending...done
Mark set [2 times]

Load-path shadows:
~/elisp/bhl hides /usr/share/emacs/site-lisp/bhl
/usr/share/emacs25/site-lisp/flim/md4 hides /usr/share/emacs/25.1/lisp/md4
/usr/share/emacs25/site-lisp/flim/hex-util hides /usr/share/emacs/25.1/lisp/hex-util
/usr/share/emacs/site-lisp/rst hides /usr/share/emacs/25.1/lisp/textmodes/rst
~/elisp/bibtex hides /usr/share/emacs/25.1/lisp/textmodes/bibtex
/usr/share/emacs25/site-lisp/flim/ntlm hides /usr/share/emacs/25.1/lisp/net/ntlm
/usr/share/emacs25/site-lisp/flim/hmac-md5 hides /usr/share/emacs/25.1/lisp/net/hmac-md5
/usr/share/emacs25/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/25.1/lisp/net/sasl-ntlm
/usr/share/emacs25/site-lisp/flim/sasl-digest hides /usr/share/emacs/25.1/lisp/net/sasl-digest
/usr/share/emacs25/site-lisp/flim/sasl hides /usr/share/emacs/25.1/lisp/net/sasl
/usr/share/emacs25/site-lisp/flim/sasl-cram hides /usr/share/emacs/25.1/lisp/net/sasl-cram
/usr/share/emacs25/site-lisp/flim/hmac-def hides /usr/share/emacs/25.1/lisp/net/hmac-def
/usr/share/emacs25/site-lisp/auctex/tex-fold hides /usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs25/site-lisp/auctex/context-en hides /usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs25/site-lisp/auctex/tex-info hides /usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs25/site-lisp/auctex/plain-tex hides /usr/share/emacs/site-lisp/auctex/plain-tex
/usr/share/emacs25/site-lisp/auctex/tex-mik hides /usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs25/site-lisp/auctex/texmathp hides /usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs25/site-lisp/auctex/context-nl hides /usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs25/site-lisp/auctex/toolbar-x hides /usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs25/site-lisp/auctex/tex hides /usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs25/site-lisp/auctex/tex-jp hides /usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs25/site-lisp/auctex/bib-cite hides /usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs25/site-lisp/auctex/multi-prompt hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs25/site-lisp/auctex/font-latex hides /usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs25/site-lisp/auctex/prv-emacs hides /usr/share/emacs/site-lisp/auctex/prv-emacs
/usr/share/emacs25/site-lisp/auctex/tex-style hides /usr/share/emacs/site-lisp/auctex/tex-style
/usr/share/emacs25/site-lisp/auctex/context hides /usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs25/site-lisp/auctex/preview hides /usr/share/emacs/site-lisp/auctex/preview
/usr/share/emacs25/site-lisp/auctex/tex-font hides /usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs25/site-lisp/auctex/tex-bar hides /usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs25/site-lisp/auctex/latex hides /usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs25/site-lisp/auctex/tex-buf hides /usr/share/emacs/site-lisp/auctex/tex-buf
~/elisp/bhl hides /usr/share/emacs25/site-lisp/bhl
/usr/share/emacs/site-lisp/debian-startup hides /usr/share/emacs25/site-lisp/debian-startup

Features:
(wdired animate misc ruler-mode cal-x debug two-column iso-transl
apropos pcmpl-gnu pcmpl-linux macros python tramp-sh tar-mode
anything-config anything woman mailcap undigest cal-move smerge-mode
diff doc-view vc-cvs pcmpl-unix json-mode rx cus-edit wid-edit cus-start
cus-load rmailedit tabify man vc-dir ewoc tramp-cache shadow emacsbug
reporter debian-bug mail-extr texmathp nxml-uchnm rng-xsd xsd-regexp
rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse
rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode nxml-outln
nxml-rap nxml-util nxml-glyph nxml-enc xmltok tmm ido etags xref project
ediff-merg ediff-wind ediff-diff ediff-mult ediff-help ediff-init
ediff-util ediff js json map imenu ispell log-view pp visual-fill-column
locate dired-aux eieio-opt speedbar sb-image ezimage dframe find-func
timezone rmailsort face-remap url-handlers nero cl rect log-edit
pcvs-util add-log grep dabbrev diff-mode vc rmailkwd mailalias rmailout
shr-color color time-stamp misearch multi-isearch server parse-time
unrmail qp url-util shr dom subr-x browse-url jka-compr cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
bibtex info sh-script executable sgml-mode vc-filewise vc-rcs image-mode
view conf-mode generic octave smie latexenc plain-tex vc-dispatcher
vc-svn preview prv-emacs tex-bar tex-buf toolbar-x noutline outline
font-latex latex easy-mmode edmacro kmacro tex-style tex dbus xml crm
tex-mode compile rmailmm message rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mailabbrev gmm-utils mailheader mail-parse rfc2231 desktop
frameset term/xterm xterm pot skeleton rmailsum rmail warnings sendmail
rfc2047 rfc2045 ietf-drums mime-compose epa-mail mail-utils epa derived
epg cal-china lunar solar cal-dst cal-bahai cal-islam cal-hebrew
holidays hol-loaddefs vc-hg appt diary-lib diary-loaddefs cal-menu
calendar cal-loaddefs tramp tramp-compat tramp-loaddefs trampver
ucs-normalize shell pcomplete comint ring format-spec bhl
switch-to-shell openwith hi-lock xt-mouse ffap thingatpt url-parse
auth-source cl-seq eieio eieio-core cl-macs gnus-util time-date mm-util
help-fns mail-prsvr password-cache url-vars scroll-in-place filladapt
ansi-color advice time quail dired-x dired generic-x disp-table
finder-inf package epg-config seq byte-opt gv bytecomp byte-compile
cl-extra help-mode easymenu cconv cl-loaddefs pcase cl-lib debian-el
debian-el-loaddefs w3m-load vm-autoload vm-autoloads vm-version vm-vars
vm-init preview-latex tex-site auto-loads mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting font-render-setting x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1184154 169326)
 (symbols 48 60527 97)
 (miscs 40 8635 7089)
 (strings 32 176775 28569)
 (string-bytes 1 5262742)
 (vectors 16 87949)
 (vector-slots 8 2220318 72858)
 (floats 8 1437 1323)
 (intervals 56 171752 2504)
 (buffers 976 239))

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Eli Zaretskii]

> Date: Fri, 13 Jan 2017 15:15:23 +0100
> From: Francesco Potortì <pot@gnu.org>
> 
> When mail-user-agent is set to 'sendmail-user-agent, email saved to FCC
> files does not include a Date: header.

Works for me here, so some other factor is at work in your case.

Can you produce a full recipe, starting with "emacs -Q", and including
the necessary customizations of mail-related variables, until you can
send a message which exhibits this problem?  I'd like to compare that
with what I have here.

Thanks.

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Francesco Potortì]

>> Date: Fri, 13 Jan 2017 15:15:23 +0100
>> From: Francesco Potortì <pot@gnu.org>
>> 
>> When mail-user-agent is set to 'sendmail-user-agent, email saved to FCC
>> files does not include a Date: header.
>
>Works for me here, so some other factor is at work in your case.
>
>Can you produce a full recipe, starting with "emacs -Q", and including
>the necessary customizations of mail-related variables, until you can
>send a message which exhibits this problem?  I'd like to compare that
>with what I have here.

Here you are:

$ emacs -Q
M-: (setq mail-user-agent 'sendmail-user-agent)
C-xm
add a recipient, subject and text at will
C-cC-fC-f
specify an FCC file at will
C-cC-c
transport RET

Now if I look at the FCC file, I see that the sent mail does not contain
a Date: header.

Thank you for looking into this

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Eli Zaretskii]

> Date: Fri, 13 Jan 2017 15:41:07 +0100
> From: Francesco Potortì <pot@gnu.org>
> Cc: 25436@debbugs.gnu.org
> 
> $ emacs -Q
> M-: (setq mail-user-agent 'sendmail-user-agent)
> C-xm
> add a recipient, subject and text at will
> C-cC-fC-f
> specify an FCC file at will
> C-cC-c
> transport RET
> 
> Now if I look at the FCC file, I see that the sent mail does not contain
> a Date: header.

I did the above, and I do see the "Date" header.  However, I needed a
few additional settings before I could proceed after "C-c C-c".  Here
are the variables I needed to set:

  user-full-name
  user-mail-address
  smtpmail-smtp-server
  smtpmail-smtp-service
  smtpmail-stream-type

Not sure if any of the above is relevant to the issue at hand.

Can you step through sendmail-send-it, and see what happens there with
the date header?  According to my reading of the code, by the time
mail-do-fcc is invoked, the Date header is supposed to be already in
the mail buffer, and that buffer is inserted verbatim into FCC.

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Francesco Potortì]

>> Date: Fri, 13 Jan 2017 15:41:07 +0100
>> From: Francesco Potortì <pot@gnu.org>
>> Cc: 25436@debbugs.gnu.org
>> 
>> $ emacs -Q
>> M-: (setq mail-user-agent 'sendmail-user-agent)
>> C-xm
>> add a recipient, subject and text at will
>> C-cC-fC-f
>> specify an FCC file at will
>> C-cC-c
>> transport RET
>> 
>> Now if I look at the FCC file, I see that the sent mail does not contain
>> a Date: header.
>
>I did the above, and I do see the "Date" header.  However, I needed a
>few additional settings before I could proceed after "C-c C-c".  Here
>are the variables I needed to set:
>
>  user-full-name
>  user-mail-address
>  smtpmail-smtp-server
>  smtpmail-smtp-service
>  smtpmail-stream-type

Hm.  It seems you did not send using sendmail (which is what 'transport
does in my recipe), but you used 'smtp instead.  In my running Emacs,
the last three variables are not even defined.

user-full-name is initialised (if I am not wrong) from /etc/passwd

user-mail-address, in may case, is taken from the $EMAIL env var

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Eli Zaretskii]

> Date: Fri, 13 Jan 2017 17:17:05 +0100
> From: Francesco Potortì <pot@gnu.org>
> Cc: 25436@debbugs.gnu.org
> 
> >> Now if I look at the FCC file, I see that the sent mail does not contain
> >> a Date: header.
> >
> >I did the above, and I do see the "Date" header.  However, I needed a
> >few additional settings before I could proceed after "C-c C-c".  Here
> >are the variables I needed to set:
> >
> >  user-full-name
> >  user-mail-address
> >  smtpmail-smtp-server
> >  smtpmail-smtp-service
> >  smtpmail-stream-type
> 
> Hm.  It seems you did not send using sendmail (which is what 'transport
> does in my recipe), but you used 'smtp instead.

This is MS-Windows; there's no sendmail here.

But that's the MTA, which should not matter in this case, I think.
The mail-user-agent is set to sendmail-send-it here, which is what
should matter.  The important question is why Date is not generated in
the mail buffer in your case.

> user-full-name is initialised (if I am not wrong) from /etc/passwd
> 
> user-mail-address, in may case, is taken from the $EMAIL env var

Yes, but that shouldn't matter.

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Glenn Morris]

Eli Zaretskii wrote:

> The important question is why Date is not generated in
> the mail buffer in your case.

Are you sure?
emacs -Q
C-x m
  -> No Date: header in mail buffer.

This seems correct, because if Date was added then, it might have no
relation to when the message was sent.

smtpmail-send-it adds Date: at sending time.

sendmail.el has always relied on "sendmail" adding Date:, AFAICS.
mail-do-fcc only adds Date: for Babyl files.
Perhaps the date in the mbox From line was assumed to be good enough for
the alternative case.

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Francesco Potortì]

>Eli Zaretskii wrote:
>
>> The important question is why Date is not generated in
>> the mail buffer in your case.
>
>Are you sure?
>emacs -Q
>C-x m
>  -> No Date: header in mail buffer.
>
>This seems correct, because if Date was added then, it might have no
>relation to when the message was sent.
>
>smtpmail-send-it adds Date: at sending time.
>
>sendmail.el has always relied on "sendmail" adding Date:, AFAICS.
>mail-do-fcc only adds Date: for Babyl files.
>Perhaps the date in the mbox From line was assumed to be good enough for
>the alternative case.

Ok, thanks for checking.  That's exactly what I was thinking, but had no
time to verify.

As far as I understand, the only change needed is in mail-do-fcc:


--- sendmail.el	2017-01-13 14:46:08.715407789 +0100
+++ sendmail.el.orig	2017-01-13 14:46:18.735581196 +0100
@@ -1408,7 +1408,6 @@ just append to the file, in Babyl format
 	(require 'mail-utils)
 	(insert (mail-rfc822-time-zone time) " ")
 	(goto-char (point-max))
-	(insert "Date: " (mail-rfc822-date) "\n")
 	(insert-buffer-substring mailbuf)
 	;; Make sure messages are separated.
 	(goto-char (point-max))

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Francesco Potortì]

Date: 13 Jan 2017 19:38:41 +0100
>Perhaps the date in the mbox From line was assumed to be good enough for
>the alternative case.

By the way, the problem would be much less important if
rmail-sort-by-date resorted to the date from the From header when a
Date: header is not found, rather than assuming a date of -Inf.

bug#25436: 25.1; sendmail does not add a Date: header in FCC

[Eli Zaretskii]

> Date: Fri, 13 Jan 2017 19:38:41 +0100
> From: Francesco Potortì <pot@gnu.org>
> Cc: 25436@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> 
> >smtpmail-send-it adds Date: at sending time.
> >
> >sendmail.el has always relied on "sendmail" adding Date:, AFAICS.
> >mail-do-fcc only adds Date: for Babyl files.
> >Perhaps the date in the mbox From line was assumed to be good enough for
> >the alternative case.
> 
> Ok, thanks for checking.  That's exactly what I was thinking, but had no
> time to verify.
> 
> As far as I understand, the only change needed is in mail-do-fcc:
> 
> 
> --- sendmail.el	2017-01-13 14:46:08.715407789 +0100
> +++ sendmail.el.orig	2017-01-13 14:46:18.735581196 +0100
> @@ -1408,7 +1408,6 @@ just append to the file, in Babyl format
>  	(require 'mail-utils)
>  	(insert (mail-rfc822-time-zone time) " ")
>  	(goto-char (point-max))
> -	(insert "Date: " (mail-rfc822-date) "\n")
>  	(insert-buffer-substring mailbuf)
>  	;; Make sure messages are separated.
>  	(goto-char (point-max))

Thanks, I preferred to use message-make-date instead.

I'm closing the bug, feel free to reopen if something else is missing.