keeping a copy of sent mail locally

keeping a copy of sent mail locally

[Alex Ghitza]

Hello,

Many thanks to Marten and Carl for the advice on using scripts for
assigning tags automatically.  It works like a charm.

The next hurdle seems to be dealing with sent mail.  I would like each
message that I send to be saved in my local mail folder and treated the
same as all my other messages -- so it will get indexed and put in the
right thread, etc.  (For example, right now the thread that started with
my question about automatic tags only has the two replies in it, and its
subject is "Re: [notmuch] automatically...")  Bcc-ing myself on every
sent message is suboptimal for a number of reasons: (1) gmail throws
away the bcc-ed copy since it has the same message id as the one sitting
in the gmail sent mail, and so the bcc-ed copy never makes it back to my
local mail;  (2) even if this was working, it would be an unnecessary
waste of bandwidth. 

After looking around for a little bit, the only other option I could see
was to use the FCC mail header.  Unfortunately this wants a filename to
save to (rather than just a directory); so I have to manually add the
FCC: header, put in a filename that doesn't yet exist, type 'y' to
confirm that I want the file to be created.  It would be great if I
could just set the directory where sent mail should go to as a global
option, and then everything would happen automatically without any more
effort from me.

I realise that this is more of an emacs question than a notmuch
question, but I'm hoping that somebody on this list has an elegant
solution to this.


Best,
Alex


-- 
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

Re: keeping a copy of sent mail locally

[Jed Brown]

On Sat, 19 Dec 2009 15:41:14 +1100, Alex Ghitza <aghitza@gmail.com> wrote:
> Bcc-ing myself on every sent message is suboptimal for a number of
> reasons: (1) gmail throws away the bcc-ed copy since it has the same
> message id as the one sitting in the gmail sent mail, and so the
> bcc-ed copy never makes it back to my local mail;

I agree it is suboptimal, but are you sure this is true?  I send
messages via gmail and those having Bcc to me are marked as new and
downloaded by the next pass of getmail (via POP).

Jed

Re: keeping a copy of sent mail locally

[Alex Ghitza]

On Sat, 19 Dec 2009 02:55:16 -0800, Jed Brown <jed@59A2.org> wrote:
> On Sat, 19 Dec 2009 15:41:14 +1100, Alex Ghitza <aghitza@gmail.com> wrote:
> > Bcc-ing myself on every sent message is suboptimal for a number of
> > reasons: (1) gmail throws away the bcc-ed copy since it has the same
> > message id as the one sitting in the gmail sent mail, and so the
> > bcc-ed copy never makes it back to my local mail;
> 
> I agree it is suboptimal, but are you sure this is true?  I send
> messages via gmail and those having Bcc to me are marked as new and
> downloaded by the next pass of getmail (via POP).
> 

Aha!  Your reply prodded me to investigate this further.  The problem
seems to be in the way I'm working with the emacs "Message" mode.
Namely, Bcc works fine when I *reply* to a message from notmuch in
emacs: while composing the reply I see the automatically added Bcc
header, and after sending through gmail and getting the new mail via POP
I get the message back. 

However, if I'm using notmuch in emacs and press 'm' to compose a new
message, then the Bcc header is not added automatically and I guess it's
not surprising that the rest doesn't work as expected.

So where is the difference between 'm' and 'r' coming from?  I have
(setq mail-self-blind t)  in my .emacs, and I thought that should take
care of things, but it obviously doesn't.


Best,
Alex

-- 
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

Re: keeping a copy of sent mail locally

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

On Sun, 20 Dec 2009 11:24:52 +1100, Alex Ghitza <aghitza@gmail.com> wrote:
> However, if I'm using notmuch in emacs and press 'm' to compose a new
> message, then the Bcc header is not added automatically and I guess it's
> not surprising that the rest doesn't work as expected.

Yeah, sorry about that. That's a known bug.

> So where is the difference between 'm' and 'r' coming from?

The difference is that 'r' actually runs the "notmuch reply" command and
that command explicitly puts your own address into the Bcc. So this is
happening outside of any emacs code.

When you hit 'm', though, there's not any notmuch code involved,
(neither C code, nor emacs lisp), as it's currently just running
`message-mail'. So we need to figure out how to configure (or hook) that
to insert the Bcc, and then we can fix notmuch.el to do this without any
user configuration.

> I have (setq mail-self-blind t)  in my .emacs, and I thought that should take
> care of things, but it obviously doesn't.

Ah, interesting.

I'm guessing that that variable would apply to the "mail" mode stuff in
emacs, but perhaps not the "message" mode stuff we're using.

Yes, emacs mail support is a huge pile of very similar, but incompatible
systems.

Hopefully someone can fix this soon, since I really don't like the fact
that I'm currently not getting copies of several messages that I'm
composing and sending.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: keeping a copy of sent mail locally

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 368 bytes --]

On Sat, 19 Dec 2009 18:37:12 -0800, Carl Worth <cworth@cworth.org> wrote:

> So we need to figure out how to configure (or hook) that
> to insert the Bcc, and then we can fix notmuch.el to do this without any
> user configuration.

Just call:

(message-mail nil nil '(("bcc" "keithp@keithp.com")))

instead of (message-mail)

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: keeping a copy of sent mail locally

[Alex Ghitza]

On Sat, 19 Dec 2009 18:50:17 -0800, Keith Packard <keithp@keithp.com> wrote:
> On Sat, 19 Dec 2009 18:37:12 -0800, Carl Worth <cworth@cworth.org> wrote:
> 
> > So we need to figure out how to configure (or hook) that
> > to insert the Bcc, and then we can fix notmuch.el to do this without any
> > user configuration.
> 
> Just call:
> 
> (message-mail nil nil '(("bcc" "keithp@keithp.com")))
> 
> instead of (message-mail)
> 

Good.  So we can put this in notmuch.el and Keith will get everybody's
sent mail. :)

It looks like we need a way to get the primary email address from the
config file.  Actually, while we're at it, we can consider making this
more flexible and adding a new option to the config file (e.g. bcc=...)
which would take a semicolon-separated list of email addresses (in case
someone wants to always bcc an address other than the primary one).

Is there a nice clean way of getting the config variables in notmuch.el?
Or should this go into a new file notmuch-compose.c?  Note that the
latter would imply having a new command like "notmuch compose".

Ah, I just realised there's another issue related to the fact that 'm'
immediately defers to Message mode while 'r' does some preprocessing:
after pressing 'r', various headers are set in the appropriate manner
(From:, for instance, using the user name and email address from the
config file).  If you press 'm', you would get something like

From: ghitza@artin.i-did-not-set--mail-host-address--so-tickle-me

due to the fact that Message mode doesn't know the name or the email
address.  This can be fixed by adding to .emacs something like

(setq user-full-name "Alex Ghitza")
(setq user-mail-address "aghitza@gmail.com")

It's not a big deal, but now the same thing must be set in two different
places (.notmuch-config and .emacs), and can lead to confusion if, say,
the user changes their primary email address in one place but forgets it
needs to also be changed in the other.  So I guess I would prefer it if
notmuch would do a very minimal amount of preprocessing before heading
into Message mode when composing mail.  In fact, we could make sure that
there is consistent behaviour between 'm' and 'r' by making "notmuch
compose" return a few preset headers (From:, Bcc:, whatever else is
deemed appropriate), and making "notmuch reply" call "notmuch compose"
first and then add the reply-specific content.

Sorry about writing a lot of prose and no code!  I'd like to get a sense
for people's reactions before trying to write a patch for this.


Best,
Alex


-- 
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

Re: keeping a copy of sent mail locally

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

On Sun, 20 Dec 2009 15:29:15 +1100, Alex Ghitza <aghitza@gmail.com> wrote:

> Good.  So we can put this in notmuch.el and Keith will get everybody's
> sent mail. :)

That seems sub-optimal, at least for me...

> It looks like we need a way to get the primary email address from the
> config file.

We actually want to let the user *select* an email address from the
config file, and then automagically set the bcc: flag as
appropriate. Without that, I'd end up bcc'ing all of my mail through my
home address, which would end up sending work email unencrypted to my
house. Sub-optimal

> (setq user-full-name "Alex Ghitza")
> (setq user-mail-address "aghitza@gmail.com")

You can set everything in the message-mail argument list:

(message-mail nil nil '(("bcc" "keith.packard@intel.com") ("from" "Keith Packard <keith.packard@intel.com>")))

Alternatively, we can hack up message mode so that when the From address
is set (using completion from the notmuch config), the Bcc line is copied.

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: keeping a copy of sent mail locally

[Marten Veldthuis]

On Sat, 19 Dec 2009 21:02:18 -0800, Keith Packard <keithp@keithp.com> wrote:
> We actually want to let the user *select* an email address from the
> config file, and then automagically set the bcc: flag as
> appropriate. Without that, I'd end up bcc'ing all of my mail through my
> home address, which would end up sending work email unencrypted to my
> house. Sub-optimal

There's a message-send-hook, which we should probably use. Something
like:

  (add-hook 'message-send-hook 'notmuch-always-bcc-sender)
  (defun notmuch-always-bcc-sender ()
    (message-add-header (concat "Bcc: "
                                (message-fetch-field "From"))))

Though I've just scrabbled this together from the docs, I think it
should work (haven't tried it though).

-- 
- Marten

Re: keeping a copy of sent mail locally

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

On Sun, Dec 20, 2009 at 07:22:55PM +0100, Marten Veldthuis wrote:
> There's a message-send-hook, which we should probably use. Something
> like:
> 
>   (add-hook 'message-send-hook 'notmuch-always-bcc-sender)
>   (defun notmuch-always-bcc-sender ()
>     (message-add-header (concat "Bcc: "
>                                 (message-fetch-field "From"))))
> 
> Though I've just scrabbled this together from the docs, I think it
> should work (haven't tried it though).

Hi, folks.  I've been following this thread a bit and I wanted to put
in my argument that using Bcc is *NOT* the way to save local copies of
sent mail.  I really don't think that we need to require that the mail
actually get send out to the world wide net just to save a copy of it
locally.  I think there must be better ways of doing this.

There must be a way to tell emacs message-mode to save a copy of
outgoing mail locally.  Mutt does this with it's Fcc commands
(ie. "file carbon copy").  I think we should look for a solution
like this.

I definitely want to save copies of all of my outgoing mail locally,
but I think that doing that with Bcc is the wrong way to do it.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: keeping a copy of sent mail locally

[Adam Sjøgren]

On Sun, 20 Dec 2009 13:31:13 -0500, Jameson wrote:

> There must be a way to tell emacs message-mode to save a copy of
> outgoing mail locally. Mutt does this with it's Fcc commands (ie. 
> "file carbon copy"). I think we should look for a solution like this.

Gnus uses Gcc for this (see gnus-message-archive-group and
gnus-message-archive-method); I think message.el also supports Fcc; eg. 
see message-fcc-handler-function:

,----[ C-h v message-fcc-handler-function RET ]
| `message-fcc-handler-function' is a variable declared in Lisp.
|   -- loaded from "message"
| 
| Value: message-output
| 
| Documentation:
| *A function called to save outgoing articles.
| This function will be called with the name of the file to store the
| article in.  The default function is `message-output' which saves in Unix
| mailbox format.
`----


  Best regards,

    Adam

-- 
 "Are you talking to us from yesterday?"                      Adam Sjøgren
                                                         asjo@koldfront.dk

Re: keeping a copy of sent mail locally

[Alex Ghitza]

On Sun, 20 Dec 2009 13:31:13 -0500, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> Hi, folks.  I've been following this thread a bit and I wanted to put
> in my argument that using Bcc is *NOT* the way to save local copies of
> sent mail.  I really don't think that we need to require that the mail
> actually get send out to the world wide net just to save a copy of it
> locally.  I think there must be better ways of doing this.

Agreed, and I think that directly saving to a file is the sane way to
go.  However, there might be other (more legitimate) uses for always
bcc-ing a certain address on all sent mail, and it might be good to have
an easy mechanism for this.  


Best,
Alex

-- 
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

Re: keeping a copy of sent mail locally

[Alex Ghitza]

On Sun, 20 Dec 2009 19:51:05 +0100, asjo@koldfront.dk (Adam =?iso-8859-1?Q?Sj=F8gren?=) wrote:
> On Sun, 20 Dec 2009 13:31:13 -0500, Jameson wrote:
> 
> > There must be a way to tell emacs message-mode to save a copy of
> > outgoing mail locally. Mutt does this with it's Fcc commands (ie. 
> > "file carbon copy"). I think we should look for a solution like this.
> 
> Gnus uses Gcc for this (see gnus-message-archive-group and
> gnus-message-archive-method); I think message.el also supports Fcc; eg. 
> see message-fcc-handler-function:
> 

Adam, thanks for the info on the message-fcc-handler-function.  I'll try
to get something working based on this.  There are two questions of
design now:

1. where should the file be saved?  I'm thinking of a new setting in
$NOTMUCH_CONFIG, along the lines of sentmail_path=...
Should there be a default if this is not set, e.g. a subdirectory "sent"
of the directory given in the config variable "path"?

2. of course, filenames need to be unique.  Do we want/have to follow
the maildir file naming conventions listed at
http://cr.yp.to/proto/maildir.html
or is it enough to use the Emacs lisp make-temp-file?


Best,
Alex


-- 
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

Re: keeping a copy of sent mail locally

[Marten Veldthuis]

On Mon, 21 Dec 2009 21:08:22 +1100, Alex Ghitza <aghitza@gmail.com> wrote:
> 2. of course, filenames need to be unique.  Do we want/have to follow
> the maildir file naming conventions listed at
> http://cr.yp.to/proto/maildir.html
> or is it enough to use the Emacs lisp make-temp-file?

I'd very much prefer a real Maildir, because that would sync back
correctly with OfflineIMAP etc.

-- 
- Marten

Re: keeping a copy of sent mail locally

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

On Mon, Dec 21, 2009 at 08:57:40PM +1100, Alex Ghitza wrote:
> On Sun, 20 Dec 2009 13:31:13 -0500, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > Hi, folks.  I've been following this thread a bit and I wanted to put
> > in my argument that using Bcc is *NOT* the way to save local copies of
> > sent mail.  I really don't think that we need to require that the mail
> > actually get send out to the world wide net just to save a copy of it
> > locally.  I think there must be better ways of doing this.
> 
> Agreed, and I think that directly saving to a file is the sane way to
> go.  However, there might be other (more legitimate) uses for always
> bcc-ing a certain address on all sent mail, and it might be good to have
> an easy mechanism for this.  

I certainly have no problem with there being the ability to specify an
address that is always Bcc'd on sent mail.  I think that's a fine
idea.  I just think that saving local copies of sent mail directly is
an important thing to do, and we should figure out the best way to
support it correctly.  Every mailer that I know of has some way to
save local copies of sent mail, and I think most users will expect
such a feature.

An important thing to consider is that all of this locally saved sent
mail should also be immediately incorporated into the database so that
it can be immediately searched and displayed with it's relevant
threads.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: keeping a copy of sent mail locally

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

On Sun, 20 Dec 2009 15:29:15 +1100, Alex Ghitza <aghitza@gmail.com> wrote:
> It looks like we need a way to get the primary email address from the
> config file.

Yes, we definitely need that.

> Actually, while we're at it, we can consider making this
> more flexible and adding a new option to the config file (e.g. bcc=...)
> which would take a semicolon-separated list of email addresses (in case
> someone wants to always bcc an address other than the primary one).

And that would be fine too.

> Is there a nice clean way of getting the config variables in notmuch.el?
> Or should this go into a new file notmuch-compose.c?  Note that the
> latter would imply having a new command like "notmuch compose".

I think we want a new C file to make it easy to ask for options out of
the configuration file. The operation of "notmuch compose" seems simple
enough to not be necessary, (though maybe it would make it easier to get
standard behavior among different interfaces).

> It's not a big deal, but now the same thing must be set in two different
> places (.notmuch-config and .emacs),

Right. I definitely want to get away from that. I want people to put
stuff into .notmuch-config and then have our emacs code "just work".
Our emacs code could, for example, set variables like user-full-name and
user-mail-address from the notmuch configuration, (it wouldn't override
these if already set I think).

> Sorry about writing a lot of prose and no code!  I'd like to get a sense
> for people's reactions before trying to write a patch for this.

No problem at all. Thanks for the thoughts and ideas.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: keeping a copy of sent mail locally

[Adrien BUSTANY]

On Mon, 21 Dec 2009 09:27:04 -0800, Carl Worth <cworth@cworth.org> wrote:
> On Sun, 20 Dec 2009 15:29:15 +1100, Alex Ghitza <aghitza@gmail.com>
wrote:
>> It looks like we need a way to get the primary email address from the
>> config file.
> 
> Yes, we definitely need that.
> 
>> Actually, while we're at it, we can consider making this
>> more flexible and adding a new option to the config file (e.g. bcc=...)
>> which would take a semicolon-separated list of email addresses (in case
>> someone wants to always bcc an address other than the primary one).
> 
> And that would be fine too.
> 
>> Is there a nice clean way of getting the config variables in
notmuch.el?
>> Or should this go into a new file notmuch-compose.c?  Note that the
>> latter would imply having a new command like "notmuch compose".
> 
> I think we want a new C file to make it easy to ask for options out of
> the configuration file. The operation of "notmuch compose" seems simple
> enough to not be necessary, (though maybe it would make it easier to get
> standard behavior among different interfaces).
What about adding a way to get/set config variables from the notmuch
command
line ? Would that be bloat ?
That would even allow changing the internal format of the conf. file
without
breaking the clients...

[Snip]

Cheers

Adrien

Re: keeping a copy of sent mail locally

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

On Mon, 21 Dec 2009 20:39:01 +0100, Adrien BUSTANY <madcat@mymadcat.com> wrote:
> On Mon, 21 Dec 2009 09:27:04 -0800, Carl Worth <cworth@cworth.org> wrote:
> > 
> > I think we want a new C file to make it easy to ask for options out of
> > the configuration file.
>
> What about adding a way to get/set config variables from the notmuch
> command line ? Would that be bloat ?

That's actually exactly what I meant above. I think we'll need this.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]